Assessing Skill, Risk and the Erik Karlsson Contract

Updated: June 20, 2012 at 8:41 am by Kent Wilson



Yesterday when the Erik Karlsson contract was announced, but before any numbers were divulged, my immediate reaction was “the Senators are going to overpay”. It was a somewhat glib, but it’s a gut feel developed from watching NHL GM’s overreact to splashy offensive seasons for years. In addition, as a Flames fan, I suffered through Dion Phaneuf’s metamorphosis from “perenial Norris candidate to!” to “uh, this contract is an albatross” in the space of a couple seasons. I learned the painful lesson that even in younger players, extrapolating an upward trend line in linear fashion is a fraught with peril.

There’s no question Erik Karlsson is a talented offensive defenseman. And there’s no argument that what he accomplished last season was extraordinary. But at $45.5M over seven years, the Senators are taking on an enormous amount of risk.

The Counting Numbers

Last year, Karlsson ran away with the scoring race amongst defenseman. HIs 78-points were 25 more than second placed Dustin Byfuglien and Brian Campbell. In fact, only four other blueliners broke the 50-points barrier, let alone 60 or 70. His is the highest single season total by a defender since NIk Lidstrom scored 80-points back in 2005-06.

What’s even more remarkable is that Karlsson did most of his damage at even strength. Lidstrom’s 80-point effort consisted of just 30 points at ES six years ago. Karlsson, in contrast, scored 50 points at five-on-five this past year, 16 more than Kevin Bieksa who sat second in the league.

That’s the highest single-season ES point total by a defender in the post-lock-out era.

The only guy to get close since 2005-06 to that number is Duncan Keith during his Norris trophy campaign – he scored 48 at five-on-five for a stacked Blackhawks squad. Most league leaders hover around the 35 ES points typically.

So the 21-year old kid had an incredible year. Those are the kind of results that sell jerseys and make GM’s heart’s beat faster and agents bank accounts grow larger by orders of magnitude.

But the thing that makes numbers like that so flashy, so notable, so exciting and so profitable for both player and agent is precisely the thing that should make the guys writing the checks much more skeptical. Karlsson’s season is such a grotesque outlier that it becomes necessary to ask the question: just how repeatable is this?

Chance and Circumstance

Incredible output like Karlsson’s can often obfuscate the fact that things like goals, assists and points totals arise through a variety of interrelated factors. Skill is one of the pertinent variables, of course, and it’s the thing each GM is betting on when he inks a young player for big dollars over many years. But often as important are things like circumstance (line mates, opposition and usage) and chance (variance around the mean). As such, it’s important to separate the skill signal from the potential noise (circumstances, luck) in the data.

Last year, Karlsson played a lot at ES and his quality of competition was one of the highest on the Senators (+0.657) so he wasn’t merely beeating up on depth players. On the other hand, his zone start ratio – the amount of shifts he began in the offensive zone relative to the defensive zone – was 57.1%. That was the 9th highest ZS amongst regular blueliners who appeared in at least 40-games last season. This isn’t necessarily an indictment of the player – coaches tend to give their big guns the high ground from the back-end – but it’s likely a contributing factor to his noteworthy season.

In terms of chance, there’s no question lady luck smiled on Karlsson to a non-trivial degree. His ES on-ice shooting percentage was 9.58, a few notches above league average (about 8%) and inside the top-20 in the league (amongst the same sample mentioned above). That’s a number that tends to regress toward the league average for most skaters, so one we can expect Karlsson’s ES scoring to deflate a bit as a matter of course. 

Individual Point Percentage

However, what was truly outlandish about Karlsson’s season at five-on-five wasn’t just how much the team scored with him on the ice, but how frequently he was involved in the scoring. Individual points percentage (IPP) is another measure that can spike or crater for individuals over a season, but also tends to regress towards a general mean over time. Scott Reynolds of Copper and Blue put together IPP charts for both defenders and forwards over a four year span last August.

Like most metrics, IPP captures a bit of everything including both chance and skill. The best defenders by that measure between 2007 and 2011 were in the 35% area (meaning, they contributed to 35% of the even strength points that were scored when they were on the ice). The league average amongst regular defenders in the sample was 28.6%.

What you’ll also notice is that IPP rates bounced around for just about every defender listed. According to NHLNumbers contributor Eric T., the year-over-year correlation for IPP over that time for defensemen was just +0.24 – meaning a positive relationship, but hardly rock solid. The R-squared (correlation coefficient) was just 0.058 which suggests about 94% of the variance in defensemen’s IPP is due to “other factors” including luck, usage, circumstances, etc.

To close the loop: Erik Karlsson was on the ice for a league high 90 goals for at even strength. He contributed to 50 of those goals, for a league leading IPP of 56% amongst defensemen. Here are some other defenders who have contributed to 50% or more of their team’s ES offense recently, plus their IPP the very next season:

  • Duncan Keith went from 0.50 to 0.26 in ’10-11
  • Ed Jovanovski went from 0.52 to 0.30 in ’08-09
  • Cam Barker went from 0.50 to 0.30 in ’08-09
  • Steve Montador went from 0.57 to 0.31 in ’08-09
  • Brett Lebda went from 0.50 to 0.21 in ’09-10 (yup…Brett Ledba)

Again, the league average for regular defenders over four years was an IPP of about 29%. Most league leaders sit around 35% on the blueline and the year-over-year correlation of IPP for rearguards is just 0.24. Last season, Nik Lidstrom’s IPP for Detroit was just 27%. Duncan Keith, who was at 50% during his aforementioned career season, sat at 35% for the Blackhawks in 2011-12. Only a handful of defensemen crested 40% (Matt Carle, Ryan McDonagh, Dan Boyle, Kevin Bieksa) and only two others broke the 50% barrier with Karlsson (Byfuglien and Keith Yandle).

So, in short – getting a an IPP over 50% is difficult, rarely duplicated and dependent to a non-trivial degree on factors outside of a player’s talent level.

Risk, The Anchoring Bias and Drawing Trend lines

There are lots of completely rational reasons GMs rush to sign big checks to guys who flashy offensive totals every year – they excite the fanbase, they sell tickets and merchandise and they can help entice other players to either stay to sign with the team. Gawdy numbers aren’t merely about the on-ice outcomes they produced at the time.

What’s more, confident agents tend to stride into negotiations with a fist full of favorable comparables and a gleam in their eyes when a client laps the field. Even for the most skeptical, most prudent general manager, the amount to pay a player has to be couched to some degree in the market established for similar players. 

That said, GM’s can be fooled into escalating salaries, bidding against themselves and shamelessly chasing after results by a couple of insidious cognitive heuristics. First and primary is the anchoring bias, which is a tendency for a single piece of information (often a number) to overly influence decisions or predictions. The bias is so persistent and unconscious that the number doesn’t even have to be related to the question to skew the resultant answer. From the wiki link above:

In a study by Dan Ariely, an audience is first asked to write the last two digits of their social security number and consider whether they would pay this number of dollars for items whose value they did not know, such as wine, chocolate and computer equipment. They were then asked to bid for these items, with the result that the audience members with higher two-digit numbers would submit bids that were between 60 percent and 120 percent higher than those with the lower social security numbers, which had become their anchor.

Obviously offensive output and a players skill level (and thus future output) aren’t wholly unrelated, so it makes sense to link the two. However, GM’s must be careful not to allow a high watermark season to anchor their perceptions of a player and his potential going forward because it will tend to have an undue effect on those predictions.

Also, people tend to assume linear trends will continue indefinitely, ie; we draw straight lines through a few data points. Of course, few things over time are completely linear, least of all the career progression of NHL hockey players. This tendency is particularly pernicious when it comes to younger players because, intuitively, youngsters generally tend to get better as they near their mid-20s.

Between those two underlying biases/assumptions, it’s impossible to shake the notion that Karlsson is going to be a 60+ point defender over the life of his new $6.5M/year contract. And if he is, the Senators are going to get value for his deal – particularly because they bought a few of his UFA years (assuming the next CBA doesn’t change those rules drastically). 

Of course, the risk is that it’s entirely possible Karlsson won’t be that productive, even on average. Defenders have a very hard time sustaining that sort of output, in part because things like on-ice SV% and IPP are dependent on factors aside from skill and tend to regress towards the league mean.

In addition, with each additional year added on to a contract, there is the additional risk of a players development stopping or regressing due to idiosyncratic factors including injuries, changing circumstances, etc. Alexander Ovechkin and Dion Phaneuf are a couple of recent examples of guys who peaked in their early 20’s and signed expensive, long-term deals only to see their production and effectiveness slump and flatten out. John Michael Liles scored 14-goals and 49-points as a sophomore. He has never duplicated the feat.

The point isn’t to compare Karlsson directly to these players – he will likely follow his own, unique development curve – but only to highlight the fact that assuming indefinite improvement, or even projecting status quo projection multiple years into the future is riddled with risk and error.


Erik Karlsson’s season was incredible and unsual last year. HIs accomplishments should be both celebrated and viewed with skepticism. The resultant contract from the Senators will be worthwhile if Karlsson can sustain a large portion of his output over the life of it’s even year term. Unfortunately, it’s highly difficult to predict if that will happen because separating skill from chance and circumstance is problematic and the future becomes more and more unknowable the further ahead we go.

It’s not impossible that Karlsson is the next, great offensive defender who establishes himself as a league leader for years to come. That said, it’s a big, $45.5 gamble based on one season which is more likely to be an outlier than the norm.