Asking the Right questions – Beware Small Samples

Updated: June 28, 2012 at 4:52 pm by Kent Wilson



(This is part one of a three part series that was originally published in April. With free agency season approaching, we figured it was worth a re-post here on NHLNumbers.)

It occured to me recently, after watching Brian Burke bloviate, Jay Feaster deflect and Steve Tambellini ramble during their respective year-end press conferences, that the true issue facing mediocre (or worse) teams is not whether an executive has all the answers – it’s if he has the wherewithal to ask the right questions.

The goal of managing a hockey team is simple – gather as many good hockey players together (for X amount of dollars) as possible. Unfortunately, there is an innumerable number of blind alleys to go down in pursuit of that destination. Player performance fluctuates year to year due to chance, circumstance, injury and career arc. The shifting free agent and trade markets continually alter the value of player assets. Throw in the fuzzy stuff both decision makers and fans tend to attach a lot of significance to (leadership, chemistry, character, truculence etc.) and you have a baffling miasma of factors and gambles to sift through every off season. Which is why if a management team isn’t endlessly curious about its outcomes or ruthlessly honest about its assets, it’s possible not only to get lost in the desert but also be driving in entirely the wrong direction.

In this series of posts, I plan to look at how the wrong questions are easy to ask and difficult to avoid. 

Larger Samples are More Meaningful Than Small Ones

I’ve noticed fans, coaches and managers of lousy teams tend to get caught up talking about relatively small stretches of games when the team tasted some success. This happened a few years ago with the Edmonton Oilers for instance. They were still terrible, but nevertheless reeled off a month of better than average results at the end of the year. This was marketed to fans by the organization that, clearly, things were looking up. Since then, they have finished 30th, 30th and 29th in the league. 

I argued against this type of thinking during Feaster’s first (half) season at the wheel when the Flames were the hottest team in the league for about two months. Similar narrative building is happening again in Calgary this off-season: both Sutter and Feaster have talked about how good the Flames were when half the line-up was made up of AHLers and kids due to injury. The obvious solution going forward, therefore, is to make the roster indiscriminately younger. Younger means more wins, right?

Intuitively, it seems to make sense to seek out periods of success to try to determine what was different/better about the team at the time and then try emulate those circumstances. Unfortunately, the truth is just about every team experiences periods of better than average results as a matter of chance. Perhaps the schedule was easy, perhaps the PP got hot for a few games, maybe the goalie was standing on his head for awhile. Think about Edmonton leading the NW division for the first six weeks of the year or Minny leading the West at the mid-way point of December, for example.

The challenge is to be consistently skeptical of hot streaks when there is a longer history of mediocrity. More data is almost always more powerful than less. If a club has a few weeks of results that is apparently out of line with an established skill level, then it’s important to examine the information from those games with the goal of discovering how sustainable – how true – those results were. In essence, a bad team on a good run should try to debunk its own (briefly held) success in an effort to determine if the improvement is real or not. If a club has established that it is average or worse, that should be the guiding assumption, even in the face of apparent, sudden improvement.

This is, of course, exceedingly difficult to do for fans and decision makers alike. The human brain looks for positive results and easy, causal explanations for things. Coaches and GMs will naturally focus on the good stuff because everyone prefers to win rather than lose and to be right rather than wrong. And during any transient outburst of success, there is bound to be some convenient, intuitive explanation for the change in fortune.

Myth Making

Last year for Calgary, it was Darryl Sutter’s departure and the subsequent change to the Flames culture that caused the improvement. This year, it was the energy and exuberance conferred by the rookies. In Edmonton, it was the emergence of the kids and the rehabilitation of Kahbibulin. In Minnesota, new coach Mike Yeo had the team playing together like never before!  Im not sure what easy explanations popped up in Toronto when they rushed hot out of the gate this year, but I’m guessing they have disappeared into the ether by now.

Even bad teams have good stretches of games. And those outcomes are not necessarily an indication that something has fundamentally changed, that the club has truly improved. The tendency to weight those periods as more meaningful than others and to fabricate fuzzy but plausible sounding explanations for them should be avoided.

The question for average-to-below-average teams should never be “why are we occasionally good?” Rather – “why are we so often bad?”