The 2012 MIT Hockey Analytics Panel: A Review

The MIT Sports Analytics Conference was founded in 2006 with the hopes of bringing together important figures from each of the major professional sports to discuss the most relevant and cutting edge developments in the field.

Bill Simmons refers to it as “Dorkapalooza” as there is a heavy emphasis on the quantitative aspect of analysis (hence the word analytics in the title).

Anyway, hockey is definitely low on the totem pole with regards to many statistical developments, but it is gaining ground in recent years thanks to the hard work of several people. Baseball has been at the forefront for a while thanks in large part to the book Moneyball. The sport lends itself quite well to statistical analysis, as it is essentially a series of one-off situations, while sports like hockey and basketball are much more team-dependent and free-flowing.

Here are some interesting sites to get some working knowledge of advanced stats in hockey:

In this post at DobberHockey, I attempted to use advanced statistics to evaluate and predict fantasy hockey performance.

The standard statistics in hockey have significant limitations. Plus-minus, even if used in the right context, doesn’t come close to telling us how well a player is performing defensively. Goals and assists are obviously important (goals win games), but there are underlying reasons that contribute to them. How much is a player shooting the puck? Where is he shooting the puck from? Is he playing against good players? Is he starting a lot of his shifts in the offensive zone?

Advanced statistics have been widely used by numerous NHL teams for the past few years, and thanks to the hard work of a few people, we as fans now have access to a wide variety of them.

 

2012 Hockey Analytics Panel

The hockey analytics panel, returning for its second year, will discuss the role of analytics in the modern era of the game, including assessing value under salary cap constraints, integrating objective and subjective scouting data, the relative value of defense, offense, and goaltending, and allocation of payroll that comprise a championship hockey team.

The panel essentially covered the spectrum of opinions on statistical analysis. On one side you have former NHL player/GM and current commentator Mike Milbury, with the old school mentality that statistics fall short when it comes to analysis in hockey. Toronto GM Brian Burke largely shared this opinion, but he had a slightly more moderate tone throughout the panel compared to Milbury.

Bruins GM Peter Chiarelli was more in the middle – he recognizes the importance of statistics but doesn’t believe they trump scouting. And at the other end was Michael Shuckers, a statistics professor at St. Lawrence University. He published a paper for the 2012 Conference on DIGR – a defense independent goalie rating system.

Tony Amonte was on the panel as well to share the perspective of a former player on statistics. Amonte’s contributions were limited compared to the other panelists.

If you don’t want to watch the hour-long discussion, here are some of the highlights.

Peter Chiarelli mentions how the Bruins look at size and height before anything else with scouting and drafting – and in fact they have done studies that have determined that weight is much more important than height with regards to projecting future performance.

Chiarelli says that the Bruins generate internal statistics by taking what is measured (goals, assists, plus/minus) and applying context and situational markers. The Bruins also create matrixes in the offensive and defensive zones so they can characterize the quality of scoring chances based on location.

This essentially amounts to counting scoring chances – a huge part of the advanced statistical movement in hockey.

*

Brian Burke says that he wants the Leafs to be on the cutting edge of all analysis. He says that he receives hundreds of papers each year, and that he and his staff do their due diligence and read them thoroughly to see if anything useful can be incorporated into what they are doing.

There hasn’t been a statistical breakthrough in hockey yet, says Burke. I’d agree, but I’m not sure there will ever be a “breakthrough.” Hockey is so free-flowing and non-static, that even a few tiny developments in statistical analysis will give whoever is using them a real edge over the competition.

On drafting prospects – “we watch them play, we try to make a projection, and we draft them on that.” Definitely old school. Burke goes on to mention how he is aggravated by the Moneyball popularity, as at its core Moneyball is a boring type of baseball – working the pitch count, for example. I guess that is an entirely different can of worms – as a pro sports organization, what is more important – winning, or entertaining? 

Burke has a point that Oakland never won a title using the Moneyball approach, but there is a lot more that goes into building a club (having the funds to compete in a free market system is a significant factor, too). What it did grant Oakland for several years was an edge to compete with teams that were outspending them significantly. To me, the word Moneyball doesn’t imply success or winning, but using tools to gain an edge over the competition, however that may be.

*

Mike Milbury, of course, isn’t a fan of Moneyball either. I understand why he was put on this panel (he is a former player and GM, and he is respected within the hockey world), but he makes quite clear that he doesn’t believe in advanced statistics at all.

*

Michael Shuckers does a good job of explaining the various research and work he has done over the years. In 2011 he developed probability surfaces for the offensive zone – some biases came up though, as shots aren’t counted the same from arena to arena (as an example, Madison Square Garden is notorious for over counting shots for the Rangers)

*

Burke tells a good story about Trevor Linden’s draft year. He said Linden was a guy they had targeted, and a few days before Linden was supposed to fly out from Medicine Hat for a draft interview and psychological testing, he called Burke to tell him he couldn’t make it. Linden’s reason? He would be busy all weekend helping brand the cattle at his uncle’s ranch. Burke asked Linden what his role on branding day was.

“He says well when the young cows come in the pen, I grab them by the neck, pull them down, and pin them while we brand them and cut their nuts off.”

“So I said, kid, you can skip the test.”

Burke used the story to explain how character is as important as any other form of evaluation. On Linden – he was a warrior, had a high compete level, a high hockey IQ, he was big, and a great skater. A can’t miss prospect.  

*

On physical testing, Chiarelli said that it is hard to get a read on an 18-year-old. And with the interviews too, as they are all coached by agents nowadays on what to say and when to say it.

*

Another interesting Burke story. He was in Prague interviewing then-prospect Jaromir Jagr before his draft year. Burke asked Jagr (through a translator) if he had ever been a captain. If a star prospect has never been a captain at any level, it should send warning signs, says Burke.

Jagr said he had never been a captain, so Burke asked why.

Jagr answered that he had always played with players three years older than him. And Burke said that was the best answer he ever got in a draft interview.

*

The next subject they touched on – league equivalency. How does the performance of a player in the QMJHL translate to the NHL, for example? Here is a read on NHL equivalencies from Gabe Desjardins.

On equivalency measures, Burke said that he has reviewed them and they have little value.

Chiarelli goes into more detail, saying that leagues change so much from year to year in terms of competition quality, that putting values on each league is very difficult.

How someone does in Quebec juniors, and how that projects to at the NHL

*

The GMs go on to discuss scouting and player evaluation. Chiarelli mentions the difficulty with scouting high school players, as the quality is so wide-ranging. Burke discusses scouting a few current NHL players.

On Milan Lucic – when Burke saw Lucic play, the big winger had such trouble skating, and was such a poor skater at the WHL level, that Burke didn’t think he would amount to much (Lucic was passed over by all 30 teams at least once during his draft year). 

On Wayne Simmonds – Burke saw Simmonds play eight times during his time with Owen Sound, and he was never impressed by him. But other scouts were – in fact, Burke said that Simmonds did nothing during the eight games he saw him, but the next scout to see him witnessed a Gordie Howe Hat-trick (goal, assist, and a fight). A great example of the difficulties with small sample sizes.

On Ryan Kesler – Burke said he watched Kesler play one shift, and he knew he wanted him.

*

Shuckers gets into zone starts a bit, mentioning Manny Malhotra. Chiarelli asks if they have statistics that measure how a player is deployed in terms of time of game and score in game (late with a one-goal lead, for example). 

Burke says that he doesn’t need to see a presentation on zone starts to know that Malhotra is good at faceoffs. While true, that appears to be an overly simplistic answer to the theory behind zone starts.

*

On the Detroit model:

Chiarelli – they mined the European leagues quite well, and they were able to convince Europeans to come over and play in the minors, something that is hard to do. Many of these European prospects are already paid well in professional leagues, and they see coming to the AHL as a demotion.

Chiarelli – the Detroit forwards are all “heavy” on the puck. In other words, they are a great puck possession team. And the numbers don’t lie – puck possession leads to shots on goal, which lead to goals.

However, does Detroit draft puck possession players and the results come from that, or do their young players learn that ability from developing in the system and being brought along slowly?

*

The panel’s discussions were interesting – I, for one, believe that the Leafs use statistical analysis a lot more than Burke lets on. He is obviously old-school in many ways (and you can’t ignore the fact that he led the Ducks to a Stanley Cup), but hockey is finally getting the attention it deserves from statistical analysis. The next five years are going to be really fascinating as statistics continue to infiltrate the hockey world.