The 2011 MIT Hockey Analytics Panel: A Review

The 2011 Hockey Analytics Panel at MIT, the first of its kind, was insightful to say the least. The speakers covered a wide range of topics, including the relevance of commonly used statistics, how the arbitration process is carried out, where the next big statistical breakthrough in hockey may come from, and much more.

I reviewed the 2012 MIT Hockey Analytics Panel here.

The 2011 panel was more interesting than the 2012 panel. The speakers seemed more engaged in the topic of statistics and evaluating hockey qualitatively, whereas the 2012 panel had more than one speaker who openly admitted to not really putting much stock into statistical analysis in hockey.

The 2011 Panel:

Stan Bowman, son of legendary coach Scotty Bowman and GM of the Chicago Blackhawks. Bowman less than a year removed from hoisting the Stanley Cup when the 2011 conference took place, and he had a lot of interesting thoughts to share, especially regarding Chicago’s salary cap maneuvering in the summer of 2010.

Don Fishman, the Assistant GM of the Washington Capitals. The Caps had undergone a significant philosophical change in the time leading up to this conference, switching from a high-octane offensive team to one that would be capable of playing at both ends of the ice.

More on Fishman:

A lawyer by trade and the team’s assistant general manager in title, Fishman is probably the most important front office employee most fans have never seen. He negotiates contracts for the organization’s two-dozen NHL players and its prospects, and advises team officials on a litany of other legal matters. But more significantly, he manages the salary cap, a responsibility that has become increasingly critical as the Capitals mature from a young and rebuilding team to a star-studded squad expected to contend in the Eastern Conference this season.

Dan MacKinnon, the Director of Player Personnel for the Pittsburgh Penguins. The Penguins, more than any other NHL team, know how important it is to find cap-friendly players to fill out a roster.

MacKinnon oversees the Penguins’ professional and amateur scouting operations in North America and Europe. He works closely with the Penguins’ directors of professional and amateur scouting to ensure that the organization maintains its strategic and comprehensive focus in player scouting, evaluation and acquisition at all levels.  MacKinnon also advises general manager Ray Shero on all Penguins’ professional and amateur transactions.

Jeff Solomon, the Vice President of Hockey Operations and Legal Affairs for the Los Angeles Kings. The Kings were about a year away from embarking on their incredibly Stanley Cup run at the time of this conference, and Solomon speaks to the “build it from the ground up” approach that the regime undertook after failed attempts to build a contender through free agency and trades.

Prior to joining the Kings, Solomon, 54 , served as an NHLPA Certified Agent for 20 years.  During that time, Solomon represented hundreds of NHL hockey players, including former Kings Tony Granato and Nelson Emerson, and Bryan McCabe and Steve Sullivan.

Solomon has extensive experience working with the NHL Entry Level System, restricted and unrestricted free agency and all aspects of the NHL Collective Bargaining Agreement and its arbitration system.

Jim Price, who runs RinkNet, a Guelph, Ontario-based company. RinkNet is used by a vast number of hockey teams at all levels as a primary scouting database. More on RinkNet:

The product includes schedules, player information and statistics across the literally hundreds of leagues that are scouted at both the professional and amateur levels. Scouts use the product to enter their own evaluations of each player, organize their travel schedules and even do their expenses.

The panel was moderated by Katie Burke, who works at Athlete’s Performance as a Director of Marketing and Partnerships.

Statistics in Hockey

Bowman has a business background, and his way of getting his foot in the door with the NHL was bringing in statistical analysis. Stats were his way into the business side of the hockey world, and it began for him around the year 2000. 

At that time, Chicago wasn’t doing much in terms of charting performance (and their on ice product reflected that). With Bowman, they started a very basic grading system – each player was given a score of 1-5 after a game from the coaches and management, and those scores were tracked over the course of the season.

Bowman discusses the difficulties with the grading system – emotion and bias were quite clear with the individual player grades, as the scores were very subjective. The coaches did respond well to the grading though, but that could have been a case of conformation bias.

The Blackhawks now have people that chart certain things each game for each player, including scoring chances for and against. The focus is now on finding things that are both quantifiable and valued by the coaching staff (statistics are great, but they need to be something that the coaches will actually use or the tracking is simply a waste of time). Bowman says that one main reason for the more objective measures – coaches tend to like predictable, coachable players, who are generally less talented. Their ratings tend to be consistent, and perhaps unfairly propped up, as their expectations are much lower compared to those of a star player or goaltender.

Jeff Solomon agreed with the above point, using Drew Doughty as an example. Often times the Kings are evaluating Doughty against himself instead of against his peers – the comparisons have to be relevant.

Solomon goes on to mention how the Kings incorporate analytics into contract negotiation and the salary cap. As a team gets closer to the cap (like the Kings after their Cup win), it is important to allocate scarce resources (cap space) as effectively as possible.

Dan MacKinnon first broke into the NHL with Nashville. His duties include organizing field work for amateur and pro scouting. He believes the major headway in hockey analytics will be in the “replacement player” valuation – finding a way to accurately assess the current labor supply outside of your own team. Can teams accurately gauge the impact of a substitute player on a team?

In terms of analytics, MacKinnon is a believer in shot location as a measure of a productive player. Regardless of what system a team employs, a player that gets more high quality shots on net is going to have a higher probability of generating goals.

Jim Price: the classic scouting of going to the arenas and watching talent will never go away, but the importance of metrics can’t be ignored. There are key statistics that teams should be looking at, based on their needs.

The next subject the panel dives into is extrapolating player performance at lower levels.

Solomon believes that the area requires further exploration, as he has read some interesting studies and research papers. Football appears to be well ahead of hockey in terms of draft pick valuation (a second round pick is worth a player of a certain level, for example) – that type of valuation isn’t as common in hockey just yet, but it should be, as every year at the draft GMs are bombarded with trade offers involving both picks and players. 

MacKinnon brings up Malcolm Gladwell’s 10,000 hour rule – it takes that amount of time for someone to become an expert at something. And amateur scouts are only seeing a fraction of a player’s performance, and of that fraction, the scouts may only record a part or portion of it. There is a high level of risk associated with projecting 16 and 17-year-old hockey players.

MacKinnon believes that the sport would benefit significantly if the developmental leagues would start to count a wider array of statistics (most notably shot quality and scoring chances) so NHL teams could use analytics more effectively when evaluating prospects.

Bowman believes that the analytics at the NHL level are pretty good right now, but the other leagues have a lot of work to do. The AHL started tracking ice time in 2011.

As a former agent, Solomon has the background of arbitration cases from the perspective of a player. The use of analytics is starting to creep into negotiations, as agents are starting to use analytics more to get a handle on how valuable their players are. The downside – the information in arbitration is discoverable, meaning that there is an obligation to turn over any research done to the other side – this would remove any competitive advantage held by an agent or by a team if they have developed their metrics in-house.

Another point – many times the arbitrators don’t know the sport of hockey well, so the statistics used tend to be very simple to understand and explain (goals, assists, plus-minus, and so on).

Solomon goes on to discuss the strategy that the Kings are using to build their team. The fact that they won the cup later after orchestrating some huge trades (Mike Richards and Jeff Carter) goes to show that they were eon the right track with their asset management. He mentions how the Kings brought up young players too quickly (Oscar Moller is an example of this), and they tried to move a little quicker than they should have – this includes trades for Ryan Smyth and Jeff Halpern. The team went back to the drawing board and completely rebuilt their foundation around Jonathan Quick, Drew Doughty, Dustin Brown, and Anze Kopitar.

The next challenge will be paying their players now that they have all experienced success at the NHL level.

MacKinnon discusses the James Neal trade (Pittsburgh acquired the big winger from Dallas in February of 2011). The Penguins could have held off and made the trade in July, but they wanted to show their players, who had been battling hard all season, that they were committed to winning now (with a deadline trade instead of one in the summer). There is a human element to trades, and players are often motivated by the threat of a trade, or the improvement of their team through a trade.

Overall, the panel did a very solid job of discussing the current state of analytics in hockey, what works and what doesn’t, and where the statistics industry needs to go to make further inroads into the sport.