By The NumbersHockey Analytics... the Final Frontier. Explore strange new worlds, to seek out new algorithms, to boldly go where no one has gone before.
I don’t think that anybody has done a comprehensive study about playoff save percentage. I thought it was an important enough topic to spend a couple of hours analyzing the data.
To adjust playoff save percentage, two adjustments are required. First, saves are normalized to a 90.5% save percentage environment. This is calculated for each goalie each year, and the goalie’s shots and saves are removed from the league for the purpose of that calculation. Second, these numbers are adjusted to an environment where goalies face 28.6 shots per game. This won’t impact save percentage in any year (as shots and saves are adjusted by the same amount), but it ensures that a goalie’s performance in a year that features many shots per game (such as 2011) is not weighed more than a goalie’s performance in a year that features few shots per game (such as 2001) when calculating career averages.
I haven't attempted to account for the fact that a goalie on a strong team will be able to play more games due to having better teammates, facing an easier first round opponent, having home ice advantage, etc. These are important things to consider, but I can't quantify them.
I’ve used data from 1984 to 2011. All numbers are taken from hockey-reference.com. I realize that playoff save percentage exists going back to the 1950s, but this is the only usable data that I have. If someone wants to continue this project going farther back, I’d welcome it.
I’ve stated before that save percentage is, in my opinion, the single best statistic to measure goalie performance. That being said, I think that save percentage is more reliable in the regular season than in the playoffs for a few reasons. First, the sample sizes are much larger, which means that one can have more confidence in the numbers. Second, the strength of opponents varies widely in the playoffs (a goalie can play 300 minutes of hockey entirely against the best team in the NHL). Third, many teams play more defensively in the playoffs – on average, I think that many teams surrender less dangerous shots, which would, all things being equal, overstate save percentage in the playoffs. Still, to the extent that save percentage is used, it should be adjusted for era.
My purpose isn’t to present one number which is a perfect representation of a goalie’s performance. Rather, I want to improve on what has already been quantified in conventional statistics.
Last edited by Hockey Outsider: 05-07-2012 at 11:58 PM.
This table shows why it's critically important to take the era into consideration when evaluating goalies' playoff performances. For example, Grant Fuhr posted a seemingly unimpressive 89.9% save percentage between 1984 and 1988, when he helped the Oilers win four Stanley Cups in five years. Adjusted for era, Fuhr stopped 91.8% of the shots he faced during those four seasons. That's not quite elite, but it's a very strong peformance over a large sample size (79). That doesn't even take into account the strong likelihood that Fuhr faced tougher quality shots than average due to playing on a run-and-gun team.
Keep in mind that career save percentage is, by definition, a career average. Tom Barrasso had a few rough playoffs at the start and end of his career, and that dragged down his average. His career average of 90.8% is barely above average; if one focuses on his prime from 1988 to 1996, Barrasso's save percentage rises to a very strong 91.6%.
Patrick Roy is tied for the second highest career save percentage out of any goalie who faced at least 1,000 shots (Roy faced nearly five times as many shots as Thomas). He's also faced 33% more shots than the next closest goalie (Brodeur). No goalie during the past thirty years has surpassed (or even approached) Roy's combination of an extremely high level of performance, and longevity.
Osgood is slightly above average at stopping the puck. I think the Hall of Fame should be a balance between ability and accomplishments. Osgood has is consistent and durable, but is only slightly above average ability. In my opinion, he shouldn’t get a spot in the Hall due to having the fortune of spending most of his career playing behind the best franchise of the past two decades.
Update for 2012: Fleury has the second-worst adjusted save percentage after a number of horrific games in the spring. Thomas' save percentage drops slightly, but he still has the highest adjusted playoff save percentage 1983-present (minimum 1,000 shots).
Last edited by Hockey Outsider: 10-16-2012 at 04:25 PM.
I realize that 93.0% is an arbitrary threshold, but it's a pretty good summary of the best playoff performances of the past thirty years.
As I said in the previous post, there is little doubt that Roy is the greatest playoff goalie of the past three decades. He has three of the top seven performances, and five of the top thirty-one. He performed at an exceptionally high level on five different occasions where his team made the Stanley Cup finals, and he was a major reason why they were victorious four times.
Brodeur doesn't get enough credit for his spectacular performance in 1995. His 92.7% save percentage looks strong on paper, but it's even more incredible when you consider that the league average was only 89.3% that year (88.9% after removing Brodeur's shots and saves). I am adamantly opposed to the idea that Brodeur deserved the Smythe in 2003, but arguably he deserved it in 1995.
Update for 2012: there are two additions to this list. Smythe winner Jonathan Quick ranks 12th all-time. Mike Smith also had a very strong postseason.
Last edited by Hockey Outsider: 10-16-2012 at 04:51 PM.
Only eight goalies have won the Stanley Cup while posting an adjusted save percentage of 93.0% or higher. Six of those netminders won the Conn Smythe. As for the other two - as incredible as Barrasso was in 1991, I don't think he was more valuable than Lemieux's insane 44 point performance. The more I think about it, the more I think that Brodeur should have won the 1995 Smythe.
Detroit won three Stanley Cups in six years with only above-average goaltending. Vernon, Osgood and Hasek played well enough not to cost the powerhouse Red Wings any series, but rarely stole any games. I was critical of Osgood in a previous post but, to his credit, he played very well in 2008.
Only one of the past twenty-seven Stanley Cup winners posted a below average save percentage: Marc-Andre Fleury. He finished 0.7% below the league average in 2009. It's easy to bash Fleury after his abysmal performance this spring, so I won't belabour the point.
Update for 2012: Quick was phenomenal this spring. He has the fifth best adjusted save percentage among Stanley Cup winning goalies.
Last edited by Hockey Outsider: 10-16-2012 at 04:29 PM.
Over the past 28 years, ten goalie have won the Conn Smythe. The first seven listed above were strong selections; although there were other strong candidates in some years, it’s impossible to argue that the actual winners were poor selections.
Cam Ward was very good in 2006, but I would have given the Smythe to Chris Pronger (no defenseman has ever won the Smythe while failing to win the Stanley Cup). I probably would have given the Smythe to Brind’Amour ahead of Ward as well.
Wayne Gretzky deserved the Smythe in 1997, but probably fell victim to the impossibly high expectations others had of him. Although his 34 points in 23 points are staggering, it was weaker than his previous four playoffs! Ron Hextall’s save percentage is almost certainly understated since he faced so many shots from the incredibly dangerous Oilers.
I don’t think that Mike Vernon had a good case for winning the Smythe in 1997. He was solidly above average, but that’s not Conn Smythe material. There were several better candidates including Fedorov (led Wings in scoring and provided exceptional two-way play), Shanahan (second on Wings in scoring and was credited for giving the Wings the toughness and grit they lacked in previous years) and Lidstrom & Murphy (for shutting down Lindros so effectively in the Cup finals).
Update for 2012: Quick was phenomenal this spring. He has the fifth best adjusted save percentage among Conn Smythe winning goalies.
Last edited by Hockey Outsider: 10-16-2012 at 04:30 PM.
This is a concept developed by Taco McArthur – link. Essentially, it shows how many games a goalie would be expected to win, had they played on an average team. I’m not sure if I like this or Wins Added more (the latter is a statistic I created), but TM’s statistic is far easier to calculate and gives fairly similar results, so let’s go with his! The chart above shows the results for all goalies with 30+ decisions.
Roy’s dominance continues. He has the most Support Neutral Wins by a massive margin. He also has the best win percentage out of any goalie with 60+ decisions. Once again, there is little doubt that Roy is greatest playoff goalie of the past three decades.
Tim Thomas has a staggering SNWL record. He has an exceptional win percentage, and his 50 decisions are a substantial number – only 27 goalies have earned the decision in more than fifty playoff games.
Fuhr actually went 90-47 in the playoffs (from 1984 onwards). The numbers suggest that he would have been 72-65 on an average team. That means that Fuhr was a very good goalie (and even better during his prime), but he clearly benefited by playing on the highest-scoring team of the modern era.
For all the criticism he's received, Joseph ranks 5th in playoff wins and has the 9th best win percentage of the 30 goalies who earned 50+ decisions. I don't think he's a HOF goalie, but he's often unfairly criticized (especially in Detroit, when his teammates scored just 1.88 goals per game, despite finishing 1st and 2nd in regular season scoring those two years).
Update for 2012: Brodeur extends his lead over third place. Despite a first-round exit, Thomas breaks the 30 win barrier.
Last edited by Hockey Outsider: 10-16-2012 at 08:39 PM.
Brodeur doesn't get enough credit for his spectacular performance in 1995. His 92.7% save percentage looks strong on paper, but it's even more incredible when you consider that the league average was only 89.3% that year (88.9% after removing Brodeur's shots and saves). I am adamantly opposed to the idea that Brodeur deserved the Smythe in 2003, but arguably he deserved it in 1995.
I came to the same conclusion about 14 months ago.
Adjusting to the regular season average made more sense to me though. For one, it's a larger sample size that includes all NHL goalies. More than that, it eliminates the issue of a 1995 Dominik Hasek going from an NHL best .930 to a .863 and helping throw the average off from where one would expect it to be from having watched the season. After all, 1995 was one of those rare three seasons in which the playoff save percentage was lower than the regular season save percentage (which really shouldn't happen for any reason other than sample size, given that half the leagues worst teams are out).
Very interesting stuff - I'm curious to see how it compares to what I've got in my database.
Agreed on Brodeur's non-Smythe in 1995; on the whole, he probably has the right number of Smythes, but they're not allocated optimally to my tastes.
I'm also working on a concept called "par", which Bill James used to use to measure managers' success. I'm not sure how well it will work for goalies, but I like what I'm seeing so far.
Agreed on Brodeur's non-Smythe in 1995; on the whole, he probably has the right number of Smythes, but they're not allocated optimally to my tastes.
You'd have given Brodeur his zero Conn Smythes in different years than he didn't win them?
Either way, 1 Conn Smythe seems about right for his career, but that never works as a measuring stick for players anyway (unless the question is "was Patrick Roy awesome? Y/N"), since only one person on one of two teams wins them every year.
Yeah, Irbe does pretty poorly from a career perspective when you consider that includes one very stellar year.
Hah yea, the first couple of san jose years he had some great high light games...then followed with some blow outs that really hurt him. I still wish dallas would have given him a chance that year he was with them. Moog wasnt doing anything, and irbe actually played pretty well that year at times. Woulda been nice to see him get some playoff action with a team that was actually expected to win. Oh well...
I think anyone using or assuming Gausian distribution in their adjustments should take a look at the results of this large study which included NHL players.
I'm not an expert in this so I would encourage interested readers to read the study rather than simply accept my summary.
I saw some criticism but nothing that seemed more deep than a casual defense of the bell curve.
In short the idea seems to be that a small percentage of elite performers including outliers have a disproportionately large impact on the average result. The rest fall below the mathematical average of the group.
The other end of the scale, negative performances have the same impact and no they don't balance out.
I suppose one could say that the outliers at both ends are the data. Eliminating or normalizing them distorts the data.
So when examining data for a season say goals scored or save % one must look at the outliers for explanation since they are most responsible for the data. Adjusting them to make the curve smoother or bell-like is wrong. That just ignores and distorts the most meaningful data.
It appears to support the 80-20 rule. 20% of sales people are responsible for 80% of the sales. Within that group the same rule applies. Furthermore this appears to apply within a team, season or career.
Similarly a small percentage of the group is responsible for most of the negative stats.
So a season with a few really bad goalies would greatly impact the overall data just as a season with a few really good goalies.
Normalizing this data creates the false impression that the group as a whole were better or worse. It would also serve to lessen the performance of the good goalies or enhance the performance of the bad goalies.
I did see a link to using Excel to calculate power curves which may interest those of you who've amassed the raw data.
The first two paragraphs-
"We revisit a long-held assumption in human resource management, organizational behavior, and industrial and organizational psychology that individual performance follows a Gaussian (normal) distribution. We conducted 5 studies involving 198 samples including 633,263 researchers, entertainers, politicians, and amateur and professional athletes. Results are remarkably consistent across industries, types of jobs, types of performance measures, and time frames and indicate that individual performance is not normally distributed—instead, it follows a Paretian (power law) distribution. Assuming normality of individual performance can lead to misspecified theories and misleading practices. Thus, our results have implications for all theories and applications that directly or indirectly address the performance of individual workers including performance measurement and management, utility analysis in preemployment testing and training and development, personnel selection, leadership, and the prediction of performance, among others.
Research and practice in organizational behavior and human resource management (OBHRM), industrial and organizational (I-O) psychology, and other fields including strategic management and entrepreneurship ultimately build upon, directly or indirectly, the output of the individual worker. In fact, a central goal of OBHRM is to understand and predict the performance of individual workers. There is a long-held assumption in OBHRM that individual performance clusters around a mean and then fans out into symmetrical tails. That is, individual performance is assumed to follow a normal distribution (Hull, 1928; Schmidt & Hunter, 1983; Tiffin, 1947). When performance data do not conform to the normal distribution, then the conclusion is that the error “must” lie within the sample not the population. Subsequent adjustments are made (e.g., dropping outliers) in order to make the sample “better reflect” the “true” underlying normal curve. Gaussian distributions are in stark contrast to Paretian or power law distributions, which are typified by unstable means, infinite variance, and a greater proportion of extreme events. Figure 1 shows a Paretian distribution overlaid with a normal curve."
I wouldn't expect anything in the NHL to reasonably follow a bell curve - since for anything you're measuring a hockey ability by, the sample set of National Hockey League players represents the far right end of a bell curve.
For instance, suppose that the NHL-average save percentage is 91%. There's a hell of a lot more people out there who could post save percentages four standard deviations beneath that than there are who could post save percentages four standard deviations above that.
The results of the study suggest that the data doesn't fit a Bell curve at all. No matter what end you're talking about. The data fits a power curve.
The authors reveal the software they use and say it works with excel.
"Results reported in Table 1 show that the Paretian distribution yielded a superior fit than the Gaussian distribution in every one of the 54 scientific fields. Recall that a larger chi-square value indicates worse fit and, thus, can be considered an index of badness of fit. As Table 1 shows, the average misfit for the Paretian distribution was 23,888 whereas the misfit of the normal distribution was larger than forty-four trillion (i.e., 44,199,201,241,681)—a difference in favor of the Paretian distribution in the order of 1:1.9 billion. Figure 2a displays a histogram of the empirically observed performance distribution of researchers. To interpret these results further, consider the field of Agriculture (see Table 1). A normal distribution and a sample size of 25,006 would lead to approximately 35 scholars with more than 9.5 publications (three standard deviations above the mean). In contrast, our data include 460 scholars with 10 or more publications. In other words, the normal distribution underestimates the number of extreme events and does not describe the actual distribution well."
Adjusting to the regular season average made more sense to me though. For one, it's a larger sample size that includes all NHL goalies. More than that, it eliminates the issue of a 1995 Dominik Hasek going from an NHL best .930 to a .863 and helping throw the average off from where one would expect it to be from having watched the season. After all, 1995 was one of those rare three seasons in which the playoff save percentage was lower than the regular season save percentage (which really shouldn't happen for any reason other than sample size, given that half the leagues worst teams are out).
I debated whether to compare the numbers to the regular season or playoff average. I used the playoff average because save percentages usually increase in the playoffs (teams are generally more disciplined and conservative, and arguably some of the weaker goalies don't qualify). Thus a 91.5% save percentage might be good in the regular season but merely average in the playoffs.
Quote:
Originally Posted by Taco MacArthur
Nice to see Kirk McLean up there!
Thanks! McLean benefits because substantially all of his playoff career coincided with his peak (88% of his career playoff games occured between 1992 and 1995). Contrast that with, say, Tom Barrasso, who played a lot of games before and after his peak. Still, a very impressive showing.
Quote:
Originally Posted by seekritdude
poor poor irbe. ._. And ill chip in my obligatory broken record of I think he should have won the conn smythe in 2k2.
Irbe is tough to evaluate. Brutal numbers in San Jose (86.7% adjusted save percentage), but a strong showing in Carolina (91.4%). Part of that was Irbe improving with age, but it's also partly due to the Hurricanes being better defensively than the Sharks.
I think anyone using or assuming Gausian distribution in their adjustments should take a look at the results of this large study which included NHL players.
Here's a chart showing the distribution of adjusted save percentages. I'm only using goalies with 300+ minutes; that threshold is arbitrary, but there needs to be some kind of arbitrary threshold or else we'd be looking at statistically meaningless results for goalies who faced only a few dozen shots.
The largest single category (90% to 91%) straddles the mean and features nearly 25% of all observations. Although there are more observations above the mean than below, the distribution appears to follow roughly a bell curve.
If the average save percentage was 90.5%, but most goalies were actually stopping say 85% but Patrick Roy and a few other superstars were stopping 95%, than another distribution (and therefore another method to evaluate goalies) might be more meaningful. Let me know if I've misunderstood your post.
I don’t think that anybody has done a comprehensive study about playoff save percentage. I thought it was an important enough topic to spend a couple of hours analyzing the data.
Just curious... Have you been able to also make some kind of adjustment for shots against when playing penalty killing? I would think that you (or others), even if not providing it here, may have thought about it? If you (or someone else) have looked, are there any particular things/patterns that stands out?
Just curious... Have you been able to also make some kind of adjustment for shots against when playing penalty killing? I would think that you (or others), even if not providing it here, may have thought about it? If you (or someone else) have looked, are there any particular things/patterns that stands out?
I haven't made an adjustment for penalty killing. (For those who are unfamiliar, what we mean is that a goalie who plays on a (un)disciplined team would face a disproportionately low (high) percentage of powerplay shots, which are significantly more likely to result in a goal).
All the data I used was taken from hockey-reference.com because it's well formatted and easy to use. That website doesn't have ES/PP/SH shot and save breakdowns. I know that data exists on NHL.com back to 1998, but it would be very time consuming to merge the data from those two websites. I'm not sure if that type of data even exists prior to 1998.
In summary, although I think it would be valuable to take that information into account, but I'm not sure if it exists prior to 1998, and I'm not willing to spend the countless hours it would take to merge and clean the data after 1998.
If the average save percentage was 90.5%, but most goalies were actually stopping say 85% but Patrick Roy and a few other superstars were stopping 95%, than another distribution (and therefore another method to evaluate goalies) might be more meaningful. Let me know if I've misunderstood your post.
Ultimately, it's not going to be a normal distribution, but (something) reasonably approximating the right-end tail of one. There are thousands of goaltenders out there who could post an NHL save percentage six standard deviations lower than the NHL average, and on the other hand, there are none (because none would get that opportunity).
The other poster is suggesting that it follows a power distributions, and he's probably right (I've written and spoke enough on the topic to know that power distributions handle outliers far more reasonably). I don't think that it matters too much in this context.
I haven't made an adjustment for penalty killing. (For those who are unfamiliar, what we mean is that a goalie who plays on a (un)disciplined team would face a disproportionately low (high) percentage of powerplay shots, which are significantly more likely to result in a goal).
All the data I used was taken from hockey-reference.com because it's well formatted and easy to use. That website doesn't have ES/PP/SH shot and save breakdowns. I know that data exists on NHL.com back to 1998, but it would be very time consuming to merge the data from those two websites. I'm not sure if that type of data even exists prior to 1998.
In summary, although I think it would be valuable to take that information into account, but I'm not sure if it exists prior to 1998, and I'm not willing to spend the countless hours it would take to merge and clean the data after 1998.
I can understand that. It was just that I remember someone doing it for the regular season, and that it might have been you. I haven't looked into the playoffs at all myself, and have no idea how much that kind of adjustment would change stats. For example, Philadelphia used to take many penalty minutes during the regular season, and if that meant they played more PK than most other teams, it may even further push their goalies up (Lindbergh. Perhaps Parent?) (But this is to me completely hypothetical as I don't even know if they played more PK during the playoffs than other teams.)
I think you have a few duplicates in your third table, the one showing Stanley Cup winners. I noticed it when counting the occurences of Patrick Roy's name, but also Fuhr occurs twice. Roy anyhow looks impressive! Before I started spending time here, I suspected he might have been somewhat overrated, because I've seen him make many mistakes too (especially when handling the puck behind/around the net). But in threads like this one he (or his stats) lives up to his high reputation.
I can understand that. It was just that I remember someone doing it for the regular season, and that it might have been you. I haven't looked into the playoffs at all myself, and have no idea how much that kind of adjustment would change stats. For example, Philadelphia used to take many penalty minutes during the regular season, and if that meant they played more PK than most other teams, it may even further push their goalies up (Lindbergh. Perhaps Parent?) (But this is to me completely hypothetical as I don't even know if they played more PK during the playoffs than other teams.).
Since we don't know the PPOA for the teams in the playoffs before a certain year, this would have to be estimated using team PIM. But we do have enough information available to draw reasonable conclusions: The fighting majors earned by the teams in the regular season, the number of PIMs they had, and the number of PPOAs those PIMs resulted in. You would get results that are pretty close, I think.
If anyone really wants to do the work, they can scour the HSP game-by-game to find exactly how many PPs each team faced.
You are right that facing a higher percentage of your shots from the PP is going to drag down your sv%. Adjusting for actual shot quality is probably more important than just situational adjustments, but particularly for pre-lockout seasons where that data doesn't exist I think situational adjustments get us closer to the truth than further away.
Ultimately, it's not going to be a normal distribution, but (something) reasonably approximating the right-end tail of one. There are thousands of goaltenders out there who could post an NHL save percentage six standard deviations lower than the NHL average, and on the other hand, there are none (because none would get that opportunity).
The other poster is suggesting that it follows a power distributions, and he's probably right (I've written and spoke enough on the topic to know that power distributions handle outliers far more reasonably). I don't think that it matters too much in this context.
Thanks for the explanation - I have a better understand of that now.
Quote:
Originally Posted by plusandminus
I can understand that. It was just that I remember someone doing it for the regular season, and that it might have been you. I haven't looked into the playoffs at all myself, and have no idea how much that kind of adjustment would change stats. For example, Philadelphia used to take many penalty minutes during the regular season, and if that meant they played more PK than most other teams, it may even further push their goalies up (Lindbergh. Perhaps Parent?) (But this is to me completely hypothetical as I don't even know if they played more PK during the playoffs than other teams.)
Yes, I looked into it with my recent post about 2009-2012 regular season stats. I think shot situations are worth taking into account, though of course there's a trade-off. In this case, I decided that the additional information that this adjustment would produce is not worth the dozens of hours it would take me to accumulate, organization and analyze the data. If anyone is interested in doing this, I can send them what I've worked on thus far.
Quote:
Originally Posted by plusandminus
I think you have a few duplicates in your third table, the one showing Stanley Cup winners. I noticed it when counting the occurences of Patrick Roy's name, but also Fuhr occurs twice. Roy anyhow looks impressive! Before I started spending time here, I suspected he might have been somewhat overrated, because I've seen him make many mistakes too (especially when handling the puck behind/around the net). But in threads like this one he (or his stats) lives up to his high reputation.
Thanks, that's a good catch. I updated the table.
Had you asked me five years ago, I would have said that Hasek was the greatest goalie of all-time by a sizable margin. Although I still have Hasek first, I now rank Roy a very close second. Relative to their peers, Hasek was only marginally better at stopping the puck than Roy (though I do recognize that Hasek played against a better peer group).
