Testing Projections for 2011

Each year, baseball fans and commentators acro s the nation make bold predictions about what they expect in the coming year. They often times make outlandish claims like Adam Dunn will hit 50 home runs in Comerica Park! or This is the year that Joe Mauer finally hits .400! but such predictions are far more apt to be Terron Armstead Jersey high than low. Sure, if you said Jose Bautista was going to summon greatne s going into 2010, you looked pretty smart, but anyone who predicts performance seriously recognizes that you need to hedge your bets. While frequently accused of being overly pe simistic about whoever your house Nine are, typically, they land high about as often as they land low. Seo of projection systems grows by the year, but there are significant differences together. Today, Ill evaluate their 2011 projections for hitters and pitchers.Firstly, lets peak in the candidates:MARCEL: Tom Tangos free projection system, intentionally utilizing a simple formula as a challenge to forecasters.PECOTA: Baseball Prospectus projection system available by subscription, run by Colin Wyers.OLIVER: The Hardball Times projection system available by subscription, operated by Brian Cartwright.ZIPS: Baseball Think Factorys free projection system, run by Dan Szymborski.CAIRO: Revenge from the RLYWs free projection system, run by SG.STEAMER: Free projection system, run by Jared Cro s, and his former students, Dash Davidson and Peter Rosenbloom.You can find out more about these projection systems here.HITTERSThe projection systems differ significantly with respect to their standard deviations of wOBA, with some hitting projection systems being particularly more risky in estimating the performance of players. The more risky a projection system, the more likely it will likely be wrong by a lot, which hurts its performance, particularly regarding its Root Mean Square Error. Thus, riskier projection systems might be right more regularly, but when theyre wrong, theyre very wrong. So, before we all do anything, lets rank the projection systems in terms of how risky they're:ProjectionStDev of WOBAOliver.0309Steamer.0289ZiPS.0287Cairo.0283PECOTA.0278Marcel.0234Marcel is going to have fewer big mi ses than Oliver will, so well want to look at both RMSE (which will punish risky gue ses) and Correlation (that will reward better player rankings), in addition to average absolute error (that will fall somewhere in between in terms of punishing and ignoring risky projections).Here may be the RMSE table, weighted by PA, and just including guys with a minimum of 200 PA. As you can see, PECOTA, a comparatively safe projection arrives ahead, even further ahead than Tim Lelito Jersey Marcel which is even safer. Ill likewise incorporate a row for last years stats to see how predict they are.ProjectionRMSEPECOTA.0317ZiPS.0318Oliver.0321Steamer.0322Marcel.0330Cairo.0333Last Years Stats.0388Oliver fared pretty much, despite its risky nature. It takes a step forward whenever you take a look at absolute average error.Absolute average error and root mean square error are differing in terms of just how much they punish bad performance. Take System A that mi ses on Player X by 20 points of wOBA and mi ses on Player Y through the same amount. Take System B that gue ses Player X exactly but mi ses on Player Y by 30 points. Average absolute error will favor System B, but RMSE will favor System A.ProjectionAAEZiPS.0244Steamer.0247Oliver.0247PECOTA.0248Marcel.0257Cairo.0264Last Year.0303ZiPS may be the champion of AAE, with its somewhat risky projections. They may be wrong by more when theyre wrong, but theyre Brandin Cooks Jersey right more often.If then we jump forward and look at correlation, we get a whole new winner. Correlation will probably be different because all correlation likes you is rankings for the most part. Should you projected Ryan Braun to have a .530 wOBA and Adrian Beltre to have a .430 wOBA, you'd have had a great projection year using correlations, even though Brauns wOBA was closer to .430 and Beltres was closer to .380. Correlation just wants you to rank the guys well. Using correlation, we obtain the following rankings.ProjectionCorrel.Oliver.6151ZiPS.6139PECOTA.6136Steamer.6039Cairo.5685Marcel.5614Last Year.4740Oliver arrives in front if you use correlation. Despite having perhaps overly aggre sive estimates of talent level, scaling back your Oliver projections may have been the best way to predict hitters.PITCHERSWhat B.W. Webb Jersey about pitchers? Well, the leaderboard will look quite different there. Following a number of these work, I include some ERA Estimators among pitching projections. This time around, Ill convert them into projections by regre sing ERA in 2011 against 2010 and 2009 versions of the ERA estimators. This produced the following formulas:SIERA_proj = .59*SIERA(10) + .26*SIERA(09) + 0.47xFIP_proj = .65*xFIP(10) + .24*xFIP(09) + 0.29FIP_proj = .43*FIP(10) + .30*FIP(09) + 0.94tERA_proj = .38*tERA(10) + .29*tERA(09) + 1.08The projections now have the following standard deviations of ERA among all pitchers with 40 IP in 2011:ProjectionStDev of ERAZiPS.7322PECOTA.7238Oliver.6356Cairo.5314Steamer.5207Marcel.4453SIERA_proj.4188xFIP_proj.3854FIP_proj.3829tERA_proj.3807Starting served by RMSEwhich should punish riskier projections, we see that it does exactly that:ProjectionRMSESteamer.8324Cairo.8736SIERA_proj.8746xFIP_proj.9014FIP_proj.9033tERA_proj.9050Marcel.9066PECOTA1.024ZiPS1.030Oliver1.042Last Years Stats1.282ZiPS, PECOTA, and Oliver all had the riskier projections and all sorts of fared the worse. Interestingly, despite being more risky than scaled back ERA estimators, Steamer and Cairo outperformed them at RMSE.What about average absolute error? The rankings look similar, though several projections swap places.ProjectionAAESteamer.7067SIERA_proj.7281Cairo.7331FIP_proj.7333xFIP_proj.7360tERA_proj.7361Marcel.7474ZiPS.7749PECOTA.7905Oliver.8009Last Years Stats.8766Steamer again comes out ahead. Moving to correlation, we have seen exactly the same kind of thing, though surprisingly, Marcel does better and Oliver does worse with correlation, despite its punishment of conservative projections.ProjectionCorrel.Steamer.4581Cairo.4213SIERA_proj.4089xFIP_proj.3763Marcel.3744FIP_proj.3739tERA_proj.3715PECOTA.3705ZiPS.3701Last Years Stats.3265Oliver.3163But on the 3, Steamer arrives ahead. Gurus Jared Cro s what was making his projections so good, and that he explained he was using velocity (as well as handedne s) in the pitcher projections Michael Hoomanawanui Jersey , which was giving them an advantage. He wasnt alone to point out doing something like this. I only started thinking seriously about this recently, but I think it truly is the next big thing in pitcher projections. Unlike hitter projections which seem to get down to which metric you want to use to test them, pitcher projections return Steamer in all three tests. Perhaps more interestingly, the better-known projections such as Oliver, PECOTA, and ZiPS, despite doing the best on hitters, they fare the worst with pitchers. Perhaps being good at projecting both pitchers and hitters is as rare to be good at doing each of them.Of course, these are all just one-year tests, there is a lot of luck involved for any of those. However, as all these systems moves forward to their next race, this is when they stand.