Rene Ritchie on Consumer Reports’s MacBook Pro Testing

Rene Ritchie, responding to Consumer Reports’s scathing but incredibly inconsistent battery life tests on the new MacBook Pros:

If I were running the tests, that right there would be a red flag. A huge, glowing, neon red flag.

Those results make very little sense and I’d take apart my chain, link by link, until I found out what was going on. I’d check and re-check my tests, I’d watch the systems like a hawk, and I’d do everything possible to find what was causing the variance. I’d even — gasp — try testing different machines and something other than web pages to see if that revealed more information.

Inconsistent results from battery life tests, for responsible publications, aren’t a reason to rush out a headline in time for the holidays. They’re a reason to start questioning everything, and to diligently retrace every step along the way, until you can get repeatable, reputable results.

I do think Consumer Reports rushed this out. There’s a lot of “We have no idea what’s going on” here. But something is going on.

Anecdotally, reports from DF readers are all over the map. Many are complaining that battery life is poor — not based on the “time remaining” estimate that Apple removed from the battery menu item in 10.12.2, but on real-world usage. Some though, are getting excellent battery life (as I did in my review, mostly using a Core i5 13-inch model with Touch Bar). Others are claiming they were getting poor battery life but it has greatly improved after upgrading to MacOS 10.12.2.

A friend pointed out the other day that this is where we really miss the old magazine testing labs, like Macworld’s. They’d buy all the various hardware models, test them thoroughly (and document the exact nature of the tests), and copiously report the results. It was a very useful service, and they were trustworthy.

Update: Phil Schiller, tweeting a link to Ritchie’s story:

Working with CR to understand their battery tests. Results do not match our extensive lab tests or field data.

Matthew Panzarino, in a series of tweets about Consumer Reports’s results:

Apple hasn’t given me anything on this, but I’ve had folks in know tell me that big data scoop (all MBP users) is NOT showing these results.

★ Friday, 23 December 2016