Consumer Report responds to criticism of their review

These guys at CR are acting like children. I'm surprised the big dogs would even allow this to be published. It reminds me of a 4 year old saying "neener neener neener! Your toy is bad! Hehehe "

 

When I woke up this morning, I remembered how Consumer Reports tests for three drivers I used to know:

• Al - a former Navy fighter pilot, he drove like a carrier pilot: max/min and darting into every gap in traffic. Riding with him, I started to get car sick.
• My wife - who believes the speed limit the minimum and delights in talking out of traffic tickets.
• Butch - drove a pony car and converted a 750 mile, 14 hour trip, into 12 hours by running 10 mph over the speed limit and only one stop.

My wife no longer drives but we used to share driving and kept the other, as passengers, awake. She would drive so fast the engine, wind, and rocking would keep me from taking a restful nap. But when she was a passenger, she would tell me 'a truck is passing you' even though I had the cruise control set to the speed limit.

Consumer Reports tests for drivers who don't really care about mileage and efficient driving and we've seen those type of drivers show up in Prius forums from time to time. Folks for whom they never get the MPG, hate the handling, and find the seat uncomfortable . . . a square peg trying to fit in a round hole.

We can use these characteristics to more quickly identify them and suggest 'the Prius is not for everyone.' If they are willing to change their driving style, the car rewards at the pump. But if they need to get there as fast as possible and in comfort, there are other cars that are designed for that goal. The sooner they recycle their car into someone's new ride, the better.

Bob Wilson

 

Uh oh, don't say they are the same, you are going to get yet another droning about the imperceptible and irrelevant fraction of a second superiority of the Liftback. From a test which no one witnessed and that fraction is within the margin of error, because there are "drag racers" on this thread, and they're experts in something you obviously don't care about anymore than I did. No point trying to be logical with people who demonstrate their bias precludes a rational understanding that their hobbies are unimportant to the subject of whether CR's metrics are relevant to the car class or target segment.

You are right overall, this thread should be about ridiculous CR response, I actually had no intention of posting about the tiny differences in acceleration other than it was one of many I found that didn't interest me as a target buyer in the least, 1/4 mile racing is one of the many metrics which CR weights too highly that appear part of an overall unapplicable and flawed methodology for evaluating a subcompact designed for high mileage. Frankly, I'm surprised with 2 spikes of over $4 for gas that someone higher up in this organization hasn't pulled this Tom aside and had them improve their testing methodology.

In many ways this Tom Mutchler reviewer is much like the "drag racers" on here, and their lack of self awareness in the way they argued with me could actually be useful examples into understand Tom's stubborn defensiveness. Being a "gear head" for so long he is certain that acceleration is a big deal, and resents the rest of the world questioning the relevancy of that viewpoint for evaluating this kind of car. After all, he knows cars, and insists the rest of the world acknowledge that on his terms. Rather than acknowledging a predisposition for attributes virtually irrelevant for a high-mileage vehicle, Tom doubled down in posting such a petulant, unscientific rationalization of why his viewpoint was right that I'm shocked his editors allowed it to be posted because he made CR look small.

I've been researching the C for months, (yes that means I'm biased the other way) but I read the 1st review with an open mind and am still convinced the C is a great value and the car for me, as literally nothing in the review gave me pause, or even struck me as relevant. I'd have the exact configuration I want today if all these other people who must agree weren't buying up all the supply. Hopefully for the sake of the buying public, the editors of CR will see Tom's response for the embarrassment that it is, and make a broader minded test for cars clearly pointed towards efficiency. Their own polling should show that that is now become the #1 attribute new car buyers consider, might be something to weigh more heavily than how quickly it reaches 80 mph, I'm just sayin.

 

Dude, you're so ridiculous far off base it's sad. Take a step back because your fanboism is clouding your reasoning ability. Look at what I drive. I've got nothing against the C. I'm also not a Prius fanboi either, so I can be objective. I'm telling you that the NOTICEABLY slower acceleration on the C was a factor in my purchase decision. I did strongly consider the lift back due to the superior acceleration which I noticed the second I drove the C, weeks before I saw any magazine test times.

You made a poorly informed statement that the acceleration difference is imperceptible, and it's definitely not, so I called you out on it. It's as simple as that. My issue is with your incorrect statement, not the C and it's performance.

 

AWDstylez you've said nothing to persuade me that I'm "offbase" other than your opinion, countless times, that you can discern the difference, and it is of some important to you, along with my offbassness for not agreeing with you about the importance of .4 seconds.

Let's agree to disagree, because I've heard all I needed to. You think like the author of this circular reasoned review, I don't value the same things you do and I'm not interested in discussing this minor point to death, let's move on.

 

I have to agree that Consumer Reports is influential. I've been browsing their car reviews for years just to see what's recommended at the moment. I knew that the Prius C was ranked 25 out of small cars BEFORE I went to test drive one to see if I would like it.

I would say I am a fairly average small car driver. I like to go fast, I like to accelerate fast, but I also enjoyed coercing my 2000 Saturn SL2 ever so nicely to "go, little car, go." But in the end, I think the crappy acceleration of the C (to be blunt it is) is a small price to pay for the mpg and for driving a hybrid which has fewer emissions and is a step in the sustainable living direction. Just owning it puts me in a better mindset to be eco-conscious which is something we all should be striving towards if we want our planet to continue to be enjoyable for future generations.

I actually did not like the interior of the Ford Fiesta, it felt too sporty and the backseat wouldn't fit anyone over 5' 10" probably. I liked the Honda Fit which continuously gets good reviews, but the 17 mpg higher rating on the C is enough to pay the extra $1500 dollars. The C seemed quieter and better designed. I love the touch screen and hands-free Bluetooth and navigation. It has lots of extras that I payed nominally more for. The seats were equally comfortable. The Fit's seats fold down a little better, but the C seemed to be the better car for me. And maybe it's not as fun to accelerate in, but it is more fun to drive, trying to learn to get better mpg and bragging about money saved. It is fun to see how your driving habits affect mpg. Of course, if I was in any other car, I would be yelling at me driving so slow off traffic lights, but once I get up to speed on the highway, I cruise just like anyone else. I drive it 72 mph on the highway normally or with traffic speed; hypermilers may scorn, but I like driving fast so I do. And I still get way better mpg than I would in any other car. 1.5 hours to LA, I normally get about 57 mpg downhill and 47 mpg uphill on the way home.

 

From the article:

Curious comments from a company that does not use mileage in their vehicle scoring . . . mileage hypocrites.

Bob Wilson

 

By scoring, I presume you mean the overall score ( 20-100) CR gives a car from designated road tests. While it seems clear that CR does not weight fuel economy high, how did you conclude it is not weighted at all ?

 

Read more: http://priuschat.com/threads/consumer-report-responds-to-criticism-of-their-review.110425/page-7#ixzz1xg7hNaXw​

These mpg numbers are CAFE, so 62 mpg production cars have been on the roads of the US since 2001. Is he giving the domestics 25-30 years to catch up ?​

 

Early this morning, I plotted CR scores as a function MPG:

Sources:

• Consumer Reports Buying Guide 2012, James Guest Editor, Volume 76, No. 13, pp. 144-145. Excludes electric vehicles Leaf and Volt from tables "HATCHBACKS: FUEL EFFICIENT," "HATCHBACKS: SUBCOMPACT," and "HATCHBACKS/WAGONS: SMALL."
• Consumer Reports, James Guest Editor, June 2012, pp. 51, 55, 57, 59.
• Consumer Reports, James Guest Editor, July 2012, pp. 58, 60-61, 63-64
• The linear trend line has a lower R**2 value than the power function.

Absent someone posting Consumer Reports formula for assigning scores, we can treat CR scores as coming from a 'black box'. Analysis of the independent versus dependent score, we can, with enough data points, derive a functionally identical formula. This will take time (at least a week) but give us insights to what CR has already reported in text descriptions. Thanks to "cwerdna" published links, we know MPG is not part of Consumer Reports scoring algorithm.

Looking at the June-July 2012 subset of this data, the Consumer Report score appears to give greatest weight to "noise" and "driving position" and lessor weight other metrics. Furthermore, there appears to be a step function based upon 0-60 mph time with the boundary inclusive between 10.1-11.2 seconds.

Bob Wilson

 

Read more: http://priuschat.com/threads/consumer-report-responds-to-criticism-of-their-review.110425/page-7#ixzz1xgTxHe3D​

Yep, steps in 0-60 times is my impression too.​

It does seem reasonable to me that fuel economy would not be in the score, since it is deemed important enough to have its own separate carve out in the review. It does come together with the road scores in 'overall value.' I think the same can be said for price.​

 

Great job with the graph! I love your logo (pic). I often think of the old cartoons and perhaps Little Rascals when they use a magnet to propel their vehicle. When a bigger vehicle comes along the highway, I usually try to draft it at about a second behind. It does wonders for the MPGs.

 

Great graph. Clearly you love statistics. What do you do for a living?

 

Hey-la-day-la! Bob Wilson's Back!

He went away and you hung around
And bothered me, every night
And when I wouldn't go out with you
You said things that weren't very nice

Bob Wilson's back and you're gonna be in trouble
(Hey-la-day-la Bob Wilson's back)
You see him comin' better cut out on the double
(Hey-la-day-la Bob Wilson's back)
You been spreading lies that I was untrue
(Hey-la-day-la Bob Wilson's back)
So look out now cause he's comin' after you

(Hey-la-day-la Bob Wilson's back)
(Hey, he knows that you been tryin')
(And he knows that you been lyin')

He's been gone for such a long time
(Hey-la-day-la Bob Wilson's back)
Now he's back and things'll be fine
(Hey-la-day-la Bob Wilson's back)
You're gonna be sorry you were ever born
(Hey-la-day-la Bob Wilson's back)
Cause he's kinda big and he's awful strong
(Hey-la-day-la Bob Wilson's back)

(Hey he knows I wasn't cheatin'!)
(Now you're gonna get a beatin'!)

(What made you think he'd believe all your lies?)
(Wah-ooo, wah-ooo)
(You're a big man now but he'll cut you down to size
(Wah-ooo, wait and see)

Bob Wilson's back he's gonna save my reputation
(Hey-la-day-la Bob Wilson's back)
If I were you I'd take a permanent vacation
(Hey-la, hey-la Bob Wilson's back)

etc....

Welcome back Great Warrior of y=mx+b. Give 'em hell.

 

I went to the library to look up past issues:

	model	0-60 sec	noise (1-5)	driving position (1-5)
1	VW Golf	9.4	4	4
2	Prius four	10.6	4	3
3	Lexus 200h	11.0	3	3
4	Insight	11.8	3	4
5	Prius c	11.3	3	3
6	Smart ForTwo	14.6	2	3
7	Caliber SXT	10.5	3	3

I've included the Caliber SXT (the outlier on the left) which had low avoidance speed, 49.5 mph, and long stopping, 149/157 ft suggesting one or both may also have a 'threshold' limit similar to 0-60 mph time.



Based upon this initial set of data, it looks like 11.0 seconds is "in the zone" and 11.3 seconds, out. Back of the envelope, missing the 11.0 seconds by 0.3 seconds gave the Prius c an ~18 point penalty in the Consumer Reports "score."

Bob Wilson

 

Hated Diff Eq - had a professor in college, back in the day, that all she did was start writing on the front boards and continued writing until she ran out of space. Then she would go to the side boards and do the same thing. When she filled those boards, she would erase the front boards and start the process all over again. And people wonder why they hate math!

I think she was afraid to actually teach, because she could have written all of that crap on ditto papers (Yes, that old!) and actually tried to teach us the material. Maybe she felt that when she passed out that many ditto sheets to everyone, that they would get high from snorting them. Yes, those were the days!!!

 

It had no use in my professional career beyond teaching me that 'there can be more than one equation that defines a result.' So when I read of some new 'physics' theory that promises to uproot and make all physics out of date, I know someone has found a new 'differential equation'. <grins>

I just appreciated how it tied all of calculus together ... another math fun to study but utterly impractical outside of Physics and some civil and mechanical problems.

Bob Wilson

 

You hit it right on the head, I did my undergrad in Civil Eng.

 

A couple decades ago, there was another car panned by Consumer Reports because of nitpicks, given a low enough overall rating such that it wasn't "Recommended," despite good objective test scores. Even so, it maintained above average reliability ratings and its road test results and other evaluations, taken on average, were as good or better than the highest rated cars in its class. Discussions like this one followed online, with the mathematically inclined trying to figure out what "best fit" of all the published data points could result in such a low overall score. My recollection is that none was ever found, because some other models that had low scores in the same categories did relatively well overall. I liked my Saturn SL2. It certainly wasn't a great car and had its share of issues. On the other hand, at the time anyway, it compared favorably with other models in its class. So, it's CR overall rating was a mystery, with the possible exception that the overall rating was pretty typical for any model from a non-Japanese manufacturer at the time.

Sometimes, CR's overall ratings really are a black box based on subjective quibbles and don't seem to reflect some weighted average of the objective results they publish. I suspect it will always be like that, as any magazine has editorial biases, even those that don't accept advertisers. Ultimately, like any magazine, you find you either agree with their editors or not. If you're like me, you take their advice as only one of many resources. In my experience, their top rated products are usually a decent bet, but I've been burned enough times to know that their opinions don't always reflect what I find after buying a product.

I will say one thing I find interesting. Historically, CR tends to gush over Toyota and Honda models. While Subaru and Mazda have moved up recently, all four are still the top rated makes on average. It's not all that common to see such a low rating on one of their cars. The new Civic and Prius C really seem to buck the trend of glowing reviews for most cars from these makes. Yes, I know there are other exceptions, like Yaris and FJ, but most other makes have much worse. Consider that the Prius C's rating of 53 is about the same as the overall average rating for all models from GM (56) or Chrysler (51), according to the April 2012 issue. You'll find a lot of very similar discussions on GM, Ford, Chrysler and VW forums...

Good to see you back, Bob W!