Consumer Report responds to criticism of their review

We have 73,000 miles now on our Prius. The lowest MPG calculated was 40.8 mpg, that was
in winter, low temps in 20's, unusual in the Northwest, short trips under one mile and warmups
before leaving in the cold.

My wife mainly drives and she does not drive for MPG, and still we are averaging 53 mpg calucalted for the
last month or so. Winter places mpg's in the mid to upper 40's range, all calculated of course. This
is with reg gas with 10% Ethanol. Summer in low to mid 50's, trips will bring that figure to the
upper 50's mpg's.

I don't know how Consumer Reports was driving, but they must have been driving like a bat out of, well
you know what I mean....

 

You know this thread is about new Prius C just came out in March, right?

 

therein lies the "bias". they are testing the car in a manner that is inherently disadvantageous to hybrids. YET they make NO mention of it. is it ignorance? arrogance (in their own test cycle)? or perhaps it's intentionally turning a blind eye? THAT is what i have a problem with.

so if they use ONLY their test cycle numbers, and not real world numbers like we all get, and other reviews for that matter, what are they left to judge? the crappy interior? the high price? of course the review is going to turn out the way it did.

However, if they forgot about their test cycle, and are getting the mpg number like we the actual customers are getting. perhaps their perspective will change. 43 mpg vs 55+ is a huge difference. more than enough to overlook the cheap interior and wind noise....

in the end, their reliance on their own, FLAWED test cycle is the cause of all this controversy...

 

Thanks! This is what I sent them:

There are car reviewers who pander to the 'performance car' owner. But other than smiling at the waiting room copies, I don't buy them. I do buy AutoWeek because it has fairly and accurately covered new cars including hybrids. But Consumer Reports has pretty well defined their review standard, a different set of standards than I choose to follow today.

Bob Wilson

 

Yes they can, and must. Anything else would be totally unscientific.

They buy all the cars they test and run every car through the EXACT SAME TESTS. This is the only way to produce a true comparison of cars.

 

So what do you propose they do? Create a new test cycle that is geared toward hybrids? Then wouldn't that dis-advantage the non-hybrids?

Even if their test cycle disadvantages hybrids, which I'm not sure is true, if they were to change it now, it would make it more difficult to compare cars based on the new test vs the old test. I'd rather see them keep the test the way it is, if for no other reason than for continuity with previous tests.

 

Nope. It's the opposite of scientific. It's not scientific because, 1) They're not driving it like people in the real world (neither casual nor hypermiler hybrid owners) do, which leads to, 2) results that are completely different than 99% of the results seen in the real world.

If a "scientist" created conditions in the lab- even if they were consistent- that showed gravity pushing objects away from each other rather than pulling them together, that wouldn't be good science, because that's nothing like what everyone else sees. Sorry, but gravity is real, and the Prius c gets well over 50 mpg (when not even trying), not 43 mpg. If you say 43 mpg, you're simply wrong. Wrong does not make for good science.

You wouldn't take a Peterbilt out to the race track and measure its handling at 100 mph, because those aren't the conditions under which the truck is going to be used by 99% of the people out there. Nor do people care about that. So why would you go out of your way to destroy the measure of the primary purpose (mpg) of a hybrid?

Coffee makers and space heaters are both home appliances, but you wouldn't measure how well a coffee maker keeps your house warm.

What they need to be doing is considering the demographic and purpose of a car, then going out and using the car for that purpose- i.e., evaluating the car under the same conditions that it will be used in the real world- by the CONSUMER. THAT is scientific- keeping the conditions under which the car is evaluated, and then used by the consumer, the same.

 

Unless you propose that they change the test for ALL CARS, and test all cars the same way, the results of such a test would be totally useless, unscientific information.

Another problem with what you are proposing is that as soon as you try to create a specific test for a certain demographic, you run the risk of creating a bias. Who would be the authority that would decide what the demographics and purposes of different cars are and what different tests should be done on them?

They may actually already have a special test that is only run on certain cars. This would be the extreme rock climb test that they run on certain 4 wheel drive vehicles. The only reason they would not run that test on all cars is because it might damage cars that are not designed to be driven on that course.

If you want to propose that they devise a special "hybrid demographic" test and run that test only on hybrids, and publish those results SEPARATELY, then fine. But they would still need to run the hybrids though the same exact tests that all cars are run through. Otherwise there would be an obvious bias.

 

They could add a special self adhesive fanboy section to their automotive reports and the fanboys could stick it over the normal report so they can't see the normal report.

 

The difference here is that one of you is looking at this as a scientific comparison of all cars, in which case, yes, the test should be exactly the same for all cars. (You might want to consider, though, whether the current test is biased towards a certain type of car. Just because it's the one they've always been doing doesn't mean it's the right test. But that's a whole seperate issue.)

The other is looking for reviews geared towards the specific types of cars/demographics likely to purchase the cars. In this case, there should be different tests for different types of cars. Ones that show off the strengths, but also provides insight on the weaknesses.

Without first agreeing on the purpose of CR reviews, you will never agree on what the tests should look like or accomplish. The argument is pointless.

 

Or, they could keep it the way it is and put a disclaimer at the top that says, "WARNING: COMPLETELY USELESS, INACCURATE INFORMATION THAT WILL HAVE NO RELEVANCE TO YOU IN THE EVENT THAT YOU ACTUALLY BUY THIS PRODUCT."

They could even change the title of the magazine to "Non-Consumer Reports," since they don't measure their products with regard to the uses for which they are intended. I'd look forward to their review of how well each brand of scissors cuts the average lawn. One must make sure that scissors and lawnmowers are evaluated on an even playing field.

 

Consumer Reports already categorizes the cars that they review. The criteria for the review is not usually an issue. However, with any hybrid or all electric vehicle, the manner in which the data is gathered and then interpreted, is the issue. Starts and stops in these vehicles may or may not be indicative of real world usage. If you have had experience with these cars, you will understand that CR's testing will NOT be representative of real world results. However, if you drive this car like a non-hybrid, CR's results could very well be representative of what you will get.

As for the other "knocks" on the Prius c, they are the typical reactions to the "value engineering" car manufacturers are resorting to these days, especially at the entry level price points Toyota is trying to hit with this car. I certainly wish my Prius c 2 took the bumps a little gentler, but it isn't terrible! Love the car!

 

Everyone knows that we would all drive a Ferrari exactly the same as our Prius c and vise versa. Come on, how silly is this kind of thinking? I bought a Prius to save money on long trips and commutes. To do this, we drive our Prius gently as traffic allows in the city and at steady speed as possible on the highway. We get great MPG and that's what we have the Prius for. I may buy a Pc as a second car or my main car, since wife uses the Hatchback for commutes.

I've written about this in another thread... CR does a fine job testing and comparing things like vacuum cleaners, washer/drier combos, blenders and other electronic wonders, but when it comes to some of their vehicle tests, they miss the boat completely.

If I wanted a car that held 6 people, our mountain bikes and gear, I would probably look for a big van. If I wanted to go autocrossing, I probably would leave that big van at home and choose a Miata or a Lotus Elise or even a McLaren F1, if I had that kind of money. If I wanted a city/suburban car that got exceptional MPG in stop and go traffic I would choose a Prius c.

So how do you test cars using the exact same set of parameters for comparison? How would this Consumer Report go?...

"Gee, a Prius c holds one more person and has more cargo space than the McLaren F1 tester we drove. And the McLaren engine makes so much noise! Way more noise than a Prius c engine could ever generate. And then there's the way the McLaren F1 handled on the track, it was so blasted fast on the test track that it frightened us. Thus, unfortunately we here at Consumer Report cannot recommend the McLaren F1, and must now go change our underwear."

Apples and Oranges.

 

Anybody who is interested in how Consumer Reports actually tests cars can find more info here: How we test

 

Hmm...a little defensive eh Consumer Reports?

I think it's a fallacy that most of the criticism of their review came from the line that those considering a Prius c should buy a used Prius instead.

I know in my response to their review, I actually thought that observation was one of the fairer things they did say. I think it is fair to suggest that someone looking to buy a Hybrid consider a Used Prius, or regular Prius....of course I would say compare all 3....but at least I think it's a fair statement to make.

No sorry Consumer Reports, that's not why you are being criticized. You are being criticized because the review was so obviously biased.

I feel no inclination to repeat my entire rather long litany of all the specific unfair spin I think they included in their original review. But I just have to laugh. Their defense? Is basically to once again attack The Prius c, with such hard hitting, and tangible "facts" as "nobody likes to drive it". Their tagline of "It's a cheap car with an expensive drive train, doesn't even make sense. If it has an expensive drive train? It's not a cheap car.

Yes, it's a subcompact, built to the fit, finish and amenity level of similar subcompacts. But in terms of a personal climate control system, Multi-Informational Display, and of course Butt Kicking Hybrid efficiency it surpasses all of it's almost non-existent competition. That's what Consumer Reports fails to recognize.

I own a Honda Fit...which is generally loved by reviewers. I've sat in and looked at a Prius c...and while I'd say YES, I do like the interior layout, fit and finish of my Honda Fit better....if that is all you are comparing, I'd also say, the two vehicles from that P.O.V. are pretty close. There are minor differences but they are very similar.

No Consumer Reports, IMO you are not being criticized because evidently nobody on your staff enjoys driving the Prius c, In my opinion you are not being criticized for recommending a used Prius instead.

You're being criticized because your review is unfairly negative, and spun so obviously to ignore positive attributes of the Prius c, and accentuate negatives.

I've heard "feels like a penalty box" used before to describe and disparage Hybrids. What does that even mean? It's the same rehashed criticism and terminology that used to get thrown at The standard Prius years ago. The same Prius they now recommend, and recommend instead of The Prius c.

Real criticism of The Prius c can be raised. As long as it's raised fairly. It's not "one line" in their review that is unfair....it's the whole review itself.

 

But they do not give us the numbers from the individual tests. Instead they use some function to reduce everything to a single number and declare that to be "the score."

Credible reviewers provide metrics like 0-60 time, lateral G force, turning radius, e.t.c. I don't fault Consumer Reports for performing a consistent set of tests. I fault them for "hiding" the data behind some formula that mixes everthing into a number, a single number, ... 42(?).

Bob Wilson

 

In fairness, I think Consumer Reports is right. I set out to buy a C. Setup a test drive. Did three test drives at two dealers. Was going in for one more test drive and to buy a C and I had the dealer setup a test drive in a Prius III because he said he could move more on the Prius III.

It was like Yugo to Mercedes. The salesperson and I sat down after just riding in a C and at the same time said "This is really nice!". The budget feel of the C was noticeable. In fairness it is a budget car but CR is right, in comparison to other cars at the same price level, they too would get the "This is really nice!" reaction in comparison to the C.

I also think CR is right, get a used Prius for $20K vs. the C though for $20K you'd get a 45mpg Prius 2009 or older and mileage is the key to hybrids.

 

>But the criticism about comparing it to a used Prius totally misses the bigger point. We rank cars against others you can buy on the new-car market. Compared with other models you can buy, the Prius C proves to be really disappointing.

The circular logic in this "response" makes me lose all the respect I once had for the seriousness of Consumer Reports, I thought they were non-emotional, logical and scientific, clearly if this reviewer is given a stage to complain, the magazine also includes petulant, defensive, self righteous, and lacking in intellectual curiosity. At no point in this rambling article did they introduce one factoid that could be considered a valid data point.

>Acceleration is slow.
No different than the Liftback or V, which your organization recommended, how is that logical?

>The cramped driving position is flawed.
Demonstrably false, compared to all other cars with its wheelbase and size, it has a shocking amount of room, state the flaw specifically or don't say it, (this line is a prime example of the petulance.)

>The ride is always busy,
Is this a complaint about too much technology? The word choice makes no sense, if it is attacking too much detail about the hybrid drive, there is but one response "Go to bed old man."

>and the cabin is loud.
how is that even possible, when most of the cars in its class have a gas engine running when idling, and this is literally make no noise whatsoever?

>Plus it reminds us that not all small cars have nimble handling.
Back to the idea that it is a slow hybrid...and that matters to the target demographic why exactly? Why not point out that the car fails to provide seating for 6, or how disappointing you found the C's towing capacity. It's a total non-sequitar, either the reviewer isn't smart enough to understand the city commuter segment, or the manager of the Prius C product line slept with his wife. I'm leaning towards the latter because this response may be even more desperately irrational than the original.

I haven't even bought the C and I find this "article" to be intellectually insulting.

 

You are giving "science" a bad name, if a botanist tests the age of a bush by cutting it down and trying to count the rings, like they do for all trees they've worked with, they can't expect to find the age of the bush, and certainly can't call that science.

>Another problem with what you are proposing is that as soon as you try to create a specific test for a certain demographic, you run the risk of creating a bias. Who would be the authority that would decide what the demographics and purposes of different cars are and what different tests should be done on them?

The specific test...actually accounts for the driving behavior of most city drivers in America over the past 50 years. You're missing the point, there is no need to create a special test for hybrids, let alone one where they do exotic behaviors like hypermilling. If said "scientific" test cannot reproduce this major component of the American market, MIGHT be a problem with the model, not with detractors of it.

Take it from someone who's done some analytical modeling, you can call it scientific till you are blue in the face, if reality and the model consistently disagree, as their 43 and the numbers of Fuelly and gov's mpg do, time to give up the assumption that the model is statistically valid. After all, that's what good scientists do. Don't get me started on the other problems with their initial review's "data points."

Bad scientists, conversely, try to convince you how dumb you are and how authoritative their model is.

I used to wholeheartedly believe that CR was a dispassionate group of consumer advocates utilizing the scientific method, this pedantic response to valid criticisms of their initial incompetent review has caused me to reject my hypothesis.

 

Not sure why "Mr. Scientific" here, who is also a non-hybrid owner, is defending Consumer Reports so vigorously here when there is NO SUCH THING as a scientific car review. How does one measure, with a high degree of reliability and validity, how good a car is? Also, we've yet to see any third-party confirmation that Consumer Report performs their tests the same way for ALL their cars. Besides being a troll, you're not fooling anyone.

What is the relevance of the Yugo comparison here? I'm so very happy you like your Prius III but don't patronize a large community of proud Prius C owners.