"Gilbert Report" on accelerator weakness

Oh, boy. What a mess.

So, they put in a 0.8 V difference between the active and duplicate sensor outputs, then allowed 0.02 V (!!!!) difference to be the lower limit on the trip? Twenty millivolts?

During the hearing I heard Prof. Gilbert state that both of these signals ended up on one device at the receiver. Well, guess what: Build 500,000 parts and you will get shorts between adjacent pins on some devices. And not ones that can be detected during factory test, either, but develop over time.

Another very interesting question I'd have is what the input signal conditioning looked like at the receiver box. ESD and EMI are real hazards in a car environment - it wouldn't be a bad idea to have MOVs/TVBs, chokes, and other EMI-killing components on the receiver. In fact, I'm kind of surprised that the entire EMI killer on the leads leaving this thing is a couple of filter caps on each input. I wonder what a 10-W handheld radio in the passenger compartment operating at, say, 800 MHz, would do to the sensors? Those wires don't look shielded in any way.

I do not know about I2C and SPI. I2C has some minimal error checking but depends upon 10KOhm pull-up resistors. In an RF noisy environment (which the passenger cabin is) that means bit errors would be a constant hazard. SPI is driven to logic 1 and 0, which is better, but has no built-in error detection by design.

I do not know. If it were me, I'd do something like this:

1. Drive a 10 kHz waveform whose pulse-width modulation ranges from 10% to 90% as the pedal is plonked back and forth. There's plenty of four-pin microcontrollers that are good for industrial range temperature that could do this, complete with integral A/D conversion for the Hall sensor voltage. Don't know about automotive range temps, but I'd be surprised if these kinds of things weren't available.
2. Have the standby do the same thing, but at 15 kHz.
3. Both signals go into ESD-protected optical isolators at the receiver. That way, changes in ground potential don't kill you.
3. Receiver picks up the PWM signals, use a simple R-C filter followed by an op-amp to derive a DC voltage on each input, referenced to local ground.
4. Error detectors:
a. Must see full-range swing on both optoisolator outputs.
b. Must see 10 kHz and 15 kHz +-10% on the respective inputs.
c. Filtered voltages must agree, assuming, say, a 0.5 to 4.5 V range, within 20 mV or something.

That would do it. At about twice the cost in raw parts, maybe. Any fault has to persist for a second before alarm, to allow for the random EMI/ESD event. You'd want to keep this around 10 kHz or thereabouts so as to keep out of the high-frequency EMI/ESD range. I guess.

But what the heck: I design communications equipment, not car stuff, so what do I know.

KBeck.

 

This is newer info than I had before, and good to have. Gilbert's
recommendation of different slopes for the two outputs would be
just as easy to spoof as what they've got now.
.
The PWM idea is intriguing, and I'll add that the six outputs
from the hybrid ECU out to the inverter for its 3-phase control
into MG1 and MG2 are straight PWM outputs that drive the high
and low transistors for each pair in the inverter rack. So it's
not like passing PWM around the car is foreign to Toyota or anyone
else for that matter.
.
However, I'm still fairly confident that Toyota made a sound
engineering decision that linear voltages were entirely adequate
to pass from pedal to ECU in a well-protected cabin interior
environment. I'm iffy on the 0.02V detection threshold as
that *is* almost a direct short, but it may eventually emerge
why they decided to allow that much slop.
.
_H*

 

I still think that the accelerator pedal might be simply a red herring, and that people should examine much more closely the cruise-control circuitry. A fault in that system may not register any faults, has nothing to do with the accelerator pedal, but can request wide-open-throttle with the accelerator pedal in idle position. If, for whatever reason, the cruise control circuit fails to heed the 'off' switch or the brake cancel input then the engine would attempt to reach setpoint speed (say 120mph?) and ignore driver input (including further action on the accelerator pedal because it can't accelerate any faster than it's currently trying).

Lots of these questions could be answered with the simple addition of a digital "black box" in the event of a crash, stores say the last 1 second of vehicle data prior to airbag deployment. Then one can be more confident of identifying causes and ruling out sub-system failures. It can't be that difficult nor expensive to design and implement.

 

Hobbit, I think I have a little problem with your "well-protected cabin environment" comment. I build telecom equipment for a living and we have to live with craft who open up cabinets, lean forward, push switches back and forth, and, at the same time, say into their walkie-talkie/cell phone the magic words, "Did that do it, Charlie?"

Ahem. 10-50 W of walkie talkie at 5 inches, or 1.5 W of cell phone at the same distance, at anything from 1 MHz to 10 GHz is not going to help matters with EMI. We shield everything, and if we don't, we filter the heck out of the leads going into sensitive components so that induced electronic garbage doesn't cause malfunctions.

And, just to make sure we got it right, standard tests involve taking off all the covers, putting the equipment in a EMC chamber, and blasting away from DC (well, not quite, but you get the idea) to daylight.

Now, I'm pretty familiar with telecom standards, but my understanding is that the standards for cars are a lot tougher. Heck, we have locked spaces into which random gonzos with overpowered kilowatt CB radios can't get into. Try keeping them out of cars and flipping the switch on the interstate..

Given that everybody from CBers, Ham operators, people with cell phones, walkie-talkies, etc., not to mention driving past airports with radars (those windows are 'way too big to stop radar waves, microwave towers, etc., I'd say the interior of the cabin is anything but well protected. Yeah, a lightning strike is safer when you're inside a car (Faraday cage games), but I bet that such a strike throws E-fields around inside the car that, while they wouldn't kill you, probably stresses the heck out of the electronics. Throw into the mash people with snow all over their boots, mud likewise, and the extremes of temperature of an Arizona summer and a Point Barrow, Alaska winter, and I'd venture to say that the cabin of a car, and the area of the gas pedal in particular, would be one of the places on Earth that I'd rather not be. Especially if I were something electronic. Automotive electronics engineers have got my respect, they've got a tougher job to keep things working in an environment that's probably nearly as bad as operating on Mars.

Simplicity and toughness are probably virtues in such an environment. I thought of that one with PWM with about fifteen seconds thought. A real automotive engineer would probably throw me out the window because the chances of failure with the additional complexity would be too much.

Toyota's web site actually has a video talking about just how tough their cars are with respect to EMI. Well, I believe them: that they passed testing. But it doesn't mean that some blame third-order effect isn't leaking through somewhere, due to a failure or otherwise.

The real failing on Toyota's part is that when these out-of-control car events started happening, they didn't grab those cars and deconstruct them to a fare-thee-well. I listened to the testimony of the Smiths. From what they said, it sure sounds like the Toyota techie looked for error codes, didn't find any, blew them off and ran away, and didn't listen to what they had to say. Other testimony at that particular hearing said that some of these racing-out-of-control engines had been observed at dealerships and Toyota, apparently, didn't investigate those, either. When you got a whacko problem you don't run away from it. And that's Toyota's error.

 

I recall that when I worked for a Class 8 truck manufacturer is the late 80's, the first electronically controlled Caterpillar truck engines used a PWM throttle control. Of course, in non adjusted dollars, that engine was sold for about the same amount as a base level Prius.

 

gilbert did introduce two fault points: 1) crossing two particular wires and 2)increasing the voltage through that same cross. What are the odds of this happening? Especially when in the diagrams in the report it seems that the wires that have to be shorted are two wires apart and not next to each other making it impossible for a cross to result from a soldering issue. Gilbert testified in response to a question whether he sabotaged the vehicle in his test he said he did not sabotage the vehicle to achieve the result of sudden acceleration. In my opinion 1 & 2 faults are sabotaging the vehicle.

Furthermore, gilbert in the abcnews spot used the prius to demonstrate sudden accelaration problem which the prius doesn't even have. Using the prius to demonstrate an issue it doesn't even have sends the wrong message confusing people when they are looking for clarity. ABCnews and gilbert have done a disservice to the people by this.

In the abcnews demonstration and I believe in his report also, gilbert says the malfunction indicator lamp (MIL) didn't even go off. Well why would the MIL go off if the voltage sensor lines (VPA & VPA2) were crossed? When crossed, both VPA and VPA2 would show the same voltage. Is this really a fail? It is not. A fail is if the VPA sensor is going up and VPA2 is going down or the VPA and VPA2 are not the same value beyond a certain limit. Then there is a problem for the MIL to light up. Why would the MIL go on if both sensors are showing the same voltage as would happen in a cross? And again why would the MIL go on when he introduces the second fault which is a voltage increase which would be again a normal event. I think he also said something to the effect that there needs to be a redundancy implemented. There is a redundancy and he sabotaged the redundancy.

 

Looking forward to reading it - I'm mostly interested in learning what specific fail-safe mechanisms are employed in the cruise control circuitry since, according to the webinar, it is an integrated part of the ECM, and how do they enforce disabling CC on user input in spite of failures?

Wouldn't this result in a catastrophic failure, though, where I'd imagine the totally confused ECM would simply shut down the engine, perhaps fuel cut-off/ground sparkplugs?

 

There is good information in the thread.

Definitely have to wonder how rigorous toyota testing has been to this point. There was no "proof" in the demonstration, but it did overcome the barrier of "doubt". It is enough to suggest Toyota has not adequately ruled out electronic and software issues.

This may be one reason toyota is finally going to add brake interlocks. In the rare circumstances one of these things things happen, this possibly faulty software or hardware needs to be over ridden. I am not sure if they are doing the brake overide in a safe manner.

I am sure that the prius acts differently, so this fault will be unlikely to present itself to us in the same way.

I could easily design electronics that do not have this issue, but I have not designed such things in years. I'm sure Toyota has engineers that could do this more easily than I could. I am unsure of the cost per car though. My guess is with lawyers involved, the engineers are not really free to fix the problems in a straight forward way.

 

First things first: I listened to that hearing. One of the things that Gilbert said was that the outputs of the throttle sensor went to one device in the engine control box.

Ha. Now, I don't have any schematics in front of me, but with two analog inputs going to the same device, the chances of a short between wires in the silicon in the device are not zero.

OK. Again, I don't know the device, but I do know standard engineering principles. Typical input on an ESD-protected device (almost all are protected that way) is a reverse biased diode to ground, a small resistor (100 Ohms), a reverse biased diode to power, another reverse biased diode to ground, and then the wire goes to the gate input, transistor base input, or whatever.

The idea here is that a surge in voltage above power level forward biases the diode connected between the signal and power, thus limiting the maximum positive voltage. A surge below ground on the input wire forward biases the two diodes connected to ground, thus limiting how negative the voltage interior to the chip on the signal lead can get.

However, too much current and the diodes get fried. A fried diode either melts or vaporizes; I've seen both. Further, we're talking about itty bitty distances here - the whole chip in question is likely smaller than an eighth inch square. When the silicon melts and refreezes, it provides a lower impedance path to random (and I do mean random) other circuits around it. Further, chips like these depend highly on silicon doping levels (that is, the amount of carefully applied non-silicon impurities) to work properly. When an ESD or surge event occurs over the limit, those doping levels go all over the place.

So, what do you want after a surge? Vaporized parts blown off the board? You betcha. Parts that look good, But Just Don't Work (tm)? Anytime. Parts that work right until you give them a good thump, then they don't? No kidding. Parts that appear to work right, but fail six months down the road? All the time. In a real manufacturing facility, everybody wears light blue anti-ESD jackets, funky-looking heel covers that lead to strings that go into the shoes and are strapped on, or maybe special slippers, and special wrist straps with telephone-cord-like wires that go to banana plugs that plug into special jacks, and, when people go into the facility, they step on a plate, hit a button, look for the green light, and log it in the log book. Quality control engineers in factories live and die by ESD protection and training, training, training. And you wonder why there's all those warnings in the computer manuals about grounding yourself when you put in more RAM? They're not kidding, bucko. You might also ask yourself why, when you look at a car radio, the thing looks like a solid metal box with no openings. Besides keeping out interference, there's a solid EMI/ESD reason for that, you know. Metal's not cheap.

Once the parts are on the boards there are typically discharge paths on the boards that mitigate ESD and surges of various flavors. For wires that come from Places That Might Be Bad capacitors, resistors, metal oxide varistors (MOVs), zener diodes, ferrite beads, special filter caps, transorbs, and other stuff that I can't remember just now are all possible tools to keep the EMI/EMC under control so parts don't go blooey.

OK, fine. The throttle sensor has two separate drivers, two sources of ground and power, all good. But Gilbert said the receiver for both signals are on one chip. I heard that and my jaw dropped. I'll do stuff like that on a board sealed inside a box, or maybe if it's not a safety critical signal that can be detected in other ways, but on something that's pretty much hanging out in the middle of the air and is safety critical? Those wires shouldn't have touched on a common piece of silicon until after the Analog to Digital (ADC) conversion.

Gilbert's no dummy, believe me. And what he found is a real hardware bug, no joke. If what he said about both signals going into a single component in the receiver is true, then one incident with a zap somewhere and you get both-wires-have-resistance-to-each-other-as-well-as-power and your car is off to the (unintended) races. Or, if the zap is repeated, maybe you just get the loss of error detection on the first zap and Whoa, Nellie! on the second. Or fourth. Never mind the possible zap some part got in the factory or just defective silicon (it happens) and just now decided to fail a bit more decisively. (Look up the term, "Metal Migration" in Wikipedia. Fun. If you want even more fun, look up "tin whiskers" and "dendrites".)

As to whether that's the cause of all the unintended acceleration incidents that don't involve floor mats, who knows? Ms. Smith said the cruise control light went on, so that sure sounds like something else.

However, Toyota's claims that, "nothing can possibly go wrong with the throttle electronics" are hereby blown out of the water. Remember: This took a couple of hours of staring by an engineer who was fairly uninformed about throttle controls when he started. You'd think that Toyota has engineering staff that would know better, and that they check each others' work.

But forget that - stuff happens. People make mistakes. Designs are sometimes flawed. People sometimes die because of those mistakes. That's life, hard as it is. But when people start kvetching about stuff, and that stuff they're kvetching about doesn't sound anything like the-little-old-lady-hitting-the-gas-instead-of-the-brake, then it's Toyota's job (and the NHTSA's) to go looking for causes. That, apparently, didn't happen, and that's why people are upset with Toyota.

Kbeck.

 

Hi All,

Never heard of a 50 watt walkie talkie. 10 watts is the max down below about 50 MHz, and above about 5 watts. The batteries for a radio that powerfull would weight 5 pounds or so, which is a little heavy for a hand-held radio. Cell phones are limited to .6 watts, and few run at that power level continuously. Most mobile radios that are above 5 watts are operating on external antennas, and thus have sigificant shielding to the internal cabin.

Most people who operate hand held radios in cars know that below about 150 MHz, the frame of the car is doing significant attenuation. So, if you drive by an FM broadcast station the ambient field will be very much down in the seating position of the cabin, and even more attenuated down in the foot well. AM Broadcast stations would have even less signal in the car, by a factor of .01 or less.

Now there are areas of the country where there are 1 GHz and 1.2 GHz radars around on one hilltop, shooting over another hilltop. In the flatlands this is a non-issue because the pattern of the radar antenna will attenuate the signal at ground level dramatically. The hilltop-to-hilltop situation at 1 GHz and above with megawatt pulse ERP (Effective Radiated Power) will have a chance at getting into the foot pedal well, in my opinion.

There are a few of us Ham Radio operators with Prius. I have never been one to operate out of the car. But friends report no problems with the Prius while transmitting on Ham Radio equipment. Including one who is in the IEEE EMC Society.

Not that these issues do not happen. I remember being in a traffic jam on I74 just west of Champaign, IL, with a State Trooper out of his car in the left median, a trucker in the left lane, and me lined up looking at the trucker and trooper in the right lane. I saw the trooper pull his microphone out of the car bring it too his mouth, push the button, and instantly the rear brakes of the trucker locked up. The trooper radio was probably 100 watts, and right next to the rear wheels of the truck when it started transmitting. I have been in an EMC facility that can fit an 18 wheeler truck, and they use it to test braking systems on the trailers....

 

To the Engineers on this thread,
From someone who is not:

Please tell me if my reading of the Gilbert preliminary report, plus what all of you have to say, below, is right, and what parts of it might be wrong.


Gilbert is not claiming that he knows how the unintended accelleration occurred. All he's claiming is that he has created a condition where it might occur.

You engineering types seem to be convinced that this is not out of the range of possibility, given the Toyota electronics, but you don't agree on how likely this might be. For example, Bob Wilson has a speculative hypothesis about what might have happened; whereas hobbit seems more confident about Toyota's original engineering choices.


Both you engineers and Gilbert seem to be focussing on the possibility of what you might call a "real world" electrical/electronic problem (a short circuit), rather than a traditional "bug" in the computer programs.

Austingreen says that we need to know more about the Toyota testing program. Does anyone (hobbit maybe??) disagree with that?

 

Have a look at this pic; the whole bundle of 6 wires between
pedal and ECU is shielded, and point A goes to some ground point
under the right hand side cowl/firewall in the same area. This
is out of the '04 electrical diagram.
.
But by "well protected" I wasn't even considering EMI, I was
simply thinking that the whole mess is inside out of the weather
such that a low-resistance "almost short" between any of the two
wires is less likely than if it was underhood. I did my own
little RF test on this stuff too, as pointed to in the "storm"
writeup, with the full understanding at the time that it wasn't
comprehensive.
.
Now, if we insisted that every analog input to a typical automotive
ECU get its OWN separate A/D chip, how much bigger and more expensive
do you think that would make the unit? Ya gotta draw the line
somewhere. Plenty of other devices utilize chips with multiple
A/Ds on board, including ones offered for automotive use. Like
the ATMega88 Automotive I pointed to in some other thread. Were
their designers "horrified" by the idea? Doesn't sound like it.
There are thousands of similar MCUs on the market.

I like how this thread is bringing all the real techies out of
their caves. We need more of this.
.
The cruise control is pretty much entirely a software entity in
the Prius and indeed any other TBW setup ... the only real hardware
is the resistance-ladder based control lever. Now *there's* a
rock-solid reliable control path that can never go wrong, huh??
I bet Gilbert would have had a field day with that if he'd been
smart enough to think of it. Decade-box it to read "jog up" all
the time, disable the brake pedal switch, and hang on!
.
_H*

 

Harold, that depends. Modern uP's and uC's have watchdog timers - if the timer isn't strobed periodically, a full reset is applied to the microprocessor/controller and (metaphorically speaking) all hell breaks loose in the Error Recovery routines. So, if some kind of surge gets into the uC, RAM, or whatever, it usually makes that watchdog timer go off and then, of course, we'll have a record of it happening.

The usually part is because, given the right kind of surge, it might not be sufficient to blow away all the operating system code. The strobe for the watchdog timer depends upon a software timer tick; the software timer tick executes code in the response to another hardware timer; and if just that stuff is working, but practically nothing else is, Strange Things Can Happen. And, yes, I've witnessed stuff like this from time to time.

What can go wrong to make things like this happen? Well, bad software is a favorite; component variations vs. threshold voltages of voltage monitors is another. However, if you really to start having willies, think about ROHAS (that's the lead-free electronics initiative) and the growth of atomic-width tin whiskers. Little tiny wires, microns wide, slowly growing across voltage gaps, until... ZAP!

I know I've been posting a lot, recently, and positing all sorts of evil things that can happen to electronics. Hey, I like my job, designing gear. But after 30 years of doing this stuff, one gets a certain bloody-mindedness about the way things fail.

KBeck

 

You really should watch the videos from the "webinar" that Bob Wilson posted the link to.

http://priuschat.com/forums/prius-hybrid-news/76831-electronic-throttle-control-webinar.html

Spoiler: Toyota also uses redundant pieces of Si; there is a main processor and a sub processor which both receive throttle commands and perform cross checks. If the logic doesn't agree, they shut down the system.

 

Look: Gilbert found himself a hardware design flaw. If he's right about those two signals going to a common analog piece of hardware in the (I guess) engine controller then a single hardware fault would do it. Given a couple million cars out there the statistics are eventually going to give you that fault: Call it the Law of Large Numbers.

On the other hand, Smith's report talks about the cruise control light going on. So, that implies that the Devil Car in question had some sort of computer-related problem. However, that problem could be hardware; it could be software; or it could be both.

The only way, really, to find out what's happening is to grab, say, 20 of those cars and do a full-court press. All the boxes come out. They go through factory test. The engineers who designed these things get pulled out of the design labs. Hypotheses are made: 99.9% of them are expected to be garbage. Do a Sherlock Holmes on the ones that look likely. Test, Test, Test. Look for outliers. Eventually, the right rock(s) will be overturned.

Those of us posting here are armchair kibitzers. It's fun recalling all the Evil That Has Happened To Us, but the people who should be doing this work are the people who made the cars.

By the by: Companies hate do do this. It blows all their schedules out of the water when the R&D engineers are fixing field problems. I don't mind doing this kind of work, especially if it could be construed to be life threatening, but I've run into more than one engineer who'd be perfectly happy if all the troubleshooting problems would go away so he/she could design in peace. (And, yes, I have problems with that attitude..)

Final point: Am I convinced that there's something electronic that made Ms. Smith's car go down the highway at high speed? YOU BETCHA. She was believable; the professor was believable; and the comments by the safety expert about the reports that he had seen were completely believable. And Mr. Toyoda's claim that it was strictly floor mats is the stuff that comes out of a horse.

KBeck

 

Curiousity question: What is the range of resistance value that fits the window between MIL and no MIL? Is it really wide, narrow or really narrow?

For this to occur in the wild, you would need a resistive short circuit between the VCA and VCA2 circuits, it would have to be stable within the resistance range long enough that another source of 5V from another circuit would then short to the shorted VCA and VCA2, correct?

 

I think you're over-reacting kbeck. Automotive parts generally have very high ESD ratings exactly because of the kinds of concerns you raise. While there is a non-zero probability of the two signals in question getting shorted together because they are on one IC, for every one case where this magically happened you would see a huge number of inputs shorted to rail or ground as that would generally be be a much more common failure mechanism. Its my understanding that such a failure would result in a signifigant imbalance in the two sensors and throw a code. If that were true Toyota dealers would be over run with cars being towed in for failed throttle controls. What needs to be shown is not just that this sort of failure is possible, but that its possible in a way that would not produce other symptoms or failures that have not been observed. You also seem to forget that you still need two failures to occur. You need both the two sensors to get some how shorted together, but then you also need for the data from those sensors to be faulty, indicating full acceleration when its not requested. All of this also needs to happen such a way that it sometimes works fine, and then suddenly freaks out, and then goes back to being fine in order to fit many of the complaints.

Crazy things do happen, and all the things you mention are possible. However, in a low current sensing path like this many of them are highly implausible.

 

Even if it should turn out that there is a real issue, its hard to blame Toyota for responding as they have. Unfortunately, the fact that there are so many false claims of unintended acceleration it makes it very difficult to see something like this coming. For example, randomly picking Cheverolet models from 2006, there are at least 30 claims of unintended acceleration in the NHTSA database. Does that mean GM must also have a real throttle control problem, or does it just mean that a lot of people blame their own screwups on their cars. Has GM acquired every one of these vehicles to disect and make sure there isn't a real problem, or do they just blow it off because they realize the vast majority of such claims are bogus?