"Gilbert Report" on accelerator weakness

They were testing the Gilbert theory, not the ECU.

 

Yes, I've been through the ECU a bit with an ohmmeter and have
done a bit more testing on the pedal inputs over the past few
days. [Yay! It's springtime, heralded by wires hanging out of
the car's dashboard again!]
.
The two pedal signals are brought through a small resistor network,
39K to ground and 6.8K in series (see the pic), surrounded by a
few chip caps to sink transients to ground too. From there they
possibly go through another r/c network and pass the typical
reverse-diode-to-power limiters, not 100% sure about that area
as things dive in and out of the vias pretty freely. Eventually
the signals wind up at two followers built around a quad op-amp,
near as I can tell; searching for "C451G" returns hundreds of useless
pages at various data-sheet pirates and no actual specs, although it
may be a custom-branded Nat Semi part. The way the outputs are tied
straight to the negative inputs matches the typical pinout of when
you do this on a 14-pin quad op-amp package like a TL084 or the 324s
I use. That chip is powered from 12V, not 5. Those two buffered
outputs then go through a couple more resistors and land at two
adjacent pins on the big Fujitsu Ten TDFP11 processor that's probably
the master brains of the whole thing.
.
None of this should be taken as absolute truth since it's very easy
to get lost wandering around this thing, and keep in mind this is a
hybrid ECU that sat under salt water for some amount of time in
Katrina and will never run again.
.
In my '04 Prius, the voltage tolerance between the pedal leads is
0.4 volts, not 0.02. That probably represents a design improvement
and a bit of increased paranoia that may not have showed up in other
ECU designs around the same time. The pedal itself also has internal
active feedback on its outputs, and on the scope I can see them
visibly fight attempts to jigger them up or down via resistors.
Within limits, of course; if loads get down in the neighborhood of
200 ohms, the output driver amps can't hold them steady anymore.
.
_H*

 

Hobbit, THANKS!

The part is an NEC uPC451G. It's an analog quad op amp, bipolar. It has some interesting characteristics:

1. Input voltage can go from 0.3 V less than the negative supply terminal to 32 V above the negative supply terminal. If you go below, it doesn't look good. Interesting that the positive overvoltage is not dependent upon the positive supply voltage, which is a good thing in case of power faults, I guess.
2. It can take a short out the output of the op amp forever. That's good. And a bit unusual in that regard.
3. Temperature range on the standard part is -40 to +85C; the extended range part goes from -40 to +125C, not shabby.
4. Bipolar part, so no CMOS latch-up.

Your analysis of the input circuit sounds good. With those 6.8K series resistors in line with the op amp inputs and the random capacitors and diodes around the input it sounds like it would take serious effort (like, say, a cross to 400V AC or something) to blow the op amp.

The good news is that the op amp itself and the input resistor network make this thing robust against ESD, EMI, and, probably, ground faults. (It's amazing how much of a ground fault you'd need to get a milliamp through that 6.8K resistor...).

The bad news: It's still not a really great idea running both input signals through the same quad op amp. It's not like there aren't tons of single op amps in a SOT-5 package. (For those who don't know, that's the kind of package where it's not a good idea to inhale near an unmounted one before you sneeze. Tiny.) Finally, they're apparently using the built-in A/D converter(s) in the uC to sample the signals. NASA wouldn't go that route but, on the other hand, we're not launching Camrays to Mars.

So: A sufficiently busted op amp here could still give us Gilbert's result. But the probability of this happening is low (I hope), and the likelihood of such a problem happening intermittently, and lots, without people noticing, is even lower.

Of course, this is the design for a Prius which, if memory serves, nobody has accused of running out of control yet. But such is life.

As you put it every so succinctly, let's go on to the next cockamamie hypothesis and let this one lie for a while. And, seriously, thanks for taking out the probes and having at it.

KBeck

 

Thanks to hobbit and kbeck for hacking into the Prius (Gen II?) ECU and giving us a view of what's going on there.
I even think I understand a bit of what they are saying.

Despite kbeck's plea that we go on to other cockamamie theories, I want to ask this question, which I asked in another thread, but no one seemed to be answering:
I've now looked at the Exponent Report as well as Gilbert's Report.
As I recall from the news stories, the Exponent Report was vetted by Toyota's lawyers, as well it should have been. I know what I might advise Toyota, and I have ideas about advice that other lawyers might give Toyota. Therefore, I would suggest that we have not yet reached the point of full disclosure of what Exponent might have to say. (Which in my view is OK, unlike the views of some others.)

As a non-engineer, I am left with two unanswered questions, before I am willing to put Gilbert to bed:

1. Exponent says there is no evidence of the wear, exposed wires, etc. that you would expect to see if there had been a short of the type that Gilbert induced. But I don't remember Exponent saying that it had access to the cars (or even to pictures taken by Toyota at the time) which experienced unintended accelleration. Am I missing something? Or is this really missing from the Exponent report? Surely this is a very important question.

2. In the other thread on the Gilbert Report, kbeck says:
"They didn't report anything about the ECU, beyond saying "nice doggie" when they talked about it.

"That's odd. Very extremely odd, since the ECU is the most likely place for something to go wrong with the reception of signals from the pedal assembly. I would have expected some analysis and/or statement saying that there was redundant silicon in there, or that no single-point fault in the ECU could cause unwanted acceleration without a DTC. Those statements, or something like them, are notable by their absence." [end kbeck quote]
NOW, I'm changing what I asked in the other thread, to account for the later hobbit/kbeck dialog on hobbit's actual work on an ECU rescued after Katrina:
Can we rule out a quad op amp failure here? If not, do hobbit and kbeck have a sense of how low the probability of this failure, without setting an error code, would be?

 

My comments below.

First and formost: I've read that report. That report is a hack. As in, "I hacked something up to make the people who ordered the report happy."
Gilbert never said that the wires would short out. He said, specifically, that if one put some relatively low resistance between the wires, Bad Things Would Happen.

So, where can low resistance happen in this case? Mind you, what follows arenot conclusions: It's counting on the fingers.

a. It can happen inside the pedal assembly
b. It can happen in the wiring harness.
c. It can happen in the ECU.

I don't have enough knowledge on any of this stuff (I'm talking in-front-of-me-disassembled-knowledge, not the internet kind) to know any of this. And nobody, right now, on this forum, does either. With the likely exception of Hobbit who seems to have a semi-disassembled 2008 Prius in his hands. But, from the Internet I-can-theorize-anything-if-it-looks-likely, I'll take a swing.
a. Pedal assembly: Two, duplicated, separate power supply types of circuits. Hokay, circuit boards can short out, and it appears to be one circuit board. Possible, but not a high runner. There's not a lot of parts.
b. The wiring harness. OK, wiring harnesses can sometimes Do Serious Evil. There was a jumbo airplane that went into the drink off the shores of Long Island a few years back, all hands lost, because the wires in the fuel tank frayed, shorted, and, then with an explosive atmosphere in the tank, blew the plane apart. So, the report spends 90% of their analysis time talking about how the connectors aren't that tight, not broken, the wires are in mid-air, there's no moving parts involved, etc. OK: On this one, I'll believe them. But they could have left out the garbage about sabotage. That's just trying to smear Dr. Gilbert.
c. The ECU. Big, complicated monster. Lots and lots of circuit board traces. Potted, according to the report. (Proper potting is where one submerges everything on a circuit board with something that resembles bathroom silicon rubber. It makes a hard to have shorts on a board, never mind fungus.) However we have a picture from Hobbit of that 2008 Prius ECU he has that shows no potting at all. Unless Hobbit used some chemical means to get the stuff off. So what potting are we talking about? Around the case? They didn't say.

How much analysis? Out of, what, a 20-page report? Two inches of paragraph?

We can't rule out a quad op amp failure. It's a real possibility. The problem is that failures of that op amp would have to be intermittent.. Maybe. Question for the crowd at large: For cars that has experienced unwanted acceleration, have any of those cars been driven around after the fact, successfully?
Hmm... Wasn't there some report that that cop who had the spectacular crash in California in that loaner car, that that car had experienced unwanted acceleration before, which is why it was at the dealer?

I might have also added that we couldn't count out a microprocessor hardware failure, either, seeing as both the primary and secondary pedal sensor went into the same A/D built into the microprocessor. However, there's supposed to be a second microprocessor with, presumably, a second A/D in it, and that those two microprocessors would cross-check each other for faults.

OK, that last is one that Hobbit could answer. Hobbit, there's supposed to be two microprocessors in there. Is that what you see? Does a single output channel of one of the op amps go to both microprocessors? Or does it just go to one?

If that last is the case, we have a serious potential smoking gun here.

Finally, about that report.
It's bleedingly obvious that the report should have concentrated at least as heavily on the ECU as anything else. It's also bleedingly obvious that, if there was any investigation of the ECU performed, no mention of it was made of such an investigation in that report. This kind of looking around for obvious causes is not rocket science. It's the same level of technical expertise that a radio repairman uses when fixing a radio.

I've got this stainless steel ring on my finger. It's from the Order of the Engineer, administered at Purdue, 1981. I took an oath not to endanger the public. And, by and large, I've lived by that oath since then.

There are dead people lying around because of unwanted acceleration in Toyotas. That's a fact. And these guys wrote what is clearly an incomplete report. I, personally, wonder where their ethics are.

KBeck.

 

You mean like Dr. Gilbert who was hired by Mr. Kane who was hired by plaintiffs' attorney?

From what I've read, Rhonda Smith's car was driven for another 27000 miles without incident. In the Lexus, the prior borrower had reported to the dealer that the floormat (from a Lexus SUV) had interfered with the gas pedal the day before it was again loaned to the CHP officer.

Thanks, judge and jury. Even the media admit that it is not a fact that anyone has died because of unwanted acceleration in Toyotas. I find it difficult to believe that you are still endangering yourself and everyone else on the roads by driving a Toyota.

 

Look: Gilbert's report was strictly technical. He stated exactly what he had done; how he had done it; and what the results were.

Further: He had started investigating the Toyotas on his lot before contacting the lawyers. They then paid him for his time in writing up the report.

It's the Toyota report that used the loaded word, "sabotage". And there's still the point that that report talked very little about the ECU, a glaring omission if ever there was one.

There's that ex-Toyota lawyer who's been speaking to the likes of CNN about Toyota's "Books Of Knowledge" that, apparently, have never seen the light of day. He directly claims cover-up.

Finally, I keep on hearing the words, "The dealership didn't find anything wrong" when talking about brakes, etc.

Yeah, I'm still driving my Prius. I'm going to continue to do so. But I've asked my wife not to, not until some of this underbrush gets cleared out. Would I buy a Prius, new, right now? Durned if I know, I don't work well with hypotheticals like that. And the odds seem to be in my favor, what, 20,000:1?

I'm glad to hear that the board is potted. That gets rid of some possibilities with dendrites, I hope. And, while everything is a possibility until the problem is nailed, I still don't think that a Gilbert-style fault (and that includes running both signals through a single physical op amp) is the highest possibility, here: I think software is more likely.

ECU's on ebay, eh? I'll take a look.

KBeck

 

Where in the circuit is the analog voltage converted into the digital domain?

A potential issue could be once the data becomes digital, there can be timing faults in this conversion process, which can be transient with very low statistical probability.

Inside chip designs we use very special circuits called synchronizers whenever data crosses from one time domain to another. It's highly likely that there is a time domain crossing in this process.

Say the A/D converter was sampling at 50Mhz, and the uProcessor was sampling that data at 500Mhz. Lets say this is an 8 bit bus. What happens if the sample is captured at the exact same time the data value is changing?

In reality, digital signals are really analog, and a synchronizer is designed to be tolerant to sampling a signal when it's at the 1/0 threshold.

Anyway, this is why I was interested in datalogging the CAN data, as this will very likely be what the processor is operating on and making decisions based on. And that digital data will be at the end of the whole transformation process from pedal angle to analog voltages to one resolved digital value.

It might already be in this thread, but what's after the op amp, where is the A/D converter, and what is the actual micro processor which is receiving the data and running the engine? I would like to start looking into that aspect of the process.

Thanks, John

 

The op-amp outputs run to two pins on the TDFP11, which presumably
is the main processor on the ECU. I see no evidence of two equal-
priority CPUs on this thing although there may be some outboard
stuff for running the motors.
.
Let's suppose the op-amp lost its ground pin, through a solder
crack or whatever. Its outputs would likely rise past 5V no
matter what the pedal inputs said. But the software reading
the A/Ds inside the TDFP11 is supposed to immediately go "ut oh,
that's way outside the operational envelope" and invalidate
the pedal input. What's the chance that something like a flakey
ground could raise the op-amp outputs, still staying together
at the correct offset, to just *under* the 5V limit and hold
them there long enough for, say, a 20-minute 911 call??
.
I will grant that the entire setup may be different in other
cars; I only have this one ECU and my own car's pedal electronics
to examine.
.
_H*

 

What about the input from the brake system, what path does that take?

I'm postulating that this part is the place where all info comes together, and all decisions are made.

I'll try to grab some specs for the TDFP11, but I didn't find them on a quick search. It looks like a custom toyota/denso part.

I'm going to pursue the data logging also, I think there is some good data to be gathered.

Thanks, John

 

Hobbit,

Thanks again for looking at this.

So, the Toyota video that said two parallel processors with either able to kill the engine is probably right, but the duplex part doesn't seem to include the pedal inputs.

I still think the most likely thing is in software. But, since you mentioned the op amp..

Most problems are in the wires, and, in the case of ICs, "cold" or cracked solder joints to the circuit board. The result of these can be:
1. Normal. Well, you have metal-to-metal contact. It might go umpteen years, especially on a low-current input, before anybody notices.
2. Higher resistance than normal, possibly varying over time, and usually increasing. This happens in vibrating environments. A tin-tin connector I heard about once had this problem when the cables were a bit too tight. When the connector was initially shoved together the tin (at the microscopic level) on one side of the connector shmushed into the tin on the other side of the connector, leaving a gas-tight joint with an oxide forming over the outside of the shmushed area. When the vibrations hit, the two connectors would move back and forth against each other, forcing tin oxide into the shmushed zone. Eventually the shmush was more tin oxide than tin metal, the resistance zoomed, and the circuit failed. A bad solder joint, either cracked or "just touching" might do this.
3. Intermittents. Flexure of the board might give you contact, no contact, etc.. Depending upon the vibrations one might even get contact/no contact/contact on the microsecond level.

Bed-of-nails testing, JTAG testing, automated visual inspection, and x-ray inspection (particularly of Ball Grid Arrays) can catch most clearly bad solder joints. But it's kind of a gimmie that some will escape the manufacturing process. Most of the production lines I'm involved with (which aren't for consumer goods, anyway) also do thermal testing, where we take the Device Under Test (DUT) and, while running banks of the DUTs, run them, while operating under some kind of test program, to min and max temperatures, pulling out any that fail. Yeah, and, for these things, assuming that Toyota does it, from -40 to +80C (as a guess) or so.

Even with all this, you'll still get the occasional DOA. Or the one that fails after a year in service.

OK. Now I'm going to talk about FITs. To those of you that have had probability theory this is going to sound interesting. To those of you that haven't, hang on to your hats.

"1 FIT" is defined as an averaged single failure in ten to the ninth hours, or, 1 failure in 10e9 hr. A typical single resistor has a failure rate of 1 FIT; that is, given a population of one billion resistors, we'd expect one failure per hour in that population. Given that failure rates for devices tend to follow a Poisson distribution, that failure rate tends to be a constant over the life of the component.

I haven't looked recently, but a single quad op amp is on the order of 1.5 FITS. So, then, how many failures should we expect in a year?

The equation is: (Number of systems)*FIT*(number of hours in a year)/10e9

Leave out the number of systems for now. So, our rate would look like:

1.5*(365*24)/10e9 = 13.14e-6.

To get one failure per year, we would have to have the inversion of that, so, the number of cars that would have to be around to give us a failure rate of one car per year would be:

1/(13.14e-6) = 76,104 cars.

That is, with that op amp, I'd expect with a population of roughly 100,000 cars on the road that there's be one failure per year. If there's 200,000 cars on the road, there'd be 2 failures per year.

Now, that's a straight failure, of any kind: Solder, bad silicon, metal migration, etc. And this is after manufacturing, too. And I guess this doesn't include the FIT rate of the microprocessor, which, in this kind of circuit topology, would be added to that of the op amp. (But it would have to be just that part of the uC that does A/D conversion. Maybe. Don't know.)

What this equation doesn't speak to is whether the failure is detectable or not. A detectable failure is not going to do anything besides throw a DTC. The problem is, I think, that some possible failure modes in that op amp could result in a non-detectable error. Is there a big enough population of Toyotas with this technology on the road to give us the calculated failure rate? Umm. I don't think so. And FIT rates tend to be a little too pessimistic, real failure rates (as in, returns from the field) tend to be less.

Now, this is where 1+1 redundancy kicks in and does wonders. There's some stuff about mean time to repair, but the basic idea is that it would take a failure of both the service and protection input at the same time; that would look, roughly, like the FIT rate squared, so a (1.5 failures)/10e9 goes to 2.25/10e18, a truly small number.

I can state for a fact that that's how telecom gear and networks are built. Lots of redundancy.

Well, that's why the duplex sensors on the gas pedal. But, based upon the Toyota videos, I thought that they had duplex hardware paths through the ECU to a pair of duplex processors. And that appears not to have happened.

If they can guarantee that 90% of all op amp/CPU A/D faults are detectable, then my practical objections to that design vaguely melt away. However, Toyota has made no such claims, and, given that this kind of stuff is ridiculously proprietary, probably never will, unless forced by an exceedingly knowledgeable lawyer/judge. And FIT rates of software are usually a lot worse than hardware, anyway.

Finally: FIT rates aside, if there's a true design flaw, then the error rates can take off for the moon, and the genteel march of entropy on silicon would be overwhelmed by said purported flaw.

I think I'll go and work on my ulcer tonight. After I re-up my subscription to the Toyota repair manual documentation and start reading the details of the gas pedal and the brake circuits.

KBeck

 

Something doesn't make intuitive sense here. The calculations you performed for a 1.5 FIT Op-Amp would have to be multiplied by the number of Op-Amps, correct? Add in each 1 FIT resistor, etc, and one would probably compute an expected failure rate much higher than 1/100,000/yr - all of which goes against common sense/experience.

There are millions of vehicles out there, and there are many with problems (mechanical/electrical or otherwise), but by your calculations I would expect any car I purchase to require a repair within 1 month if not sooner, and I hate to even start thinking about my 3 year old computer (or my 5 year old computer) with the millions of transitors, Op-Amps, resistors, etc.

What am I missing?

 

Not much. Take your random PC. I'd guess at maybe 10 big ASICs, each with about 10 FITs, a couple hundred resistors, each with one FIT, and so on. I'd guess a typical PC runs around 3000 FITs, and that may be on the low side since moving parts (hard drives, CD/DVD drives, fans, etc.) typically have higher FIT rates than than non-moving parts. The boards I've designed and built have had FIT rates between 150 (the sparse ones) and 2000 (switching power supplies are not good for FIT rates, believe me).

FIT rates calculations are not perfect. Actual return rates typically run 5X less than what FIT rates would predict, and that's well known in the industry. However, if you've got two pieces of gear, and one has a FIT rate that's 2x another piece of gear, then the one with a higher FIT rate is deemed to be 2x less reliable. And since everybody calculates FITs of overall systems the same way (there's standards and auditors involved), it makes the buying public (or, at least the public that buys from the likes of my crowd) better informed.

Wild guess: I'd guess that there's maybe 200 parts inside the ECU: resistors, IC's, what-all. In some cases, like resistors, we can talk about 1 FIT per resistor. For IC's, there's a sliding scale, with the 16-pin quad op amp at 1.5 FITs, the CPU probably around 50 or so. Call the whole thing 400 FITs. (There are manuals: op amps get this, IC's with yea many pins or transistors get that; high power transistors get this other number and, if said tranny runs hot by design, a bigger number; and so forth.)

400 failures/10e9 hours * 365 days * 24 hours/day = 3.5e-3 probability of failure per year. So any one car has a .35% chance of failure in a given year; two years gives 0.7%; and so on.

Invert it: 1 failure per year in 285 cars. Actual rate: five times less, or 1 failure per year in roughly 1,400 cars. I do not know, to me, that sounds roughly right. The big point: All those failures have to be detectable. And what I just wrote about is the hardware, only, not the software. I don't know what calculated FIT rates for software look like, I've never had a reason to check.

One final thing. Engines parts, these days, are very reliable. Things like a piston or crankshaft have failure rates well below a FIT.

There's a story (real) that dates back, I think, to the mid-70's or so when men were men, women were women, and ICs and the like were hardly every found in an automobile. The Big Three called a conference and invited the likes of Fairchild Semi and Texas Instruments. The automotive guys held up a piston and said something like, 'We get one failure in 100,000 pistons when we build them, and one failure in 10,000,000 pistons over the life of the car. If you can't give us ICs that match that, we're not gonna put that junk in a car." At the time, it wasn't unusual to have one bad part per 100, as shipped from the silicon manufacturers.

As a result there was a serious quality increase in the semiconductor industry.

KBeck

 

All right! So, we have two new pieces of data:

1. Toyota shows more than 255 alternating applications of the brake and gas pedal on Sikes's car.

2. One of the members (don't remember which post) points out that if the throttle was jammed, somehow (note: Not the gas pedal) and the brakes were held down, MG1 would have been smoked. which didn't happen.

My working hypothesis of the moment: Mr. Sikes thought that he'd pull a scam. Cranks the car up to full speed, calls 911. Then (my guess) applies both gas and brakes and discovers! that the override works. Oops. Gotta cover this - so starts doing gas pedal/brake/gas pedal/brake down ye road.

Well, this sure wears down the brakes some, but he can't pull that stunt with the cop right next to him - the cop will notice. So, he puts on the emergencies, hits the brake one more time, and comes to a stop.

Now he's worried. He might get found out. Doesn't want a lawsuit, but does want a new car. And, for all his googling before the fact, didn't know that Toyota could pull that kind of detail out of the ECU.

We'll see where the final report takes this. But, for the moment, told the wife she can drive the car again, and she's just as disgusted as I am. (She's also an engineer, though not the EE kind.)

Doesn't mean that other Toyotas might not be undergoing unwanted acceleration. But I think that lets Toyota off the hook on this car.


As mentioned in another thread the airbags deployed on that Prius in New York, so they will have the gas pedal/brake information on that car. I'm looking forward for the report.

KBeck

 

Sorry I haven't been keeping up with the news on this. Could someone point me to the source of this info? Thanks!

 

Video: Toyota Shares 5 Videos of Preliminary Findings in Sikes Case | PriusChat

 

Question about possibility of bogus pedal data

Question about possibility of bogus pedal data (see end of post)

The videos posted on priuschat do not have the info but I finally found it here:
Toyota: Sikes "intentionally misused" brakes to overheat them | PriusChat

and then this:
http://priuschat.com/forums/prius-h...nce-today-release-findings-6.html#post1086801

and then this:
http://priuschat.com/forums/prius-h...ce-today-release-findings-11.html#post1087760

Still no original source of the number "250" nor the action of "alternating between brake and accelerator" but I assume it is in the original unabridged video somewhere.

My question: If the car computer went crazy and decided to give unintended acceleration, wouldn't it be possible to also record some bogus pedal application data? (Similar to the case when the 12V went weak all kinds of bogus error codes come up) Thanks!

 

Where's the full-length press conference video?

That's my question too. I've found excerpts of the video on CNN and the AP, but no full-length video, including at Toyota.

KBeck