After losing verdict, Toyota settles in sudden acceleration case

If service brake is the parking brake, then it wouldn't be much help with an accelerating car, or one just cruising under power. It is a manual brake, and wouldn't be affected by a computer failure(but that is starting to change). Being manual means it is just cable operated. There is no vacuum or other boost to multiply the force applied by the driver on the handle or peddle. I've used the hand brake to check my speed while OECing 35mph downhill. That was all it was good for.

More importantly, the parking brake only operates the rear wheel brakes. It is easy for a FWD car to overcome the them from park. if the brake keeps the rear wheel from spinning, it will just be dragged. A car in motion has more force for a brake to overcome. The brake likely did work for those people, it just was being asked to do something beyond its capability. It's a parking braking after all. Only designed to keep an unpowered car from rolling while parked.

 

Automated code analyzers have come a long way. But design for proof and traceability are concepts that have been implemented successfully in things as complex as operating systems. The only problem is you have to start from scratch with those concepts integral to the design, wait the extra time that they take initially to code and test, and pay the $ cost of the time to market while you are doing it. I was lucky enough to manage such an effort and it took sneaking millions past the budget hounds to hoard enough resources to do it right. Small group, immensely talented, communicating constantly often working in someone's basement because they got more done there. The pay off comes when the OS is feature complete as testing and debugging become so much more simple/cheap/fast. Not a bad way to implement security too, BTW.

I'd wager auto makers start with someone else's chip and the RTOS the chip maker put together under tremendous time to market pressure. The demos look good and we have to get the car out the door in 24 months and don't have the 3 years to develop our OS, we'll just slap some application code on what the chip maker sells us, get a few fixes and see if it can work in the time frame to test the system in the test cars.

We have had computers controlling our cars for 20 years.

 

Basic deal: When the throttle is open, vacuum assist to the power brakes is minimal. If the driver comes off the brakes at all and pumps, the effort to stop the car has gone up by huge and is >unusual<. At this point the car needs a lot of work to stop (not good if one is bumping over curbs and running into things), and any other minor mistakes results in overheated brakes and one can't stop. On this issue, in the testimony, the defense attorney for Toyota attempted to kick dirt on the particular case at trial: It was a long on-ramp, the throttle shouldn't have been wide open, etc., etc.

The problem for the defense was:

1. Mr. Barr had identified one particular fault that, by flipping a single bit in a RAM location just above a stack that (a) was darn near full under normal circumstances and (b) because of the characteristics of the underlying software (recursion routines present) that had the characteristic of stopping the "X" process; that particular fault would cause the throttle in the engine to become stuck - but not go wide open. And, with that one particular fault, coming completely off the brake, then back on, would inadvertently (though the checking processor to one side), stop the engine in a second or two.
2. The defense lawyer harped on the the details of this one fault. But, importantly, if there really was a stack overflow, a lot more junk would likely have been mangled than just that one bit. And Barr pointed out that the panoply of multiple stuck processes (none of which would be detected since the watchdog was hooked strictly to the timer tick, which would very likely keep going), random unprotected OS variables (of which there were tons, including the bit that would allow/disallow Process X from running), upwards of 10,000 global variables that were unprotected and subject to bugs, including the one that controlled the actual throttle angle, all resulted in situations where the car could take off on its own; and, even if you didn't consider all that other possible failure modes, the one failure mode that he did decide to investigate in somewhat more detail could have caused the crash, just on its own. And, even with that, it wasn't guaranteed that the off-the-brake, on-the-brake "fix" would work - the check processor wasn't designed to catch the error it would unreliably catch, so, the driver could be screwed anyway. This wasn't a 20-minute panic driving down the interstate - it was a 5-10 second drive down a longesh off-ramp, with a major panic in the last few seconds followed by Death Of Passenger.
3. And, in the further panoply of things that Could Have Gone Wrong, there are buffer overflows, dereferenced pointers, just plain bugs in the code, and cosmic ray upsets. Using semi-standard code verification software and eyeballs, they found tens of thousands of the things.
4. A better CPU with ECC would have caught a lot; a watchdog rigged so that all running processes had to check in for the watchdog to be kicked would have stopped this one in its tracks; stack overflow checks (for that one class of fault) would have helped; mirroring (another check procedure, incidentally, one I'd never heard of before this, but makes perfect sense) might have stopped another whole class of faults; better initial analysis of the stack size might have prevented the stack from overflowing, assuming that stack overflow was an issue; and an MR system to record testing and field reports so that trends could be detected and analyzed instead of ignored on a case-by-case basis would have been key.

Whew. I have to calm down again.

KBeck

 

It isn't the parking brake, it is the main braking system.

 

Ok. With the main brakes it will be situational whether or not they will stop the car. Without a brake interlock, it's going to come down to how heavy the acceleration is and the speed at which the incident starts at. Heavy acceleration will be difficult for the brakes to stop the car. With a FWD car you have the strength of the driver's leg, multiplied by the brake booster, trying to stop the engine. In most cases, I believe the engine will win. In this emergency situation it would probably be better to have a RWD car. In that case, the main stopping brakes aren't on the same drive axle.

If the acceleration is low, the brake force might be able to overcome the force of the engine's output. Depending on the speed of the incidence, the brakes might still overheat and fail though.

Keep in mind that the braking system on a car wasn't designed with having to fight against the propulsion system.

Going into neutral will allow the brakes to work without fighting the engine, of course. It requires the driver not to panic, and there to be enough time before collision.

It is sounding like Toyota wasn't using basic debugging techniques, safety checks to prevent and fix errors in the system while in use, and a reliable way to record any errors that occurred. With that and NASA's investigation not being thorough since they were working under an assumption from the wrong or miscommunicated information, it would seem that saying the brakes and transmission shifter will work fine during the event isn't possible with a high degree of certainity.

 

Your points are valid, but a look the big picture of the present state of technology clearly shows some other options.

There is no law of god requiring all or any ECU calculations to be done in SW at all. All the logic can be in a soft coded FPGA or hard coded logic. This alone would eliminate 50 to 95% of the failure mechanisms created by a single thread system as well as a huge amount of the processing overhead of SW checking itself. I've been directing this for a long time in many avionics systems. A huge advantage is the number of different vendor chips, tools, compilers, code checkers, debuggers, device drivers, etc. are fantastically reduced. Most all my schedule problem are do to some vendor's SW being faulty and I have to assign my staff to fix their stuff. The time to market differences going to direct logic is substantial. All the discussion about how to improve vehicle SW needs to include a look at abandoning the 100% SW approach to simpler technologies.

However, your point about culture and skill level will always be true regardless of technology.

 

Bob,
I have to admit: I've done a total of four "microcontroller" designs in my lifetime, practically nothing.

Bob,

I've programmed microcontrollers or equivalents only often enough to be dangerous, but not an expert. I've programmed a couple of stand-along DSPs to do DTMF/MF tone recognition and generation; another DSP for ABCD bit signalling conversion from one US standard to an ITU standard; and, finally, a controller that ran and monitored a bunch of muffin fans. Control of these things was performed variously through CPU addressable registers, memory-mapped dual-port RAM, and serial PMBus. The OS in all cases was strictly home-grown round-robin (wait until the next interrupt/clock tick), although all of them had "interesting" inter-CPU communications methods.
The first two DSPs didn't have or use a stack, but the latter two did, and I spent lots and lots of time sweating stack worst case usage, worrying about state machines (who set what, and cleared what, and when), and that whole semaphore who-gets-to-write-or-read-the-buffer-and-when stuff. By the by, the fan controller chip was in the MSP430 class and, as far as I know, is still Out There in use. Not a bad chip, once one gets through all the errata sheets.

My basic training on all this stuff dates back to Purdue University back in the late 70's, long before people were messing with virtual machines. And, frankly, the kind of things I was working on were nothing like an engine controller. However, from my reading between the lines in Barr's testimony, nothing major seems to have changed except for better code checking and putting numbers on what appears to me to be standard common sense (don't make your subroutines too complex, darn it!). And I kind of doubt virtual machines are some kind of panacea; motor controllers are about as real-time as one can get, and VM context switches take time. Although a real expert can, I'm sure, make hash of this argument.

I saw the comments from some poster somewhere around here stating that using downloadable VHDL into FPGAs seemed to be a better way of running an engine controller. I rather doubt that: Putting ones state machines into hardware just seems like another way to make what one is doing more obtuse, having worked on cleaning up somebody's attempt to do just that on another hardware project. Yeah, FPGAs are lots faster, but the equivalent of spaghetti code is even nastier to troubleshoot. (Says the guy with a couple of major ASICs under his belt.. and who has done what, in retrospect, looks like way too much bug chasing in his and other people's code.)

All of the above was part of what made my hair catch on fire. I freely admit I don't have a fraction of Barr's experience in these matters, but I can recognize Bad Design Practice from a mile away, having been burned. I find it amazing that Toyota got as far as it did without somebody ringing the Big Bell and screaming, "Start Over!" (And, yeah, I've done that once or twice in my career.) When that doesn't happen.. Well, management. And really stupid decisions.

I don't work on man-safety stuff like car motor controllers. But, in 30 years of banging on hardware, there have been a couple of incidents where I said the magic words, "If this doesn't get fixed, there's the potential for a fire." When I did that, management sat up, took notice, and Things Got Done. Quickly. Without argument. Didn't people at Toyota realize that issues with the engine controller were worse than that? Sheesh.

As regards the Prius: "For a man with a hammer, everything looks like a nail." Yeah, brake pauses may have been associated with funky spaghetti code. However, Barr's testimony has a couple of tidbits for the Prius people. First, he does state somewhere that the Prius uses a controller with ECC memory. Second (and I may be wrong on this) I think I remember something about him saying that there was a brake override in the Prius.

There's the shadow of a hint that, as time moved on, Toyota got better at their firmware. And also kept on adding stuff to what they had, including fly-by-wire on the throttle to firmware that was already controlling spark plug timing and such. Given just how different the Prius is from standard gas engines, I have this faint opinion that they might have started over from scratch, or nearly scratch, which would imply less funky code. That's not proof by any means, and is probably wishful thinking.

KBeck

 

Let me clarify a couple of points. I've worked the software, hardware, and VHDL side of high reliability electronics. The vast majority of it flying on aircraft. Each has it's advantages and disadvantages, so I'm not advocating there is one answer, but there certainly are bad answers. What I can say without reservation is that when a bunch of functions that have to occur in parallel are forced to be put into a single processor, a huge amount of extra code must be written to make it work reliably. When all the separate functions are put into programmable logic, the complete separation and independence of the functions has some huge payoffs. The biggest payoff is the complete elimination of an operating system and all the tons of stuff (and cost) to support it. The second payoff is a huge amount of SW driven failure mechanisms are eliminated. One can not overflow the stack if there is no stack. The third payoff is independence of the functions. If one were to occur a fault, it would be limited to that function and have no impact (functionally) on the others. These are real payoffs used on real aircraft flying now.

 

No argument, especially with stuff flying about. I guess rather recently, a couple of years ago, I ran into a design situation where someone had put a number of no-kidding hardware state machines into an FPGA, whose only purpose in life was to drive a much larger ASIC with a zillion control, alarm, and consequent action bits. Initially the idea was that the FPGA, which was originally on a dumb pack, would do all the initialization, control, and alarm reporting of this monster ASIC (as well as some other large hunks of hardware), and provide a specialized data bus back with status and config to a smart controller elsewhere.
When this "interesting" design was transferred to a different project, it took about a month for those of us trying to adapt it to this new project to realize just how unwieldy it all was: Every time a software type would state that some control bit, or worse, some alarm state machine, had to be configured differently or exercised for the first time, it required major remangling of the VHDL state machines, VHDL register instantiations (i.e., more of them), and the VHDL read/write routines. All of this was, unfortunately, Not Easy to Extend. At the time I thought that if the original designer had put in one or two 8051 macros into the FPGA (they would have easily fit, along with RAM/ROM), the whole thing would have been a heck of a lot easier to deal with. As it was, the new platform did have a local CPU - so, when deadlines were being extended two weeks for each week that went by, I rang the Big Bell, we essentially ripped out the VHDL state machines and provided a direct memory interface to the monster ASIC. Software initially complained - until they realized that changes in their code would no longer require a lengthy VHDL design change with associated compile and that their local CPU had 'way more bandwidth than needed to handle the load.
They then went to town on their requirements and had it all done in a lot less time. And, six months later, when it was discovered that a single configuration bit had been set the wrong way, it didn't require a dive into the VHDL to fix it, just a read through the MR, a scan into the 1800 page ASIC manual ) ), and a single line of C++ code change.

But all the above is an unfortunate war story about hardware/software architecture and too many layers. The above isn't what I would call true real time processing: You're right, VHDL math and registers could do a better job for a straight multiprocessing math job.

One other thought. I sort of skimmed pieces of the NASA report. What I vaguely remember was a report that the Toyota developers were using Mathcad (or some similar tool) to develop the complex engine control algorithms. I think I remember that there exist Mathcad to C/C++ translators. I wonder.. Did they go that route and foul themselves that way? A translator like that would qualify as a kind of compiler. And compiler bugs can be nasty. And, if one is tempted to clean up the C/C++ that comes out of such a beast, then one wouldn't want to, for fear of breaking the original Mathcad algorithms. Hm.

KBeck

 

My hair would be set on fire by a runaway engine causing uncontrollable loss of braking. This is what some crash victims alleged in Audi 5000s a generation ago, and in Toyotas a few years ago. This is what NASA was looking for.

Loss of power assist? Been there, done that several dozen times in old stall-prone leaky-accumulator cars as I was learning to drive. This was controllable by reverting to old-fashioned manual braking mode (no power assist), putting full body weight into the pedal. See 49 CFR 571.135 - Standard No. 135; Light vehicle brake systems, which requires the vehicle to stop within 551 feet, from a speed of 62.1 mph, when the driver applies a brake pedal force of 112.4 pounds, under various brake failure modes. Unfortunately, many modern day drivers have never experienced manual brakes and won't respond correctly to what are still legal brake fail-safe modes.

Cars with electric brake boost no longer lose the power assist when the throttle opens or engine stalls. This includes most EVs, full hybrids, and some non-hybrids too.

Runaway engine? Been there, done that too. Just once, in a classic fatigue-induced 'pilot error', the overwhelmingly most common cause. That was controllable too, by disengaging the engine from the wheels. Unfortunately this ordinary skill or reflex of the past is very rapidly disappearing among modern drivers.

The Toyota SUA fiasco also highlighted a hole in the NHTSA brake standard -- it doesn't cover full engine power while in gear. This hole has expanded with modern-day horsepower bloat, as shown by the San Diego CHP (Saylor) crash in a Lexus ES350. I'v never heard evidence to determine whether the failure to shift to Neutral was the fault of the driver or of the car. But I also have no desire for an engine of that power.

Previous threads have indicated that brake systems are independent and isolated from the engine and throttle systems. So when any of the many ECU programming and architectural sins described here lead to a runaway engine, the fault should not cascade to an uncontrollable loss of braking or runaway vehicle. NASA found no evidence of an electric fault path. This thread focuses almost entirely on creating a runaway engine, not on finding a cascaded path from engine to brakes. The sole path mentioned (loss of vacuum) is not electric, does not exist in the Prius, is (partially) covered by current federal regulations, and is otherwise controllable unless something else locks the transmission out of Neutral and prevents shutdown.

I strongly agree that these ECU design sins should be fixed. Any control upset adds driving risk, and today's drivers are increasingly unable to handle them -- in part because these imperfect ECUs are nevertheless more reliable that the previous era's mechanical controls, providing much less opportunity to practice with failures.

But the design sins and fault paths described so far should still be controllable. My hair may stand on end at some of them, but won't catch fire until someone shows an uncontrollable path.

 

We do have this in the Saylor case. Previous driver likely damaged the brakes, while having them stop the car. The dealer, decided he didn't need to do anything about the complaints of unintended acceleration, as the policy by many dealerships is to pretend problems are occuring. The dealer gave the damaged brake car to Saylor and the brakes did not stop it in accelerated mode. Lots of complaints that dealers told people the cars didn't have unintended acceration, even though the NHTSA database flaged it, caused them to attempt to get toyota to put a brake interlock in to kill the engine to protect drivers like Saylor. Now I don't think the cause was electronic here, but when the policy is for dealers to ignore complaints, then there were likely more cars our there with damaged brakes.

 

We would if the transmission was somehow locked out of Neutral, an unconfirmed allegation I've heard for a different incident. But in this case I haven't heard anything to distinguish such a car fault from a user error / bad user interface problem. The later was clearly the case with the push button power switch, where he didn't know about three second time needed to shut off the car, in a type of case where that was also too long to wait.

But the dealer ignoring reported problems when handing him the car is another red flag. I don't believe the NHTSA braking regs go that far, requiring additional stops after a previous driver 'used up' much of the required performance.

 

So say just for theoretical discussion purposes that a Prius has a runaway acceleration incident. And shifting to neutral doesn't work. What other option does one have beyond holding the off button down for many seconds? No mechanical key to disconnect things any more. No hand powered parking brake.

 

Also on the foot parking brake. The rear brakes don't contribute much compared to the front, but the parking brake is cable activated. So it goes around the hydraulic system if that isn't working right.

 

This takes so many cascading failures that I suspect that it is far more likely the driver missed something. My belief here is reinforced by first hand Sudden Unintended Acceleration experience in a previous car, where I was very certain at the time that my foot was pressing the brake pedal. Only a hindsight review noticed some anomalies indicating otherwise. The most important of these clues would go untested or unnoticed by most American drivers.

(1) Is the foot really on the brake, not the gas? When I reflexively 'pumped the brake', my foot landed on the correct pedal, the car responded correctly to the 'pump', and the underfoot feel of the pedal magically changed. When hearing interviews of other SUA victims, I cannot blindly accept their claims of certainty about being on the brake without filtering through my own experience with 'certainty'. (NB for vacuum-assisted non-hybrids: if your foot was really on the correct pedal the first time, pumping wastes stored vacuum that you might need for the next step. Pick your poison, no single response is best for all situations.)

(2) Are you really pressing hard enough on the brake? With both feet? Federal passenger car braking regulations are written for 112 pounds-force (500 newtons metric) to overcome a variety of brake system failures. More is better, especially when fighting against the engine, so try to apply your full body weight. Don't assume that lack of braking effect with 'regular' foot pressure means that the whole braking system has failed, it usually hasn't.

(3) Did you really shift to Neutral? Try more than one option. Prius imposes a delay when intentionally shifting to N, but goes here immediately when one commands an improper shift to Reverse or Park at significant speed. I prefer R because it is quickly performed with a blind motion of an open hand, similar to my manual transmission experience, no looking for that little P button.

(4) Under power, a press-and-hold of the Power button imposes a delay of about three seconds before the car is supposed to turn off. Starting with the 2012 Prius (at least on the Liftback, I don't know about the others), several quick stabs of the button should do the same.

(5) I'd still shove the parking brake to the floor. Even if it isn't enough to stop the car, some slowdown is still a benefit.

If all these fail, I'd be looking for the softest available bog-down or sideswipe or collision target. And quickly, if speed is still increasing.

What other actions would other readers suggest?

 

Try to lift the accelerator with your foot in case it snagged the floor mat, or pull on the the mat.
Beyond that and opening the windows to increase air drag, I think you covered them all.

 

The original Toyota Camry unintended acceleration event was with that cop in LA. This car, with two passengers, went for a wild ride through the freeways of Los Angeles until the car whammed into something, killing all passengers.

Toyota promptly claimed the cause was a floor mat on top of a floor mat, that the pedal must have been stuck, the runaway had nothing to do with the electronics, and the next thing you know everybody's got floor mat inspections.

But the guy driving the runaway car was an experienced emergency driving instructor. It took long enough before the car crashed for at least one passenger to make more than one panicked call the 911.

And now we find out that the ECU on these cars really could be faulty. It was Toyota that was running around claiming that there was no way this could be an ECU problem. They were wrong. Dead wrong.

It may be they really believed that the ECU was spot-free of errors. And I guess one could technically prove with enough jamming around that a gas pedal could get stuck under the mat. But that also assumes that this driver, an experienced cop, driving a relatively high-power car, with passengers who were relatives, was having fun flooring it on and off the freeway. And, further, with a pedal stuck down, it didn't occur to him to unstick the thing. I do not know. I think that accident should be re-investigated in light of Barr's study of Toyota's code.

KBeck

>As pointed out by several posters, I got the city wrong: It wasn't LA, it was San Diego!

KBeck