1. Attachments are working again! Check out this thread for more details and to report any other bugs.

After losing verdict, Toyota settles in sudden acceleration case

Discussion in 'Prius, Hybrid, EV and Alt-Fuel News' started by bwilson4web, Oct 27, 2013.

  1. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,661
    15,662
    0
    Location:
    Huntsville AL
    Vehicle:
    2018 Tesla Model 3
    Model:
    Prime Plus
    Rather than start a new thread, I dug up this corpse thread:
    source: Okla. judge: Settlement reached in Toyota case | Courier-Post | courierpostonline.com

    This is one case that I may see about getting a copy of the court transcript. I am curious as to what evidence was presented.

    Bob Wilson
     
  2. randyb359

    randyb359 Member

    Joined:
    Feb 28, 2008
    430
    38
    0
    Location:
    Pittsburgh
    Vehicle:
    2008 Prius
    Didn't NASA check the software and find it was not at fault?
     
  3. austingreen

    austingreen Senior Member

    Joined:
    Nov 3, 2009
    13,602
    4,136
    0
    Location:
    Austin, TX, USA
    Vehicle:
    2018 Tesla Model 3
    Model:
    N/A

    No! Nasa tested the system and could not find an electronic fault with working cars, that was causing the problem beyond a reasonable doubt. That does not mean there was not faults that caused the accidents. Toyota prevented many investigations on autos by refusing to read black boxes.

    That was a source of major anger at Toyota, and Toyota has now agreed to actually read black boxes, and have them work.

    We do know that Toyota gave a wink and a nod to dealers that there were no problems, and directly worked against the NHTSA in their effort to have a brake interlock installed. They also hired former regulators to lobby against the NHTSA.

    We have 2 things that toyota has admitted to problems with.

    Doubled floor matts installed at dealers entrapping the pedal, leading to uninteneded acceleration.

    Sticky pedals, that Toyota illegally hid from the NHTSA, and has now paid fines for hiding safety information.

    Toyota did enough bad in the past to create reasonable doubt in lawsuits. We may never know if there were really electronics or software problems, but we can definitely know that if toyota had installed brake interlocks and not hidden safety information, fewer deaths and accidents would have occurred. They are best off settling, changing their behavior, and putting the incidents behind them.
     
  4. a_gray_prius

    a_gray_prius Rare Non-Old-Blowhard Priuschat Member

    Joined:
    Jun 13, 2008
    2,927
    782
    0
    Location:
    IL
    Vehicle:
    2008 Prius
    Model:
    N/A
    After losing verdict, Toyota settles in sudden acceleration case - chicagotribune.com


     
    austingreen likes this.
  5. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,661
    15,662
    0
    Location:
    Huntsville AL
    Vehicle:
    2018 Tesla Model 3
    Model:
    Prime Plus
    Ok, now we have some details:
    Source: Toyota Case: Single Bit Flip That Killed | EE Times (requires registration)

    Having read more than few crash and stack dumps (and currently having a Java anomaly on my list,) these are not trivial technical challenges. For those who subscribe to embedded computer journals, it looks like the next months will have some very interesting articles.

    For what it is worth, the comments on this EE article are almost as enlightening as the original article. Mission critical software is not trivial and latent defects can come from the most obscure sources. For example, 'tin whiskers' are real and at least one Camry accelerator with 'tin whiskers' was found by NASA Goddard to be part of a failure that reduced accelerator control authority. But don't set your 'hair on fire.'

    Our Prius have:
    • Independent brake ECU - you can stand on it and it will slow and stop the car (although we don't know if it is a mechanical override or just a signal to the engine controller to 'back off')
    • Independent transaxle ECU - separate from the engine controller, it will shift into "N" or shifting into "R", more quickly
    Having been an operating system programmer and now dealing with a COTS software package that does not come as a source-code, package . . . bugs-be-real but they always have been there. So no, I'm not selling our Prius but I will take it in for any Toyota patches . . . after the pioneers. I still have my notes from the "brake pause" problem.

    Bob Wilson
     
    walter Lee and dave77 like this.
  6. Mike500

    Mike500 Senior Member

    Joined:
    Mar 1, 2012
    2,593
    764
    0
    Vehicle:
    2012 Prius v wagon
    Model:
    Two
  7. jdcollins5

    jdcollins5 Senior Member

    Joined:
    Aug 30, 2009
    5,131
    1,340
    0
    Location:
    Wilmington, NC
    Vehicle:
    2010 Prius
    Model:
    III
    It is interesting to me that we have not heard much from sudden acceleration since this case and the media blitz that followed, along with the copy cats that followed. As far as I know Toyota has not modified the software in all of their vehicles in the meantime. So if this was the fault of Toyota's software would not one expect many more events such as this?
     
  8. bisco

    bisco cookie crumbler

    Joined:
    May 11, 2005
    110,133
    50,049
    0
    Location:
    boston
    Vehicle:
    2012 Prius Plug-in
    Model:
    Plug-in Base
    i would.(y)
     
  9. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,661
    15,662
    0
    Location:
    Huntsville AL
    Vehicle:
    2018 Tesla Model 3
    Model:
    Prime Plus

    There is one class of failure, intermittent, that are d*mn difficult to diagnose. When we first heard of the 'brake' problem, it took a lot of collaboration to gather scattered reports and add instrumentation before I could reproduce it at will. Happily, Toyota came out with a fix about the time I was finally able to reproduce it at will:
    1. Speed ~18 mph AND
    2. Slippery driving surface AND
    3. Regenerative braking force, not mechanical AND
    4. Speed bump or equivalent pot hole
    To call it a fault, hind-sight is always 20/20, fore-sight is less precise. I understand the problem and it is not trivial. So tonight I'll will spend several hours trying to figure out an obscure problem that seems to fail about 1 out of 80 times. But then that is why I'm paid 'the big bucks.' <grins>

    I am sympathetic to the technical challenges but these problems, once known, need diagnosis and correction.

    Bob Wilson

    ps. I still believe when the engine finally runs out of gas it should at a minimum raise the "Check Engine" light. Silently running down the traction battery and then raising the "Power Steering" light is the wrong response. Having an undocumented loss of "power flow arrows" in the energy display is less bad than nothing.
     
    walter Lee and dave77 like this.
  10. jdcollins5

    jdcollins5 Senior Member

    Joined:
    Aug 30, 2009
    5,131
    1,340
    0
    Location:
    Wilmington, NC
    Vehicle:
    2010 Prius
    Model:
    III
    The Prius brake transition delay problem described above is a totally different issue than the sudden acceleration that was supposed to be the topic of this thread.

    I get the brake transition issue every now and then but accept it as a characteristic of regen braking and do not really see this as a problem.

    I fully agree that Toyota could definitely improve on their software. I am sure that would be true for all manufacturers.
     
  11. kbeck

    kbeck Active Member

    Joined:
    Feb 10, 2010
    420
    275
    0
    Location:
    Metuchen, NJ
    Vehicle:
    2010 Prius
    Model:
    III
    Bob,

    Back when I first started hanging out on Prius Chat, I thought long and hard about what was going on in the ECU, listened to the Senate hearings on runaway Toyotas, and thought about that lady with the runaway Camry and that trooper in California. I noted the rumored existence of "Black Books" in Japan, which detailed problems in cars made by the Japanese automakers, which could not be subpoenaed across the Pacific.

    I listened to the NASA story with regard to their analysis, and noted the very careful wording: That the software could not be "proved" correct, but that they didn't find anything.

    I showed up here and spoke my mind about the subject; analyzed what I could from the Gen II ECU that was brought to our attention, watched videos from Toyota and others and thought about it. At that time I said that it sure sounded like firmware/software bugs. I only shut up on the subject when it was pointed out that Priuses have to have an engine limiter and brake override, lest they blow their motor-generators. And this is Prius Chat, after all.

    I think I stand vindicated. And what really gets my knickers in a twist is that the black box itself was faulty and didn't record data properly.

    And, as a result of all this, people have died.

    I wonder how those officials at the NHTSA who bad-mouthed all the unintended acceleration cases on Toyotas feel now? Do they get demerits on their performance reviews by being too buddy-buddy with Toyota?

    Sheesh.

    KBeck
     
  12. fuzzy1

    fuzzy1 Senior Member

    Joined:
    Feb 26, 2009
    17,557
    10,324
    90
    Location:
    Western Washington
    Vehicle:
    Other Hybrid
    Model:
    N/A
    A major problem in identifying these events is that they are mixed in with similar events having other causes, and it is often difficult to separate them by cause. There is strong statistical reason to believe that 'driver error' causes many more of these events than do software, hardware, and other mechanical causes combined.
     
    xpcman, bwilson4web and jdcollins5 like this.
  13. SageBrush

    SageBrush Senior Member

    Joined:
    Jun 4, 2008
    11,627
    2,531
    8
    Location:
    Southwest Colorado
    Vehicle:
    2012 Prius v wagon
    Model:
    Two
    Good enough for me, but then I don't spend any time chasing the tooth fairy even though you would say it has never been proven she does not exist.
     
  14. jameskatt

    jameskatt Member

    Joined:
    Apr 7, 2011
    148
    50
    0
    Location:
    Monterey, CA
    Vehicle:
    2012 Prius
    Model:
    Four
    Verdicts like this make me salivate at the chance that a Google autonomous car runs over a pedestrian. Such a car should never get onto a accident. It would be completely Google's fault.
     
  15. SageBrush

    SageBrush Senior Member

    Joined:
    Jun 4, 2008
    11,627
    2,531
    8
    Location:
    Southwest Colorado
    Vehicle:
    2012 Prius v wagon
    Model:
    Two
    Are you volunteering ?
    We should probably use a dead pedestrian and have the accident in space so that pedestrian or road cause can be reliably excluded.

    Now, how to keep all the space junk out of the way.
     
  16. kbeck

    kbeck Active Member

    Joined:
    Feb 10, 2010
    420
    275
    0
    Location:
    Metuchen, NJ
    Vehicle:
    2010 Prius
    Model:
    III
    Agreed. However, and this is the picky point: Sometimes the only way to nail a software problem is to attack it from two directions:
    1. Look at the external symptoms and say, "What could have caused this". Write down a gazillion reasons. Since the engine controller is directly involved, both in hardware and software, it's high (but not proven) on that list.
    2. Since the software is on the list, and high on that list to boot, really take a look at the software.
    NASA or no NASA, that kind of work takes time, people with a vicious mindset, and the source code. And not just the top and medium application code, but the low level OS code as well.

    In case you were wondering, there are reasons why Microsoft software doesn't end up in mission and life-safety software. Partly because the OS is Microsoft's deep, dark secret: Partly because it is (from all reports) spaghetti code dating back to DOS 1.0.

    Apparently, from the EE Times article, NASA never got the OS code that runs on the engine controller. They also did not get a lot of time. And, once the Evil Software Engineers started seriously looking, they found that bit flips could cause unintended acceleration.

    I'm mainly a hardware engineer by trade and training. So, during and after I do a circuit design, I pull out my evil eye, look at every trace I'm laying down, and ask questions:
    1. What happens if this lead is shorted to ground?
    2. What happens if this lead is shorted to VCC?
    3. What happens if this lead is open?
    4. What happens if this lead is shorted to the adjacent pins, if any, in either direction?
    The idea is that when one of those things happen, the hardware/software system should, with high probability, detect the fault. Silent Failures are Evil. Designing parity and similar tricks so that stuck-at faults can't escape are very good ideas, especially if one can have interlocking parity areas (that is, the parity generated in one area has an end point after the beginning of the next parity area). Background diagnostics (those things that run at the same time as the system) that detect Funny Stuff are very useful: So is software that does diagnostics on boot.

    And, when one writes and designs the software that diagnoses all this garbage, one does it one careful step at a time. Building fault boards to verify these algorithms and actually adding hardware that induces faults so one can detect stuck-at faults in hardware are important, too.

    And everything I mentioned above is pretty much Standard Design Practice. What the NASA guys do when they send a satellite across the cosmos makes the above look silly. (In general, there are no repair trucks in space.)

    For software, bit flips is what happens when Things Go Wrong in an otherwise correct design. I am not kidding now: Cosmic rays can and do come down from on high and flip bits, especially in DRAM. Especially if one is not using RAD-hard hardware. And, yeah, exposed to cosmic rays, some DRAM cells might flip, some might not, and on a sample-by-sample basis, the results one gets from this kind of thing vary all over the map. Note: This stuff is not a deep, dark secret, and it's not new, either.

    So, OF COURSE one designs around these problems. Firmware loads have checksums. One can write software that, in the background, checks the RAM area for load checksums and such. Parity and ECC memory are well-known, and paying attention to the error bits when they flip up are important.

    Amongst the many things in that EE article, which, unfortunately, seems to have been written by somebody who thinks that all software is an "App", was that independent software routines that should have been running could die and stop - and the overall OS wouldn't notice. That's double-plus ungood, as you might guess.

    By the by: The watchdog timer mentioned in the article is an old-time favorite: If the software doesn't reset the watchdog register from time to time, the watchdog times out and resets the hardware. If having N processes all running is critical to safe operation, then there are things One Can Do that guarantee the Watchdog only gets reset when everything that is supposed to be running is running. That's another one.

    The thing was: The guy who did the analysis didn't say that there was, say, a single fault: He said that there were Lots. And the equivalent of Open Barn Doors to drive them through. It sounds like he got tired of the Nth iteration through, "this mission critical byte bit-flip doesn't get detected", called a halt to it, and wrote his report - all 800 pages of it. I suspect that there were 800 pages not because the fault-finding was that complex: It was likely because he kept on repeating himself over and over and over again. Lawyers like that, but I'd hate the be the one doing the cutting, pasting, and doublechecking, especially since you could bet the automaker would be looking for misplaced commas.

    And, really? Toyota never knew any of this? Sheesh.

    KBeck
     
    walter Lee and bwilson4web like this.
  17. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,661
    15,662
    0
    Location:
    Huntsville AL
    Vehicle:
    2018 Tesla Model 3
    Model:
    Prime Plus
    That is why many of the comments had more insight than the article author. But the 800 page report might be interesting. Also, they mentioned some sort of simulator.

    I would be more impressed if they'd mounted the computers on a test fixture with stand-in sensors and then shown how to exploit the faults. To toss an 800 page technical report at a typical juror who may barely know the 'three finger salute.'

    Bob Wilson
     
  18. austingreen

    austingreen Senior Member

    Joined:
    Nov 3, 2009
    13,602
    4,136
    0
    Location:
    Austin, TX, USA
    Vehicle:
    2018 Tesla Model 3
    Model:
    N/A
    Toyota has since

    - Carved gas pedals and replaced sticky ones
    - modified sticky pedals
    - Changed software in many existing cars that would except it
    - Modified new cars, so that all new cars would have a brake interlock
    - made black box readers available to check out unintended acceleration claims
    - Media has made it almost criminal when toyota dealers doubled up floor matts- putting stories on local tv (this was the case with the fatal lexus case - car dealer pretending the untintended acceleration was fake, sent it out again with a different driver with doubled matt).

    Toyota is acting completely differently than before the NHTSA called them out for lying about unintended acceleration being a non issue. Don't sugar coat the bad behavior on toyota's part. They covered up illegally unsafe conditions. We just don't know how far the cover ups went.

    Gen III prius never had any of the reported problems, although some drivers did report unintended acceleration, none was validated, and afaik no fatalities occured from a gen III prius UI acceleration.
     
    ftl likes this.
  19. jdcollins5

    jdcollins5 Senior Member

    Joined:
    Aug 30, 2009
    5,131
    1,340
    0
    Location:
    Wilmington, NC
    Vehicle:
    2010 Prius
    Model:
    III
    I am not trying to sugar coat anything here. I am just asking an obvious question about the software. If it was a software problem why have we not seen many more such sudden acceleration accidents?

    Most of what you said above has nothing to do with the software.

    Don't get me wrong, I am not a Toyota fan and think their response was totally wrong. I just do not understand why we have not seen additional incidents.
     
  20. austingreen

    austingreen Senior Member

    Joined:
    Nov 3, 2009
    13,602
    4,136
    0
    Location:
    Austin, TX, USA
    Vehicle:
    2018 Tesla Model 3
    Model:
    N/A

    I have no idea if there are software or electronics problems. I doubt that they occur often if they even exist.
    We do know past behavior by Toyota, along with real possibilities of problems, mean that a jury would be much more likely to find against toyota, and long court cases are not good for the company. It needed to settle and put this behind them.

    I do know that if there are these problems, reading the black box immediately afterwards will help find faults. We also know that having a break interlock will prevent accidents if software or electronic problems were occurring. Toyota because of the deaths and court cases have now changed their cars to minimize the problems. The NHTSA had asked them to do this in the past and they had refused.