Toyota to Recall 1.9 Million Priuses to Update Software

Good question. About agreeing, I mean. I only play power engineer part time, mostly with telco battery plant follies.

I've had some experience with power MOSFETs going blooey. As I'm sure you know, power FETs are actually built of a huge array of much smaller transistors on the same silicon die; when a MOSFET gets a jolt of current/power, some spots are naturally a bit warmer than others. The transistors in that warmer spot will have the Vth threshold voltage drop, causing those transistors to carry more current, which makes them hotter, and one gets a thermal runaway which melts the subject transistor. Once that happens, it's a short, sweet trip to vaporized silicon and digging around in the carpet for the blown-off device lids. (It took a lot longer than that to convince the original designer that he had a Problem, partly because said designer didn't have a stiff enough power supply to induce the fault.)

Hence, the SOA (Safe Operating Area) associated with power transistors, or at least with MOSFETs. Interestingly, when talking about SOA, it's not precisely that one has violated the power dissipation of the device; rather, the temperature has not had a chance to spread out across the device so that all the transistors in the array properly share the current!

What I guess I'm saying is that, in my experience, when a power FET takes an overcurrent, which would probably violated the SOA stuff, catastrophic failure swiftly follows. But that's not what the NHTSA documents talk about; they talk about deformation (physical!), cracks in the solder, broken off devices, but not actual blown transistors. And, yeah, I'm talking about MOSFETs, and what's actually in there is an insulated-gate bipolar.. And I don't know, offhand, if Beta (current gain on a bipolar transistor) or the equivalent increases with increasing temperature or not.

Further, the letter to the NHTSA, while short on the fine details, details quite a bit of searching on Toyota's part, complete with false starts and retries. Once they had narrowed it down to cracks in the solder (!), then they started looking at deformation and such, and found a solution. Finally, it seems to be identified as a corner case.

Your hypothesis could certainly be correct. I'd like to think that mine is still in the running, but you've got more experience with IGBJTs. My hypothesis is that up to the point that the device cracks off the board, no damage has actually been done; with yours, I think, weakening of the devices with electrical overstress and eventual earlier failure is more possible.

If your argument about pulse widths are correct, then there shouldn't be any drop in power after the fix; in addition, since power's not being dissipated in semi-shorted current paths, the overall car efficiency should rise! But it's probably too small to notice.

Either way, glad they found a firmware solution!

KBeck

 

Thanks to he EE's here! I'll have to read the posts when I have some time and am not drowsy.

I've submitted all 3 pages of this thread (so far), to Internet Archive: Digital Library of Free Books, Movies, Music & Wayback Machine, in case it somehow gets nuked again.

I still wonder what happened to the original thread. I'm guessing that a moderator made a mistake and deleted the wrong thread or made some mistake in trying to merge some threads. I didn't think that I'd need to submit the page containing kbeck's earlier post to archive.org, before it got nuked, so I hadn't.

 

Here are some pictures of the 3rd Gen Prius inverter module and schematics. Taken from the Oak Ridge National Laboratory publicly available document "Evaluation of the 2010 Toyota Prius Hybrid Synergy Drive System".

Inverter Schematic.



Pictures of the IGBT and heatsink structure. Obviously heavy duty. Silicon attached directly to water cooled block. Notice how the boost converter has the fewest IGBT and the traction motor has the most.



Close up of boost converter IGBT modules. You can see how the leads are attached to minimise thermal stress. It is also immersed in a gooey coolant not illustrated here.



Improved power electronics cooling when compared to the Gen2 Prius inverter. Note, fewer layers and lower thermal resistance for the power electronics.




If you'd like to see the full analysis of the Toyota Gen3 Hybrid Synergy Drive, PM me and I'll dig up the link.

 

Thank you!

If you're talking about the link to the above document, thanks to your keywords, I found it by Googling. It's at http://info.ornl.gov/sites/publications/files/Pub26762.pdf. If it's something else and it's a publicly available document somewhere on the web, just post the link.

 

Freaky!!! Inside that PCU is pretty scary...

The Oak Ridge National Laboratory document is fascinating though. Toyota must be feeling a little miffed - I know I would be...

 

About the analysis, tests and reverse engineering? I'm not sure they would.

You can bet that for virtually every vehicle model out there that's sold in large numbers, competitors have bought a copy or two and dissected it. They obviously wouldn't publish their findings publicly though.

 

Beaut. There's the transistor to ground, the inductor, the rectifying diode, and the filtering cap components in all their glory. There's an IGBJT in the upper leg; however, I happen to know that, unlike MOS transistors, Bipolar Junction Transistors (BJT) are unidirectional devices; that is, current can only flow one way. Since these appear to be NPN transistors (the arrow points out on the emitter leg, rather than in), current can only flow from the collector through to the emitter. Therefore, the upper transistor is used as part of a buck converter; that is, its purpose is to takecurrent from the motor/generators and dump it into the batteries.

The pictures are interesting but a bit confusing, especially trying to match up the stack-up with the picture. In the stack-up, the only place there's solder is directly under the IGBJT/diodes; but, in the picture, I'm not seeing die, but something else. In any case, it appears that that one batch of solder is where things are breaking off; I think I'm still going along with the instantaneous thermal stress theory.

Boy, those are some small, naked leads there on the right! Looks like bond wires on a die..

So, when they inspect these things, do they take the lid off and look for physical damage? Wow!

KBeck

 

Hi cwerdna,

Great work finding that info!

I agree with you completely that competitors would buy each others' products and privately pull them apart for their own enlightenment.

Yeah, I was really meaning about the publication. For myself, as a slightly knowledgeable member of the buying public, I think the inverter construction looks extremely crude and hand made (scary!). To think they have made a few million or so of these already is amazing to me. Now, I certainly don't feel nearly as confident, owning or driving a car that depends almost entirely on something this unrefined.

Those leads (especially the small signal ones on the right) looks so random and roughly formed. No precision, automated, robotic manufacturing there, obviously. If I was turning those out and had this photo published here, I would certainly be embarrassed!

The electronic design is also very disappointing to me. Do they moderate the DC rail (below 650V DC) depending on load, or does it always run at full voltage, does anyone know? Maybe it's in the Oak Ridge document but I pretty much only "looked at the pictures" so far. ;-) If so, that would mean 650V is always being applied to the motor windings, in a metal contaminated oil bath. Hardly seems like a good combination to me. I now start to wonder if the high aluminium levels seen in the Transaxle ATF Oil Analyses are due to electrolytic corrosion, and maybe some of the high iron (and aluminium) levels are due to EDM (Electric Discharge Machining) very commonly seen in electric motors (in their bearings) running off (or near) Variable Speed Drives.

I would much prefer to see Dual DC Rails, one for each MG, and for these rails to be individually voltage regulated, according to the power demand. But maybe, it's much easier to feed the output of one MG straight into the other this way. Hopefully, the voltage is moderated at least. I guess it has to be, thinking about the regen voltage graphs I glanced at last night. Maybe not so bad then.

 

Hi there KBeck,

Call me picky! However... for the sake of accuracy:

Sorry to disagree, but ordinary Bipolar transistors (as the name implies) can pass current in either direction. Certainly, they are meant to be controlled by passing current through the Base-Emitter junction, but they CAN be put in-circuit the other way around (CBE instead of EBC) and still operate. That used to be a bit of a manufacturing problem, because the resulting product might still work, but badly. (Not to be confused with reversing the circuit supply voltage, which usually results in burning something out.)

BJT transistors are designed and optimised for the Base-Emitter junction to provide the ideal Collector current gain, and running them back to front invariably ends up with very poor current gain, but it might still be enough for them to "work". After all, they are only a 3-layer device (N-P-N or P-N-P), and without knowing which end is which (as here), the only way to work it out is test the current gain either way, then surmise that the lower current gain is the wrong way around.

Of course no designer would normally design a BJT to be operated back to front, but it's good to be aware that these current paths exist.

Oh, and I can't say either way without checking, whether it's possible for an IGBT to be operated back to front, but I suspect not...

You're kidding, of course! Apparently, the top side is filled with some kind of coolant gel!

 

I'm pretty far removed from power electronics, but it does remind me of some of the things I've done in the consumer electronics business... Like running some accelerated life testing; taking a chip and running it well past the rated voltage and temperature limits, which accelerates the various aging mechanisms that can make silicon chips fail over time. The current required to power the chip at these high temperatures and voltages was so high that we were melting the inductors off of the board. In other words, the inductor was getting to something a little over 200°C, at which point the solder would melt, the inductor would fall onto the ground (and often embed itself into the tile floor a little bit), and the power supply would stop working, (thus allowing the system to cool off again ). From that experience, we learned that we needed to beef up our power supply circuits and the cooling provided to them while doing this sort of testing. Until we did that though, I went to Fry's and bought a bunch of fans, which we zip-tied above the inductors to keep them cool.

And kbeck, I know that when we do failure analysis of ICs, we can literally sand away layers within a chip, and look for cracks in the metal. Or, maybe what they had to do to find this failure, saw through it vertically to find the hairline fracture in the solder.

I'm not surprised it took them this long to find the issue - it seems like a fairly rare failure, resulting from fairly unusual conditions, that tends to lead to this issue.

 

Can someone put in layman's terms whether or not were going to be ok with all this. I'm at the point where I will just take a loss and get rid of mine. There's a lot of non hybrid/electric cars out there that get 30+ mpg's or better I could live with.

 

Get the software upgrade done from Toyota and carry on as normal. Don't worry about the transistors and how they work. The new software will build in even more margin of safety to protect the electronics from heat damage.

According to the recall, 450 inverters have failed in 1.9million cars worldwide. Your chances of killing an inverter are now pretty small with the update.


iPhone ? - now Free

 

Hokay. Summary: In my opinion, with the firmware fix present, the car will likely last as long as it would have with a perfect design from the get-go.

You may get different opinions from other engineers.

More complicated: In my experience of high-power transistors, when something internal to the transistor goes bad, it doesn't go partially bad, it catastrophically fails. What Toyota is talking about isn't that. What they're talking about are mechanical stresses due to the die of transistor expanding with heat faster than the chunk of aluminum it's fastened to. Too much stress and the die pops off and that's that.

I've read, or at least skimmed, the NHTSA document describing the inverter. There are temperature and current monitors all over those transistors. When Toyota says "inspect", I'm reading that as "run the engine, let it stabilize, and check current, voltage, and temperature levels. Anything out of whack gets that inverter replaced on the spot."

I'm thinking that the stress under the die is sideways, so, if a crack develops, it's a sideways crack. That would definitely change both the resistance of the transistor to the substrate, making it go up and reducing the current through the transistor, and heating things up some (current*current*resistance = power, and power dissipation heats stuff up). Both of these things are testable in-circuit by the electronics inside the inverter; the NHTSA document specifically says temperature and current are monitored in the transistor die itself.

So, if the tests are good and the firmware's in, you should be good to go.

KBeck

 

Macman, been there, done that, and I do do that. One of my major jobs is dissecting dead packs/ICs/what-have-you and figuring out Why. And pushing changes as fast as they can go through the factory, as needs be.

The fun part are the failures that resemble Rube Goldberg machines: First this dies, that that does that, and then a humidifier malfunctions and spreads white grit over everything. Fun.

The letter from Toyota said they were taking x-rays under stressed modules, saw cracks, and then were able to replicate the problem on the bench, but it took some work to get that far. The team that nailed this one deserves an atta-boy.

KBeck

 

Relax, we're talkin' shop about the power electronics. Other than some personal quirks, everything is good and the firmware patch will resolve the problem.

What is not clear is whether those who have had inverter failures (i.e., GrumpyCabbie and others) are going to get some relief for early failures. But one thing makes sense, cab service would lead to the traction battery getting warmer over the shift and this appears to be part of what leads to the inverter failure.

In spite of the commercials, a Prius is not for everyone and you should be happy enough with your ride. There hasn't been a vehicle I've ever owned that I didn't have a 'bitch list' and that includes the 1929 Model A Ford my Dad thought was good for teaching mechanics . . . more like "DO NOT DO THIS!!!"

A lot of choices in life are between what s*cks less. If you want an excuse to change from the Prius, by all means take it and find happiness (or another place to complain.) With gas prices going up, you should get a good price, especially in about 30-45 days when the gasoline prices start to peak out.

GOOD LUCK!
Bob Wilson

 

Have a question. The inverter and boost converter are under the hood. The battery is below/behind the rear seats.

Why would the temp of the battery have anything to do with the damage to the inverter?

 

Batteries perform differently depending on temperature. Warmer batteries can supply more current than cold/cooler batteries since the charge is supplied by chemical reactions which are temperature dependent. That's my guess, anyway.

SCH-I535

 

Bingo, that's why I keep my camera batteries in my inside coat pocket, next to my nice warm torso, when I'm out on a winter night photography session.

 

Hi Fore,

I can't really speak to the life expectancy of the other parts of your car, and my impression is that the Toyota Prius IS a little on the delicate side. But get this firmware update, treat it nicely, change the lubricants as recommended here, and it should last you well. Don't worry about all this back and forth about the Inverter or the Boost Converter.

Some of the guys here have missed the main point. The Prius V uses this same Boost Converter and DOESN'T HAVE THIS PROBLEM. But it does have different firmware, which virtually proves that the Hardware is okay, and the Gen3 Prius firmware is not quite on the money. Trust them (Toyota) that the fix is good and will be fine. If you like your Prius, treat it nice and stick with it.

As for the question about why the Battery Temperature has anything to do with this:

The HSD system closely monitors Battery Temperature, and increases the battery cooling fan speed to compensate, as letting the battery get too hot will substantially shorten its life. The Gen1 had problems like that. However, that may not be enough, so the HSD system ALSO adjusts to reduce the battery charge rate when the battery is getting hot. This means changing ("reducing") the electronic signals that go to the Buck/Boost Converter that all this discussion is about. And it seems that the reduced Battery Charging Signals are where things went wrong. Somehow, when they made this software adjustment to compensate for higher battery temperature, they created a bug. That's probably why this problem only happens rarely, as most of the time, Prius batteries are not running hot, and as the "normal run" software works just fine, no problem arises.

But, like Grumpy Cabby found out, if you drive the Gen3 very hard, you can blow your PCU (Power Control Unit), aka Inverter, etc. No doubt, at that moment, his battery was running hot, and the bad section of firmware was running and frying his Transistors, to the point where one or more want BANG.

Hope this Helps!

 

Do you an URL to the 450 number?