Toyota to Recall 1.9 Million Priuses to Update Software

This one has the description I was looking for:

source: ibid

Because the temperature of the battery is involved, this particular aspect is not likely to be reproducible. Worse, if it did show up, we could be in "safe home mode" or worse. But we're also making the assumption that the only change is in this one aspect of the control software. The hybrid ECU pretty well handles all the control laws and accumulated fixes for other aspects may be in the software. Still I wonder if our UK friend, 'GrumpyCabbie' might have been stung by this one.

About six months ago, Toyota lost a case that centered around the control software possibly leading to 'run away acceleration.' Imagine one byproduct was to implement improved software quality . . . say improved code tools that added new warnings and improved variable checking. It is entirely possible that the code fix for the inverter circuit heating includes these additional features.

Speculation, additional code repairs may have no known, overt failure symptoms. They are potential bugs, lurking for the right combination of events. But without a list of all changes, not just the ones that led to IGBT overheating, we'll never know.

Bob Wilson

 

Bob,

Be sure to monitor the Boost Converter voltage. I would guess this will be included in the software updates. Currently it goes to 650VDC during increased power demand, including hill climbs, and increased regen braking.


It will be interesting to see how they may limit the use of the 650V Boost.

It will also be interesting to see how the cruise control works on a downhill descent since it has already been reported that the speed seems to increase more after the software update. One of the benefits to me was the ability to maintain CC speed on descents.


Thanks,

Dwight

 

I agree, most likely the software limits the boost voltage to under 650V, say 600V or less.
This will affect full power performance, only the 0-60 time will tell.
The boost voltage varies with power demand and speed of MG2.

 

Since this doesn't involve the the prius phv, which uses lithium, it seems fairly clear that the problem has to do with the SOC, temperature, and charge and/or discharge rate of the nimh pack. The gen III ice is more powerful which means its able to charge the battery more easily than gen II. They also increased maximum discharge rate from 25 kw to 27 kw. Perhaps under some conditions software is not appropriately taking care of possible problem conditions, or the new gen III hardware needed more constrainsts.

If your software qc process leads to "show stopper bugs", and this is a show stopper, then there is something very wrong with how they are issuing updates of software. Since my 2009 car is in this recall, I would say this problem was over looked when they first released software for the gen III.

If somehow after all the unintended acceleration accidents, somehow they did not test that new code could cause unintended acceleration, then I would fire the entire quality control team, after hiring a new one. I don't think this is the case.

No they are bugs, that just haven't been exposed. The hardware is not that coplicated, so these can get caught in simulation as long as inverters, ice, and battery are properely modeled. I prefer to think this condition was overlooked until it happened in enough cars to get toyota corporates attention. After that it should not have been difficult to find the defect. We found in many cases with unintended acceleration the toyota culture kept it from the engineers that could fix it. Akido Toyoda promised to fix this problem. At least us Gen III owners will be getting a fix before anything bad happens to us here.

 

Hat tip to Agent J and Bob Wilson as they wrote in the original Grumpy's post about his car failure (Possible Gen3 Prius Transaxle Failure at 70,600 miles | PriusChat), which occurred shortly after braking on a downhill:

"I'm guessing inverter as well now that you've mentioned the higher-than-normal whine under long and hard regen braking. When that gave way, the ICE shut down, hence the sudden jolt, as a fail safe to the inverter. whatever charge that's left in the traction battery is what kept the car moving at safe mode" Agent J

"Braking while descending a hill at high speed has been the best way to put a maximum charge, the most electrically stressful load on MG2, the inverter, and the traction battery. The reason is high-speeds give a significant kinetic energy source exceeded only by the potential energy of the hill. Worse, charging NiMH batteries is exothermic and will heat them up in a hurry. This is why I recommend using "B" when descending steep grade, 8%, tall 160 m hills. This both moderates the speed and dumps the excess heat out the exhaust instead of stressing the electronics and battery.
The high-frequency noise heard may have been the failing power electronic inducing noise in the 12V system. When the CD was turned off, the noise went away" Bob Wilson


Seems I will start using "B" in a 7% 200m hill my Prius C transits everyday, sometimes with a full stop at the bottom. Until today I used the brakes to fully load the battery and cover several hundred meters in EV. I hope the lower voltage in the C inverter (max 520V) avoided any issue but I prefer to be on the safe side

 

Well what I would expect...and I mean this as a compliment to the many involved and intelligent people that participate in Prius Chat, is eventually probably as a group effort- exactly what Toyota is changing and has changed with the software update will become known, as well as a better definition of specifically what the problem was...

( Sadly I guess I have to say, I expect this knowledge will come to light....assuming it is allowed to be posted in Prius Chat-Disappearing Thread Syndrome).

What I'd ask, is that eventually a comprehensive simple synopsis be given. I want to know, at least on a rudimentary level what is going to happen to my Prius when Toyota wheels it into the dark corners of the service bay. I want a definition in more depth than the current Toyota provided..."Possible Overheating Transistors/Software Update".

And to be honest, sometimes, the brilliance and respectable knowledge of my fellow posters is too much information.
You lose me with graphs showing specific current and voltages...

Because basically as an owner...I'm in it. Not really much choice here. Toyota has voluntarily recalled my vehicle for this update.

I suppose I could ignore it, but that IMO is really not an option. If I ever did have a failure, it would be not only potentially dangerous to me...but a nightmare as far as warranty support.

I guess I'm dissatisfied not because Toyota has done this recall, or even that the recall has happened. I'm dissatisfied because I think the public announced definition of both the problem and the solution is lacking in revelation. I don't need "exact current or voltage specifics" but I want more than potentially deformed or damaged transistors and "software update".

 

Odds are they reprogram the power limits on charging or discharging the battery in certain conditions. I doubt it has anything to do with power going from mg1 to mg2. I doubt toyota will be transparent about what the defect actually is, and whether it is a hardware problem or a software problem, but if it is a hardware problem they can work around it in software.

 

Just had the update completed. No hardware change, just a flash software update. Took 1 hour at the dealer including a free car wash.

Service invoice notes:

PERFORM EOE RECALL
PERFORM EOE RECALL
OP AGG09A
CAMPAIGN COMPLETED

No noticeable change in engine performance or MPG on the 25 mile drive home.

 

Guys,

I have no idea what happened to the Original Post either, but as it happens, I managed to nab one page of it - one of the better pages. If I'm lucky, I'll be able to upload it for all to grab, though if I have the same problems I have with uploading photos now, that isn't going to happen. But I do have page 3...

 

Guys,

I have no idea what happened to the Original Post either, but as it happens, I managed to nab one page of it - one of the better pages. If I'm lucky, I'll be able to upload it for all to grab, though if I have the same problems I have with uploading photos now, that isn't going to happen. But I do have page 3...

 

Nope, it didn't attach. Let me try once more...

 

If the problem is related to higher battery temperatures, could that explain why the cooling fan in my 2010 Prius IV seemed to run faster and more often than it does in my current v?

 

could a ScangaugeII Xgauge be use to monitor the Booster Converters?

Toyota/Lexus/Scion Specific : Linear Logic : Home of the ScanGauge

DC/DC Upper Converter Temp 07E22174 056106740000 3008 00090005FFD8 DUC x F
DC/DC Lower Converter Temp 07E22174 056106740000 3808 00090005FFD8 DLC x F

 

The DC/DC Converter is the 200V/14V converter for charging the 12V battery and powering the car's 12V systems.

The Boost Converter boosts the 200VDC to 500VDC and 650VDC for MG1 and MG2. The SGII XGauge:

Voltage After Boost (Battery) 07E22174 010702EA0321 3010 000100020000 BVB xxx V Voltage after Boost

 

Having this recall performed as we speak.... :/ hope I don't notice any change but risk of not having it done seems too great

 

I just had my 2012 v in for its 30k (18 month) service. The dealer confirmed that there were no outstanding recalls on my vehicle.

 

Ho, boy. I read the NHTSA document. And this sounds like real fun. I see the problem; I see the probable reason; and I probably see the fix.

First, the disclaimer: Whilst I have a decent working knowledge of transistors, inductors, capacitors, and do muck with them from time to time in my day job, I've never actually seen the guts of the inverter assembly, and am about to guess and golly my way through what sounds like what happened. There may actually be some connection to the truth: Those of you who have taken apart inverters are more than welcome to chime in and say where I've gone wrong.

Next: We're going to discuss a standard boost power converter and what makes it go. Since I'm assuming that a fair number of readers here don't muck with electronics, I'll take it slow. First, the "interesting" component du jour: An inductor.
For those of you who don't know, an inductor is, generally, a coil of wire. Literally. Take a length of copper wire, find a nail somewhere, wind it around the nail, and, ta-da! It's an inductor. When one runs current through a length of wire, the moving electrons make a magnetic field then goes around the wire, circularly. When one makes a coil of the wire, the circular fields add, making a stronger magnetic field.
Let's draw a little circuit diagram with a battery, a resistor to limit the current, and an inductor:


So, the DC resistance of the inductor is 0 ohms (just a hunk of wire, right?), the current is limited to 1 Amp (200V/200Ohms, Ohm's law), and we have a magnetic field around the inductor. Current flows in the left side and out the right (OK, I'm a EE, I think that way. Techies think of terms of electron flow, which goes the other way, but I digress.) There's 200V across the battery, + to -, 0V across the inductor (no resistance, right?) and 200V across the resistor, + to -.
Now comes the interesting question: What happens, precisely, if the wire is broken between the inductor and the resistor?
Well, a novice would think that the current, like, stops. They'd be wrong. There's that magnetic field, suspended in mid-air if you like, and without something to sustain it, it's going to collapse. And when it does, it's going to do so in a fashion that keeps the current going in the same direction that it was going in. The sign of the voltage across the inductor will swap, with + on the right and - on the left; and the voltage across the inductor will get very, very large until something gives way.
Any of you who have unplugged a running vacuum cleaner (motors are full of coils, right?) knows what happens next: there's a big spark, a snap, and all the energy that was in the coils in that vacuum cleaner goes into heating up the air. The current going through the coil decays exponentially at a rate of I = Iiniitial*e**(-t*L/R), where e is 2.7 something, "**" means "raised to the power of", R is our resistance (pretty big when the plug is pulled out of the wall - it's the resistance of ionized air, yeah, the voltage across the inductor gets that big), L is the value of our inductor, and t is time (as in seconds, microseconds, etc.).
Another way to think of this: Inductors do not like to see instantaneous changes in the current going through them. Yes, one can make the current go up, one can make the current go down, but step changes in ideal (and, pretty much, non-ideal) inductors just don't happen. The inductor will arrange the voltage across itself to ensure that no instantaneous change in current will happen.
So long as we're at it, here's a couple more components. First, a capacitor:

A capacitor is two conductors (usually thin and big in area) separated by a non-conductor: glass, mica, ceramics, or tantalum oxide. They store energy, too, but in the form of an Electric Field (the kind one gets when one makes ones hair stand on with a brush. Unlike inductors, a capacitor not connected to anything but with a charge on it doesn't do anything dramatic like throwing sparks, although it can lead to situations where a tech can put a 200V charge onto a cap, disconnect it from the power supply, then toss it to a victim, yelling, "Hey, catch!". Ahem.
Where inductors hate to see instantaneous changes in current, capacitors hate to see instantaneous changes in voltage, and ideal caps will source or sink enormous amounts of current (i.e., electrons flowing madly in and out) in order to keep step changes in voltage from occurring.
Next, two components: A FET (Field Effect Transistor) and a diode:

That FET, which happens to be of the NMOS (N-type Metal Oxide Silicon), has the Gate terminal on the left, the Drain terminal on top, and the Source terminal on the bottom. When the Gate (on this transistor) is positive, the Drain and Source are connected to each other, usually with a very low resistance (milliohms).
That diode, "D1" is a EE's version of a one-way device. Current can flow through the anode (terminal on top) to the cathode (terminal on the bottom) when the voltage across the device is positive, from anode to cathode. When the voltage is reversed so the anode is negative with respect to the cathode, No Current Flows. (Although, if the voltage gets big enough, then current flows, as well as flying bits here and there, but I digress.)

So, here's the simplified schematic of switching boost regulator:

Now, we get to the good stuff.
Case #1: We're not doing a boost. The controller keeps the FET, M1, off. Current flows out of the battery, through the inductor, through the diode, and into "Rload", which I'm going to claim is everything that makes AC voltages for the motors. (I'll do how you make DC into AC at a later date. Yes, it involves lots more trannies, inductors, and what-all.)
Case #2: Making 600 from 200. Hang onto your hats, here we go.

1. Start: We turn M1 on, ideally, really, really fast. Suddenly, L1 has 200V on one side and 0V (the on transistor) on the other.
2. Remember that bit where inductors don't like step changes in current? Well, that happens here. The current going through the inductor starts increasing madly, also making the magnetic field around the inductor bigger and stronger.
3. The voltage on the anode of the diode, D1, goes to zero volts, right along with the inductor voltage. This is where the diode shows its mettle: There's still roughly +200V on the cathode of the diode, thanks to our handy capacitor which isn't letting its voltage droop much. The D1 is reverse biased, so no current flows from right to left.
4. Current is flowing like mad out of the capacitor into the load, but that's what capacitors do. The cap is keeping the output voltage stable for the moment.
5. I want to note at this moment that, while we're having fun putting energy into the magnetic field of the inductor, no power is actually being dissipated. (Well, a little in the transistor, but power = I*I*R, but R is as tiny as the engineers can make it.) If nothing is actually getting warm (and a changing magnetic field doesn't warm anything, particularly), then we're not dissipating power.
6. Now comes the magic: We turn the transistor Off. Let's just say that we have twice the current going through the inductor at the moment we do this than in the steady state in Case 1.
7. So, what happens? Well, the current though the inductor has to go somewhere, and that somewhere is through that diode. The inductor suddenly develops a voltage, negative on the left, positive on the right, and
8. Whammo! that diode becomes forward biased right quick, the energy in the magnetic field that we just stuck in there gets converted into a current that (a) keeps the load happy, with one-half of our current out of the inductor going there and (b) the other half of that current going into the capacitor, charging it up and making the voltage across it bigger.
9. Before the inductor's quite finished discharging, we repeat the process.
10. Result: The voltage across the capacitor and load goes up. Since I said that we switch the transistor off at the point where the current out of the inductor is twice the normal load current, the voltage will rise towards 2X the original voltage, 400V. If we wait for the current to reach 3x the nominal load current in Case 1, we rise towards 600V. (Niggling experts will point out that it's the average current though the inductor that counts.)
11. Usual trick on the controller is that it generates a pulse train at a fixed frequency. When it wants more voltage/power coming out, it increases the pulse width of the pulse train, forcing more current through the inductor when M1 is on. This is one reason that a boost converter like this is called a PWM (pulse width modulated) power converter. (There are frequency-type converters about.. Doubt if the Prius is using one.)

Weirdly enough, the efficiency of a switching boost power supply like this, PowerOut/PowerIn, can be up in the 90% range. Discharging and charging magnetic and electric fields are, in the first order, non-lossy things to do.

There's a couple more things that were probably done in the Prius's boost supply. First, real diodes, like the ones one buys at Radio Shack, have a fixed, 0.7-1V drop across them when they're forward biased. Since P=I*V, that voltage makes the diodes hot. A common trick is to replace the diode with another FET that is controlled from the controller; by turning it on and off at the appropriate spots, it can be made to act like a diode, and FETs have much lower voltage drops when on than diodes do. Another advantage that occurs to me is that FETs are, by nature, something of a bidrectional device: That is, when they are on, current can flow from the drain to the source or vice versa. It wouldn't take much work to take the above topology and convert it into a scheme where current flows, in general, out of MG1/2 into the battery, as a buck (rather than boost) regulator.
Second, nobody every said that there is a single inductor, transistor, diode set in the Prius. A nickle says that they'd have several of these in parallel, all chugging along in sequence.

Another comment: When one is doing something like this, in general, the higher the switching frequency, the better. A fast switching frequency means that one needs a smaller inductor; smaller inductors tend to have lower losses and don't make the wheels bend down in front.

Now, the good stuff: What about the losses? Where do they come from?

1. First and foremost: Voltage drop across M1. In the Prius documents on file with the NHTSA, they state that these are insulated-gate, bipolar transistors. They have a gate, but what lives inside is a bipolar transistor, not a MOSFET. The advantage of such a beast is that they are rugged. Due to the way that they are built, the physical area in the transistor that gets hot when power gets dissipated is more spread out, keeping hot spots from melting the silicon. The disadvantage is that when they are On, they have a saturation voltage of around 1-2V. P=I*V, and if V is up a bit, then one is going to see lots of power dissipated in the transistor. Not as much as is going to the motors, but still. Which is why they only boost up to 650 or so when you smash the gas pedal; you get more losses, but maybe the designers thought that those losses would be a decent tradeoff in comparison to getting run over by a semi. If you were wondering why the Prius has a separate cooling system for the inverters, well, now you know.
2. Inductors can get warm. Wouldn't be surprised if they were bolted to the heat sink inside the inverter, too.
3. Same for the big capacitors. When capacitors get too warm, they don't work as well and they have their nonlinearities, as well.

Now, why the recall?

The NHTSA report from Toyota says, basically, that the transistors were breaking off the circuit board. Man, I am Impressed.

I have an image of how this "circuit board" is built. First, we got serious power floating around loose in here. What they probably do is solder the transistors to a slab of metal that, in turn, is bathed in radiator coolant. That's good: The coolant isn't going to get much above boiling, and bog-standard silicon transistors can get up to 125C or larger before they go blooey. Second, that's a slab of metal: When I muck with heat sinks, my general feeling is that if one is connected to a slab of metal, that slab of metal is going to be the same temperature throughout. The heat conductivity of metals is pretty high, and the specific heat (how much energy it takes to heat up metal) is very low.

The second trick here has to do with differential expansion. OK: Say I put a device on a circuit board made of fiberglass, and said device has a bunch of solder contacts on the bottom. (These are called, no joke, Ball Grid Arrays, BGAs). If the device and the board get warm with all the heat coming out of the device into the board through those contacts, both the case of the device and the board are going to expand with the heat. Device manufacturers go Out Of Their Way to make the coefficient of expansion of the cases of their devices with heat the same as the coefficient of expansion of the fiberglass circuit board. If one doesn't do this, then the fiberglass board and the case expand at different rates.. And the mechanical stress on those ball grid array contacts can break the contacts right off the board. Been there, seen that, got the tee shirt, not much fun.

Now, consider what a typical one of these power transistors looks like. It ain't no ball grid array:

That's a slab of metal on the bottom of the transistor: Your actual transistor in the Prius may not look exactly like this one, but that Big Metal Tab on the bottom is not for looks, the silicon transistor is pretty much welded to it, under the black plastic. The idea is that the silicon transistor dumps it heat into that slab of metal, and the slab of metal is soldered to the heat sink. Metal everywhere, got it? Great thermal conductivity to the coolant, right?

So, what they're getting is cracked solder joints on the bottom of the power transistors.Amazing. What that means: When the boost is first turned on, so much heat gets dropped onto the transistor tab that said tab expands, before the heat sink warms up and it expands, too. Stress builds up, and enough stress to break the transistor off the heat sink. Which also breaks off its electrical connection as well.

First thing that happens: The Prius electronics detects that Evil Hath Occurred, lights up every warning light in existence, and puts the car into limp mode at the side of the road.

Worst thing that can happen: Enough arcs and sparks (200-600V, remember?) fly around so that the power controller becomes reset, at which point, dead car.

Frankly, I am flabbergasted. The designers over at Toyota must have been, too: Nickel bet says that they got their differential expansion rates right, but didn't consider the dynamic case, where things are just beginning to heat up. Tracking this one down was difficult, I bet, and the documents on the NHTSA web site pretty much say that. I'm not surprised that this is a function of battery voltage and temperature; my gut feel is that the power transistors for the booster likely work harder when the battery voltage is low, so a sudden acceleration under that corner condition with a discharged, or cold battery, might have done the trick.

Solution seems fairly straightfoward, although I guess we'll never know: Don't boost the voltage straight to 650 right off, but sneak up on it for the 3-5 seconds (wild guess) it'll take for the heat sink and transistor temperatures to stabilize. Nice to have the controller under full computer control so they can do this.

Fun problem.

KBeck

 

With the last recall issued for the brake booster, it took quite awhile longer up here in Canada before the dealer had information that even confirmed the situation.

My guess is that Toyota Australia may be like Toyota Canada... a little slower of the mark than the USA outfit, but trust me, they get there in time.

Sounds like for most a trouble free fix.

Roland

 

Thanks for that information Kbeck. Definitely increases my understanding of some of the processes going on inside my Prius' converter.

Also gives me pause as to what I can expect after the update is flashed to my car. It may be that it will still allow full power to flow when needed, but only after a very short period of time where temperatures are allowed to increase on the board to prevent damage. If that is the case, then regenerative braking down some of our long mountains here, or the odd time you have to pass a huge semi truck on a windy road will not be hampered in any way by this recall, if I properly understand.

Roland

 

Hi KBeck,

That's a pretty good description in simple layman's terms, well done!

I've worked in Electronics (Back in the Day, mostly), but recently worked for a company that sold 3-phase 400V Mains Inverters (Variable Speed Drives), saw a few in pieces and chatted with our Technical Manager about them.

The structure of the IGBT "Power Plate" of the modern VSDs (at least the ones we sold) is not really as you describe. I never saw inside one, (power plate) but the transistors (IGBTs) seemed to be fully embedded in the one piece, extensively finned heat sink that formed the back section of the drive. The IGBTs certainly weren't in TO-220 style cases as per your picture. Rather, the inner face of the Power Plate merely had the transistor leads (really, small solder studs) protruding from it. If one transistor blew, that was it for the power plate. This structure is obviously to achieve maximum heat / power dissipation with minimal cost, and presumably, all the thermal expansion calculations and compensations had already been "done and dusted", and embedded in the design.

Anyway, quoting Bob Wilson's post, I'm going to beg to differ regarding the reasons behind this problem.

Bob's post included the quote:

Therefore, I am going to suggest that the Boost Converter problem in the Gen3 is not hardware related.

I agree that it almost certainly uses IGBTs (rather than diodes) to control the power out of the inductor into the capacitor as per your diagrams and explanation. This will presumably allow for the regeneration mode to operate, but even if not, it provides for greater efficiency and more control. So, for the sake of the discussion, let's say that D1 is now replaced with M2, another IGBT that controls the current out of the Inductor (L1) into the Capacitor (C2).

What I'm going to suggest is that the issue is a timing problem with the switching of the IGBTs, somewhere.

I agree that it's likely to be a PWM system, and that the pulse width of the signal to M1 increases to increase the power / voltage to C2. With D1 replaced by M2, another signal is required which switches M2 On at the correct times to allow the voltage / current from L1 to discharge into C2.

Now, what say they made a miscalculation in the timing of the signal driving M2?

For example, what would happen if, in going to full power, as the signal driving M1 gets wider (longer, as it will), the signal to M2 doesn't move out correspondingly?

At some point, both M1 and M2 will start to overlap, so that both devices are momentarily ON together. This may only be for a nanosecond or so, but that is not good. Instead of a controlled flow of current out of the inductor into C2, what will happen is that 650V DC from C2 will start to flow in reverse through M2, and through M1 back to Ground (the other end of C2). This, effectively, is a short circuit across C2, through M2 and M1 in series, and as I know you can well imagine, massive currents will start to flow.

This is the kind of current that could easily lead to "deformation" of the IGBTs, which I interpret as marketing speak for transistors going BANG!

AND, it may only happen in extremely rare conditions, due to a tiny software bug, which can be so easily fixed by a software (firmware, if you will) update, as they are doing.

I can't say that this is the specific problem - it could also be the opposite, where M2 is being turned on late, allowing excessive inductive spike voltages to be developed across M1 (and to some extent M2), but this fits less with the description involving "deformation". This kind of overvoltage on the IGBTs would cause high voltage breakdown, which damages the transistors in a slightly different, often more subtle way, leading to loss of performance, and eventually, most likely, a complete breakdown as above.

Whatever the problem is, it doesn't really matter, so long as the fix is correct, and there is some kind of testing to determine that the IGBTs haven't been degraded, and there is some kind of Warranty Extension to cover failures in this area. Would you agree?