My Project Lithium Battery Caught Fire

FYI: @jacktheripper is now libeling my name in private communications with other users. It's sad to see Jack stoop to this level.

As Jack's comments are intended to disparage my qualifications as it pertains to electric vehicles, I would like to make the following factual statements regarding my professional career (warning: these are not humble because I am defending my reputation):
1: I am an electrical engineer specializing in energy storage systems and power conversion products. I graduated Magna Cum Laude from Vanderbilt University.
2: I began my professional career at National Instruments as an analog design engineer, where I learned the engineering process from several legends in the industry.
3: Eventually a startup I was working on at night took off, so I left NI and worked there for over a decade. During this time, I designed numerous products as the startup's primary design engineer. To date, we haven't issued any recalls on any products we've shipped; not even one. Company is still going strong today, with over $100M in revenue. I still advise said company, but am no longer involved with day-to-day.
4: I 'retired' from said startup in 2022, so that I could focus full time on my next passion project: designing a safe lithium conversion product for the G1 Honda Insight. I want to stress that this is a passion project. I am not in this for the money. I am fortunate to be in the position that money is no longer a concern.
5: Prior to beginning this project, in 2022 I supplemented my formal engineering training by reading six separate textbooks focused solely on lithium energy storage design theory (Fundamentals and Applications of Lithium-ion Batteries in Electric Drive Vehicles, Battery Modeling: Battery Management Systems Volume 1, Battery Management Systems of Electric and Hybrid Electric Vehicles, Equivalent Circuit Methods: Battery Management Systems Volume 2, Battery Management Systems: Accurate State-of-Charge Indication for Battery Powered Applications, & Battery Management Algorithm for Electric Vehicles).
6: I absolutely love engineering. It is my one true calling.
7: Almost everything I design is open source, including the entire G1 Honda Insight product I sell (for way below market value). I absolutely encourage others to remix my designs as they please, and I encourage and support those users that choose to do so. As soon as I can post links, I will post the entire schematic for my battery management computer... I encourage everyone to compare my schematic to any other design on the market.

Please use the above background when you're evaluating any statements Jack may send your way.

...

Despite Jack's statements, I want to reiterate that I'm rooting for Jack and his project. Unfortunately, there are some critical safety issues with his existing design that I feel morally obligated to post publicly.

When I first discovered these issues a few months back, I decided that since nobody had yet reported a thermal event, I would only post a single video outlining my concerns (search youtube for "Concerns with Project Lithium's Toyota Prius NexCell Lithium Module Design"). However, now that at least one thermal event has occurred, I've decided to more vocally raise my concerns in this community, with a primary focus of informing the public about these design concerns.

 

Finally, a couple people have asked (both publicly and privately) why I didn't end up working with Jack.
A: Initially, I left the ball in Jack's court; he chose not to continue pursuing a partnership, and;
B: after reviewing their tech stack, I decided it had negative value (i.e. it was a liability). The primary component Project Lithium was offering to bring to the table was their lithium cell design, which performed poorly in my personal testing. Since I was bringing everything else in the design to the table, I didn't see a reason to partner with them at that time and;
C: Project Lithium isn't the only potential partner in the 3rd party hybrid vehicle market. More details to follow at the appropriate time, but in short: I've moved on.

So then to specifically answer the question "why didn't mudder end up working with Jack?":
Somehow, I was still on the fence until mid-December. At that time I had my first (and only) conversation with Jack (prior to that I was amicably discussing issues with his staff). To say our conversation didn't go well is an understatement. It's unfortunate that Jack's takeaway from that conversation somehow fails to recall just how concerned I was for his customers' safety. Yes, I was direct in that regard. And here we are.

My primary motivation for posting in this thread is to provide technical guidance to the community, based on my professional engineering experience. I fully concede that I'm not a truly unbiased party in this matter. After all, I'm a potential competitor, but right now that isn't relevant to my intent: I want to make sure this community is purchasing safe products.

To that end, I once again offer to review Project Lithium's schematics, if Jack will make them available. For example, I would love for @jacktheripper to show us his apparently now-discontinued "Signal Soother" schematic - it's no longer mentioned anywhere on his website - so that I might offer a 3rd party analysis on the design intent and implementation. Note that I have zero intention to "rip off" this product, both because it's ill-conceived; and also because I intend to entirely replace the OEM Prius computer (just as I did in the G1 Honda Insight).

One final point regarding any concerns that I'm only disparaging Jack because I'm a potential competitor:
When lithium modules catch on fire, that affects the entire aftermarket lithium conversion space, not just the actual company that designed it. Project Lithium is making my job harder not because they might eventually compete for sales, but because their reputation - good or bad - will ultimately rub off on any other company that attempts to enter the market. It is for this reason that I hope Project Lithium pulls a rabbit out of their hat with their next hardware design (V3?)... I don't want to end up cleaning up their mess.

...

Overall, I do hate to come out swinging here, but I absolutely will not tolerate someone disparaging my name. I have sent @jacktheripper a direct message stating the same, and I strongly encourage him to heed it. Jack, you are letting down your community.

 

My whole point this entire time has been, versus pointing blame and trying to shame, what are we doing to resolve the issue. Thats been my stance in all of my words. Considering I was the one who let him know about the reddit post, I can say he didn't speak ill of you when he had more than one opportunity. I go back to the energy surrounding all of this, and like I stated before... as a beta tester, you don't get the freedom to talk negative madness and if you believe X, Y and Z to be A then why not assist in making the batteries what the community needs? Which is a reliable alternative to OEM.

The notion of shaming someone on the internet who is just as fallible as everyone here just baffles me. To then hear a guy talk down on himself being a mailman and not smart enough yet, he feels his words and his intellect is smart enough to say how wrong someone is while attempting what others didn't. Am I just looking at this in a different light?

There is nothing in the previous two posts before this one that does anything to get us all to a resolution of comfort. It just more salt on the wound and slows Jack down with all the static in the air. heh.. and I get called out for taking longer than some think I should to provide refunds and yet they discourage people from making a purchase because they believe they know exactly how I operate and do business based on assumptions and imaginations of grandeur.

If you aren't here to help get to a resolution then you are in the way making this all harder than it needs to be for the rest of us.

 

I understand you aren't replying to me directly, but I'm choosing to respond to your statements as if you had.

I outlined my proposed steps to resolution in my first post in this thread. Jack still has not addressed my concerns.

Correction: "He didn't publicly speak ill of you".
I prefer a public bashing, wherein I can at least defend myself.

My intent is to do precisely that, just without partnering with Project Lithium. See my previous post for my rationale.

This isn't a public shaming, per se.
Rather, this is the fallout from an insufficiently engineered product that caused a thermal event in a customer's car. I'm aware that Jack offers top notch private refunds whenever there are customer issues. However, his lack of public transparency is the primary contributor to any perceived shaming you see herein. Couple that with three released hardware revisions so far (V1/V2/V2.5), for a SIL2 safety critical product that also (briefly) had a "Signal Soother" man-in-the-middle, and there's a design pattern screaming for attention.

I'm anxiously aware that eventually one of my customers (either Honda Insight, or later Toyota Prius) will experience a thermal event in their car... it's bound to happen as the user base grows and time marches on. And when it does, I will own it publicly, and keep everyone up to date on what happened, and not trash talk people who know more than I do behind their backs, and I will admit my shortcomings (of which there are many).

Jack can end all this right now by making a public statement identifying the numerous shortcomings in the existing product (or even just refuting those I outlined previously in this thread); outlining how customers who have existing hardware can safely continue using it until a safer design exists (if possible, e.g. remove Signal Smoother, monitor pack with Dr.Prius app, etc); and explaining how he intends to fix said shortcomings with a future design. Once again I will suggest that Jack's silence is telling.

 

I have no dog in this fight, but I do have some strong feelings, as I've been debating about purchasing one for the last several months. Heck, I was ready to pull the trigger on the one that was for sale on the forum by the member in Florida, but someone beat me to it after he updated his post with more info....

Back when this was first being brought into production, we had a member who had significant battery manufacturing experience that made his opinion known, including about liability and insurance. He was ignored and bashed a bit. All we've heard for years from those same people is how great this battery is and how it's the future of the Prius and on and on and on and please buy one using my link. Now, suddenly this other information is coming into the light and one has to ask how much was known by this group prior to this thread and while hawking how great everything is and how 'Upgrade V.x" is the bomb, with no mention of the true 'why' about what made the 'upgrade' needed.... I can say my purchase interest is on hold, depending on how this gets resolved.

It wasn't that long ago, that some celebrities who endorsed a certain crypto business got their 'buttocks' sued off when that business was found to be lacking. I'm curious what the liability would be for an 'endorser' who knowingly convinces people to buy a product, and earns a fee from that purchase, if they know it's flawed (and possibly has safety circuits bypassed?)....I suppose one bad fire, accident, injury or death + one lawyer would show what liability is all about...if it's actually the case, a safety being bypassed in a commercially available product could be absolutely catastrophic for everyone involved.

This is a truly interesting thread...

 

I can empathize with you just like I can those who stand outside of buildings protesting how they feel about something. If you need me, ill be here with open arms and large ears to listen. Protesting, That is your right, but at what point can a human set aside their feelings and beliefs so as to not cause discomfort to a random person trying to get to work... and you are lying on the highway blocking traffic? Heck, what if someone was on their way to the hospital and passed away because of you wanting to express how something made you feel by protest or lots of words on the internet? goes back to everything I've said in all this, WHAT ARE WE DOING TO HELP VERSUS KICK DOWN? I wonder if Tesla owners are at Elons neck on the internet or if other vehicle owners are proceeding the same way if there is a battery guy for them like Jack? just all this noise is bonkers really, but hey, its the internet! Yeah, your energy drowns out the words and it becomes a burden to read.

heh, you are faulting a man for releasing updates? I mean, isn't that what the companies who make the software you are using to type on do as well? Where were you beating down Toyota's door when you had to replace the head gasket? In the grand scheme of things even this is just selfish.

maybe I am the only one that has this visual from the energy behind your words.


even the one who had the battery hiccup isn't as a flame on energy, at least that is my perception.

 

I like your analogy, just think with all the speculation and assumptions by those who might not be intertwined into development as they thought or simply not at all that gets mixed with freedom of speech... you end up with a group of people with PTSD from prior experiences unintentionally making a situation far worse then it truly is. I don't say that with any experience at all, just something I read on a blog recently.

Separately, its pretty insane that as humans we naturally gravitate to the worst. It's as if no one has witnessed positivity out of the hurdles that lead to a battered reputation. hahaha next thing you know you have someone thinking they could get a battery in Florida and truly excited about it to them wondering if what's going on is synonymous with a crypto business celebrities lost their money on. and now, I am stuck with that thought in my head. Boy I am glad the words of those online don't influence me, one could be a Terrified Terry and swear off nothing not OE just on that thought alone!

 

When you manufacture a product for public consumption, you better have your sht together, because regardless of what your 'energy' thinks, this is a sue happy world. There's a HUGE difference in liability between a product causing a problem due to an unknown defect and a product causing a problem due to a known defect that was defeated by bypassing a safety device.

You buy a smoke alarm and install it in your house. Unfortunately, you find out it triggers even when you smoke a cigarette in the next room. The manufacturer realizes it made 5000 smoke detectors with the wrong sensitivity settings, so they send everyone a zip-lock bag to put around the smoke detector circuit board so it will stop alarming when you smoke that cigarette. Two weeks later the house burns to the ground while everyone is sleeping. There's some injuries. Hope there's a lot of positive energy about how to help that company.

 

I never disagreed with this point at all, but the fact remains nothing is 100% perfect and flawless. We are driving cars that a manufacturer appeared to have their shit together and here we are left replacing head gaskets at a significantly high average rate. But hey, I miss the 80's, the Eddie Murphey RAW days where a joke didn't offend anyone and people didn't seem to be as sue happy. Either way, all the assumptions are for the birds.

So while I have my 2013 Plugin and my 2008 running grade B cells from the manufacturer in question, I am not over here shaking in my boots that something could go wrong. For me, I encourage shit to pop off. This way, I can figure out a way to resolve the weak link.

 

Thanks for reposting the pics. The pics in your quoted post are much clearer than when I first saw them at the pic hosting site.
I got to the OP before there were any comments.... When I went to look at the set of pics on the other site my browser crashed and computer locked-up. Nothin new about my machine locking up on certain web sites because it's an really old machine with an unsupported, bleeding edge, customized installation.
The pictures I see in the quoted post as not at all what I remember seeing on the other site, especially the pic of the (middle connector blade) which looked like it was the most damaged and I was thinking what might have caused that?
In the OP and on the other site I couldn't even tell what that gray was. Now I see that gray is smoke.
@sworzeh I hope your new OEM pack works out well for you.
@jacktheripper thanks for posting the analysis of the malfunction.

I see that @ mudder is now a priuschat member, I read your post at insightcentral about your desire to make a prius lithium pack, and that it wouldn't be to hard to compete with the Nexcell pack. I just hope none of your lensight packs ever have a safety issue like the one in this post and your development process continues unharmed with those lithium packs.

Jack, I wish you a quick recovery from this incident, and getting all things in your life back to normal. .

 

Hmm, flurry of posts since I looked last.

The Signal Soother is still of key interest to me.

TheLastMojojomo has one specific idea of what the Signal Soother does:

That idea may be right, but I can't add my own opinion behind it before I've seen more of the Signal Soother. For now, I have to allow at least the possibility that it could have more interesting internals and be doing something more than just that.

mudder and I have now both offered to examine a Signal Soother if anybody has one to examine.

At this point, it seems there's a lot that could be learned from examining that thing.

Certainly, if anybody else has an adverse event, I'd advocate for hanging onto the Signal Soother, rather than just discarding it, or sending it back right away.

If anybody has an event that incurs an insurance claim, the Signal Soother maybe should be provided to the insurer. An insurer might have someone they could contract with to examine and report on it.

If anybody has one of these batteries and just stops using it for some reason, or the car gets crashed or something, a chance to examine that Signal Soother would be nice.

There's one way we can keep any discussion of the Signal Soother a little bit simpler:

It's sufficient to say "altering voltage readings". Only voltage readings arrive at the battery smart unit via that voltage sensor harness; therefore, voltage readings are all the Signal Soother (which is just interposed into that harness) can directly alter. Internal resistance values are derived in the ECU, using those voltage readings and the battery current sensor reading the ECU also has.

Naturally, any way that the Signal Soother changes the measured voltages, that will also affect the ECU's calculated internal resistance values. But sometimes we write as though the Signal Soother is involved in 'transmitting' both voltage and IR readings, and that makes it sound like something extra complicated. All it does is sit in the middle of a bunch of wires whose voltages get measured.

 

Tesla and Toyota doesn't implement any kind of cell balancing. The NiMH are all in series and they can last 500k miles without implementing a balancing circuit board. Tesla recommends to charge to 100% in certain intervals to aid in cell balancing. I believe simple is the best way to make something last long. I don't think LFP needs balancing. When a cell is over charged it should dissipate excess energy as heat as long as it is not dangerously overcharge. Toyota hybrid systems will only charge to a certain point max and that shouldn't cause harm to a pack. Unless a module fails which should trigger alerts to pull over. I think the problem is the slight mismatch in overall voltage and keeping them in range per modules for Toyota specs to see them as normal.

 

Mine are not "feelings and beliefs", but rather facts and concerns.
If I am "lying on the highway", it is to bring public awareness to engineering design deficiencies. However, I'll suggest I'm "standing outside of buildings protesting what I think about something" (underlined word substitution is my own).

As it pertains to your analogy, I am not physically impeding Jack's ability to perform commerce with willing customers. Instead, I am informing passers-by about possible safety issues with his product. Of course, Jack can dispel (or confirm) my assertions simply by sharing any recent schematic (e.g. V2.5, V3 in progress, and/or Signal Soother). Hiding behind the "trade secrets" mantra is laughable at this point. Who reading this thread would walk away thinking "yeah, I'm gonna copy that design!". Once again I'll note that my entire open source design is available for free right now. I suggest any potential competitors should start there instead.

I've actually done a lot behind the scenes, as I've outlined previously. My sincerest apologies to Jack that his inability to publicly address our concerns - with even a cursory statement regarding the facts - has compelled me to discuss this publicly.

Five seconds on teslamotorsclub should thoroughly answer your question.

What words? Jack isn't saying anything. There's nothing to "drown out". Apologies my writing style is verbose; I'm an analytical engineer.

Incorrect. I am faulting a company that failed to understand even basic concepts regarding how to safely design SIL2 products. I'm repeating myself, but one need not look further than either the now discontinued "Signal Soother" or my V1 hardware design analysis video. Both of these products contain numerous Consumer Safety 101 level defects. Based on my engineering experience while reviewing the publicly available information, I am almost certain that V2 and V2.5 hardware has not resolved these issues (that I previously outline).

To be clear, then: I am absolutely onboard with releasing properly engineered updates as the design improves. However, my primary concern here is that engineering best practices were thrown out the window while developing this product. As others have mentioned previously in this thread, it's one thing to DIY your own battery solution if you don't know what you're doing... but it's something else entirely when you commercialize said product and distribute it to unsuspecting customers.

Let's briefly review how I've handled my similar G1 Honda Insight product:
-My RevA prototype PCB had several non-safety-related design issues. It never shipped to customers. Maybe this is Jack's "V0" hardware?
-My RevB PCB fixed the issues I had discovered during my validation testing. At this point I announced a "public alpha", to properly message that the product was still a prototype. Maybe this is Jack's "V1" hardware? Unfortunately, my alpha testers discovered one major (non-safety) issue that intermittently caused check engine lights in some vehicles. That is the desired outcome with a SIL2 level design. Before I had a chance to make a new PCB, I used my engineering experience to safely temporarily fix the issue on the existing hardware, and then sent replacement hardware to every alpha customer. Specifically, I did not release a "Signal Soother" to mask my product's design defects, but rather I redesigned the errant subsystem and then reworked it onto the RevB PCBs. I even helped one alpha user reinstall it in her car while she was driving cross-country to Canada.
-My RevC PCB fixed the issues discovered during the alpha period. To date, no additional hardware issues have been discovered. In fact, I still ship RevC units (at a slightly discounted rate).
-My RevD PCB only exists due to parts shortages resulting from the COVID panic buying.
-Note that today I am still representing my product as an "open beta", to properly message to customers (and potential customers) that there could be issues with the design. At this point the primary reason I'm still in beta is due to shortcomings in the existing firmware. I am actively working on hitting firmware version 1.0.0 ASAP.

Absolutely, but once again for safety-related designs, these companies are (hopefully) using rigorous engineering design best practices, including test fixtures and fault insertion testing. I will concede that my product's test coverage isn't as broad as I would like, but it certainly covers every safety critical component that could cause a product-induced thermal event.

Note however that "companies who make the software you are using to type on" aren't SIL2 safety products. In fact, if you read any consumer level computer's legal agreements, you will find a statement similar to "don't use our product in mission critical applications, where a failure for the product to perform could cause death or serious injury".

FYI: Prepare to spend big $$$ if you want to buy a computer that omits this disclaimer. These safety systems typically run realtime operating systems designed from the ground up with safety as the number one priority.

This isn't the first time I've spoken up with my engineering hat on. Whenever I'm an affected party, I will absolutely lead the charge. In fact, I would counter that this is precisely what I'm doing here now. In researching several lawsuits filed against Toyota, I absolutely love how applicable the various preliminary statements are to this thread: "Vehicle manufacturers have certain basic rules and procedures that must be followed. When a vehicle manufacturer sells a vehicle, it has a duty to ensure that the vehicle functions properly and safely for its advertised use and is free from defects. When a vehicle manufacturer discovers a defect, it must explicitly disclose the defect and make it right or cease selling the vehicle."

This is an excellent AI image. Unfortunately, it's an ad hominem attack, and I don't really see how it applies here. Maybe this is my own blinders. Maybe if the baby cried in such a way that they presented logical arguments in a public forum?

I think ultimately the general layperson might misunderstand just how much engineering effort is required when designing a product that can cause death or serious harm.

Still waiting on Jack to say anything else in this thread to address the numerous issues I've brought up. Instead, he continues privately telling people whatever he can to tread water. I love his private statement to me that "You don't just put NMC chemistry battery in a Prius and put a computer on top to control it, it is suicide". That's probably news to every EV manufacturer doing precisely that.

And yes, NMC is more dangerous when a thermal event occurs. Spoilers: the trick is to design a battery management system that can keep the battery within its safe operating area, or else fail in a safe manner when something goes wrong. It's not perfect; EVs still catch on fire. The difference is the engineering effort most manufacturers put into their designs to minimize the chances these thermal events happen. And of course also to inform the public when the designs are insufficient, and then of course update the designs.

 

thank you for saying this! now here is someone that knows something good.

 

This is what i dont agree with the most... why introduce a bms when the car didnt come with one from the factory? one should simply be able to connect alternative cells that are within the same volts and the system will accept them. mind you, my only knowledge in this is from the PHEV standpoint and I'm going for a lot more battery capacity then everyone else. I could be partial to knowledge that only applies to the PHEV and its additional ecu with all the extra data space on board. Just no time to figure out how to PHEV convert a Prius just yet.

 

I, too, hope this never happens. See my previous statements regarding that eventuality, which I've reposted below for your convenience:

 

Exactly right.

 

I disagree with that statement. Every single Tesla BMS I've reviewed incorporates per-cell voltage monitoring and balancing circuitry. The latest Tesla product I've reviewed in the Powerwall 2, which also had per-cell balancing circuitry. If Tesla has removed these elements from later designs, that's news to me (and also it would be terrifying). Please post links to any first hand knowledge you have regarding this alleged removal.

Typically the balancing element is a very low current passive element, which is sufficient for well-matched cells. And certainly if the cells eventually end up unbalanced the Tesla BMS knows about it and can make safety calls if things get too bad.

I agree with this statement. As I stated previously, most NiMH cells have an internal chemical process that allows top-balancing without external circuitry.

I don't have any notes as it pertains to this conversation. Ok.

Clarification: The safety issue I'm discussing isn't directly due to the presence or lack of balancing circuitry. Instead, my focus is on cell voltage monitoring. However once you implement that into the design, adding cell balancing is trivially easy, hence pretty much every commercial lithium product on the market incorporates both.

So then from a safety perspective, LFP doesn't require balancing circuitry, but consumer products should certainly have per-cell voltage monitoring circuitry. Without it, there's no guarantee that a massively imbalanced cell (e.g. that has failed) will be detected by the system.

Of course, there's always the argument that LFP is safer when overcharged... and yes, that's true. But it's still not absolutely safe, and of course over/under charging LFP dramatically reduces cell lifetime, so it boggles my mind that you wouldn't include a safety system to properly detect cell failures and then report them to the system and/or user.

I disagree with this statement. LFP cells do not contain internal self-balancing chemical properties. Specifically, they lack the hydrogen evolution reaction, which is then catalytically recombined with oxygen produced at the nickel electrode. As long as the charge current is low enough, this safely results in cell heating, which effectively prevents the NiMH cell from overcharging.

LFP cells do not contain this chemical process. External electronic circuitry is required to balance LFP cells.

This is only true if the cells remain balanced. Suppose the system is QTY3 cells in series, and each cell safely operates between 3 and 4 volts (for example only). If you're only looking at the total stack voltage, you would conclude that the system is safely operating if the total voltage is between 9 and 12 volts. However, without looking at each cell, it is possible to achieve a safe stack voltage (e.g. 10 volts), wherein one or more cells are operating at an unsafe voltage (e.g. 2, 2, and 6 volts as an extreme example).

Bingo. That's the key safety issue that's lacking here. This is why lithium battery management systems measure every single cell voltage in the stack.

Correct: The underlying issue is ultimately due to mismatched cell voltages. Based on both my personal testing with these modules, and also what I've heard in my discussions in this community, I would propose that a contributing factor here is designing/specifying a cell that cannot handle the extreme demands of a hybrid traction battery.

...

I will once again reiterate that Jack can easily put any uncertainties I have to rest by sharing any existing Project Lithium schematics. If that's too much to ask, an abstracted high level diagram outlining the overall system architecture would suffice. Said abstraction would require enough transparency to understand the general per-cell battery management strategy.

 

I think my jaw just hit the floor.......

 

I think your jaw landed on my jaw.