Since my last post, MeXy has given joy rides/drives to three more of her contributors, Newton (Jeff Owen) the brains behind most of the mechanical engineering that went into her, and
Tritium engineers James Kennedy and Alex Rudzki who tested and supplied the brilliant 750 volt WaveSculptor motor controller and the two open-source EV Driver Controls units (EVCUs), and helped us whenever we managed to screw things up with our software or our motor tuning parameters.
Two posts back I mentioned a destroyed cell. It's time to tell that story. Like all great engineering disasters, three or four things had to go wrong at once to bring it about. Not that this was a great engineering disaster.
One Sunday night, many weeks ago, I convinced my wife to let me drive us in MeXy, to a birthday party for her mum at Pinjara hills. Regular readers of this column may remember what has happened on other occasions when my wife has been a passenger. The word "Jinx" comes to mind.
MeXy was not fully charged, but the "fuel" gauge showed about 70% and it was only a 35 km round trip (MeXy has easily a 130 km range). We got there without incident, but on the way home I noticed that the cell stress gauge (re-purposed oil gauge) was showing increasing stress spikes on acceleration and hills, and soon our EVCU software began to dial back the WaveSculptor's power limit. As soon as I could do so safely, I pulled over to consider our options. The fuel gauge showed about 60% and yet at least one cell was clearly going undervoltage. How could we be running flat with the gauge showing 60%?
It was around 9 pm. Janelle resignedly suggested we just phone the RACQ and get a tow-truck, but I said "Where's the fun in that?". We were near a major shopping centre, and I remembered Jeff Owen's stories, so we crawled into the shopping centre and crawled around the outside looking for power points. Way around the back of the shops, where normal people don't go, we found one and plugged in. I went to find a loo, leaving Janelle with the car.
A little while later a cleaner came out, looked at the lead coming out of MeXy's closed boot, followed it with his eyes down to the ground and up to the power point, and said, "What's going on here?". Janelle replied, "It's an electric vehicle. We just need a little power to get home." I came back and said with a smile, "Yeah, it's an electric vehicle, want to see under the bonnet?". He replied with the immortal words of
Sargeant Schultz, "I don't want to see anything. I didn't see anything.", and went back to work.
We noticed there was a security camera, with its ring of infrared LEDs, looking pretty much straight at us. I figured we needed to charge for an hour to get home, and thought it was best if we weren't there if someone else discovered the lead, so we went for a walk to circumnavigate the shopping centre looking for other power points in case we got moved on. And indeed, when we got back, the cleaner was waiting, and said he was leaving now and asked if we would please do the same. So we moved to where we'd found a DVD vending machine plugged into a double power point, and plugged in beside it.
At one stage a security vehicle came past and stopped for a long hard look. The charging lead was not visible from their point of view, Janelle was reclined in the passenger seat with her eyes closed and I was about 20 metres away studiously pretending to use my phone. They drove on. After the hour's charging at 10 amps (240 Vac) we drove gently home without further incident, and with only the tiniest indication of cell stress as we drove up the final slope. Hoorah!
I did not plug it in to charge when I got home, because I suspected a badly out-of-balance cell and a complete charge would have balanced it. Although that was kind of silly because every CMU (Cell Management Unit) records its cell's lowest and highest voltages and temperatures until manually reset via a laptop.
Because of various other commitments it was almost a week before I got to look at the problem. I hooked up the laptop and was told by the CMU on cell A62 that it had not heard from cell A61 for some time. And when I finally got physical access to A61 I found it at 1.5 volts (too low for its microprocessor to operate).
Any lithium ion type cell that has been below about 2.0 V (continuously in its rested state as opposed to briefly under load) loses capacity and becomes an unpredictable danger, and so it had to be replaced. Why? Search on "copper" in this document:
http://americansolarchallenge.org/ASC/w ... ontrol.pdf
The cell had also bulged with pressure due to whatever non-condensing non-dissolving gasses are formed under these conditions, presumably by breakdown of the electrolyte or its solvents. It beats me what the supposed vents on the tops of these (Sky Energy) cells are for, since they never seem to relieve this pressure. So I needed to completely remove the battery box to undo the cell-row clamping-rods sufficiently, to get the cell out. Cell A61 is in the forward fuel-tank box, and this requires first craning out the rollbar and its battery box. Thanks again to Hunter (my son) and Kurt (offgridQLD) for helping me with this process.
The question of course is how did it get down to 1.5 V? It clearly wasn't anywhere near that when we got home. We'd have to have been crawling the last several kilometres in turtle mode.
I will eventually describe all the compounding causes, but the final stage of its demise was "Death by CMU". As I jokingly said to Coulomb, "I hate it when Jack Rickard is right".
We clearly have a problem with the MOSFETs that we use to switch the bypass resistors on and off, because they occasionally and randomly go leaky and start drawing about 10 mA from the cell (averaging about one failure out of the 218 every 6 months). Normal CMU current is only about 1.6 mA. So as well as the cell, I had to replace the MOSFET on its CMU (an
Si2312BDS).
Any ideas about the cause of this in-service onset of leakage, and what to do about it, would be welcome. The MOSFET simply switches 6.6 ohms of resistors across the cell, via a fuse and a diode. It has a 47k gate pulldown and its gate is driven directly from the MSP430 microcontroller powered from a regulated 2.5 V supply.
A clue to the second partial-cause came when I asked my recent off-grid solar power system customer (the one with Black Monolith #1) how his system had fared during recent consecutive days of overcast weather. He said the State-of-Charge meter had never been below 80% and "Could there be something wrong with the meter?". Combine this with my own question, "How could MeXy be running flat with the fuel gauge showing 60%?", and a light-bulb came on (a highly efficient LED light bulb
).
The
Black Monolith uses the same BMS as MeXy, including the IMU (Current Monitoring Unit), but because of the much longer time-constant of its PI-control-loop (which, in the end, we couldn't use), it samples the current-shunt and sends status-bytes at a leisurely 1.9 Hz compared to MeXy's frenetic 15 Hz. Because of this, I modified the coulomb-counting code to handle different update rates differing by powers of two. The coulomb-counter is reset at full-charge (when all cells are bypassing) and so it really measures Depth-of-Discharge (DoD), which is converted to State-of-Charge (SoC) for display by subtracting it from 100%. During testing of the Monolith's coulomb-counting code, I decided it was overestimating the Depth-of-Discharge by a factor of two, and thought I must have been out by one shift-instruction, and so I "fixed" it. This code was later propagated back into MeXy.
So what was happening with both MeXy and the Monolith was that they were underestimating the DoD by a factor of two. 50% SoC on the gauge was really 0% SoC. And 60% on the gauge was really only 20% SoC.
The third contributing factor was my failure to fully recharge, and hence balance, the "A" half-pack for weeks prior to the run-flat, which allowed cell A61's leaky (10 mA) MOSFET to pull it 20% below the other cells. But why didn't I fully recharge the A-pack?
Contributing factor number four is the fact that even when MeXy is not being driven she is programmed to turn on the traction battery contactors as needed, so the DC-DC converters can recharge the 12 V auxiliary battery. Except that I still haven't got around to replacing the microcontroller in the EVCU for the "B" half-pack with one whose analog section isn't blown up, and so it fails to participate in these events, and only the "A" half-pack expends itself in maintaining the auxiliary battery. This leads to the A-pack taking longer to charge than the B-pack, particularly since MeXy was not being driven very often back then, and when I did decide to drive her I was impatient.
So there you have it.
Behavioural lesson: Fully recharge your EV at least once a week so balancing can compensate for any differences in discharge between CMUs.
Technical lesson: Find out why these MOSFETs are failing, or failing-that, just replace them with devices having better values of all the specs that could matter. And otherwise try to ensure that all CMUs always draw the same small current.
One of the fathers of MeXy the electric MX-5, along with Coulomb and Newton (Jeff Owen).
