PIP-4048MS inverter

Introductions, general chit chat and off-topic banter.
dakoal
Noobie
Posts: 5
Joined: Sat, 08 Jul 2017, 21:56
Real Name: Bernhard STrunz
Location: Europe/Austria

Re: PIP-4048MS inverter

Post by dakoal » Thu, 12 Oct 2017, 03:04

That's great.

THX

User avatar
weber
Site Admin
Posts: 2229
Joined: Fri, 23 Jan 2009, 17:27
Real Name: Dave Keenan
Location: Brisbane
Contact:

Re: PIP-4048MS inverter

Post by weber » Sat, 14 Oct 2017, 11:52

Hacking the PIP/Axpert firmware to implement a dynamic charge current limit, has turned out to be more difficult than we hoped, but still not as difficult as it might have been. We still believe we can do it without having to patch the SCC firmware. I'll try to explain what we're attempting to do. I must warn you that the following may only be of interest to those with a computer or communications background, but I'll do my best to make it clear to anyone sufficiently interested to put in the work of following it.

Unlike lead-acid cells, lithium-ion cells cannot tolerate overcharging. So we want to be able to send a new maximum current limit to the PIP every few seconds, so that a lithium ion battery management system (BMS) can throttle back the charge current, based on the voltage of the highest cell, and eventually reduce the current to that which the BMS can bypass around the cells which are already full, thereby allowing the other cells to catch up, without damaging the already full cells.

Here's a diagram showing the normal pattern of communication between the PIP/Axpert's main processor, sometimes referred to as "the DSP" (digital signal processor) (a 16 bit TMS320C28x), and the secondary processor in its built-in solar charge controller, "the SCC" (an 8 bit HCS08x).
DSP			SCC
	--RST----->		reset (command)
	<----(ACK--		acknowledge (response)
	<-----QRI--		query rating information (command)
	--(RI...-->		rating information (contains settings) (response)
	<-----QGS--		query general status (command)
	--(GS...-->		general status (contains measurements) (response)
	<-----QGS--
	--(GS...-->
	<-----QGS--		this repeats indefinitely 
	--(GS...-->		until the DSP sends another RST command
	...
In simplified terms: The DSP sends a reset command. The SCC acknowledges it. The SCC then requests the "rating information". The DSP responds with an "(RI" packet, or message, containing the various settings such as float voltage, absorb voltage and maximum charge current. The SCC then repeatedly requests (about once a second) the "general status". And each time, the DSP responds with a "(GS" packet containing the latest measurements of the battery voltage and current, and the stage of charging (off, bulk/absorb, or float). I have omitted some other message types, sent once after a reset, because they are not relevant to our story.

The DSP has two serial ports. The one that it uses to communicate with the SCC, as above, and the one it uses to communicate with an external computer (in our case the BMS master unit), using the protocol described here:
uploads/293/HS_MS_MSX_RS232_Protocol_20 ... pgrade.pdf
You will notice that the protocol used between the DSP and the SCC is very similar to that used between the DSP and an external computer. An external computer can request the "rating information" with a QPIRI command, and the "general status" with a QPIGS command. In both cases (SCC and external computer) every packet (command or response) must end with two bytes of cyclic-redundancy-check (CRC) that allows checking for errors, and a carriage return. That's what makes it a "packet" rather than merely a message. And in both cases a response always begins with a left parenthesis "(", while a command does not. There is no matching right parenthesis.

When an external computer sends one of the commands that change the "rating information", such as PBFT (float voltage), PCVV (absorb voltage) or MNCHGC (maximum charge current), then the DSP's present way of informing the SCC of such a change is to send a RST command to the SCC, which causes the SCC to request the rating information again. Unfortunately, a RST command has the horrible side-effect of causing the SCC to stop charging for about 40 seconds, apparently due to to restarting its maximum power point tracking sweep from scratch. This is completely intolerable if we want to send a new maximum charge current every few seconds.

So we are attempting to change the operation of the MNCHGC command (which we pronounce as the "munch" command), so that instead of sending a RST command to the SCC, it just sends an "(RI" packet containing the new max current, without having been asked for it.

There is no a priori reason why the SCC should take any notice of an "(RI" packet that it didn't request, but fortunately, the SCC code is written in such a way that it does take notice. However, we have implemented the above, and it doesn't work. Sure, the SCC no longer does that horrible reset, with its 40 seconds of no charge, but it doesn't change its charge current either.

Figuring out the reason for this, was a scientific adventure spanning several days. It involved repeated cycles of forming hypotheses and testing them, by (a) discussing them with each other (mostly by email), (b) mentally simulating the existing code in both the DSP and SCC, and (c) performing experiments after loading patched code into a PIP and crossing our fingers that there would not be a loud bang, or a whimper.

There was also one very important case of (d) a lucky accident, where coulomb put a wrong number into the code and sent the "(RI" packet to the external computer instead of to the SCC, and we got to see that it "collided" with an "(ACK" packet that was correctly being sent to the external computer in response to the MNCHGC command! We received "(RCK" followed by the normal CRC and carriage return of the "(ACK" packet, followed by the last few numbers of the "(RI" packet and another CRC and carriage return.

When we fixed this bug, and it still didn't work, and while coulomb was recovering from his hard disk crash, I wondered whether the "(RI" packet might now be colliding with the "(GS" packets that are repeatedly being sent to the SCC. This was confirmed by experiments where, by sending 20 MNCHGC commands at random intervals about a second apart, causing 20 "(RI" packets to be sent to the SCC, the SCC would change to a new current limit, although it wasn't always the current limit I had specified in the MNCHGC command. Presumably, by sending enough times, eventually one "(RI" packet got through, but sometimes with a corrupted maximum current value. But apparently with a CRC that said the message was valid! :o

Some reading of the disassembled code, painstakingly commented over many months by coulomb, showed that the reason for the collisions was that the out-going packet buffer, a region of memory in the DSP, was not implemented as the usual circular buffer which would have allowed two or more messages to be queued up, one after the other, but instead overwrote each message with the next. It assumed that there was a long enough delay between messages, that a message would have been completely transmitted at 240 characters per second, before the next packet overwrote it. But we could not guarantee such a delay between our "(RI" packet and the preceding "(GS" packet.

But I reasoned that the original "RST" packet had exactly the same problem, of possible collision with a preceding "(GS" packet. So all we had to do was figure out how the designers of the DSP firmware avoided such collisions in the case of the "RST" packet, and apply the same method to our unrequested "(RI" packet. Our various hypotheses about how they did that, turned out to be wrong, until eventually coulomb showed that in fact they did not avoid collisions when sending RST packets. They just kept on sending the damn things until one finally got through! They would know one had made it thru when an ACK was finally returned. So it was like this.
DSP			SCC
	--(GS...\		crash
	--RST---/
	--(GS...\		crash
	--RST---/
	--RST---\		crash
	--(GS.../
	--RST----->		reset (command)
	<----(ACK--		acknowledge (response)
	<-----QRI--		query rating information (command)
	--(RI...-->		rating information (response)
	<-----QGS--		query general status (command)
	--(GS...-->		general status (response)
	...
:o Extraordinary! There was no way coulomb and I were going to implement something that awful for our unrequested "(RI" packet. For one thing, it is a much larger packet, due to all its settings numbers, and it would be sent far more often. And for another thing, coulomb and I have some pride in our workmanship. :geek: :ugeek:

So yesterday we got together for another 12 hour intensive brainstorming and coding session, fueled by tea and pizza, and we believe we have something that has a good chance of working. Unfortunately, it was out of the question to implement a proper circular buffer, as it would have required patching every command in the system, So we just patched the sending of the "(RI" packet so that it would check if there was another packet in the buffer, still being sent, and if so, disable sending temporarily while it added itself after the end of the existing packet, instead of overwriting it, and fooled the sending code into sending them both as if they were a single packet, except it had to ensure that the second CRC was calculated only for the second packet, not the combination of the two. This is so complicated that coulomb will first check it in a TMS320 simulator, before we are brave enough to test it on the real thing. :)
One of the fathers of MeXy the electric MX-5, along with Coulomb and Newton (Jeff Owen).

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Sat, 14 Oct 2017, 14:12

So if you get it working, we'll be dynamically setting the voltage, right? Not the current.

Which makes sense: when we want to stop charging, we want the current to be 0A (at float voltage), not 2A (the minimum current you are able to specify).

User avatar
weber
Site Admin
Posts: 2229
Joined: Fri, 23 Jan 2009, 17:27
Real Name: Dave Keenan
Location: Brisbane
Contact:

Re: PIP-4048MS inverter

Post by weber » Sat, 14 Oct 2017, 14:40

Hi andys. I'm afraid we really are working on dynamically setting the current, not the voltage. But it will not be limited to the values you can presently set. You will be able to set it to any number of (whole) amps from 0 to 140 inclusive.

Our BMS has a bypass current of 0.65 amps, so in this way of operating, when the first cell becomes full and we are balancing the cells, we would be telling the PIP to charge at 1 amp for about 65% of the time, and 0 amps for the rest of the time, so the average would be 0.65 amps. So about every third command would be MNCHGC0500 while the others would be MNCHGC0501.

This would happen automatically due to the operation of the PI control loop we would be running in the BMS master, to keep the voltage of the highest cell constant, until all the cells are full. During that time, it wouldn't matter what the battery voltage setting of the PIP is, provided it is higher than the actual battery voltage.

We already use this method successfully in charging an electric vehicle with 208 LFP cells in series. However this is done with a TCCharger that already had dynamic current control, not a PIP.
One of the fathers of MeXy the electric MX-5, along with Coulomb and Newton (Jeff Owen).

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Sat, 14 Oct 2017, 15:20

Thanks for the explanation weber. I can still work with that, but if it stops receiving commands (something broken or an emergency), could it default to 0A instead of 2A?

User avatar
weber
Site Admin
Posts: 2229
Joined: Fri, 23 Jan 2009, 17:27
Real Name: Dave Keenan
Location: Brisbane
Contact:

Re: PIP-4048MS inverter

Post by weber » Sat, 14 Oct 2017, 16:36

You make a good point -- that you might well want it to go to zero current if communication was lost -- and indeed that's what the TCCharger does. But up until now, we have been writing the PIP patches so that it falls back to whatever is set in the EEPROM, in other words, whatever was last set using either the LCD and its buttons, or a standard MNCHGC command (i.e. one whose argument has not had 500 added to it). I thought we were going with hennejg's suggestion in that regard, but in fact he wrote:
hennejg wrote:For robustness reasons [they] would remain in effect only for a few seconds unless refreshed continuously which would let the charger fall back to safe defaults [if] the charger/BMS communication was lost.
And the EEPROM value is not necessarily a "safe default". In fact the lowest value of maximum total charge current that can be set with the LCD or a standard MNCHGC command is 10 amps. You are probably thinking of the maximum utility charge current setting (or the MUCHGC command) when you mention 2 amps. We pronounce that as the "much" command as opposed to the "munch" command. You can think of the N as standing for "entire", i.e. solar plus utility, as opposed to the U for utility alone. We're not planning to fiddle with the "much" command at all.

So we'll give your suggestion some serious thought. If anyone has any ideas about when it should go to zero current versus when to go to the EEPROM value, or whether we should try to allow the EEPROM value to be set to zero, please let us know.
One of the fathers of MeXy the electric MX-5, along with Coulomb and Newton (Jeff Owen).

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Sun, 15 Oct 2017, 05:46

Zero would be great. I always thought it was a serious deficiency that they don't allow you to simply stop charging completely (except by swizzling voltages).

(There really should be a physical input terminal that disables all charging)

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Yesterday, 08:47

Does anyone know if the remote on/off inputs can be "daisy chained" ? Can I switch on/off two or more inverters at the same time by connecting their NC/NO contacts in parallel or series and a single switch?

paulvk
Groupie
Posts: 122
Joined: Fri, 23 Oct 2015, 23:45
Real Name: Paul
Location: Sydney

Re: PIP-4048MS inverter

Post by paulvk » Yesterday, 12:02

I would use a 4 pole switch and 8 wire cat5 cable you can do 4 that way.

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Yesterday, 12:08

Sure - but I actually wanted it to be computer controlled and 4-way or 4x relays is overkill unless absolutely necessary :)

paulvk
Groupie
Posts: 122
Joined: Fri, 23 Oct 2015, 23:45
Real Name: Paul
Location: Sydney

Re: PIP-4048MS inverter

Post by paulvk » Yesterday, 12:27

Computer control easy I can do that already , you only need a relay with 4 poles less than $5.
Do you want control with a web page so any device can be used eg smart phone?

andys
Groupie
Posts: 61
Joined: Wed, 20 Jul 2016, 19:44
Real Name: Andrew Snow
Location: Sydney

Re: PIP-4048MS inverter

Post by andys » Yesterday, 13:34

paulvk wrote:
Yesterday, 12:27
Computer control easy I can do that already , you only need a relay with 4 poles less than $5.
Do you want control with a web page so any device can be used eg smart phone?
No, I've got a little embedded computer which will power up extra units depending on time of day, load, and battery %soc .. do you have a part number for a cheap 4P relay?

Post Reply