Pico crash/reboot on EEPROM.put()

[myklemykle]

Hi,

My Settings object uses a struct "_Settings" to store misc settings in NVram, using the Arduino-Pico EEPROM emulation. It looks like this:

// slow, heavily logged version:
bool Settings::put(){
  Dbg_println("going to put settings"); 
  Serial.flush();
  delay(5000);
  EEPROM.put(0, s);  // object member s, a _Settings struct
  Dbg_println("put settings");
  Serial.flush();
  delay(5000);
  EEPROM.commit();
  Dbg_println("committed settings");
  return true;
}              

And this has worked through several versions of that Settings struct. But today, after I've modified the struct, all further calls to EEPROM.put() seem to cause my Pico to restart. (I only see the first line "going to put settings", then it reboots.)

The new struct "s" is much like the old struct, but with a new boolean member & a 32-bit integer changed to 16-bit.

// New Settings:
typedef struct {
  uint16_t _flag;
  uint16_t _version;
  unsigned long measureLen;
  unsigned char outs[4];
  bool imuTested;
} _Settings;

/*
// previous Settings:
typedef struct {
        uint16_t _flag;
        unsigned int _version; 
        unsigned long measureLen;       
        unsigned char outs[4];
 } _Settings;
*/

Is there some reason why EEPROM.put() would choke on this structure? Could some content of the variables themselves cause this? Is there a size limit? Any other gotchas?

[earlephilhower]

Can you make an MCVE (minimal sketch to reproduce)? That would help others to look into it.

[myklemykle]

Working on it. For the record, tho: does calling EEPROM.put() from core 0 require that core 1 be available? I have presumed not. But in my getting-more-minimal sketch it appears that even if setup1() and loop1() are both empty, pausing core 1 gives me the bug, whereas allowing core 1 to run does not. If EEPROM.put() uses both cores, that would be one explanation.

[myklemykle]

Tell me this: is it safe or unsafe to call EEPROM.commit() from core 0 if I have a custom setup1() and a custom loop1() ? Because i'm definitely doing that.

[earlephilhower]

Yes, it's supported. That's the whole reason for the idleOtherCore/flash-write/resumeOtherCore. If it wasn't, there'd be no need to touch the other core. :)

[myklemykle]

So to clarify: you seem to be saying it is unsafe to call EEPROM.commit() from core 0 if I have a custom setup1() and a custom loop1() unless i idle the other core first and resume it after. Is that right?

[earlephilhower]

No, the exact opposite. As a user you can run your own setup1/loop1 and never ever call rp2040.idle/resumeOtherCore and write to flash via EEPROM or LittleFS. The Arduino core plumbing here does all that for you. In fact, I'd say 99.999% of users should never call rp2040.idle/resumeOtherCore.

If you try and freeze the other core while it's already frozen (i.e. if you called rp2040.idleOtherCore in your setup() and then tried to do an EEPROM.commit(), then the commit would wait forever since core1 will never respond to the interrupt...).

[myklemykle]

In my sketch I have an audio subsystem running on core1. setup1() sets up PWM audio on gpio pins, starts DMA transfers to PWM, installs an ISR to write the audio buffers the DMAs read, and starts another PWM to trigger the ISR.

I find that if I don't call rp2040.idleOtherCore() at the beginning of setup(), and rp2040.restartCore1() at the end of setup(), the audio subsystem fails to initialize. I didn't dig into the real cause, because adding those two lines fixed the problem. My thinking was just "wait until setup() is finished before running setup1()" .

There are those two calls in my setup() routine, but there are no other moments in the entire codebase where idleOtherCore() is called. So I don't get how that effects EEPROM.commit() when I call it ten seconds later. Core1 is definitely not frozen at that time. But it's true: if I leave out those two commands in setup(), then EEPROM.commit() works fine. But on the other hand I have no audio.

I am just wondering if there is something else I have to do from core0 to make core1 leave flash alone during EEPROM.commit() .

[myklemykle]

Ok for instance: I'm using a pico-sdk command to set up the ISR on core 1.

irq_set_exclusive_handler(PWM_IRQ_WRAP, &RP2040Audio::ISR_test);

Could that result in arduino-pico not disabling the ISR during the commit?

[earlephilhower]

All the FIFO freeze-core code is in https://github.com/earlephilhower/arduino-pico/blob/master/cores/rp2040/RP2040Support.h

As long as you don't touch the multicore FIFO interrupt, all is fine. But, you need to make sure interrupts are enabled on the 2nd core so the FIFO interrupt will be serviced at some point. And be aware you can't service your audio IRQs while the core is frozen for the EEPROM write (i.e. underflow will occur).

[myklemykle]

MCVE: https://github.com/myklemykle/syncmaker_EEPROM_troubles

[myklemykle]

OK, I've replaced that idle & restart strategy with having setup1() wait to start until setup() signals its done via the FIFO. With that, everything seems to be working. I don't even hear any skipped samples during the write. Thank you for elucidating all this.

It still looks to me that doing either freeze&continue or freeze&restart of core1 during setup() can cause EEPROM.commit() to be flaky later on. If that's not expected behavior, I can try again to get a smaller working example of the problem. If it is expected behavior, I'd like to add it to the docs.

[earlephilhower]

It still looks to me that doing either freeze&continue or freeze&restart of core1 during setup() can cause EEPROM.commit() to be flaky later on.

Your call if you want to dig into it, but this is definitely not expected nor does the IRQ code have any side effects as far as I can see. The codepath really is just a normal IRQ on the slave core:

Core0: Send an IRQ, wait for ack, <do fast work>, clear busy flag
Core1: Get FIFO IRQ, set ack, busy-wait until busy flag == false, return from IRQ

[myklemykle]

OK, good to know. Thanks again for all your help!

[myklemykle]

FWIW, in the end this turned out to be a really basic out-of-memory problem. When I reduced memory allocation elsewhere in my sketch, the problem went away. So it wasn't really related to EEPROM flashing at all, except that EEPROM routines apparently use RAM.