Summary

Rapidly power-cycling an STM32H7 flight controller can tear a flash write and leave a 256-bit word with a double-bit ECC error. When PX4 reads that word while loading parameters during early boot, the H7 raises an uncorrectable ECC fault that hard-faults the board on every boot thereafter. An MCU mass erase is the only fix. The proper fix, merged in PX4 1.18, is a bootloader scrub that DMA-scans flash and erases any sector with an uncorrectable error before the app runs. If you run PX4 on an STM32H7 board with flash-based params, update your bootloader using SYS_BL_UPDATE.

Have you ever spammed the on/off button on your power supply just for fun? I was bench-testing an ARK FPV, rapidly power-cycling it to chase down an unrelated power sequencing race condition, when out of nowhere the flight controller started hard-faulting on every boot. I reflashed the app firmware and the bootloader, but the hardfault kept occurring… exciting… This was a failure mode I’d never seen before. Naturally, I dropped everything I was doing.

Reading the backtrace

When PX4 hardfaults it dumps the backtrace to the console:

arm_hardfault: Hard Fault escalation:
arm_hardfault: PANIC!!! Hard Fault!:arm_hardfault:      IRQ: 3 regs: 0x380032a4
arm_hardfault:  BASEPRI: 000000f0 PRIMASK: 00000000 IPSR: 00000003 CONTROL: 00000000
arm_hardfault:  CFSR: 00008200 HFSR: 40000000 DFSR: 00000000 BFAR: 081eac20 AFSR: 00000000
arm_hardfault: Hard Fault Reason:
up_assert: Assertion failed at file:armv7-m/arm_hardfault.c line: 173 task: nsh_main
...
arm_registerdump: IP: 00000000 SP: 38003378 LR: 08160e03  PC: 0816e322

The fault status registers tell the story:

RegisterValueMeaning
CFSR0x00008200Precise data bus fault (BFARVALID + PRECISERR)
BFAR0x081eac20Data address that faulted
PC0x0816e322Instruction that triggered it

CFSR flagged a precise data bus fault, BFAR holds the address of the load that faulted, and PC points at the instruction that triggered it. Cross-referencing BFAR against the linker script puts the faulting address in the last sector of flash — the sector used for parameter storage on boards that don’t include an FRAM. Mapping the PC back through the .elf file lands on find_entry(), called from parameter_flashfs_init(): the fault happens while loading parameters from flash early during app boot. To confirm the root cause, I hooked up an ST-LINK and read the flash status register, FLASH_SR. It had latched a double-bit ECC error — DBECCERR.

Wtf is a double-bit ECC error?

A double-bit ECC error is two flipped bits in a single flash word — enough for the hardware to detect the corruption but not to fix it. On an STM32H7, that turns an ordinary flash read into a bus fault.

Flash on the STM32H7 doesn’t store raw bytes. Memory is organized into 256-bit words, and each word carries ten extra ECC bits computed when the word is programmed. On a read, the hardware silently corrects any single flipped bit, and it can detect — but not correct — two flipped bits. That second case is the double-bit error.

So how does a word end up with two bad bits? The H7 programs a full 256-bit word in one operation, writing the data and its ECC bits together, and a word can only be programmed once before its whole sector has to be erased again. If power drops partway through that program, the ECC no longer matches the data and the word is left inconsistent. The next read detects the mismatch and flags an uncorrectable error. This is a torn write.

Why reflashing doesn’t fix it

Flashing a new app or bootloader doesn’t touch the parameter sector. The only thing that actually clears it is erasing the affected sector. On every H7 board that stores parameters in flash, this is one torn write away.

How do we fix this?

My first instinct was to throw Claude at the problem. It came back with an in-app hardfault handler that catches the condition, erases the bad sector, and reinitializes parameters. It looked like it would work, but it felt fragile: you’re recovering from inside the fault handler of a running RTOS, the repair only fires if and when the bad word is actually read, and recovery means a silent parameter reset. This was not an adequate solution.

So I went digging to see if ArduPilot had hit this. Sure enough, in 2024 someone reported the same thing: rapid power-cycling, then a hardfault on every boot. Bingo! Their fix was a DMA flash scrubber in the bootloader, and that’s the approach I mirrored.

There’s one trick that makes this possible. You can’t scan flash for the bad word with the CPU, because the act of reading it is what causes the fault. But a bus fault is a CPU exception — it fires because the core issued the load. A DMA engine is a separate bus master. When DMA reads that same corrupt word, the flash controller just latches DBECCERR in its status register and the transfer completes — no fault. This is exactly what ST documents as the way to find ECC errors in flash without tripping the core (see AN5342).

Cortex-M7 CPU DMA engine separate bus master Corrupt 256-bit flash word torn write → inconsistent ECC Precise bus fault → HardFault, every boot FLASH_SR: DBECCERR set transfer completes — no fault load read

So the bootloader now scrubs flash before it ever hands control to the app. A mem-to-mem DMA read walks the application and parameter sectors in chunks; after each chunk it checks FLASH_SR, and any sector that comes back with an uncorrectable error gets erased. What gets erased decides what happens next: a scrubbed parameter sector re-seeds on the next boot — from the SD-card backup or from defaults — and a scrubbed application sector fails its image check and waits in the bootloader for a reflash.

On an ARK FPV (STM32H743) the scan adds about 50 ms to a clean boot. When it actually finds corruption, recovery runs about 850 ms. The error condition is now self-healing, and you’ll never even know that it occurred in the first place.

The full change — bootloader scrub plus the fault-injection tooling — is in PX4 PR #27642.

What you should do

If you’re running PX4 on an STM32H7 board with flash-based parameters, update your bootloader. PX4 1.18 ships this fix for all STM32H7 boards with flash-based params. You can update straight from the running app: set SYS_BL_UPDATE to 1 and reboot, and the app reflashes the bootloader on the next boot.

ARK FPV flight controller

Affected board · STM32H743

ARK FPV Flight Controller

USA-built, NDAA-compliant, and the board this bug was found and fixed on. Ships with PX4 by default — update to the PX4 1.18 bootloader and the scrub-and-recover is automatic.

Now, back to spamming that power supply…

Jacob Dahl

Jacob Dahl maintains PX4 Autopilot and leads software engineering at ARK Electronics, a US-based manufacturer of NDAA-compliant drone and robotics electronics. Find him on GitHub.

Leave a Reply

Your email address will not be published. Required fields are marked *