Solana outage caused by JIT bug

On February 6, 2024, at 09:53 UTC, Solana Mainnet Beta halted block finalization due to a bug linked to the JIT caching system in the validator software.

The issue was traced to a legacy loader program triggering an infinite recompilation loop, which stalled validator nodes and disrupted consensus across the network.

Engineers released a patched version v1.17.20 and coordinated a restart from slot 246,464,040, with consensus resuming at 14:55 UTC.

The root cause involved how the LoadedPrograms system handled legacy deployments, leading to repeated recompilation and blocking validator progress.

A mitigation was applied by disabling the legacy “v2” loader, preventing the conditions that caused the bug. A full fix will be included in future releases.

FAQ

What caused the Solana outage?

A bug in the JIT cache triggered an infinite recompilation loop due to legacy loader behavior.

How long did the outage last?

Approximately five hours, from 09:53 to 14:55 UTC on February 6, 2024.

What was the fix?

A patched release disabled the legacy loader and allowed validators to restart from a safe slot.