It started with a crowd. A swarm of tiny mechanical bots poured through the town gate, all shouting the same request at once, again and again and again. The couriers got buried, the validators couldn’t hear each other over the racket, and then all over town the clocks stopped.

Mr. Croak immediately opened a stall selling Pre-Owned Seconds. Nobody bought any. They were all watching the clock tower, where Vera, Pim and a crowd of engineers were climbing up with lanterns and toolboxes.

This chapter is a story, but the stops were real. On 14 September 2021, Solana’s mainnet beta stopped producing blocks for about seventeen hours. Bots flooded the network with transactions during a token launch, validators ran out of memory and crashed, and they couldn’t agree on the state of the chain. Engineers wrote a patch, and operators coordinated an upgrade and a restart.

It happened again after that. On 6 February 2024 the network stopped for about five hours because of a bug in how validators cached compiled programs. Most of the stake was running the same version of the software, so hardly anyone could vote until a fix came out and operators restarted.

Both times the work was unglamorous. Find the bug, write it up in public, fix it, and change things so the same problem is less likely to come back.

Up in the clock tower the engineers pinned a big card to the wall that said WHAT NEXT. Everyone had a part to play, and the steps only work if you do them in the right order.

Help them pick each next step. Pick wrong and Vera raises one eyebrow. (Vera’s eyebrow has restarted more networks than most engineers have.)

At last the clocks shuddered, coughed and started ticking again. A cheer went up. Pim delivered seventeen hours of letters in strict order, and down in the square, people who’d never met before started talking about what they wanted to build next.