Solana stakeholders scrambled to fix the network Saturday night after what one whistleblower called an “incredible amount of data” flooded the proof-of-stake chain, pulling validators out of consensus and halting block production. .
The bots invaded the popular non-fungible token (NFT) minting tool known as the Candy Machine early Saturday with an unprecedented tsunami of incoming traffic: four million transaction requests and 100 gigabits of data every second, a record for the network. , a person at the Solana Foundation said.
For reasons that remain unclear, this swarm kicked out the consensus validators. Block production became impossible and the network shut down at 4:32 PM. ET. At 11 p.m. m. ET, the validators (coordinated via Solana’s Discord channels and a Google document created by one of the validators) rebooted the cluster in slot 131973970.
Co-founder Anatoly Yakovenko, who said he was on the road for much of the fight, credited the validator community for spearheading the mainnet recovery. He had been criticized on Twitter on Saturday for allegedly being “MIA” during a network crisis.
Unlike the 17-hour outage last September, Saturday’s hard fork restart was not resolved by the new and improved code populated in validators. It just picked up where the network failed seven hours earlier.
In preparing for the reset, validators considered whether to implement code that would temporarily block Candy Machine transactions. Some debated on Discord whether such a move constituted censorship. It would only be effective, however, if two-thirds of the validators opted to participate. Few seemed to do so on Saturday night.
Other parts of the ecosystem moved quickly to bolster their defenses. At 23:36 ET, Metaplex, Solana’s key NFT infrastructure manager and one closely related to Candy Machine, tweeted that it would soon be implementing a 0.01 SOL “botting penalty” to help NFT projects stop excessive traffic. .
Solana ecosystem services such as the Phantom wallet and decentralized exchange Mango Markets struggled to hold their own after RPC node providers came back online.
The outage contributed to a bloody, albeit brief, drop in the SOL markets. The native Solana token plunged to a 24-hour low of $83.13 roughly three hours after the outage before rising back to $89, according to CoinGecko.
Solana’s core developers have yet to diagnose what went wrong on Saturday or how the apparent bot attack bypassed existing safeguards to hinder consensus, a foundation member told CoinDesk.
