Harmony Protocol Went offline for 7 hours on Friday as heavy spam traffic halted the main shard of the network.
The official Twitter handle of Harmony explained the issue on Saturday morning, January 15, after resolving the issue. According to the core team, the network suffered multiple block propagation issues for the last 30 hours due to heavy spam traffic. This led to shard 0, the main shard of the blockchain, stopping working. Shard 1 was also affected.
1/ We are undergoing an outage due to multiple block propagation issues. Heavy spam traffic in the past 30 hours has halted the network.
— Harmony 💙 (@harmonyprotocol) January 14, 2022
Our core team is working with validators & internal leader nodes to resolve the outage.
Minimum gas upped to 30 gwei to prevent spam attacks
What Happened Throughout This Outage?
The official Reddit of Harmony explains the cause of the outage as:
“An internal leader node of the Harmony blockchain became overwhelmed by the spam traffic following a network upgrade and ended up forking the chain which then resulted in a consensus not being able to be met and the network halting.”
To lower the spam traffic, the team increased the network gas fees and asked validators “to downgrade their node binaries to a previous version as the bandwidth of validators with V4.3.2 were overburdened.” Furthermore, the team sat down to work on a “special version of validator node binary to reduce bandwidth usage” was testing internally the new versions of shard 0 to restore the blockchain.
At 3:00 AM Pacific Standard Time (PST), the Harmony core team announced that it had provided to the validators of shard 0 and 1 with a new version of the node binary. The network remained halted but the bandwidth was reduced.
4/ Progress Update – 3:00 AM PST
— Harmony 💙 (@harmonyprotocol) January 14, 2022
Validators on Shard 0 and 1 were provided with a new version of the node binary by the Harmony Core team.
Network remains halted but bandwidth usage has reduced.
At 5:00 AM PST, the team announced that it had also designed a specific validator binary to combat spam and noted that “the spam that halted the network was NOT of arbitrage trading bots.”
According to the Reddit page, at 9:00 AM PST:
“New build was released to validators on shard 0 and 1. The blockchain can now be run but needs to achieve 67% consensus. Waiting on external validators on blocks 0 and 1 to execute the upgrade. We are getting close! Currently at 54% consensus.”
At 9:40 AM PST, Shard 0 reached consensus for a single block, briefly reaching 67% consensus. However, the consensus percentage dropped again and the shard 0 was unable to reach consensus for the next two blocks: 21705524 and 21705525.
At 10:35 AM PST, the team said:
“We were able to reach consensus once again getting us to block 21705526. Validators are still actively working with harmony to get shard 0 off the ground again.”
At 12:42 PM PST, the team announced that shard 0 and 1 were producing blocks but had not achieved 2 second block times. It was 10 seconds block time at that time.
Finally, at 2:49 PM PST, the Harmony team rolled a new binary to all four shards that restored that block time. “Harmony is starting to appear as if it has recovered,” reads the Reddit page.
The current status for all four Harmony shards now shows “online”.A post mortem report will follow.
Visit the official Reddit page of Harmony to see the full details of actions taken by the Harmony core team during this incident.
