This blog post reveals a threat to the Ethereum network that existed between the Merge and the Dencun hard fork. Prior to the merge, different message size limits were put in place to prevent denial-of-service (DOS) attacks. These limits were applied to messages received via HTTP endpoints and were carried over to the engine API, which plays a crucial role in connecting Execution and Consensus Layer clients during block production. However, this also meant that blocks could be produced that surpassed the RPC size limits of some clients but still fell within the acceptable range for others.
This created a vulnerability where an attacker could craft a message that exceeded the size limit of the client with the lowest setting, while still adhering to the gas limit requirements. This could result in a situation where some clients would accept the block as valid, while others would reject it, leading to HTTP error code “413: Content Too Large.”
The impact of this attack is significant as it could allow an attacker to force the majority of nodes (especially geth) to reject blocks that a minority would accept. This would cause forks in the blockchain and the proposer of the block would miss out on rewards.
Initially, it was thought that this attack could only be carried out using modified clients or builders. However, it was later discovered that it was possible to trigger the limit by having a client with a higher limit propose the block, and the CL requesting validation of this proposed bigger block.
To address this issue, a solution was proposed to temporarily lower the RPC limit on all clients to the lowest value (5MB). However, it was later discovered that an attacker could create a block that would hit the 5MB limit with smaller transactions, resulting in a chain of forks and reorganizing of the blockchain.
In response, it was decided to increase the RPC limit on all clients to prevent this attack. The timeline of events leading up to the discovery and resolution of this issue is as follows:
– 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) try to submit a specific transaction to the network that could result in up to 2.7 MB blocks when snappy compressed.
– 2024-02-06 13:25: Pari receives errors from his local Geth node, indicating that the transaction should be valid.
– 2024-02-06 15:14: Justin successfully puts the transaction in a block and submits it through the Besu client.
– 2024-02-06 20:46: Sam (EF) alerts the team about certain Sepolia nodes struggling.
– 2024-02-06 21:05: The team confirms the bug with Marius from Geth.
– 2024-02-06 21:10: The team begins debugging.
– 2024-02-07 23:40: It is decided to limit the RPC request limit to 5MB for all clients.
– 2024-02-07 6:40: It is discovered that there may be a bigger issue with the attack being possible with smaller transactions.
– 2024-02-07 10:00: It is decided to increase the RPC request limit for all clients.
– 2024-02-07 21:00: The fix is merged in Geth.
– 2024-02-09: Geth is released.
The issue was resolved in subsequent releases by the individual client teams, including Geth v1.13.12, Nethermind v1.25.4, Besu 24.1.2, Erigon v2.58.0, and Reth v0.1.0-alpha.18.
