When The soft fork did not go according to plan. Not only did the consensus protocol have unforeseen side effects (DoS vulnerability), but a data-race was also discovered in the rushed implementation, making it a ticking time bomb. With a rapidly approaching hard fork deadline, a new strategy was needed.
The A postmortem was conducted in order to identify the root causes of the issues, and to provide recommendations to prevent it from happening again. An interesting discovery was made during the autopsy: the soft forking code inside [go-ethereum](https://github.com/ethereum/go-ethereum) was robust from all angles; it was fully covered by unit tests (3:1 ratio), reviewed by six foundation promoters and even manually tested on a private network. However, there was still a fatal race for data that could have caused a serious network disruption.
It Turns out that the flaw could only be detected in a network with multiple nodes and miners, all minting blocks simultaneously. Even if these scenarios were true, there was a slim chance that an error would occur. Unit tests and code reviews are not enough; manual tests are unlikely to uncover edge cases. The conclusion was that developers needed more tools to test complex interactions between multiple nodes in a concurrent network environment. Manual checks of such edge cases are too cumbersome to be effective, and rare bugs will not be identified without continuous tests in the development process.
And thus, Hive was born…
What is the Hive?
Testing Ethereum implementations had become a difficult task. Unit tests can be useful for validating quirks and multi-client compatibility, but validating that a client meets a certain quality is not easy. Hive is an extensible test harness, allowing testers to use any language they are comfortable with to write tests. The harness ensures that the client is up and running before the tests are executed, and it provides checks for compliance with specifications and behaviors in different conditions.
Most importantly, Hive was created with the purpose of being integrated into any customer’s CI workflow.
How Does Hive Work?
Hive’s body and soul is [docker](https://www.docker.com/). Each client implementation is a dockable image, each validation suite is a dockable image, and every network simulation is a dockable image. With the Ethereum clients available as docker images, Hive can help developers create the right environment (dependencies, tools, configuration) for the client. Several instances of the same application can be activated, each running independently on Linux systems.
Test suites are also dockable images. Testers can use any programming language they are familiar with. Validation clients are also docker images. They can be used to verify that a client has the desired behavior. Network simulations, finally, are also defined by docker images. Unlike tests, simulators can not only run code against a client, but they can also start and stop clients at will. These clients can run on a virtual network and connect to each other, creating an Ethereum network on demand.
What Role Did Hive Play in the Fork?
Hive is not meant to replace comprehensive review or unit testing. For a successful implementation of any feature, these practices are essential. However, Hive can also provide validation beyond what can be done by traditional methods. For example, extensive tests in complex environments can be run to check for possible corner cases, something that would take hours to set up manually.
In the case of the DAO hard fork, it was important to ensure that, in addition to all the consensus and unit testing, the network was divided into two separate subsets: one supporting the fork and one opposing it. Three network simulations were implemented with Hive to verify miners, the splitting of the network when the fork arrived, and the ability of new nodes to sync and light-sync the chain of their choice.
Did Hive catch the error? Did it just provide additional confirmation that everything is fine?
