Among 31 teams from 12 organizations competing in this year’s SONiC Hackathon, one project earned exceptional enthusiasm from the developer community. The Most Wanted by Devs Award was awarded to the SONiC COWBOYS team from Alibaba Cloud’s Network R&D Department: Jiayi Hu (team lead), Yubin Li, Fengsheng Yang, and Shijie Yang. Their submission, COWBOYS, is a highly flexible, parallelized virtual testing framework that dramatically accelerates SONiC development workflows.
Introduction
SONiC COWBOYS is a highly integrated and efficient virtual framework that operates 24/7. It consists primarily of three components: BMS (Bare Metal Server), VendorSIM, and Jenkins. Traditionally, running a full test suite on a single T0 testbed using modules, fibers, fanout, servers, and switches takes more than 30 hours. However, COWBOYS can simultaneously execute up to eight testbeds per BMS, significantly shortening test time. It can also simulate a wide range of testbed topologies and run test cases under different environments, including large-scale and production scenarios.
The team’s solution was selected by SONiC developers across the community, earning the Most Wanted by Devs Award.
The Problem They Solved
First, let’s address the problems the team was facing. In today’s development and testing environment, there are several challenges. The two major problems are as follows:
1. Lack of resources.
Single-machine functional testing requires a significant amount of physical equipment. For example, building a standard ASW topology from sonic-mgmt requires modules, fibers, fanout devices, servers, switches, and more.
2. Lack of flexibility.
Current tests cannot easily change the test topology at any time. Because of the resource limitations above, teams must design a “perfect” topology that can support many test requirements with minimal changes. In reality, it is difficult to find a single topology that fits all situations.
There are also some minor problems, such as difficulty debugging in a terminal, long execution time (a single full test round can take more than 30 hours), and heavy migration effort from existing tests.
Their Hackathon Solution
To address the problems mentioned above, the team designed and built COWBOYS as the solution.
COWBOYS is made up of two main components: vSONiC and Jenkins.
vSONiC is a virtual switch running on BMS that can simulate physical devices almost 100 percent accurately, which means the team no longer needs a large amount of physical hardware. It also supports easy migration of existing test cases from physical environments to virtual ones.
Jenkins provides a platform that allows the team to quickly launch different devices and topologies. It also enables test cases to be split into multiple groups that run in parallel on different vSONiCs on the BMS, greatly reducing overall test time.
Taking Broadcom chip devices as an example, the team used BCMSIM as the ASIC model and used real AliNOS to drive it. This allows vSONiC to simulate almost the entire pipeline and all SAI/SDK logic. They also designed a framework to ensure that multiple vSONiCs can communicate with each other.
Impact & Benefits
With the basic vSONiC test framework in place, the team can build their own topologies using vSONiC and run any test they need.
They arranged all management cases and topologies for ASW/PSW/DSW and divided them into separate jobs to ensure that a single device model can complete all of its cases within four hours. Jenkins can now launch all parallel jobs with one click and run them non-stop, 24 hours a day.
In theory, the total testing time for all topologies (ASW + PSW + DSW) on a single device has been reduced from 30 hours to just 3 hours, a 90% improvement.
Lessons Learned
The team noted that it is important to focus on general, architecturally reusable efforts that improve efficiency. This mindset reflects the true spirit of open-source communities and aligns with their original motivation for building infrastructure.
Next Steps
- The team hopes to custom-create large-scale simulations to test capabilities, while also simulating online production environments to reproduce and locate issues.
- They hope to collaborate with the community to promote vSONiC testing with more chip emulators.
- They hope to continue improving the visualization of test results.
