Authors: Prabhu Sreenivasan, Partha Dutta, Shirisha Dasari, Sachin Suman at Broadcom Inc.
The SONiC Hackathon 2024 was a great success, with teams across the globe presenting innovative solutions to tackle key challenges. Today, we are highlighting the hackathon entry from Broadcom, “SAI conformance test”, which was awarded “Most Wanted by Devs” award this year.
As the world of open-source networking evolves, maintaining seamless integration and interoperability between components like SONiC (Software for Open Networking in the Cloud) and SAI (Switch Abstraction Interface) is essential. This post dives into an innovative solution presented during a recent hackathon, aimed at enhancing the SONiC-SAI integration process by automating SAI API conformance testing. By leveraging the open-source SAI specification, this tool brings a new level of reliability, vendor neutrality, and efficiency, making it invaluable for the SONiC community.
The Challenge of SAI Conformance in SONiC Integration
The rapid growth of SONiC has created a pressing need for efficient, reliable, and standardized testing of SAI implementations. Here is why this is critical:
- Preventing Regressions: Without robust conformance testing, regressions in SAI implementations can go undetected, potentially disrupting SONiC functionality.
- Vendor Neutrality: A common testing standard ensures that SONiC can work consistently across various vendors, avoiding dependencies on any specific silicon or software provider.
- Streamlined Onboarding: For new silicon vendors aiming to enter the SONiC ecosystem, having a standardized SAI testing tool simplifies the onboarding process and reduces integration complexity.
These challenges highlight the need for an automated, specification-driven test infrastructure that can validate SAI implementations across various hardware platforms and configurations.
SAI Specification-Driven Test Infrastructure
Our hackathon project addresses this problem by proposing a SAI spec-driven test infrastructure that is both vendor-agnostic and adaptable to changes in the SAI specification. Here’s how it works:
- Vendor Agnostic: The solution operates independently of any particular vendor, making it versatile for diverse hardware environments.
- Comprehensive API Testing: By validating SAI APIs against all possible attribute values and types, it ensures rigorous testing coverage.
- Future-Proof Design: Allows a seamless future adoption of those objects/attributes by SONIC.
- Adaptive to Spec Changes: If the SAI specification is updated, the test infrastructure automatically adapts without requiring code regeneration.
This innovative approach leverages the existing SAI Metadata for automated attribute validation and integrates with a newly developed tool, SAI Specification Conformance Test, which checks a SAI implementation for adherence to the official SAI specification.
Understanding SAI Metadata and the New Tool
What is SAI Metadata?
SAI Metadata comprises auto-generated data and functions derived from SAI headers, enabling validation of SAI attributes. Generated in ANSI C, the metadata is available as both source code and a shared library. It includes several essential tools:
- saimetadatatest: Tests for metadata accuracy.
- saisanitycheck: Verifies integrity across SAI modules.
- saidepgraphgen: Generates dependency graphs.
- saiserializetest: Tests serialization of SAI objects.
Introducing ‘SAI Specification Conformance Test’
The newly developed SAI Specification Conformance Test tool extends the functionality of SAI Metadata by directly testing a SAI implementation against the official specification. This tool performs CRUD (Create, Read, Update, Delete) operations on SAI objects and records the results in a JSON report, providing a transparent overview of compliance status and potential issues.
How the Test Infrastructure Works
Our solution thoroughly exercises the SAI CRUD APIs, covering a wide range of attributes and values. Key features include:
- Attribute Value Combinations: The tool tests various combinations of attribute values, ensuring extensive coverage.
- Scalability Testing: It can handle large-scale testing for selected objects and attributes, simulating real-world usage and edge cases.
- Support Detection: The tool identifies which objects and attributes a particular SAI implementation supports, which is invaluable for compatibility assessments.
- Conformance Verification: It verifies whether each attribute’s properties comply with the SAI specification, providing an additional layer of accuracy.
The resulting JSON report is versatile and can serve multiple purposes, such as identifying breakages during continuous integration, and helping quality assurance teams track coverage and conformance.
What Sets This Tool Apart?
This automated testing tool offers several unique features, including:
- Object Dependency Handling: It automatically identifies object dependencies, creating necessary default objects and storing them in a reusable object pool.
- Flexible Attribute Values: Attribute values are intelligently chosen based on data type, with default values that can be overridden by user input or randomly generated for thorough testing.
- CRUD Operations Support: It performs CRUD operations across all supported object types or on specific objects, following attribute flags like mandatory-oncreate, create-only, and read-only.
- Skip Invalid Combinations: By recognizing valid-only and conditional statements, it can skip invalid attribute combinations, streamlining the testing process.
This solution also complements the SAI-PTF, which covers functional aspects, by focusing specifically on API syntax and compliance, making it an ideal choice for thorough API conformance testing.
Benefits for the SONiC Community
This solution offers several benefits to the SONiC community:
- Enhanced Reliability: By identifying regressions early, it reduces the risk of integration issues, ensuring SONiC’s robustness.
- Time Savings: The tool automates testing, reducing manual effort and accelerating the onboarding process for new vendors.
- Increased Interoperability: With a standardized approach to testing, it supports seamless SONiC operation across multiple hardware platforms, advancing the goal of a truly interoperable network OS.
Evolving with SONiC and SAI
Looking ahead, this solution has the potential to become a cornerstone of SONiC’s integration process, expanding its scope to cover additional aspects of SAI compliance. Possible future enhancements include:
- Automated Documentation Generation: Using test results to automatically generate up-to-date documentation, providing transparency and easing the work of documentation teams.
- AI-Powered Test Optimization: Leveraging machine learning to identify the most critical test cases, optimizing test execution time, and focusing on high-impact areas. This can also be a dashboard of summary of test results for quick insight of the spec test.
Conclusion
Our hackathon project — SAI Specification Conformance Test tool—brings a new dimension to SONiC and SAI integration. By addressing the complexities of multi-vendor interoperability and automating comprehensive API testing, this solution empowers the SONiC community to maintain higher standards of reliability, efficiency, and vendor neutrality.
