The SONiC Mentorship Program brings contributors and mentors together to work on practical technical challenges and strengthen the open networking ecosystem. In this spotlight, we speak with Shale Lucas, a student at The City College of New York, about his work on improving team daemon performance by implementing a proxy between the Linux kernel and userspace.
About the Mentee
My name is Shale and I attend The City College of New York. My current majors are Computer Science and Pure Math. I was inspired to apply to the Linux Foundation as a whole because I wanted to get experience with open source development.
I saw the SONiC organization as an opportunity to learn more about programs involving computer networking and low level systems. Being able to learn more about Linux by working on a part of a NOS (Network Operating System) let me gain experience working on daemons and my own proxy server.
A simple diagram of the original setup with teamd processes and the kernel.
Q: What project did you work on, and why is it important to SONiC?
Team daemons are background programs that monitor and control network “teams” or “bonds,” which are groups of network connections combined to work as one for better speed or reliability. These daemons, often called teamd, use information from the system to keep each network team running smoothly and to react to changing conditions, like switching to a backup link/device if one fails.
When a system has many (~240) of these network teams (also called Link Aggregation Groups, or LAGs), problems can happen if the team daemons can’t keep up with the system’s updates. The Linux kernel sends out netlink messages (which report on the status of network devices) very quickly, faster than the daemons can read them. When the daemons fall behind, they might miss changes in the network, leading to trouble like ports/links/devices rapidly going up and down, also known as port flapping. The focus of the project aimed to reduce system resource usage by implementing a proxy between the kernel and userspace.
Q: What were your main technical contributions?
My mentor helped me improve the way messages move from the Linux kernel to userspace by guiding me in creating a proxy server. Instead of having each network team daemon manage its own netlink socket to receive network status updates directly from the kernel, I changed the setup so each daemon used a Unix Domain Socket instead. Now, there is just one netlink socket handled by the proxy server, which reads all the messages from the kernel and then sends the relevant ones to each team daemon through their Unix Domain Sockets.
To test this new setup, you wrote bash scripts that would repeatedly flap (bring up and down) network ports, causing the kernel to generate lots of netlink messages. For the proxy server, I used the C programming language, containerlab for network simulation, standard Linux utilities, and bash for scripting and automation.
A simple diagram of the proxy, kernel and teamd processes.
Q: What challenges did you face, and what did you learn?
The biggest challenge I faced was “getting up to speed”. Learning about a lot of the commonly used vocabulary in this environment was the first step. Once that was over, I was able to start writing simple programs to log diagnostics about network devices. Throughout the course of the program I learned more about containerization technologies like containerlab and docker. Additionally, I learned about lib-nl (lib-netlink) and its suite of libraries. I was able to improve my bash scripting skills, C skills and my use of epoll.
Q: What impact has this mentorship had, and what are your next steps?
My work serves as a starting point for other daemons to also improve their performance and reduce their resource utilization. I want to continue to contribute to open source tools in the SONiC environment and ultimately keep working on the proxy server in the future.
Q: Is there anyone you’d like to acknowledge?
Special thanks to Shenglin Zhu, Evan Harrison, the SONiC organization and The Linux Foundation for this opportunity.
Get Involved
Interested in contributing to SONiC? Join the community and get involved through wiki, mailing lists, and working groups.
