How did this packet get here?

Jeronimo Bezerra presented “AmLight-INT: In-band Network Telemetry @ AmLight” at the Internet2’s annual Technology Exchange conference which took place in New Orleans, LA on December 9-12, 2019. the event is  a premier technical event in the global R&E community, convening technology visionaries (technologists, architects, scientists, researchers, engineers, operators, and students) from around the world, for 3.5 days of presentations and discussions focusing on the next generation of federated trust and identity, information security, and advanced network design.

Session Abstract: AmLight-INT: In-band Network Telemetry

At AmLight, since 2015, we have been pursuing, developing, and evaluating solutions to mitigate network issues affecting the SDN network operation. Many of these solutions were presented in previous Internet2 Technology Exchange and Global Summit conferences, such as the Testbed Sanitizer (2015), AmLight Sniffer (2015), SDNTrace (2016), Inter-Domain SDNTrace (2017), and the AmLight SDN Looking Glass (2018). However, these solutions were created with one significant restriction: the data plane could not be changed due to hardware restrictions. The fact that the data plane was untouchable limited our efforts and efficiency.

In 2018, a new project was created to take the troubleshooting to the next level: the AmLight In-Band Network Telemetry project, or AmLight-INT. The main goal of the AmLight-INT is to evaluate how to leverage new programmable data planes to increase our capabilities to mitigate the most diverse range of connectivity and performance issues. AmLight-INT is based on the capabilities provided by the new Barefoot Networks’s Tofino chip: a fully programmable chip capable of forwarding traffic at 3.2Tbps. Using Tofino and the P4 language, the next steps would be to program the AmLight data plane to send telemetry data from each network device using a selective monitoring approach to In-band Network Telemetry (INT) collectors developed by FIU to receive, store, and process that telemetry data in real time. The INT collectors are components of the AmLight SDN framework. The reports provided by the INT collectors will guide the SDN path finders to make forwarding decisions.

The primary goal for AmLight-INT is to provide telemetry data to Collectors, in real time, so they can answer questions, such as the following:

1. How did this packet get here? The sequence of network devices a packet visited along its path.
2. Why is this packet here? The set of rules a packet matched upon at every AmLight-SDN switch along the way.
3. How long was this packet delayed? The time a packet spent buffered in every AmLight-SDN switch, to the nanosecond scale, all the way from South America to the U.S. As secondary goals, we want to address these challenges.
4. How do we report the network status back to applications and users?
5. How many applications/flows can we monitor in parallel? AmLight network engineers understand that, even though Barefoot Tofino chips can handle 3.2Tbps, network servers and collectors cannot.
6. Is it possible to dynamically enable monitoring of specific flows? With scalability in mind, AmLight SDN software developers aim to extend AmLight SDN controllers’ capabilities to trigger data plane monitoring of specific flows on-demand.
7. What is the impact caused by INT in a complex network such as AmLight? Introducing metadata on each packet crossing a switch has implications: packets get larger, and MTU might become an issue; larger packets introduce an additional delay for serialization, and larger packets use more buffers.

We understand that the future of SDN and performance monitoring and measurement are linked to the capabilities of expanding the data plane to support previously unimagined solutions and applications. Our goal with this presentation is to share with the TechEx audience our experience with programmable data planes and in-band network telemetry, lessons learned, achievements, and future steps for AmLight to use INT in production to support large scale projects, such as the Large Synoptic Survey Telescope (LSST) and Open Science Grid (OSG).

Co-Authors:
Name: Julio Ibarra
Affiliation: Florida International University
Email: julio@fiu.edu

Name: Heidi Morgan
Affiliation: University of Southern California
Email: drheidi@mac.com

Name: Arturo Quintana
Affiliation: Florida International University
Email: arturo.quintana@fiu.edu

For Jeronimo’s presentation please click here.