You might have seen this before. Periodic drops with massive ICMP echo requests (ping) against an IP of a switch or router (doesn’t matter) and someone complaining about packet loss while ‘testing’ the network.
It’s not broken and here is why.
Under the Hood
Let’s take a look at a pretty standard four port data center switch. We used some magic to be able to watch the incoming and outgoing packets, as well as different colors for user traffic (blue) and important control packets like ICMP, ARP, STP BPDU, OSPF HELLO … (red, orange, yellow).
Modern network devices have dedicated workforce to forward data, we call these guys ASIC. ASICs are equiped with special phone book like tables and can look up where a packet or frame should go in ludicrous speed. That way, the time in and out of the switch (port-to-port latency) is mostly in the range of only a few µs, yes microseconds – or even below.
BTW: If you’re chaising the root cause for application latency of multiple seconds, it’s most likely not (only) your data center network!
ASICs are good for lightning fast forwarding, they are interconnected and form the so called Data Plane of the switch. Almost none of the blue packets ever leave the date plane (unless through the front interfaces), since the ASICs know exactly how to handle them.
All other packets, however, are internally redirected to the Control Plane, the brain of the switch. It takes care of all the essential networking protocols, and the few types of packets the ASICs don’t know in their tables. For instance, the routing neighborships and updates are maintained by the control plane. And when a connected host ARPs for the MAC address of his Default Gateway hosted by this switch, the control plane responds. With ICMP/Ping it’s just the same.
CoPP – Control Plane Policing
The Control Plane is basically an x86/ARM based mini computer inside the switch with limited resources, but most important tasks, including the programing of the ASIC tables. Now, some protocols punted to the control plane are, however, more important than others to keep the whole network in a healthy state. Spanning Tree and routing protocol packets are essential to be processed if we don’t want to risk a broadcast storm or traffic black hole. To ensure that these important packets are definitely processed by the control plane, a special set of rules is stored within the ASICs – the Control Plane Policy.
Control Plane Policing (CoPP) protects the control plane and separates it from the data plane, which ensures network stability, reachability, and packet delivery.Cisco Nexus Configuration Guide
A quick lookup on a recent Nexus 9000 switch shows a limit of 360kbps for ICMP traffic with the default CoPP policy ‘strict’ active:
N9K-1# show policy-map interface control-plane class-map copp-system-p-class-monitoring (match-any) match access-group name copp-system-p-acl-icmp set cos 1 police cir 360 kbps , bc 128000 bytes module 1 : transmitted 13818516 bytes; dropped 84348 bytes;
If we do the math, it’s clear that less than five ‘jumbo’ pings fit into one second, and about 600 normal 64 bytes long echo requests. That limit can be definitely reached in busy networks. So triggered by this policy every now and then an ASIC worker has to leave his comfy chair to drop a packet into the bin. And that’s when you see a request time out with ICMP.
To summarize: Pings to a network device by no means verify the forwarding behavior, it’s just a basic reachability test and response times are dependent on the actual CPU load of the control plane. In extreme cases you could get very bad or no response, but the normal traffic flows like a charm through the ASICs / data plane.
That’s why good application performance monitoring tools exist, that take this unpredictable control plane behaviour out of the equation and measure the real path between app instances and end-to-end instead.
A recent tweet about this topic got so much attention, that I decided to write these lines and use an easy-to-understand model, to have a reference for future use. Of course, this blog post was only made possible by my sons’ generous donation of precious modelling equipment and time.