Netmap presentation
Download as PPTX, PDF2 likes1,218 views
Netmap is a framework that allows applications to access network interface controller (NIC) hardware at near line-rate speeds. It reduces packet processing costs by eliminating system calls, memory copies between layers, and memory allocation/deallocation. Netmap provides applications with direct access to NIC buffers and rings via memory mapping, allowing packets to bypass the kernel for fast I/O. It also supports multi-queue NICs and provides rings for the host stack to still use the NIC while netmap applications have access.
![Netmap Data Structure
Safe slots: [curr … curr + available – 1]](https://image.slidesharecdn.com/netmappresentation-150224181937-conversion-gate01/85/Netmap-presentation-12-320.jpg)
![Lightening fast packet forwarding
...
src = &src_nifp->slot[i]; /* locate src and dst slots */
dst = &dst_nifp->slot[j];
/* swap the buffers */
tmp = dst->buf_index;
dst->buf_index = src->buf_index;
src->buf_index = tmp;
/* update length and flags */
dst->len = src->len;
/* tell kernel to update addresses in the NIC rings */
dst->flags = src->flags = BUF_CHANGED;
...
/ conditional part](https://image.slidesharecdn.com/netmappresentation-150224181937-conversion-gate01/85/Netmap-presentation-17-320.jpg)
Netmap presentation
- 1. FreeBSD Netmap Amir Razmjou [email protected] Intelligent Communication and Information Systems Laboratory Florida Tech
- 2. What’s netmap? • Allows line-rate speed on commodity hardware and NICs (14.88 Mpps on 10G 1.48 Mpps on 1G) • Simple API but still allows to utilize hardware specific features (multi-queue NICs). • No need to mess up with Kernel coding, Netmap applications can easily be ran in userland. • Full control over host stack.
- 4. Packet Journey in Kernel ether_input(struct ifnet *ifp, struct ether_header *eh, struct mbuf *m) { m->m_pkthdr.rcvif = ifp; eh = mtod(m, struct ether_header *); m->m_data += sizeof(struct ether_header); m->m_len -= sizeof(struct ether_header); m->m_pkthdr.len = m->m_len; ether_demux(ifp, eh, m); } void ether_demux(ifp, eh, m) ether_type = ntohs(eh>ether_type); switch (ether_type) { case ETHERTYPE_IP: . . . break; schednetisr(NETISR_IP); inq = &ipintrq; (void) IF_HANDOFF(inq, m, NULL);
- 5. Packet Journey in Kernel //ISR for IP while(1) { ip_input(m); s = splimp(); IF_DEQUEUE(&ipintrq, m); splx(s); if (m == 0) return; ip_input(m); } void ip_input(struct mbuf *m) { if (m->m_len < sizeof (struct ip) && (m = m_pullup(m, sizeof (struct ip))) == 0) { } ip = mtod(m, struct ip *); ipstat.ips_toosmall++; return;
- 6. Network I/O costs • per-byte cost: amount of traffic - copying, checksum computation, encryption • per-packet cost: comes from the manipulation of descriptors (allocation and destruction, metadata management) and the execution of system calls, interrupts, and device-driver functions… • The larger packets the less per-packet cost.
- 7. Sockets Bottlenecks • IPC Communication “System Calls”. • Parsing RAW Ethernet frames into different data structure at different layers, mbufs, skubuff. • Memory Copy (inter layers) • Memory Allocation/Deallocation
- 8. Design Decisions 30 years ago. • memory was a scarce resource; • links operated at low (by today’s standards) speeds; • parallel processing was an advanced research topic; and the ability to work at line rate in • all possible conditions was compromised by hardware limitations in the NIC
- 9. Packet Handling Costs UDP System Call mbuf Memory Copy Hardware TCP
- 10. What’s netmap • “Netmap is a novel framework that employs some known techniques to reduce packet- processing costs. Its key feature, apart from performance, is that it integrates smoothly with existing operating-system internals and applications. This makes it possible to achieve great speedups with just a limited amount of new code, while building a robust and maintainable system.”
- 11. Netmap goals • Reduce system calls • Remove allocation costs • Eliminate data copies. The inspiration for data structure design comes from the way that network interface stores buffers.
- 12. Netmap Data Structure Safe slots: [curr … curr + available – 1]
- 13. netmap API • Access – open(‘’/dev/netmap’’); – ioctl(fd, NIOCREG, arg) – mmap(…, fd, 0) maps buffers and rings. • Transmit – Fill to avail buffers, starting from slot cur. – IOCTL NIOCTXSYNC • Receive – IOCTL NIOCRXSYNC – Process up to avail slots from slot cur
- 14. Example
- 15. Multi Queue NIC • Multiple Descriptor Queues – queues can be associated with specific processor cores • Receive-Side Scaling (RSS) – To determine which receive queue to use for incoming packets, – specific flow for a given packet is determined by the calculation of a hash value derived from fields • Extended Message-Signaled Interrupts (MSI-X) – Multiple interrupt vectors
- 16. Host Stack Access • In netmap mode host stack is disconnected from NIC. • But the operating system still can use NIC. • There’s two additional rings for incoming outgoing traffic of host stack. • Packets destinated to the host stack are queued netmap and then and passed to the host stack and vice versa. • The approach provides an ideal solution for traffic filters, manipulators and etc.
- 17. Lightening fast packet forwarding ... src = &src_nifp->slot[i]; /* locate src and dst slots */ dst = &dst_nifp->slot[j]; /* swap the buffers */ tmp = dst->buf_index; dst->buf_index = src->buf_index; src->buf_index = tmp; /* update length and flags */ dst->len = src->len; /* tell kernel to update addresses in the NIC rings */ dst->flags = src->flags = BUF_CHANGED; ... / conditional part
- 18. Comparsion
Editor's Notes
- #2: Thank you for watching my videocast for FreeBSD netmap, for those of you who don’t know me my name is Amir Razmjou and I’m computer science graduate student at Florida Tech. Today I’m gonna talk about netmap, a new packet capturing framework. It’s going to brief an overview over to this technology. Hopefully we will going to have a seminar later and hands-on workshop and I hope that we will have more advanced videocasts on this topic in future as well. If you had any questions you can send me an email to the email address written here I’ll be happy to answer them.
- #3: These packet rates are can saturate maximum bandwidth that network interface can handle even if you have a 64 bytes long frames. For a long time, the bottleneck for packet processing on commodity hardware were network interfaces. In early 90s CPU memory and hardware were fast enough to cope with number of packets on that time. With advent of modern network interfaces 10Gs now we are game rules changed a lot. Memory is not a scarce resource anymore, network stacks were designed to serve different type application and were very generalized. Today we usually use network stacks for special purposes, content servers, data plane and etc.
- #4: In conventional operating systems and network interfaces. The final packet handed to userland application is the result of complicated interactions between network interface and operating system. In this picture hardware representing network interface shown here… is basically a ring data structure that holds physical address for frames written to machine main memory usually through direct memory access to shared memory between operating system and network interface. As we can see these buffers cannot directly be used by applications, actually network stack does do a long processing on each one of them to make them useable for userland applications. For example consider that network interface captures a packet on wire as soon as it captures the packets it directly writes it in machine physical memory and fires a hardware interrupt to notify operating system that new packet with given physical address on memory has been received, usually operating system defers this task to later time, in order to not disturb other tasks in but in general it makes at least one copy of buffer to fit it to data structure suitable for both userland application and operating system. These structures in freebsd are called mbufs and skbuffs in their linux counterparts. Although these structures are preallocated but they have a fix size for payloads so operating system usually has to use a chain of these structures to represent a single packet.
- #8: System calls happen when userland application requests a service from operating system. It’s costly because the processor has to do a context switch. It means that it should store current state of processor to some safe place and execute the service requested in operating system context
- #10: Why is mbuf handling so time consuming? In the uniprocessor systems on the scene 30 years ago, memory allocators and reference counting were relatively inexpensive, especially in the presence of fixed-size objects. These days things are different: in the presence of multiple cores, allocations contend for global locks, and reference counts operate on shared memory variables; both operations may easily result in uncached DRAM accesses, which take up to 50-100 nanoseconds.