Final Step #7. Memory mapping For Sunday 15/05 23h59

Transcription

1 Final Step #7 Memory mapping For Sunday 15/05 23h59

2 Remove the packet content print in the rx_handler rx_handler shall not print the first X bytes of the packet anymore nor any per-packet message This is important to do as it affects a lot the performances and we ll work on performances in this step

3 Go back to mode 0 on file close When a file descriptor is closed, go back to mode 0 for the attached device. This makes the handling of module close very easier : in fact there is nothing to do. If you set the.owner = THIS_MODULE for file_operations correctly, the module will not be removed until all file are closed (as when a file is opened, a try_get_module is automatically done on your module).

4 Mmap (~6 points) Implement the mmap facility of your fastnet file_operations When the user call mmap(), it creates a memory zone mapped between user space and kernel space. To know the size of the space, you will modify the struct fastnet_register like this : struct fastnet_register { char ifname[ifnamsiz]; int mode; size_t buffer_size; unsigned int buffer_nr; }; The total size is buffer_size*buffer_nr. The default size of a buffer is 2048 bytes, the default number of buffer is 512. If the user give values == 0, the default value is used. All values shall be power of two. Pay attention to use copy_to_user to update the user fastnet_register struct to inform him of the final values. If the space given to the user by the mmap facility is different than buffe_size*buffer_nr, return an error code of -EINVAL.

5 Mmap zone layout This is not a double entry table! This is a big fat space divided in buffers. Buffer buffer_size Buffer Buffer_nr * buffer_size Buffer Buffer

6 Buffer layout Each packet content will be copied inside the buffer. The first bytes of each buffer are reserved to put some metadata. struct fastnet_descriptor { uint16_t length; uint16_t data_off; } Where length is the size of the packet inside the buffer. The real content of the packet will start at (unsigned char*)buffer + data_off; bytes (not bits!). For example data_off should be at least 4, as fastnet_descriptor is 4 bytes the packet content cannot start before the end of this descriptor zone. buffer_size leng th data _off Packet content data_off Length

7 Buffer lifecycle A userspace application can know if the first packet is available by looping on the length field of the first buffer. When the length is not zero, it means the content is available. So you should first copy the packet content, then the data_offset then the length as it will indicate to the user space app that the buffer is ready. The user application can then move to the next buffer and wait for it to be!= 0. This method is called polling, this is a non-blocking method, meaning that the application will have to check by itself from time to time if the next length field changed. Usually if there is no packet, doing a usleep() is good practice to not eat 100% of the cpu looping on the length field. When the userspace application has finished using the packet, it will release the buffer by setting the length to 0. The kernel can then reuse buffers using the same technique. No need to loop, when you need a new buffer, check if the next one has a length equal to 0. Use the buffer zone as a ring (after the last buffer you go to 0). No need to prevent fragmentation (always go to the next one and check for length)

8 Specifications If the buffer is not big enough, the end of the packet data will be lost If there is no more buffer available inside the space, packets are dropped Copy the very start of the packet with the ethernet headers, so not skb- >data but skb_mac_header(skb) Read will still work, but it will use the underlying shared memory zone If the buffer space cannot be allocated, the ioctl will return -ENOMEM

9 Mmap in practice int fastnet_mmap(struct file * filp, struct vm_area_struct * vma) filp, you already know about it. The filp->private_data should allow you (maybe through the net_device->rx_handler_data or directly) to recover the space allocated during the ioctl. vma represent the mapped zone already created for you by the kernel, this is a part of a page table entry with empty mappings, that you must set correctly. Look around remap_pfn_range and vmalloc_user/remap_vmalloc_range to create the mapping

10 Buffer space accessors struct fastnet_space { //Size of one buffer (default 2048) size_t buffer_size; //Number of buffer (default 512) unsigned int buffer_nr; //Final size of the buffer size_t size; //Pointer to the buffer void* mem; //Index of the next buffer to use for putting data unsigned int head; //Index of the next packet to read when the user is using read instead of mmap unsigned int next_read; }; void* buf_at(struct fastnet_space* space, int i) { return (void*)(((unsigned char*)space->mem) + (space->buffer_size * i)); } int next_buf(struct fastnet_space* space, int i) { if (++i == space->buffer_nr) i = 0; return i; } int is_full(struct fastnet_space* space) { return buf_len(buf_at(space,space->head))!= 0; } #define buf_descriptor(buf) ((struct fastnet_descriptor*)buf) #define buf_len(buf) (buf_descriptor(buf)->length) #define buf_off(buf) (buf_descriptor(buf)->data_off) #define buf_data(buf) (((unsigned char*)buf) + buf_descriptor(buf)->data_off)

11 Mode 3 (~3 points) + 2 bonus You will provide a generic way that any driver could implement to achieve full zero-copy. This will be the mode 3. If the driver sees that the device is in fastnet mode 3, it will use directly buffers from the memory zone to receive packets and therefore you won t even need your rx_handler as the packet will directly be received in the memory zone, and won t be processed by the kernel stack, even not calling the napi_* or gro_* and *receive_skb_core* functions. You will implement the mode 3 support for the e1000 driver. Another driver developer should be able to make its driver compatible staying totally generic (using net_device_ops functions, setting some flags, ). There should probably be a fastnet_get_buffer function to allow the driver to get the next free buffer, etc... No packet copy can happen, as you will give your buffer+data_off address to the NIC rings directly. Of course you ll have to copy the length in the buffer yourself after receiving the packet

12 Mode 3 specs If a particular device does not support fastnet mode 3 but the user tries to set it, you can go back in mode 2 and display a message in the log, and put back 2 in fastnet_register->mode You can let the register_rx_handler even in mode 3 even if it s not used

13 Profiling (~3 points) You will profile the speed to show the advantages of each methods to receive packets in userspace : A libpcap-based application A fastnet application using read, on mode 1,2,3 A fastnet application using mmap, on mode 1,2,3 Make a conclusion about the performance gain/loss, and the state of the Linux network stack for receiving raw packets, your thoughts about it knowing that in the real world, it is what is used (generally, libpcap applications). Ensure that your test VM use only 1 core Use packeth to generate 64 byte UDP packets and overwhelm your VM as much as you can, you may use multiple ethernet device and multiple core on your host until the guest is using 100% of his CPU to throw away packets. Use the linux perf tool to profile your host, and explain in your report with nice figures/tables where the CPU time is lost.

14 Compile perf cd linux /perf/tools sudo apt-get install libelf-dev binutils-dev python libgtk2.0-dev buildessential flex bison liblzma-dev libnuma-dev python-dev libperl-dev libslang2-dev libaudit-dev make sudo./perf top -a -g -k ~/OS/linux /vmlinux This will allow you to see live performances, but as your CPU is taken at 100% by interrupt handlers, perf will become very slow. Use perf record then perf report/annotate to take your time and analyse what happened after ~30 seconds of capture.

15 Report + Clean code + Re-evaluation of last steps after correction (~8 points) Follow the same rules than for previous steps (comments, patch, report, submission, obligations, ) Explain what you did, where, how, why, how you finded it. The problems you had How you solved them I will also look back at all the rest of the code as this is the final step. It should be easy to grab those points!