High-Throughput I/O Engine

• Old title: Linux Modern IO Multiplexing
• Old project: Asynchronous Web Server with io_uring

This is a fun learning project where I implemented an echo server using 3 different technologies (poll(7), epoll(7), and io_uring(7)) to see how far we have got in Linux when it comes to multiplexing IO.

Besides implementing these, I have also run benchmarks to measure the performance of each multiplexing frameworks, and went down the rabbit hole to see where my expectations were not met.

The goal of this project was to learn the frameworks (even if it feels like with io_uring I just scratched the surface), set-up CI pipeline with memory checks using Valgrind, and run relevant benchmarks on my work.

Why "Modern" in the title? Because io_uring is still quite new in terms of the Linux ecosystem and not a lot of materials out there discuss on its usage.

Contents

This project is organized as:

• ./LICENSE: A license file generated by GitHub to allow anybody to run wild if they find what I've done useful.
• ./plan.txt: A place where I note my ideas, things to do, and a general plan with future work.
• ./README.md: This file.
• ./.vscode/: My configuration folder for VS Code, I want to keep it even if not standard practice to include it in a repo.
• ./rust_echo_bench/: A git submodule with a client program written in Rust to run benchmarks on my servers, more details and it's origin are in it's repo TODO: add link.
• ./code/: Is where I hold all the code written by me.
• ./code/c/: Is where I hold my C implementations in case I get around to write a server in Rust or other languages.
• ./code/c/client/: A simple client program written in C that connects to the server on the machine's loopback interface. I used it for testing, also I was originally planning to also use it for benchmarking send and wait type of clients, and send do not wait ones. Now, this is future work. More details in the client's README.md.
• ./code/c/common/: Some headers and structures where or could be useful to both the server and the client code, so this folder is where they reside.
• ./code/c/common/Makefile.defines: A file imported by the Makefile of the servers and the client with common definitions.
• ./code/c/server/: Here reside the multiple server implementations using different Linux multiplexing technologies, together with some common code that was useful to share between the servers. More details on the servers and a nice discussion on io_uring in each of the server's README.md.

How to run

The servers written in C are built using make files. For example the epoll server can be build like: make -C ./code/c/server/epoll, or from the server's folder with make. The resulting executable is run without arguments in the server's folder: ./epoll_server.

The C client is compiled and run in the same manner, with the Makefile in the client's source code folder.

The configuration parameters of the servers and client are passed as defines at compilation and can be changed in Makefile.defines.

Memory safety

Because a good chunk of the code is written in C, memory and resource management was a must. I have checked all my work with Valgrind's memcheck tool and written automated tests in the Makefile of each server. To run these for example for io_uring, one would just need to run:

• make -C ./code/c/server/io_uring test_memory_no_clients: To test just initializing the server for a few seconds and then closing it gracefully.
• make -C ./code/c/server/io_uring test_memory_with_clients: To test with a continuously connected client and one that connects for just one message exchange.

These tests finish automatically after a few seconds, performing a graceful teardown of the server's resources, closing all file descriptors and deallocating all the dynamic memory.

Benchmarking

Methodology

I test the performance of my echo servers by connecting multiple clients to them and making measurements on the client side. The client benchmarker is written in Rust and based on the work of Harald's rust_echo_bench.

The bench code spawns a thread for each client and sends a message of a fixed size to the echo server. Then it waits for the reply before sending a new message. My modification is that I ensured all threads start measuring just after all of them connected and that they ensure the messages do not get segmented.

Message segmentation was another concern of mine. I do not want to send and receive a message in 2 system calls, at that point I would no longer measure the framework's multiplexing performance but how fast my machine can move data between kernel and user space. The server and the clients exit with failure if they detect a message is segmented.

Initial benchmarking idea:

I wanted to measure on the server side the latency to service the requests I gave up on this because in the io_uring code I can not measure the time from right before I do a receive operation, the framework notifies me that something was read. So I shifted to measuring servicing time on the client's side.

In the source code of the servers metric collection portions can be noticed. I experimented with macros to automatically remove these parts at compile time but it looks unesthetic. I will probably remove the macros at some point in the future and just not start the metric collection. The server also had a warm-up start period when metrics were not collected. Note to self: do not over rely on macros, they make the code look ugly and the flow ambiguous.

What

I have measured on the client side the number of requests serviced by my echo serves. I measure this in requests / second and throughput / second.

My benchmarking configurations are:

• number of clients: 1, 10, 100, 1000, 5000, 10000
• message size (bytes): 128, 512, 2048, 4096
• multiplexing framework: poll, epoll (edge-triggered), io_uring (simple), io_uring with SQPOLL

How

The tests were run on my machine, the clients connecting for 60 seconds to the server on the loopback interface. My machine's specs are:

• kernel: Linux machine_name 6.8.0-117-generic #117-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
• CPU: AMD Ryzen 7 6800H - 8 cores - 16 threads
• RAM: 2 x 8GB Unbuffered DDR5 - 4800 MT/s

I set my CPU's power management software to performance and set the minimum CPU frequency to the maximum frequency.

Using taskset I bound the server process to thread 1 on core 0 of my CPU. Thread 0 of core 1 is where I bound the io_uring SQ_POLL kernel thread. I wanted to make sure the process and the kernel thread do not jump around and end up on different cores. I also set clients to run on the next 12 threads to not jump around and preempt my server process. The last 2 threads are a hint to the scheduler to run there what else is needed by my system.

Data

For now only the metrics in tables, I will plot them later.

• MSG_SIZE = 128 bytes | requests / sec

framework\client no	1	10	100	1000	5000	10000
poll	55084	136805	136424	122223	1066401	105368
epoll	54490	136021	134480	119111	102494	102192
io_uring simple	53606	140722	140396	124368	65650	88411
io_uring SQPOLL	58597	113094	110955	99494	59215	76857

• MSG_SIZE = 512 bytes | requests / sec

framework\client no	1	10	100	1000	5000	10000
poll	53580	134353	134089	115984	105009	102523
epoll	53723	134712	133182	113777	100376	100711
io_uring simple	53001	140244	139672	120149	64592	87916
io_uring SQPOLL	57983	112714	109766	99188	53323	70137

Results

I can already discuss about the results, even if I benchmarked only partially the servers yet.

When message size is small, the measurement showcases more the overhead of system calls and frameworks, the cost of moving the actual data is minimal. When message size increases, we start showing the bottlenecks of moving the data between user and kernel space.

It does not look sunshine and rainbows for io_uring. In simple mode, for 100 clients and message size of 128 bytes, it beats poll and epoll just by 2.86% and 3.45% respectively. When we get into 1000s of clients, poll and epoll have an advantage over io_uring, poll showing the best performance.

My initial theory is that I wait only for one completion event before processing the completion queue and I should batch more of them. But I disregarded this theory because in SQPOLL mode the performance is even worse, and I even do hot looping on the completion queue. Or maybe the workload of an echo server is not a good benchmark for io_uring.

Another strange behavior is that the number of serviced requests / second by the io_uring servers go down for 5000 clients and then increases again for 10k clients. More digging is needed.

Acknowledgements

I must state that I realize I am running my benchmarks on a single machine running multiple clients. As much as I tried to tell the scheduler to isolate things, there are still many processes in the background that can affect measurements.

I also must acknowledge that all the servers are single threaded, except for io_uring in SQPOLL mode which also uses a kernel thread. This could be considered cheating.

Future Work

I have a big list of things yet to do and improve, but I will take a small break from this project soon.

• investigate the strange behavior of io_uring low performance and rollercoaster performance
• finish the benchmark
• plot the measurements
• write per server READMEs
• email io_uring maintainer about the man page mistakes and typos
• add a CI pipeline
• use io_uring zero copy
• use io_uring multi-shot accept
• rewrite this README
• server in Rust with io_uring
• server in Rust with tokyo

Thanks

I would like to thank everyone on Stack Overflow for the questions and answers posted along the years. The maintainers of man pages, and to the Lord of IO Uring for the amazing examples.