Welcome to the Linux Real Time Manual. The goal of this manual/wiki is to serve as platform for real-time experts to share their knowledge; and for system designers and developers to use as a guide to implement real-time applications using Linux.
As hardware has become ever more capable there has been a trend to implement real-time applications on Linux. Linux was designed from the beginning for server and desktop applications, not for real-time applications. This means that achieving real-time properties on Linux is not trivial. This document is a guide for anyone attempting to implement a real-time application using Linux.
A system with real-time constraints aims to perform work with a guarantee of the time when the work will be finished. This time is often called a deadline, and the system is designed with the purpose of not missing any, or as few as possible, deadlines.
A system where the consequences of missing deadlines are severe, for example with respect to danger for personnel or damage of equipment, is called a hard real-time system. A system where deadlines occasionally can be missed is called a soft real-time system.
The work done by a real-time system is often initiated by an external event, such as an interrupt. The nature of the work often requires participation of one or more concurrently executing tasks. Each task participates by doing processing, combined with interaction with other tasks. The interaction typically leads to one or more task switches. A deadline for the work is often associated with the completion of the work, i.e. when the last participating task has finished its processing.
There are several challenges when implementing real-time systems. One challenge is to obtain as much determinism as possible. In this way, it becomes easier to make predictions and calculations of the actual execution time that will be required. From these predictions, the risk that the system would miss deadlines can be evaluated.
When implementing a soft real-time system and using Linux as an operating system, it is important to try to characterize possible sources of indeterminism. This knowledge can then be used to configure, and perhaps also modify, Linux so that its real-time properties become more deterministic, and hence that the risk of missing deadlines is minimized, although not guaranteed.
The remainder of this section gives a selection of areas in Linux where sources of indeterminism are present. The purpose is to give a brief technical overview, to serve as general advice for projects aiming at using Linux in real-time systems.
Each section in this chapter objectively describes: The topic/mechanism; Default configuration and The real-time impact. It does not describe how to optimize for real-time behavior. That is done in the next chapter, Improving the Real-Time Properties.
Contents: 2.1 - Kernel Preemption Model 2.2 - Scheduling 2.2.1 - SCHED_FIFO and SCHED_RR 2.2.2 - SCHED_OTHER 2.2.3 - SCHED_BATCH 2.2.4 - SCHED_IDLE 2.3 - Real-Time Throttling 2.4 - The PREEMPT_RT Patch 2.5 - CPU Load Balancing 2.6 - CPU Isolation 2.7 - Dynamic Ticks (NO_HZ) 2.8 - Power Save 2.8.1 - Dynamic Freqency Scaling 2.8.2 - CPU Power States 2.8.3 - CPU Hotplug 2.9 - I/O Scheduling 2.10 - System calls 2.11 - Interrupts 2.12 - Soft IRQs and Tasklets 2.13 - Work Queues 2.14 - Threaded Interrupts 2.15 - Ticks 2.16 - Memory Overcommit 2.17 - RCU- Read, Copy and Update
Improving the real-time properties is very much about reducing latency. In a hard real-time system, it is all about worst case, while in a soft real-time system, it is about reducing the probability for high latency numbers. Often, the improvements come at a price, e.g. lower average throughput, increased average latency, reduced functionality, etc.
This chapter starts with a set of actions that are of low complexity, easy to apply and/or well established. For some systems these actions will be enough. If not, it is suggested to isolate the real-time critical application(s). The idea is to avoid sources of jitter on the real-time applications. Finally, a set of actions with larger impact are presented.
Contents: 3.1 - Evaluating Real-Time Properties 3.2 - First Attempt 3.2.1 - Configure the Proper Kernel Preemption Model 3.2.2 - Optimize Power Save 3.2.3 - Use Threaded Interrupts 3.2.4 - Optimize Real-Time Throttling 3.3 - Isolating the Application 3.3.1 - Isolating CPUs 220.127.116.11 - Setting up a Partition System 18.104.22.168 - Benchmarks for CPU isolation 22.214.171.124 - Further reading about CPU isolation 3.3.2 - Full Dynamic Ticks 3.3.3 - Optimizing a Partitioned System 3.3.4 - The CPU Partitioning Tool - partrt 3.4 - Further Actions if Needed 3.4.1 - User Space Runtime 3.4.2 - Use an RTOS
Optimizing the Linux system itself for real-time is only half the solution to give applications optimal real-time properties. The Linux applications themselves must be designed in a proper way to allow for real-time properties. This section will provide some hints how to do this.
The C function library, libc, is a part of the runtime for applications under Linux. It provides basic facilities like fopen, malloc, printf, exit, etc. The C library shall provide all functions that are specified by the ISO C standard. Usually, additional functions specific to POSIX are also supported. GNU libc is the most widely used libc, but there are alternative implementations like newlib and uClibc, all of them supporting the POSIX.1b (IEEE 1003.1b) real-time extensions.
libc provides a powerful toolbox with many useful and frequently used features, but from a real-time perspective, it must be used with some caution. The first problem to deal with is the level of real-time support in the libc code itself. The code is often considered to be proven in use and therefore used without deeper analysis. This is probably a valid assumption for a typical Linux system where average performance is more important than worst-case behavior, but for real-time systems it might be an unreasonable attitude. This should however not be a major issue since the source code is available for analysis. Another issue is the fact that the functions in libc may use system calls interacting with the kernel. Depending on the kernel preemption model, this may lead to execution of different non-preemptible sections. The kernel can also decide to execute other tasks like soft IRQs on its way back to user space.
Further application design challenges are: Memory handling (memory access latency, shared memory); Multi-threaded applications (task switch overhead, locks); I/O (blocking, driver-induced jitter).
The real-time properties of a system do not only depend on the operating system. The hardware is also important. This chapter contains a brief discussion on what to consider regarding real-time capable hardware.
Real-time capable hardware requires, at the minimum, that resources can be accessed within a deterministic time interval. If there is a risk for resource congestion, the hardware must implement deterministic multiple access algorithms which guarantee that there will be no starvation. Another typical hardware requirement in a real-time system is that CPUs have access to reliable high resolution timers.
Contents: 5.1 - CPU 5.2 - Shared Resources 5.2.1 - Shared Memory and Memory Congestion 5.2.2 - Hyper-Threading 5.2.3 - NUMA 5.2.4 - Decrease Impact of Shared Resource Congestion 5.3 - The System Management Mode (x86)
When someone states the goal to "optimize a specific Linux target for real-time" and provides a benchmark result, it is very important to be clear on what capabilities the measured system actually has. Benchmark results may be interesting to read, but they are only valid and relevant if they are somewhat comparable with each other and if the setup is relevant for real-world use cases.
This appendix states the goal to optimize for real-time, but it actually tries to reach as far as possible regarding both throughput performance and low worst-case latency response time since the use case we focus on is an embedded control system within the communications domain, which normally has both fairly high soft real-time requirements and performance requirements.
Contents: A.1 - In Reality, "Real-Time" is Not Only About Interrupt Latency A.2 - Performance and Latency Tuning Strategy A.3 - Configuring and Building the Enea Linux PowerPC p2041rdb Kernel(s) A.4 - Benchmark Description A.5 - The "Default" LLD Kernel Targeting Demo/Debug A.6 - The Performance-Optimized RELEASE Kernel A.7 - The "RT-Optimized" Mainstream Low-Latency Desktop Kernel A.8 - The "RT-Optimized" PREEMPT_RT Kernel A.9 - Summary and Conclusion
The combination of cyclictest and stress has been used for the benchmarking presented in this document.