Linux Kernel Runtime Guard
 src/modules/exploit_detection/p_exploit_detection.c from LKRG 0.9.0. | |
| Original author(s) | Openwall |
|---|---|
| Developer(s) | Adam Zabrocki (Principal developer) |
| Repository | github.com/openwall/lkrg |
| Written in | C |
| Type | Kernel security |
| License | GNU GPL v2 |
| Website | www.openwall.com/lkrg/ |
Linux Kernel Runtime Guard (LKRG) is a out-of-tree security module for the Linux kernel developed by Openwall. It does run-time integrity checks in order to stop known, and unknown, security vulnerabilities in the Linux kernel. It can log detected intrusion attempts or stop them by causing a kernel panic - resulting in a frozen machine or a reboot depending on how the kernel is configured.
The developers note that LKRG has a a performance penalty of around 2%-3% in the release notes. We found it to be slightly below 2%.
Target Audience[edit]
Who is the Kernel Runtime Guard suitable for?
"Shared/cloud hosting providers and their users. Security enthusiasts. Security-hardened distros."
Do you recommend it for casual Linux desktop users?
"Generally, no. There isn't much local security on typical Linux desktop systems/distros anyway, in the way those systems are configured and used, irrespective of any locally exploitable Linux kernel vulnerabilities. Thus, adding LKRG would result in inconsistent security. Specifically, on a typical single-user Linux desktop system there's that one user account that uses "sudo" to access root, so it is effectively root-equivalent (only different from root in terms of safety from mistakes, not really in terms of security). LKRG does provide some relevant security hardening to such systems anyway - it might detect a web browser vulnerability exploit trying to escape the web browser's sandbox - so it could help e.g. a vulnerable system withstand web browser attacks in the Pwn2Own contest. However, that's not how most end-user systems are actually compromised (when they are).
A security enthusiast can very well reasonably use LKRG on their desktop system, though, primarily for consistency with their other systems, and for advance testing of kernel+LKRG combinations prior to deployment on the servers the person manages."
Do you recommend it for and all kinds Internet-facing servers?
"Not necessarily "all kinds", but generally yes. LKRG helps on systems where there are multiple users (or at least pseudo-users for multiple services), and especially where kernels might not be updated or rebooted into in a timely manner after new vulnerabilities are discovered."
So it's not just for firewall deployments?
"No, it is primarily for servers that provide much more than firewall functionality. It isn't nearly as useful on firewalls - not much need for local security when the device does just one thing anyway. However, with web-managed firewalls and such this gets nuanced - the web server is a separate service, hopefully running in its privilege-separated context from the actual firewall functionality (wishful thinking?) and LKRG can help harden that separation."
Security[edit]
We haven't tested any kernel exploits against a machine with and without LKRG. The authors have found several new kernel vulnerabilities using LKRG during development.
"During LKRG development and testing I've found 7 Linux kernel bugs, 4 of them have CVE numbers (however, 1 CVE number covers 2 bugs):
- CVE-2021-3411 - Linux kernel: broken KRETPROBES and OPTIMIZER
- CVE-2020-27825 - Linux kernel: Use-After-Free in the ftrace ring buffer resizing logic due to a race condition
- CVE-2020-25220 - Linux kernel Use-After-Free in backported patch for CVE-2020-14356 (affected kernels: 4.9.x before 4.9.233, 4.14.x before 4.14.194, and 4.19.x before 4.19.140)
- CVE-2020-14356 - Linux kernel Use-After-Free in cgroup BPF component (affected kernels: since 4.5+ up to 5.7.10)"
It does appear that the authors of LKRG know what they are doing. You will have to judge for yourself.
LKRG can provide four different levels of security validation and for levels of security enforcement.
You can see all the configuration options it has available, and there are many, by running:
sudo sysctl -a | grep lkrg
You can permanently configure your preferred LKRG settings by editing /etc/sysctl.d/lkrg.conf (LKRG 0.9.1+ creates a example configuration file there for you).
The lkrg.profile_validate option decides "a pre-defined profile controlling whether, when, and to what extent LKRG validates system integrity and detects attacks.".
There are four predefined security validation levels (lkrg.profile_validate=0 disables security validations):
| lkrg.profile_validate=1 | Light | |
| lkrg.profile_validate=2 | Balanced | |
| lkrg.profile_validate=3 | Heavy (default) | Incompatible with VirtualBox |
|---|---|---|
| lkrg.profile_validate=4 | Paranoid | "Has unreasonably high performance overhead and poor scalability while not necessarily providing a practically relevant improvement in attack detection" |
It is possible to change the underlying settings each profile activates, see configuration below for a list of keys. The lkrg.profile_validate= will read as 9, meaning custom, if one of the settings affecting a profile has been changed.
There are also four predefined security enforcement levels:
| lkrg.profile_enforce=0 | Can be used for safe testing of LKRG, where any detected violations and attacks are logged but no enforcement is performed. It can also be useful where LKRG is meant to act as a sensor within a larger security monitoring and response setup (e.g., network-wide). |
| lkrg.profile_enforce=1 | Performs selective enforcement - log only for kernel integrity violations, varying effective actions ranging from killing a task to triggering a kernel panic for other types of violations and attacks. This mode is extremely unlikely to panic the kernel on a false positive. |
| lkrg.profile_enforce=2 | Performs strict enforcement - varying effective actions for all types of violations and attacks, including triggering a kernel panic for kernel integrity violations.
This is the default setting if you do not change it. |
|---|---|
| lkrg.profile_enforce=3 | Performs the most paranoid enforcement - kernel panic for all types of violations and intrusions. |
The default lkrg.profile_enforce=2 enforcement setting makes LKRG kill tasks in most cases, but not all. If a kernel vulnerability is used to change the EUID of a task from 1000 to 0 then LKRG will kill that tasks, log the event and leave the system running as if nothing special happened. It will respond with a kernel panic if a kernel vulnerability is used to change the SELinux state or other more serious events happen.
You will very likely want to configure the Linux kernel to reboot if a kernel panic is encountered with either kernel.panic=seconds (kernel.panic=10 to reboot after 10 seconds) or panic= in a file in /etc/sysctl.d/ if you use one of the LKRG profiles that can cause a kernel panic when a security intrusion attempt is detected.
Some Small Caveats[edit]
LKRG can be by-passed.
"LKRG defeats many pre-existing exploits of Linux kernel vulnerabilities, and will likely defeat many future exploits (including of yet unknown vulnerabilities) that do not specifically attempt to bypass LKRG. While LKRG is bypassable by design, such bypasses tend to require more complicated and/or less reliable exploits."
An adversary who is aware that a machine is running LKRG would be able to by-pass it. This is by design. It it something to be aware of, but it is also mostly a non-issue. It is unlikely that some Internet worm trying to infect as many random computers as possible will try to target computers running LKRG or even be aware that LKRG exists.
LKRG will also not provide any protection from an attacker who has root access, such an attacker could simply unload the r_lkrg module. It is not meant to protect against a evil janitor who happens to notice that someone carelessly logged into a machine as root and went home without logging out.
Potential Usability Issues[edit]
LKRG has some options that can be problematic. The LKRG setting lkrg.block_modules makes LKRG block further loading of Linux kernel modules. It is, by default, set to 0 (disabled).
Enabling lkrg.block_modules may or may not be great if you are using LKRG on some kind of simple firewall machine. It is not all that great if you are using a desktop computer and you connect external HDDs or a gamepads or other devices that triggers on-demand loading of kernel modules regularly. Your devices will stop working.
Additional security is nice, but it can also prevent you from doing things you normally do on your computer. You should consider the implications before you enable LKRG and the various security options it offers.
The default enforcement level, which makes LKRG sometimes cause a kernel panic if a serious intrusion is detected, can also a big usability issue. The alternative is to just log serious intrusion attempts. That's less secure if there is a successful intrusion, but it may be preferable when you initially install and test LKRG. You can configure the Linux kernel to reboot automatically upon kernel panics and that is something you should do if you enable a security level were LKRG can cause kernel panics.
You may want to use lkrg.profile_enforce=0 or perhaps lkrg.profile_enforce=1 for a few days if you deploy LKRG for the first time and check the logs to see if there are false positives, and if so how many, before you enable any stricter enforcement.
Performance Penalty[edit]
The LKRG module has a performance penalty, the security it provides isn't free.
Kernel compilation took 27 seconds longer with the LKRG module loaded on our Intel i7-5500U test machine and 1 second longer on our Ryzen 2600 test machine. That works out to a 1.74% performance penalty on the Intel machine.
Verdict And Conclusion[edit]
LKRG can provide additional security for servers, and it is a good tool for security researches. However, it is not for everyone, and the developers do not recommend it for casual GNU/Linux desktop users. It does harden security, which is good, and lkrg.profile_enforce=0 can safely be used without any risk of a sudden kernel panic.
We do have some concerns about the default settings that make it cause a kernel panic if anything it believes is a intrusion is detected. That can and probably will cause problems. It may be nice to know that a production server has avoided a potential intrusion when it suddenly disappears from the Internet, but it will likely also be sad to miss out on whatever revenue that box was producing while it is down. The same applies to routers, firewalls and everything else for that matter. Configuring the kernel to reboot upon a kernel panic is essential if you use one of the profiles that causes a kernel panic. That is specially true if you deploy it on a remote server.
The additional security LKRG provides at lkrg.profile_enforce=0 and perhaps lkrg.profile_enforce=1 may be a nice addition. We strongly recommend that you think twice about what you are doing before you enable the default lkrg.profile_enforce=2 setting or lkrg.profile_enforce=3 - those are the ones that make it kernel panic and freeze or reboot the machine at the first sign of trouble. That may not be what you want unless you are a security researcher or someone else who is absolutely sure that potentially random kernel panics is somehow a good thing.
Installation[edit]
Installing LKRG is pretty easy if you have the kernel header files for the kernel you are using installed. You will likely have to install a distribution-specific package with them if you did not compile your kernel yourself.
The following instructions will of course not work when a new release is made and you will have to adjust for a newer version number.
Start by acquiring the source code and the GnuPG signatures for it. Verify that you have the right tarball even if it is kind-of pointless since an attacker in control of www.openwall.com would be able to hand you a fake key, a fake signature and a fake tarball.
wget https://www.openwall.com/signatures/openwall-offline-signatures.asc
gpg --import openwall-offline-signatures.asc
wget https://www.openwall.com/lkrg/lkrg-0.9.1.tar.gz.sign
wget https://www.openwall.com/lkrg/lkrg-0.9.1.tar.gz
gpg --verify lkrg-0.9.1.tar.gz.sign lkrg-0.9.1.tar.gz
Next, extract the source code and compile it. It is strait forward if you know how to compile things.
tar xf lkrg-0.9.1.tar.gz
cd lkrg-0.9.1/
make -j$(nproc)
Run sudo make install to install it.
You can, alternatively, compile unreleased git code:
git clone https://github.com/openwall/lkrg.git
cd lkrg
make -j$(nproc)
sudo make install
LKRG will install a systemd service and give you instructions for how you can start (systemctl start lkrg.service) it and/or enable it at boot systemctl enable lkrg.service. You can alternatively start and enable it with systemctl enable --now lkrg.service.
The service file it installs is set to load or unload the module it with:
ExecStart=/sbin/modprobe -v p_lkrg
ExecStop=/sbin/modprobe -v -r p_lkrg
Make sure you disable the /etc/systemd/system/lkrg.service if you plan on loading it using a file in /etc/modprobe.d/ or other non-systemd means. Using systemd to enable it works fine, though, you do not need to make things more difficult if you don't want to.
You may want to set it up to be compiled with DKMS if you change your kernel version regularly and you like LKRG. It doesn't come with any DKMS configuration file, so you would have to make one yourself. You may want to test and ensure that LKRG does not cause unfortunate problems like a kernel panic before the machine is done with the bootup process before you do that.
Configuration[edit]
Run-time configuration is done using sysctl. You can run sysctl -a | grep lkrg to see all the available keys once the module is loaded.
These keys can be written to a file in /etc/sysctl.d to make changes permanent.
[root@keumjo lkrg]# sysctl -a | grep lkrg| sort lkrg.block_modules = 0 lkrg.heartbeat = 0 lkrg.hide = 0 lkrg.interval = 15 lkrg.kint_enforce = 2 lkrg.kint_validate = 3 lkrg.log_level = 3 lkrg.msr_validate = 0 lkrg.pcfi_enforce = 1 lkrg.pcfi_validate = 2 lkrg.pint_enforce = 1 lkrg.pint_validate = 1 lkrg.profile_enforce = 2 lkrg.profile_validate = 3 lkrg.smap_enforce = 2 lkrg.smap_validate = 1 lkrg.smep_enforce = 2 lkrg.smep_validate = 1 lkrg.trigger = 0 lkrg.umh_enforce = 1 lkrg.umh_validate = 1
(these are all the keys available in 0.9.1 with their default values)
The README within the release tarball has a detailed description of each of this keys. You should probably read through that README file if you plan on using LKRG, there is a reason is it named README and that is specially true when it comes to LKRG.
Links[edit]
- The homepage is at https://www.openwall.com/lkrg/
- There is a mailing list. The mailing list archives can be read at https://www.openwall.com/lists/lkrg-users/
Enable comment auto-refresher