Linux Kernel Runtime Guard
src/modules/exploit_detection/p_exploit_detection.c from LKRG 0.9.0.
|Developer(s)||Adam Zabrocki (Principal developer)|
|License||GNU GPL v2|
Linux Kernel Runtime Guard (LKRG) is a out-of-tree security module for the Linux kernel developed by Openwall. It does run-time integrity checks in order to stop known, and unknown, security vulnerabilities in the Linux kernel. It can log detected intrusion attempts or stop them by causing a kernel panic - resulting in a frozen machine or a reboot depending on how the kernel is configured.
The developers note that LKRG has a a performance penalty of around 2%-3% in the release notes. We found it to be slightly below 2%.
Who is the Kernel Runtime Guard suitable for?
"Shared/cloud hosting providers and their users. Security enthusiasts. Security-hardened distros."
Do you recommend it for casual Linux desktop users?
"Generally, no. There isn't much local security on typical Linux desktop systems/distros anyway, in the way those systems are configured and used, irrespective of any locally exploitable Linux kernel vulnerabilities. Thus, adding LKRG would result in inconsistent security. Specifically, on a typical single-user Linux desktop system there's that one user account that uses "sudo" to access root, so it is effectively root-equivalent (only different from root in terms of safety from mistakes, not really in terms of security). LKRG does provide some relevant security hardening to such systems anyway - it might detect a web browser vulnerability exploit trying to escape the web browser's sandbox - so it could help e.g. a vulnerable system withstand web browser attacks in the Pwn2Own contest. However, that's not how most end-user systems are actually compromised (when they are).
A security enthusiast can very well reasonably use LKRG on their desktop system, though, primarily for consistency with their other systems, and for advance testing of kernel+LKRG combinations prior to deployment on the servers the person manages."
Do you recommend it for and all kinds Internet-facing servers?
"Not necessarily "all kinds", but generally yes. LKRG helps on systems where there are multiple users (or at least pseudo-users for multiple services), and especially where kernels might not be updated or rebooted into in a timely manner after new vulnerabilities are discovered."
So it's not just for firewall deployments?
"No, it is primarily for servers that provide much more than firewall functionality. It isn't nearly as useful on firewalls - not much need for local security when the device does just one thing anyway. However, with web-managed firewalls and such this gets nuanced - the web server is a separate service, hopefully running in its privilege-separated context from the actual firewall functionality (wishful thinking?) and LKRG can help harden that separation."
We haven't tested any kernel exploits against a machine with and without LKRG. The authors have found several new kernel vulnerabilities using LKRG during development.
"During LKRG development and testing I've found 7 Linux kernel bugs, 4 of them have CVE numbers (however, 1 CVE number covers 2 bugs):
- CVE-2021-3411 - Linux kernel: broken KRETPROBES and OPTIMIZER
- CVE-2020-27825 - Linux kernel: Use-After-Free in the ftrace ring buffer resizing logic due to a race condition
- CVE-2020-25220 - Linux kernel Use-After-Free in backported patch for CVE-2020-14356 (affected kernels: 4.9.x before 4.9.233, 4.14.x before 4.14.194, and 4.19.x before 4.19.140)
- CVE-2020-14356 - Linux kernel Use-After-Free in cgroup BPF component (affected kernels: since 4.5+ up to 5.7.10)"
It does appear that the authors of LKRG know what they are doing. You will have to judge for yourself.
LKRG can provide four different levels of security validation and for levels of security enforcement.
You can see all the configuration options it has available, and there are many, by running:
sudo sysctl -a | grep lkrg
You can permanently configure your preferred LKRG settings by editing
/etc/sysctl.d/lkrg.conf (LKRG 0.9.1+ creates a example configuration file there for you).
lkrg.profile_validate option decides "a pre-defined profile controlling whether, when, and to what extent LKRG validates system integrity and detects attacks.".
There are four predefined security validation levels (
lkrg.profile_validate=0 disables security validations):
|lkrg.profile_validate=3||Heavy (default)||Incompatible with VirtualBox|
|lkrg.profile_validate=4||Paranoid||"Has unreasonably high performance overhead and poor scalability while not necessarily providing a practically relevant improvement in attack detection"|
It is possible to change the underlying settings each profile activates, see configuration below for a list of keys. The
lkrg.profile_validate= will read as
custom, if one of the settings affecting a profile has been changed.
There are also four predefined security enforcement levels:
|lkrg.profile_enforce=0||Can be used for safe testing of LKRG, where any detected violations and attacks are logged but no enforcement is performed. It can also be useful where LKRG is meant to act as a sensor within a larger security monitoring and response setup (e.g., network-wide).|
|lkrg.profile_enforce=1||Performs selective enforcement - log only for kernel integrity violations, varying effective actions ranging from killing a task to triggering a kernel panic for other types of violations and attacks. This mode is extremely unlikely to panic the kernel on a false positive.|
|lkrg.profile_enforce=2||Performs strict enforcement - varying effective actions for all types of violations and attacks, including triggering a kernel panic for kernel integrity violations.
This is the default setting if you do not change it.
|lkrg.profile_enforce=3||Performs the most paranoid enforcement - kernel panic for all types of violations and intrusions.|
lkrg.profile_enforce=2 enforcement setting makes LKRG kill tasks in most cases, but not all. If a kernel vulnerability is used to change the EUID of a task from 1000 to 0 then LKRG will kill that tasks, log the event and leave the system running as if nothing special happened. It will respond with a kernel panic if a kernel vulnerability is used to change the SELinux state or other more serious events happen.
You will very likely want to configure the Linux kernel to reboot if a kernel panic is encountered with either
kernel.panic=10 to reboot after 10 seconds) or
panic= in a file in
/etc/sysctl.d/ if you use one of the LKRG profiles that can cause a kernel panic when a security intrusion attempt is detected.
Some Small Caveats
LKRG can be by-passed.
"LKRG defeats many pre-existing exploits of Linux kernel vulnerabilities, and will likely defeat many future exploits (including of yet unknown vulnerabilities) that do not specifically attempt to bypass LKRG. While LKRG is bypassable by design, such bypasses tend to require more complicated and/or less reliable exploits."
An adversary who is aware that a machine is running LKRG would be able to by-pass it. This is by design. It it something to be aware of, but it is also mostly a non-issue. It is unlikely that some Internet worm trying to infect as many random computers as possible will try to target computers running LKRG or even be aware that LKRG exists.
LKRG will also not provide any protection from an attacker who has
root access, such an attacker could simply unload the
r_lkrg module. It is not meant to protect against a evil janitor who happens to notice that someone carelessly logged into a machine as
root and went home without logging out.
Potential Usability Issues
LKRG has some options that can be problematic. The LKRG setting
lkrg.block_modules makes LKRG block further loading of Linux kernel modules. It is, by default, set to
lkrg.block_modules may or may not be great if you are using LKRG on some kind of simple firewall machine. It is not all that great if you are using a desktop computer and you connect external HDDs or a gamepads or other devices that triggers on-demand loading of kernel modules regularly. Your devices will stop working.
Additional security is nice, but it can also prevent you from doing things you normally do on your computer. You should consider the implications before you enable LKRG and the various security options it offers.
The default enforcement level, which makes LKRG sometimes cause a kernel panic if a serious intrusion is detected, can also a big usability issue. The alternative is to just log serious intrusion attempts. That's less secure if there is a successful intrusion, but it may be preferable when you initially install and test LKRG. You can configure the Linux kernel to reboot automatically upon kernel panics and that is something you should do if you enable a security level were LKRG can cause kernel panics.
You may want to use
lkrg.profile_enforce=0 or perhaps
lkrg.profile_enforce=1 for a few days if you deploy LKRG for the first time and check the logs to see if there are false positives, and if so how many, before you enable any stricter enforcement.
The LKRG module has a performance penalty, the security it provides isn't free.
Kernel compilation took 27 seconds longer with the LKRG module loaded on our Intel i7-5500U test machine and 1 second longer on our Ryzen 2600 test machine. That works out to a 1.74% performance penalty on the Intel machine.
Verdict And Conclusion
LKRG can provide additional security for servers, and it is a good tool for security researches. However, it is not for everyone, and the developers do not recommend it for casual GNU/Linux desktop users. It does harden security, which is good, and
lkrg.profile_enforce=0 can safely be used without any risk of a sudden kernel panic.
We do have some concerns about the default settings that make it cause a kernel panic if anything it believes is a intrusion is detected. That can and probably will cause problems. It may be nice to know that a production server has avoided a potential intrusion when it suddenly disappears from the Internet, but it will likely also be sad to miss out on whatever revenue that box was producing while it is down. The same applies to routers, firewalls and everything else for that matter. Configuring the kernel to reboot upon a kernel panic is essential if you use one of the profiles that causes a kernel panic. That is specially true if you deploy it on a remote server.
The additional security LKRG provides at
lkrg.profile_enforce=0 and perhaps
lkrg.profile_enforce=1 may be a nice addition. We strongly recommend that you think twice about what you are doing before you enable the default
lkrg.profile_enforce=2 setting or
lkrg.profile_enforce=3 - those are the ones that make it kernel panic and freeze or reboot the machine at the first sign of trouble. That may not be what you want unless you are a security researcher or someone else who is absolutely sure that potentially random kernel panics is somehow a good thing.
Installing LKRG is pretty easy if you have the kernel header files for the kernel you are using installed. You will likely have to install a distribution-specific package with them if you did not compile your kernel yourself.
The following instructions will of course not work when a new release is made and you will have to adjust for a newer version number.
Start by acquiring the source code and the GnuPG signatures for it. Verify that you have the right tarball even if it is kind-of pointless since an attacker in control of
www.openwall.com would be able to hand you a fake key, a fake signature and a fake tarball.
wget https://www.openwall.com/signatures/openwall-offline-signatures.asc gpg --import openwall-offline-signatures.asc wget https://www.openwall.com/lkrg/lkrg-0.9.1.tar.gz.sign wget https://www.openwall.com/lkrg/lkrg-0.9.1.tar.gz gpg --verify lkrg-0.9.1.tar.gz.sign lkrg-0.9.1.tar.gz
Next, extract the source code and compile it. It is strait forward if you know how to compile things.
tar xf lkrg-0.9.1.tar.gz cd lkrg-0.9.1/ make -j$(nproc)
sudo make install to install it.
You can, alternatively, compile unreleased git code:
git clone https://github.com/openwall/lkrg.git cd lkrg make -j$(nproc) sudo make install
LKRG will install a systemd service and give you instructions for how you can start (
systemctl start lkrg.service) it and/or enable it at boot
systemctl enable lkrg.service. You can alternatively start and enable it with
systemctl enable --now lkrg.service.
The service file it installs is set to load or unload the module it with:
ExecStart=/sbin/modprobe -v p_lkrg ExecStop=/sbin/modprobe -v -r p_lkrg
Make sure you disable the
/etc/systemd/system/lkrg.service if you plan on loading it using a file in
/etc/modprobe.d/ or other non-systemd means. Using systemd to enable it works fine, though, you do not need to make things more difficult if you don't want to.
You may want to set it up to be compiled with DKMS if you change your kernel version regularly and you like LKRG. It doesn't come with any DKMS configuration file, so you would have to make one yourself. You may want to test and ensure that LKRG does not cause unfortunate problems like a kernel panic before the machine is done with the bootup process before you do that.
Run-time configuration is done using
sysctl. You can run
sysctl -a | grep lkrg to see all the available keys once the module is loaded.
These keys can be written to a file in
/etc/sysctl.d to make changes permanent.
(these are all the keys available in 0.9.1 with their default values)
README within the release tarball has a detailed description of each of this keys. You should probably read through that
README file if you plan on using LKRG, there is a reason is it named README and that is specially true when it comes to LKRG.