Everything you wanted to know about SELinux but were afraid to run

Years ago, when it became apparent how important Linux would become, the National Security Agency (NSA) created a mandatory access control (MAC) architecture for Linux As in other realms, security is always important, but sometimes it’s especially important. From its creation and extending to today, Security-Enhanced Linux (SELinux) has blocked tampering threats and has prevented attempts to bypass application security.

I hasten to stress that you should not adopt SELinux just because the extra-robust security functionality exists. Yes, I know how much security matters to your business, but SELinux isn’t for everyone. Running SELinux takes extra time, requires far more hands-on involvement than other Linux distributions, and demands deep knowledge of Linux architecture.

In fact, if you set it up haphazardly, you are asking for trouble. Several SELinux professionals told me that the most common problems with SELinux occur when the operating system is deployed poorly. That’s because SELinux's fundamental security approach (restrict everything, unless explicitly permitted) is the exact opposite of Linux's (permit everything, unless explicitly forbidden).

In short: If you set up SELinux right, you avoid problems. If you don't, you encounter trouble.

How do you get it done right? I’m glad you asked.

What is SELinux? 

Technically speaking, SELinux is a set of Linux patches and user tools, which add MAC security to the operating system. They defend the OS so that a hacked or misbehaving application is locked down and unable to damage data or other applications. 

MAC is a fundamentally different approach to Linux security. The previous (and still more common) Unix/Linux security methodology is Discretionary Access Control (DAC). With traditional Linux DAC security, the root user is omnipotent, for better or worse. Each process runs under a user and group. For example, the Apache web server httpd process runs as the user apache under the group apache. Thus, the httpd process has access to all Apache files and directories. If it's cracked, the hacked httpd process can access, modify, and destroy all files that belong to Apache. 

Or, as Red Hat cloud strategy evangelist Thomas Cameron explained in a SELinux for Mere Mortals presentation, "You can chmod +rwx your home directory, and nothing will stop you. We give you the gun, and there's your foot."

With SELinux MAC, the sysadmin defines how processes interact with other server components. And here's the important part: All access is explicitly granted. A sysadmin determines whether a given process can access files, pipes, ports, and so on. There is no access by default, even if you're the root user. 

SELinux defines what can be done with each object according to a set policy. For instance, by default, the Apache user can only access the /var/www/html directory.

If the DAC and MAC conflict, the SELinux policy takes priority. So, for example, let’s say that – as root – you change the ownership of httpd to anyone by running the command chmod 777 httpd; the default SELinux policy still prevents Joe Random User from killing the web server. 

However, don't think that SELinux is all about securing individual programs. As Dan Walsh, a Red Hat consulting engineer (who is known as "Mr. SELinux") wrote, "SELinux does not worry so much about executing individual programs, although it can do this. SELinux is basically about defining the access of a process type."

The confusion due to MAC and DAC’s fundamentally different security approaches causes many sysadmins to disable SELinux whenever they encounter it. Indeed, two virtual private server companies, until recently, disabled SELinux on Fedora Server instances by default.

As with many other high-performance tools, from powerful race cars to rocket engines, you only benefit from SELinux with a steady hand, dedicated training, and a real need for its features. Use SELinux when security is Job One for your company and not for ordinary line-of-business needs. For example, you might not want to use it on internal development servers, but it's another story for a high-profile website.

Labels matter

The trick to using SELinux well is to understand its differences from the Linux you know and love.

One first step in running SELinux is to make sure you have all the pieces in place. Check that you installed the SELinux troubleshooting tools: setools, setroubleshoot, and setroubleshoot-server. For some reason, I can't fathom these programs aren't installed by default in some SELinux packages. 

One hugely important concept is labels – and a lot of trouble happens when you don’t grasp this SELinux concept.

 In SELinux, every process, file, directory, and system object has a label. Policy rules control access between labeled processes and labeled objects. These labels are stored as extended attributes on the file system. The kernel manages the labels for objects that aren't files, such as processes and ports.

Unfortunately, in some contexts, a label assigned to a process is also called a domain; you won’t be the first person to be confused. However, if you run into documentation discussing SELinux domains, remember it's just talking about labels. 

The correct label format is user:role:type:level. Get to know this syntax well, because as you'll see shortly, bad labeling is the source for many SELinux problems.

Make a policy of it

SELinux governs processes with policies. Usually, a policy is made up of a labeling file, a rule file, and an interface file. These are compiled to produce a policy file, which is then loaded into the Linux kernel. You can load or unload SELinux policies without rebooting.

A policy is comprised of explicit permission rules. For instance, a httpd policy may let a httpd process run with specified configuration, log, and content directories, using a specific startup script on specific ports, such as 80, 443, and 8080. 

Policies can be very fine-grained. You can set them to grant access based on users, files, directories, sockets, ports, and more.

Permissive mode is your friend 

One way to get into trouble – particularly with policies – is to implement them without testing them thoroughly beforehand. Otherwise, you find yourself “testing the locks” by locking yourself out of the house. 

So as you establish SELinux policies, you need to make sure SELinux won't lock up vital processes. One way to see what problems lie ahead is to run SELinux in permissive mode. 

In permissive mode, SELinux allows any and all processes to run.  If you make a mistake, instead of locking down the OS with a potentially insecure process, it presents you with an error message.

The other two SELinux modes are enforcing (a.k.a. restrictive) and disabled:

• In enforcing mode, applications stop running if they're misbehaving or their policy is set incorrectly.

• In disabled mode you have SELinux installed, but it won't do anything. This is often used when SELinux has stopped a program in its tracks, which you need to keep running.

During your own testing phase, it’s a good idea to work in permissive mode. You can switch between enforcing and permissive modes if you boot with a kernel that supports SELinux Development mode (CONFIG_SECURITY_SELINUX_DEVELOP=y). If this flag is not set (it is in the /etc/selinux/config file), SELinux starts in enforcing mode.

To see which mode you're in, run sestatus. To switch to/from enforcing to permissive, run the setenforce command. This command supports enforcing/1 or permissive/0 as arguments. So, either of these turn on SELinux in full defense mode:

# sestatus 1

# sestatus enforcing

Keep in mind that SELinux's MAC policies trumps Linux's default DAC policies. A common error is to assume a given process runs as it always has. However, with SELinux you must use a policy to explicitly give the process – and all its components – permission to run. 

It's always DNS… I mean, it’s always labeling

In networking, the truism is that when anything goes wrong is that it's always DNS. (Because it is.)

When you are troubleshooting SELinux, you should always look to mislabeling for the source of your woes. 

For example, if your Apache server can't find its website files and it's throwing fits, odds are it's because something hasn't been correctly labeled. Typically, this will cause Apache, or any other program, to either stop running or to be unable to access its data.  To fix these, use the SELinux Policy Management command, semanage.

When you know the label:

# semanage fcontext -a -t httpd_sys_content_t '/srv/myweb(/.*)?'

If you know the file with the equivalent labeling:

# semanage fcontext -a -e /srv/myweb /var/www

Or, in either case you can restore the context with 

# restorecon -vR /srv/myweb

Track down misbehaving programs

Let's say your Apache instance is simply not working. What should you do next? Well, first, let's see if it's a SELinux problem. 

If a problem goes away after you set a program to permissive mode, you know the root cause is with your SELinux settings or policies. To find out, use the semanage command. 

The next step is to allow the Apache web server to run in such a way that it generates SELinux error messages rather than just failing in some spectacular but confusing fashion. Use:

# semange permissive -a httpd_t

Simultaneously, your other processes run under SELinux's restricted mode protection. This can be very handy because the error messages are more likely to give you useful debugging information. 

Once you have that process running without any error messages, you can return it to enforcing mode with:

# semanage permissive -d httpd_t

To check on the current SELinux mode status of your processes run:

# semanage permissive -l

To view everything about how a particular program is running in use the command:

# ps auxZ | grep httpd

The Z flag gives you the SELinux context for a given file or process. In this example, this tells you that the httpd process is using the httpd_t type, the system_r role, and the system_u user. Usually, you only care about the type when troubleshooting. You can also use the Z option with other shell commands. 

In SELinux terms, processes and files are labeled with an SELinux context. A context contains information such as the SELinux user, Role-Based Access Control (RBAC), type, Type Enforcement (TE), and, optionally, its Multi-Level Security level. All these factors matter with SELinux access control decisions. 

Move files

Another command you should expect to use a lot with SELinux is restorecon. This resets the default context on files and directories, and in so doing, it follows the policy you set. For example, imagine that you move a file from your development directory to the httpd directory. When the file changes location, it doesn't move with the appropriate security context. That’s because the mv command retains the existing context.

To make the file ready for Apache to use under SELinux, you need to run a command such as:

# restorecon -vR /srv/www/foo.com/html/example_file.html /var/www/html/

That isn’t the only way to fix the problem. Another is to change the context command with the label

# chcon -t httpd_system_content_t /srv/www/foo.com/html/example_file.html

Or, change the context command with the reference label:

# chcon --reference /var/www/html/ /srv/www/foo.com/html/example_file.html

As usual, check the log files

If you suspect that SELinux is blocking a program from running, check the logs. When SELinux denies an action, an Access Vector Cache (AVC) message is logged to the /var/log/audit/audit.log and /var/log/messages files or the journald daemon logs it. Some of the most common searches to find AVC error messages are:

Or, to find SELinux denials for a particular service, use the -c comm-name option, where comm-name is the executable’s name. For example, httpd for the Apache HTTP Serve or smbd for Samba.

To get a more user-friendly error report use the sealert command:

# sealert -l [message_ID]

Fill in the message_ID with the AVC message number, which you picked up from ausearch. You'll see a friendlier explanation of what went wrong, and a suggestion for a fix. These suggestions are usually worth following up. 

You can also use AVS messages to help create SELinux policies. Use the audit2allow command to gather data from denied operations and generate SELinux policy-allow rules. Or in human terms, it’s like hearing someone say, “You don’t have a key,” and responding, “Okay, then: Give me the key!”

Start by finding out why a process was denied by looking through your logs.

# audit2allow -w -a

That may be all you need. But if it isn’t, and you are unsure what rule would let the process run, SELinux can offer assistance. To see what audit2allow suggests, type:

# audit2allow -a

If you like what you see, create a custom policy module by typing:

# audit2allow -a -M example

The -M flag instructs the command to create a type enforcement (.te) file with the specified name. It also compiles the rule into a policy package (.pp). You install this new SELinux policy with the command:

# semodule -i mypolicy.pp

But audit2allow isn't a cure-all. As Red Hat points out, "Modules created with audit2allow may allow more access than required. It is recommended that policy created with audit2allow be posted to an SELinux list, such as fedora-selinux-list, for review."

When things go really wrong

You’ve tried everything. "You've tried scrubbing them out, soaking them out, but you can still come out with"—gather round, readers of a certain age, and repeat after me—"Ring around the collar," but nothing has worked.

If things go completely awry with your server, you can turn off SELinux by updating the configuration file or by using the boot parameter. But if you do that, SELinux can't generate contexts for new or modified files. Thus, when you turn SELinux back on, it won't know what to do with them and the operating system will heavily restrict how they can be run or read. In short, it will be a mess.

The fix is to do a full file system relabel when booting with SELinux enabled. This is a pain.

If you are working in permissive mode (which is likely), run the following commands:

# touch /.autorelabel
# reboot

If you've been running SELinux in enforcing mode, do this with the following commands: 

# fixfiles -F onboot
# reboot

You don't want to resort to this, but if you must, you must.

Rebooting may take a while because SELinux relabels every last file in this circumstance. As Paul Frields, Red Hat engineering manager, points out, if you’ve been running in disabled mode, it's "like wallpapering over a leak. When you remove the wallpaper, you’re likely to find water damage."

Next steps

Over time, SELinux was merged into the Linux kernel 2.6 mainline. Today, it's primarily maintained by Red Hat Linux security developers and it's in all mainstream Linux.

Now it’s time to begin using your SELinux skills – and to learn more. For a useful general overview, watch Security-Enhanced Linux for mere mortals. For ongoing advice about using SELinux, follow Dan Walsh's SELinux blog. Of course, Red Hat's SELinux documentation is essential reading. Gentoo Linux's SELinux pages are also helpful. 

Noteworthy Linux and open-source stories:

• IBM open-sources its Granite AI models - and they mean business

• How Red Hat is embracing AI to make sysadmins' lives easier

• Linux 6.9 arrives, plus Torvalds indicates Arm64 will get a bit more love