SELinux Blocking Application Troubleshooting

Symptoms

When SELinux is actively blocking your application, the symptoms can be maddeningly vague. The most common one is a flat-out permission denied error in your application logs, even though the file permissions and ownership look completely correct when you run

ls -la

. You've checked the Unix DAC permissions three times, they're fine, and yet the app still won't start or can't read its config file. Something is refusing access and it isn't showing up anywhere obvious.

Other symptoms you'll commonly see: an application fails to bind to a port even though the port isn't in use; a web server returns 403 errors for files that Apache or Nginx clearly has read access to at the filesystem level; a systemd service fails to start with exit code 1 and a cryptic journal entry; an application can't connect to a local database socket despite correct credentials; cron jobs fail silently overnight; log files aren't being written even though the target directory is world-writable.

The frustrating thing about SELinux denials is that they often surface as generic "permission denied" errors with zero indication that SELinux is involved. I've watched engineers chase file ownership and ACLs for an hour before someone remembers to check

ausearch

. If your application is mysteriously failing and the Unix permissions are clean, check SELinux first — every time.

Root Cause 1: AVC Denial in the Audit Log

This is always your first diagnostic stop. AVC stands for Access Vector Cache — it's the SELinux kernel component that makes and logs access control decisions. When SELinux denies an action, it writes an AVC denial message to the audit log at

/var/log/audit/audit.log

. Before you do anything else, read the denial. Everything you need to understand the problem is right there.

Why it happens: SELinux is doing exactly what it's supposed to do — enforcing mandatory access control policy. Your process is running in one security context and trying to access a resource in another context that the policy doesn't permit. The kernel enforces the decision, logs it, and moves on. There's no retry, no fallback, no helpful error message bubbled up to userspace. The application just sees a permission denied and stops.

How to identify it:

ausearch -m avc -ts recent

A typical AVC denial looks like this:

type=AVC msg=audit(1713456789.123:456): avc:  denied  { read } for  pid=3847 comm="nginx" name="app.conf" dev="sda1" ino=1234567 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0

Break that down: nginx (running as

httpd_t

) tried to read a file labeled

admin_home_t

, and SELinux denied it. The

permissive=0

at the end confirms enforcement mode was active and the access was actually blocked — not just logged. You can also run

sealert

for a human-readable diagnosis that often suggests the exact fix:

sealert -a /var/log/audit/audit.log

Or watch for real-time denials during a controlled test run:

ausearch -m avc -ts recent -f /etc/nginx/conf.d/app.conf

How to fix it: Read the denial carefully. It tells you exactly what was denied, by what process, on what resource, and what the source and target contexts are. The AVC denial is your diagnostic entry point — the fix comes from one of the other root causes in this article. If you see

permissive=1

in a denial, SELinux logged it but didn't block the access. Useful for testing, but don't mistake logged-but-not-blocked for a clean bill of health.

Root Cause 2: Wrong SELinux Context on File

This is the most common cause I run into, especially after someone copies files manually, restores from a backup, or creates files in a non-standard location. The Unix permissions are fine, the file exists, the process has read access by DAC standards, and SELinux still blocks it because the file's label is wrong.

Why it happens: SELinux assigns a security context — a label — to every file on the filesystem. The context includes a user, role, type, and level, for example

system_u:object_r:httpd_sys_content_t:s0

. When a file is created in the wrong location or copied improperly with

cp

(rather than moved, or copied with

--preserve=context

), it inherits the wrong context from its source location. A config file that should be labeled

httpd_sys_content_t

might end up labeled

user_home_t

or

admin_home_t

, and the web server process simply is not allowed to read files with those labels regardless of what the permission bits say.

How to identify it:

ls -lZ /var/www/solvethenetwork.com/html/

Expected output for a properly labeled web root:

-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html
-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 style.css

If instead you see this after a manual file copy from a home directory:

-rw-r--r--. infrarunbook-admin infrarunbook-admin unconfined_u:object_r:user_home_t:s0 index.html

That's your problem. The file kept the home directory label.

httpd_t

can't read

user_home_t

. Use

matchpathcon

to confirm what label SELinux expects for a given path, based on the system's file context database:

matchpathcon /var/www/solvethenetwork.com/html/index.html

/var/www/solvethenetwork.com/html/index.html	system_u:object_r:httpd_sys_content_t:s0

If

matchpathcon

shows one context and

ls -Z

shows another, you've confirmed the mismatch.

How to fix it:

restorecon -Rv /var/www/solvethenetwork.com/html/

The verbose output confirms each relabeling operation:

Relabeled /var/www/solvethenetwork.com/html/index.html from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:httpd_sys_content_t:s0

restorecon

is almost always better than

chcon

because it consults the system's SELinux file context database to determine the correct label rather than requiring you to know the right type name and specify it manually. The

-R

flag recurses into subdirectories and

-v

gives you visibility into what changed. If

restorecon

runs but leaves the file with the wrong context — meaning the path isn't in the file context database at all — you're looking at the policy module root cause covered below.

Root Cause 3: Boolean Not Enabled

SELinux policies ship with tunables called booleans. They're essentially feature flags for the policy — on/off switches that enable optional behaviors without requiring a custom policy module. The classic example is

httpd_can_network_connect_db

, which controls whether your web server is allowed to make outbound database connections. Booleans exist because not every httpd deployment needs every capability, and the default posture is conservative.

Why it happens: The base policy for a domain like

httpd_t

locks down optional capabilities by design. Things like making outbound network connections, reading user home directories, or sending email are off by default. When you deploy an application that does need one of these behaviors, you hit a wall that looks exactly like a generic permission denied with nothing pointing at SELinux as the cause.

How to identify it:

getsebool -a | grep httpd

httpd_can_network_connect --> off
httpd_can_network_connect_db --> off
httpd_can_sendmail --> off
httpd_enable_cgi --> off
httpd_read_user_content --> off
httpd_use_nfs --> off

Cross-reference this with your AVC denial. If the denial shows

httpd_t

was denied

name_connect

to a TCP socket on port 5432, and

httpd_can_network_connect_db

is off, you've found it. The

sealert

tool will often tell you directly which boolean to flip:

sealert -a /var/log/audit/audit.log | grep setsebool

If you want to allow httpd to can network connect db
Then you must tell SELinux about this by enabling the 'httpd_can_network_connect_db' boolean.
Do: setsebool -P httpd_can_network_connect_db 1

How to fix it:

setsebool -P httpd_can_network_connect_db on

The

-P

flag writes to the policy store and persists across reboots. Don't forget it. I've seen this burn people who tested with

setsebool

without

-P

, confirmed everything worked, and then spent the next morning debugging why the app broke again after a routine reboot. Verify the change immediately:

getsebool httpd_can_network_connect_db

httpd_can_network_connect_db --> on

Be deliberate about which booleans you enable.

httpd_can_network_connect

is broader than

httpd_can_network_connect_db

— the former allows outbound TCP to any host on any port, while the latter is scoped to database ports. Always use the most specific boolean that satisfies your requirement.

Root Cause 4: Policy Module Missing

Sometimes your application runs outside the well-known paths and domains that the default targeted policy covers. Custom applications, third-party software installed to non-standard directories, or software that does unusual system operations will generate denials that no boolean can fix — because there's simply no policy rule that covers that combination of domain, action, and type.

Why it happens: The default targeted policy covers major system daemons: httpd, sshd, named, vsftpd, postgresql, and others. If you've deployed a custom Go binary, a Python microservice, or a commercial application as a systemd service, it might run in

unconfined_t

or a generic domain like

init_t

that doesn't have the specific access rules your application needs. The kernel blocks the access and you see an AVC denial, but no existing boolean maps to what you need. You can flip booleans all day and nothing will change.

How to identify it:

ausearch -m avc -ts recent | grep scontext

scontext=system_u:system_r:init_t:s0

Seeing

init_t

as the source context for your application process is a clear sign the app has no dedicated SELinux type. It's running in the parent's domain rather than its own. Combined with denials for very application-specific operations — writing to a custom log path, opening a proprietary socket — this confirms you need a custom module.

How to fix it:

Use

audit2allow

to generate a policy module from the denial messages:

ausearch -m avc -ts recent | audit2allow -M myapp_policy

This generates two files:

myapp_policy.te

(the type enforcement source) and

myapp_policy.pp

(the compiled policy package). Before loading anything, read the

.te

file:

cat myapp_policy.te

module myapp_policy 1.0;

require {
        type init_t;
        type var_log_t;
        class file { write create append };
}

#============= init_t ==============
allow init_t var_log_t:file { write create append };

Don't blindly load modules. Read what

audit2allow

generated. If it's trying to grant

init_t

broad write access to a widely-used type like

var_log_t

, think about whether that's really what you want. For a well-understood application where the generated rules are narrow and specific, loading the module is perfectly reasonable:

semodule -i myapp_policy.pp

Verify it loaded correctly:

semodule -l | grep myapp

myapp_policy	1.0

For production systems, keep your

.te

source files in version control alongside your deployment configuration. An accumulation of

audit2allow

-generated modules that nobody has reviewed is a security and maintainability problem. Use

audit2allow

to get unblocked, but invest time in building proper named type definitions for custom applications that will be running long-term.

Root Cause 5: Permissive Mode Not Helping

You've switched SELinux to permissive mode and the application is still broken. This one genuinely confuses people because the assumption is that permissive mode removes SELinux from the picture entirely. It doesn't — and even when it does apply, it won't fix an underlying misconfiguration that has nothing to do with SELinux.

Why it happens: Permissive mode tells SELinux to log denials but not enforce them. There are two scopes of permissive mode: system-wide and per-domain. If you've set only a specific domain to permissive using

semanage permissive

, only that domain's actions are logged-but-not-blocked while the rest of the system remains in enforcing mode. The more important point is this: permissive mode doesn't fix incorrect contexts, missing policy, or wrong port labels — it just stops enforcing them. If your app is broken even in permissive mode, the root cause is almost certainly not SELinux at all.

In my experience, this scenario usually means one of two things. Either the real problem is completely unrelated to SELinux — a missing library, a bad config file, a wrong environment variable — and switching to permissive just stopped one error while exposing another. Or something is genuinely wrong with the SELinux policy store itself, which is rare but happens after interrupted package upgrades.

How to identify it:

getenforce

Permissive

Check whether any per-domain permissive exceptions are active separately from the global mode:

semanage permissive -l

Customized Permissive Types

httpd_t

If the system is truly in permissive mode and the app is still failing, shift your focus to non-SELinux logs immediately. Look at the systemd journal directly:

journalctl -u myapp.service --no-pager -n 50

Run the application binary manually as the service user to see if it fails outside the systemd execution environment:

sudo -u apache /usr/sbin/httpd -t

Even in permissive mode, AVC denials still get logged. Check them — they're still your roadmap for when you return to enforcing mode:

ausearch -m avc -ts recent

How to fix it:

If permissive mode reveals a completely non-SELinux problem, fix that problem. Missing shared libraries, bad configuration syntax, wrong file paths — these have nothing to do with SELinux enforcement mode. Fix the real issue and then return to enforcing. If AVC denials are still appearing in permissive mode, those denials are your guide. Address them using the techniques in this article, then re-enable enforcement:

setenforce 1

To remove a per-domain permissive exception you previously added:

semanage permissive -d httpd_t

Root Cause 6: Port Labeling Issue

Applications that bind to non-standard ports hit this regularly. SELinux maintains a database of port labels, and a process running in a confined domain can only bind to ports that carry the right label. If your app runs on a custom port, SELinux almost certainly doesn't know what to call that port, and the bind fails before any connection is ever accepted.

Why it happens: Standard well-known ports are pre-labeled in the policy. Ports 80 and 443 are

http_port_t

, port 22 is

ssh_port_t

, port 5432 is

postgresql_port_t

. Custom or non-standard ports get labeled

unreserved_port_t

by default, which most confined daemons are not permitted to bind to. The AVC denial for this is distinctive — you'll see a

name_bind

denial rather than the file read or write denials you see with context problems.

How to identify it:

ausearch -m avc -ts recent | grep "name_bind"

type=AVC msg=audit(1713456800.000:789): avc:  denied  { name_bind } for  pid=4521 comm="nginx" src=9443 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0

Check what ports are currently labeled for your service type:

semanage port -l | grep http

http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

How to fix it:

semanage port -a -t http_port_t -p tcp 9443

Verify the port was added:

semanage port -l | grep 9443

http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000, 9443

If the port is already assigned to a different type and you need to reassign it, use

-m

instead of

-a

:

semanage port -m -t http_port_t -p tcp 9443

Root Cause 7: Domain Transition Failure

When a service is launched via a wrapper script, from an unexpected parent process, or with an executable that isn't labeled correctly, the SELinux domain transition that should happen at exec time doesn't occur. Your application ends up running in the parent's domain with either too many or too few permissions — and neither version is what you want.

Why it happens: When systemd executes

/usr/sbin/httpd

, SELinux transitions the new process into

httpd_t

because the policy defines a type transition from

init_t

to

httpd_t

when executing a file labeled

httpd_exec_t

. If the binary is mislabeled as

bin_t

or

usr_t

, the transition rule doesn't fire. If the exec happens via a shell wrapper script, the script itself runs as

shell_exec_t

and the transition may not propagate correctly. Either way, your application ends up running in the wrong domain.

How to identify it:

ps -eZ | grep myapp

system_u:system_r:init_t:s0       4823 ?        00:00:01 myapp

If you expected

myapp_t

and you're seeing

init_t

, the domain transition didn't fire. Check the label on the executable itself:

ls -Z /usr/local/bin/myapp

-rwxr-xr-x. root root system_u:object_r:bin_t:s0 /usr/local/bin/myapp

It's labeled

bin_t

instead of

myapp_exec_t

. The transition rule in your policy module is conditional on the exec type of the binary — if the binary isn't labeled as the expected exec type, the transition simply doesn't happen.

How to fix it:

semanage fcontext -a -t myapp_exec_t "/usr/local/bin/myapp"
restorecon -v /usr/local/bin/myapp

Relabeled /usr/local/bin/myapp from system_u:object_r:bin_t:s0 to system_u:object_r:myapp_exec_t:s0

Restart the service and confirm the process is now running in the correct domain:

systemctl restart myapp
ps -eZ | grep myapp

system_u:system_r:myapp_t:s0      5102 ?        00:00:00 myapp

---

Prevention

The best time to think about SELinux is before you deploy an application, not after it's blocking production traffic. Most of the incidents described in this article are entirely avoidable with a few habits built into your deployment workflow.

Always run new services in permissive mode during initial staging. Set the specific domain permissive, exercise every code path the application will take in production, capture the denial baseline with

ausearch

, and build the correct policy — boolean enables, port labels, context rules, custom modules — before flipping to enforcing. This is what

audit2allow

was designed for. Use it as a development-time tool during staging, not as an emergency response tool when production is down.

Keep your custom SELinux configuration in version control. Export your custom file context rules:

semanage fcontext -l -C > /etc/selinux/custom_fcontexts.txt

Store that file alongside your Ansible playbooks or Salt states for the host. When you provision a new instance of sw-infrarunbook-01, you can replay the entire SELinux configuration from source control rather than reverse-engineering it from a running system. The same principle applies to custom port labels and boolean settings — export them, commit them, apply them from automation.

Add

matchpathcon

checks to your deployment scripts. A short verification block at the end of a deploy that confirms critical files landed with the correct SELinux context will save you debugging sessions you can't afford. If the context is wrong at deploy time, fail the deploy, run

restorecon

, and retry before the service ever starts receiving traffic.

Set up alerting on AVC denial volume. If you're forwarding audit logs to a centralized platform, create an alert that fires when AVC denial count spikes within minutes of a deployment completing. A deployment that silently introduces SELinux violations should page an engineer immediately, not surface during a post-mortem three days later when a related feature stops working.

Document your boolean policy as part of your server baseline documentation. Capture which booleans are enabled on each server role and why. When someone runs

setsebool -P httpd_can_network_connect 1

without understanding why it was off in the first place, you want that tracked in change management — not discovered during a security review six months later.

Don't disable SELinux. The pressure to just set

SELINUX=disabled

in

/etc/selinux/config

is real when production is down and the clock is ticking. Don't do it. The techniques in this article will get you unblocked faster than you think, and the fix is targeted and auditable. Disabling SELinux is a silent, permanent security regression that strips mandatory access control from the entire system. It's easy to forget about and nearly invisible until the incident it was preventing finally happens.