Maniphest T1991

www
Open, NormalPublic
Actions

Assigned To

None

Authored By

	dereckson
	Jul 31 2024, 16:21

Description

@DorianWinty has reported that nginx issue:

$ sudo systemctl status nginx
× nginx.service - nginx - high performance web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-07-31 16:11:27 UTC; 58s ago
   Duration: 5month 2w 3d 5h 12min 14.373s
       Docs: http://nginx.org/en/docs/
    Process: 2788170 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)
        CPU: 112ms

Jul 31 16:11:26 docker-002.nasqueron.org systemd[1]: Starting nginx - high performance web server...
Jul 31 16:11:27 docker-002.nasqueron.org nginx[2788170]: nginx: [emerg] open() "/var/log/www/error.log" failed (13: Permission denied)
Jul 31 16:11:27 docker-002.nasqueron.org …

Can be fixed with chcon -t httpd_log_t /var/log/www /var/log/www/*.log, but before I tried a
restorecon and the context has heen lost:

Default context for /var/log	var_log_t
Context for nginx logs	httpd_log_t

As far as I know, https://devcentral.nasqueron.org/source/operations/browse/main/roles/webserver-core/nginx/config.sls$91 takes care to set that context permanently.

Revisions and Commits

rOPS Nasqueron Operations
	D3392	rOPS4a03b8471a74 Avoid egrep in Salt code base

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T1991 Context has again been lost on /var/log/www
		Resolved		dereckson	T1992 Install patch on redhat family as part of core

Event Timeline

Issue can be repro on Dwellers:

$ cd /var/log/www
$ ls -lahZ
total 956M
drwx--x--x.  2 nginx web  system_u:object_r:httpd_log_t:s0   41 Apr 17  2023 .
drwxr-xr-x. 12 root  root system_u:object_r:var_log_t:s0   4.0K Jul 28 00:00 ..
-rw-r--r--.  1 nginx web  system_u:object_r:httpd_log_t:s0 373M Jul 31 16:21 access.log
-rw-r--r--.  1 nginx web  system_u:object_r:httpd_log_t:s0 583M Jul 31 16:21 error.log
$ restorecon -F -R /var/log/www
$ ls -lahZ
total 956M
drwx--x--x.  2 nginx web  system_u:object_r:httpd_log_t:s0   41 Apr 17  2023 .
drwxr-xr-x. 12 root  root system_u:object_r:var_log_t:s0   4.0K Jul 28 00:00 ..
-rw-r--r--.  1 nginx web  system_u:object_r:var_log_t:s0   373M Jul 31 16:22 access.log
-rw-r--r--.  1 nginx web  system_u:object_r:var_log_t:s0   583M Jul 31 16:22 error.log

Gotcha!

Complector

$ cd /opt/salt/nasqueron-operations
$ salt dwellers state.apply roles/webserver-core/nginx/config
[…]
----------                                                                                                                                                                                                                                    [3/295]
          ID: selinux_context_nginx_logs
    Function: selinux.fcontext_policy_present
        Name: /var/log/www
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2428, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1317, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/selinux.py", line 326, in fcontext_policy_present
                  current_state = __salt__["selinux.fcontext_get_policy"](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/selinux.py", line 507, in fcontext_get_policy
                  "filespec": parts.group(1).strip(),
              AttributeError: 'NoneType' object has no attribute 'group'
     Started: 16:25:51.413301
    Duration: 391.186 ms
     Changes:
----------
          ID: selinux_context_nginx_logs_applied
    Function: selinux.fcontext_policy_applied
        Name: /var/log/www
      Result: True
     Comment: SElinux policies are already applied for filespec "/var/log/www"
     Started: 16:25:51.804764
    Duration: 6.322 ms
     Changes:
----------
[…]

Already reported upstream: https://github.com/saltstack/salt/issues/65608

dereckson added a project: upstream.Jul 31 2024, 16:30

dereckson moved this task from Backlog to Bug and issues on the Salt board.

I've applied P352 to replace egrep by grep -E on dwellers and docker-002.

That needs to be done every Salt update until we prioritise merging the fix with upstream.

dereckson lowered the priority of this task from High to Normal.Aug 3 2024, 10:11

dereckson added a subtask: T1992: Install patch on redhat family as part of core.Aug 3 2024, 12:04

We can actually provide P352 as hotfix.

See https://github.com/saltstack/salt/issues/56121#issuecomment-585063805 for an example of how to apply a patch against Salt with Salt.

dereckson added a revision: D3392: Avoid egrep in Salt code base.Aug 3 2024, 13:09

dereckson moved this task from Working on to Pending review on the Servers board.Aug 3 2024, 16:32

dereckson moved this task from Backlog to To check again on the upstream board.

dereckson closed subtask T1992: Install patch on redhat family as part of core as Resolved.Aug 5 2024, 19:26

dereckson added a commit: rOPS4a03b8471a74: Avoid egrep in Salt code base.Aug 21 2024, 18:19

Same thing for Dwellers:

Oct 14 00:22:06 dwellers.nasqueron.org nginx[1665274]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Oct 14 00:22:06 dwellers.nasqueron.org nginx[1665274]: nginx: [emerg] open() "/var/log/www/error.log" failed (13: Permission denied)

dereckson moved this task from To check again to To report on the upstream board.Oct 31 2024, 18:46

Unassigning this task to make it available for others.

As I'm not currently working actively on this issue, reset the assignee.

Contributions opportunity. If you’re interested, go ahead and assign yourself. I’m available to share context, review early approaches, or pair if helpful—just tag me.

Context has again been lost on /var/log/wwwOpen, NormalPublicActions

Description

Revisions and Commits

Related ObjectsSearch...

Event Timeline

Context has again been lost on /var/log/www
Open, NormalPublic
Actions

Related Objects
Search...