Page MenuHomeDevCentral
Create Task
Maniphest T1991

Context has again been lost on /var/log/www
Open, NormalPublic
Actions

Description

@DorianWinty has reported that nginx issue:

$ sudo systemctl status nginx
× nginx.service - nginx - high performance web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-07-31 16:11:27 UTC; 58s ago
   Duration: 5month 2w 3d 5h 12min 14.373s
       Docs: http://nginx.org/en/docs/
    Process: 2788170 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)
        CPU: 112ms

Jul 31 16:11:26 docker-002.nasqueron.org systemd[1]: Starting nginx - high performance web server...
Jul 31 16:11:27 docker-002.nasqueron.org nginx[2788170]: nginx: [emerg] open() "/var/log/www/error.log" failed (13: Permission denied)
Jul 31 16:11:27 docker-002.nasqueron.org …

Can be fixed with chcon -t httpd_log_t /var/log/www /var/log/www/*.log, but before I tried a
restorecon and the context has heen lost:

Default context for /var/logvar_log_t
Context for nginx logshttpd_log_t

As far as I know, https://devcentral.nasqueron.org/source/operations/browse/main/roles/webserver-core/nginx/config.sls$91 takes care to set that context permanently.

Revisions and Commits

rOPS Nasqueron Operations
D3392rOPS4a03b8471a74 Avoid egrep in Salt code base

Related Objects
Search...

Event Timeline

dereckson triaged this task as High priority.Jul 31 2024, 16:21
dereckson created this task.
dereckson added a project: Regression.

Issue can be repro on Dwellers:

$ cd /var/log/www
$ ls -lahZ
total 956M
drwx--x--x.  2 nginx web  system_u:object_r:httpd_log_t:s0   41 Apr 17  2023 .
drwxr-xr-x. 12 root  root system_u:object_r:var_log_t:s0   4.0K Jul 28 00:00 ..
-rw-r--r--.  1 nginx web  system_u:object_r:httpd_log_t:s0 373M Jul 31 16:21 access.log
-rw-r--r--.  1 nginx web  system_u:object_r:httpd_log_t:s0 583M Jul 31 16:21 error.log
$ restorecon -F -R /var/log/www
$ ls -lahZ
total 956M
drwx--x--x.  2 nginx web  system_u:object_r:httpd_log_t:s0   41 Apr 17  2023 .
drwxr-xr-x. 12 root  root system_u:object_r:var_log_t:s0   4.0K Jul 28 00:00 ..
-rw-r--r--.  1 nginx web  system_u:object_r:var_log_t:s0   373M Jul 31 16:22 access.log
-rw-r--r--.  1 nginx web  system_u:object_r:var_log_t:s0   583M Jul 31 16:22 error.log
dereckson added a comment.Jul 31 2024, 16:27

Gotcha!

Complector
$ cd /opt/salt/nasqueron-operations
$ salt dwellers state.apply roles/webserver-core/nginx/config
[…]
----------                                                                                                                                                                                                                                    [3/295]
          ID: selinux_context_nginx_logs
    Function: selinux.fcontext_policy_present
        Name: /var/log/www
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2428, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1317, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/selinux.py", line 326, in fcontext_policy_present
                  current_state = __salt__["selinux.fcontext_get_policy"](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/selinux.py", line 507, in fcontext_get_policy
                  "filespec": parts.group(1).strip(),
              AttributeError: 'NoneType' object has no attribute 'group'
     Started: 16:25:51.413301
    Duration: 391.186 ms
     Changes:
----------
          ID: selinux_context_nginx_logs_applied
    Function: selinux.fcontext_policy_applied
        Name: /var/log/www
      Result: True
     Comment: SElinux policies are already applied for filespec "/var/log/www"
     Started: 16:25:51.804764
    Duration: 6.322 ms
     Changes:
----------
[…]
dereckson added a comment.Jul 31 2024, 16:29

Already reported upstream: https://github.com/saltstack/salt/issues/65608

dereckson added a project: upstream.Jul 31 2024, 16:30
dereckson moved this task from Backlog to Bug and issues on the Salt board.
dereckson added a comment.Aug 3 2024, 10:11

I've applied P352 to replace egrep by grep -E on dwellers and docker-002.

That needs to be done every Salt update until we prioritise merging the fix with upstream.

dereckson lowered the priority of this task from High to Normal.Aug 3 2024, 10:11
dereckson added a subtask: T1992: Install patch on redhat family as part of core.Aug 3 2024, 12:04
dereckson claimed this task.EditedAug 3 2024, 12:13
dereckson moved this task from Backlog to Working on on the Servers board.

We can actually provide P352 as hotfix.

See https://github.com/saltstack/salt/issues/56121#issuecomment-585063805 for an example of how to apply a patch against Salt with Salt.

dereckson added a revision: D3392: Avoid egrep in Salt code base.Aug 3 2024, 13:09
dereckson moved this task from Working on to Pending review on the Servers board.Aug 3 2024, 16:32
dereckson moved this task from Backlog to To check again on the upstream board.
dereckson closed subtask T1992: Install patch on redhat family as part of core as Resolved.Aug 5 2024, 19:26
dereckson added a commit: rOPS4a03b8471a74: Avoid egrep in Salt code base.Aug 21 2024, 18:19
dereckson added a comment.Oct 14 2024, 00:23

Same thing for Dwellers:

Oct 14 00:22:06 dwellers.nasqueron.org nginx[1665274]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Oct 14 00:22:06 dwellers.nasqueron.org nginx[1665274]: nginx: [emerg] open() "/var/log/www/error.log" failed (13: Permission denied)
dereckson moved this task from To check again to To report on the upstream board.Oct 31 2024, 18:46
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator