Page MenuHomeDevCentral
Feed All Stories

Aug 3 2024

dereckson raised the priority of T1993: Salt migration to 3007, 3008 and extensions from Normal to High.

I've also discovered we can't deploy secrets to Salt 3007+ anymore from a Salt 3006 server:

Aug 3 2024, 16:53 · Nasqueron Operations Squad, discussion, Salt
dereckson moved T1991: Context has again been lost on /var/log/www from Backlog to To check again on the upstream board.
Aug 3 2024, 16:32 · upstream, Regression, Servers, Salt
dereckson moved T1991: Context has again been lost on /var/log/www from Working on to Pending review on the Servers board.
Aug 3 2024, 16:32 · upstream, Regression, Servers, Salt
dereckson moved T1992: Install patch on redhat family as part of core from Working on to Pending review on the Servers board.
Aug 3 2024, 16:32 · Servers
dereckson moved T1994: Upgrade Salt repository on Debian from Backlog to Pending review on the Servers board.
Aug 3 2024, 16:32 · Eglide, Servers, Salt
dereckson moved T1998: Resolve conflict between core and shellserver roles for Vault in Salt configuration from Backlog to Bug and issues on the Salt board.
Aug 3 2024, 16:26 · Vault, Servers, Salt, Eglide
dereckson moved T1998: Resolve conflict between core and shellserver roles for Vault in Salt configuration from Backlog to Server config on the Eglide board.
Aug 3 2024, 16:26 · Vault, Servers, Salt, Eglide
dereckson added a comment to D3393: Update Salt repository on Debian.

Unit wanted to overwrite Eglide-specific Vault Salt configuration, I've opened T1998 for follow-up.

Aug 3 2024, 16:26
dereckson added a comment to D3393: Update Salt repository on Debian.
Eglide
$ salt-call --local state.apply roles/core/salt test=True
[…]
----------
          ID: /etc/apt/keyrings/salt-archive-keyring-2023.gpg
    Function: file.managed
      Result: True
     Comment: The file /etc/apt/keyrings/salt-archive-keyring-2023.gpg is in the correct state
     Started: 16:25:09.653065
    Duration: 7.356 ms
     Changes:
----------
          ID: /etc/apt/sources.list.d/salt.list
    Function: file.managed
      Result: True
     Comment: The file /etc/apt/sources.list.d/salt.list is in the correct state
     Started: 16:25:09.660512
    Duration: 1.221 ms
     Changes:
[…]
Aug 3 2024, 16:25
dereckson added projects to T1998: Resolve conflict between core and shellserver roles for Vault in Salt configuration: Servers, Vault.
Aug 3 2024, 16:24 · Vault, Servers, Salt, Eglide
dereckson triaged T1998: Resolve conflict between core and shellserver roles for Vault in Salt configuration as Low priority.
Aug 3 2024, 16:23 · Vault, Servers, Salt, Eglide
dereckson added a comment to D3392: Avoid egrep in Salt code base.
Complector
$ salt dwellers state.apply hotfixes/salt
dwellers:
----------
          ID: T1991_egrep_patch
    Function: file.patch
        Name: /opt/saltstack/salt/lib/python3.10/site-packages/salt
      Result: True
     Comment: Patch successfully applied
     Started: 16:15:53.189797
    Duration: 45.172 ms
     Changes:
              ----------
              pid:
                  4123780
              retcode:
                  0
              stderr:
              stdout:
                  patching file modules/csf.py
                  patching file modules/selinux.py
Aug 3 2024, 16:17
dereckson added a revision to T1994: Upgrade Salt repository on Debian: D3393: Update Salt repository on Debian.
Aug 3 2024, 15:44 · Eglide, Servers, Salt
dereckson requested review of D3393: Update Salt repository on Debian.
Aug 3 2024, 15:44
dereckson renamed T1994: Upgrade Salt repository on Debian from Upgrade Salt on Eglide to Upgrade Salt repository on Debian.
Aug 3 2024, 15:17 · Eglide, Servers, Salt
dereckson added a comment to T1996: Servers on hyper-001 have network issues.

From router-001 network looks good:

Aug 3 2024, 13:59 · security, Servers
dereckson added a comment to T1996: Servers on hyper-001 have network issues.

Stopped currently not needed salt and node-exporter on router-001 to see if that helps.

Aug 3 2024, 13:58 · security, Servers
dereckson renamed T1996: Servers on hyper-001 have network issues from Server outage: complector to Servers on hyper-001 have network issues.
Aug 3 2024, 13:23 · security, Servers
dereckson shifted T1996: Servers on hyper-001 have network issues from the S1 Nasqueron space to the Restricted Space space.
Aug 3 2024, 13:23 · security, Servers
dereckson lowered the priority of T1996: Servers on hyper-001 have network issues from Unbreak Now! to High.

Could be at hypervisor level. SSH failed until 13:22 where it worked immediately.

Aug 3 2024, 13:23 · security, Servers
dereckson added a comment to T1996: Servers on hyper-001 have network issues.

As of 13:18 UTC, SSH access works.

Aug 3 2024, 13:19 · security, Servers
dereckson added a comment to T1996: Servers on hyper-001 have network issues.

Also, at the same time, DevCentral is slow for arc diff or to publish this task. This delay behavior is similar as when DNS resolution timeouts occur.

Aug 3 2024, 13:14 · security, Servers
dereckson triaged T1996: Servers on hyper-001 have network issues as Unbreak Now! priority.
Aug 3 2024, 13:13 · security, Servers
dereckson requested review of D3392: Avoid egrep in Salt code base.
Aug 3 2024, 13:10
dereckson added a revision to T1991: Context has again been lost on /var/log/www: D3392: Avoid egrep in Salt code base.
Aug 3 2024, 13:09 · upstream, Regression, Servers, Salt
dereckson added a comment to T1994: Upgrade Salt repository on Debian.
Eglide
$ salt-minion --versions
Salt Version:
          Salt: 3007.1
Aug 3 2024, 13:01 · Eglide, Servers, Salt
dereckson added a subtask for T1950: Deploy PHP 8.3: T1995: PHP 8.2 and PHP 8.3 seems both to be installed on Eglide.
Aug 3 2024, 12:59 · Servers, PHP 8.x support
dereckson added a parent task for T1995: PHP 8.2 and PHP 8.3 seems both to be installed on Eglide: T1950: Deploy PHP 8.3.
Aug 3 2024, 12:59 · PHP 8.x support, Eglide
dereckson moved T1995: PHP 8.2 and PHP 8.3 seems both to be installed on Eglide from Backlog to Servers on the PHP 8.x support board.
Aug 3 2024, 12:59 · PHP 8.x support, Eglide
dereckson triaged T1995: PHP 8.2 and PHP 8.3 seems both to be installed on Eglide as Low priority.
Aug 3 2024, 12:59 · PHP 8.x support, Eglide
dereckson edited projects for T1994: Upgrade Salt repository on Debian, added: Servers, Eglide; removed Nasqueron Operations Squad, discussion.
Aug 3 2024, 12:56 · Eglide, Servers, Salt
dereckson triaged T1994: Upgrade Salt repository on Debian as Normal priority.
Aug 3 2024, 12:52 · Eglide, Servers, Salt
dereckson requested review of D3391: Bump Sphinx package name to use Python3.11.
Aug 3 2024, 12:18
dereckson added a revision to T1982: Upgrade from Python 3.9 to Python 3.11+: D3391: Bump Sphinx package name to use Python3.11.
Aug 3 2024, 12:18 · Servers
dereckson claimed T1991: Context has again been lost on /var/log/www.

We can actually provide P352 as hotfix.

Aug 3 2024, 12:13 · upstream, Regression, Servers, Salt
dereckson added a comment to D3390: Install patch on RHEL-family servers.

Ok, with wget/wget2 transition still problematic (unrelated to patch package).

Aug 3 2024, 12:13
dereckson added a revision to T1992: Install patch on redhat family as part of core: D3390: Install patch on RHEL-family servers.
Aug 3 2024, 12:08 · Servers
dereckson requested review of D3390: Install patch on RHEL-family servers.
Aug 3 2024, 12:08
dereckson added a comment to T1992: Install patch on redhat family as part of core.

patch is available on Eglide as part of build-essential, so presumed OK for Debian

Aug 3 2024, 12:06 · Servers
dereckson added a parent task for T1992: Install patch on redhat family as part of core: T1991: Context has again been lost on /var/log/www.
Aug 3 2024, 12:04 · Servers
dereckson added a subtask for T1991: Context has again been lost on /var/log/www: T1992: Install patch on redhat family as part of core.
Aug 3 2024, 12:04 · upstream, Regression, Servers, Salt
dereckson requested changes to D3387: Expose metrics for dovecot.

Logic looks good to me. Tested working fine on Prometheus.

Aug 3 2024, 11:58
dereckson updated the test plan for D3388: Scrape Dovecot metrics into Prometheus.
Aug 3 2024, 11:55
dereckson updated the test plan for D3388: Scrape Dovecot metrics into Prometheus.
Aug 3 2024, 11:54
dereckson retitled D3387: Expose metrics for dovecot from Init metrics for dovecot to Expose metrics for dovecot.
Aug 3 2024, 11:53
dereckson added a comment to T1987: Dovecot Metrics.

Live on Hervil.

Aug 3 2024, 11:51 · Restricted Project, Mail
dereckson accepted D3388: Scrape Dovecot metrics into Prometheus.

I can query from Prometheus the metrics.

Aug 3 2024, 11:50
dereckson added a comment to D3388: Scrape Dovecot metrics into Prometheus.
WindRiver
$ arc patch D3388
[…]
Aug 3 2024, 11:49
dereckson added a comment to T1982: Upgrade from Python 3.9 to Python 3.11+.

certbot against Python 3.11 should be checked on dwellers and docker-002

Aug 3 2024, 11:45 · Servers
dereckson committed rOPS7d32f4cfeeb2: Bump default versions to build ports (authored by dereckson).
Bump default versions to build ports
Aug 3 2024, 11:43
dereckson closed D3368: Bump default versions to build ports.
Aug 3 2024, 11:43
dereckson closed D3371: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Aug 3 2024, 11:42
dereckson committed rOPS67fecad1cda9: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines" (authored by dereckson).
Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines"
Aug 3 2024, 11:42
dereckson added a reverting change for D2609: Install Python 3.9 on CentOS/Rocky 8.5 machines: rOPS67fecad1cda9: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Aug 3 2024, 11:42
dereckson added a reverting change for rOPSe509029a4712: Install Python 3.9 on CentOS/Rocky 8.5 machines: rOPS67fecad1cda9: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Aug 3 2024, 11:42
dereckson closed D3385: Switch from fixes to flags in node pillar.
Aug 3 2024, 11:37
dereckson committed rOPSe0d579362756: Switch from fixes to flags in node pillar (authored by dereckson).
Switch from fixes to flags in node pillar
Aug 3 2024, 11:37
dereckson requested review of D3389: Refresh known issues.
Aug 3 2024, 11:36
dereckson moved T1992: Install patch on redhat family as part of core from Backlog to Working on on the Servers board.
Aug 3 2024, 11:30 · Servers
dereckson triaged T1993: Salt migration to 3007, 3008 and extensions as Normal priority.
Aug 3 2024, 11:16 · Nasqueron Operations Squad, discussion, Salt
dereckson lowered the priority of T1991: Context has again been lost on /var/log/www from High to Normal.
Aug 3 2024, 10:11 · upstream, Regression, Servers, Salt
dereckson added a comment to T1991: Context has again been lost on /var/log/www.

I've applied P352 to replace egrep by grep -E on dwellers and docker-002.

Aug 3 2024, 10:11 · upstream, Regression, Servers, Salt
dereckson added a comment to T1992: Install patch on redhat family as part of core.

I wanted to apply P354 to fix Salt SELinux issue with patch -p1 < ~/egrep.patch on docker-002.

Aug 3 2024, 10:09 · Servers
dereckson updated the task description for T1992: Install patch on redhat family as part of core.
Aug 3 2024, 10:08 · Servers
dereckson triaged T1992: Install patch on redhat family as part of core as Normal priority.
Aug 3 2024, 10:08 · Servers
dereckson claimed T1992: Install patch on redhat family as part of core.
Aug 3 2024, 10:07 · Servers
dereckson created P354 Salt patch to supersede egrep by grep -E.
Aug 3 2024, 10:04

Jul 31 2024

DorianWinty added a revision to T1987: Dovecot Metrics: D3388: Scrape Dovecot metrics into Prometheus.
Jul 31 2024, 17:51 · Restricted Project, Mail
DorianWinty requested review of D3388: Scrape Dovecot metrics into Prometheus.
Jul 31 2024, 17:51
DorianWinty updated the diff for D3387: Expose metrics for dovecot.

uncomment all default metrics

Jul 31 2024, 17:40
DorianWinty added a comment to D3387: Expose metrics for dovecot.
# HELP process_start_time_seconds Timestamp of service start
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1722447303
# HELP dovecot_build Dovecot build information
# TYPE dovecot_build info
dovecot_build_info{version="2.3.21",revision="47349e2482"} 1
# EOF
Jul 31 2024, 17:37
DorianWinty requested review of D3387: Expose metrics for dovecot.
Jul 31 2024, 17:37
DorianWinty added a revision to T1987: Dovecot Metrics: D3387: Expose metrics for dovecot.
Jul 31 2024, 17:37 · Restricted Project, Mail
DorianWinty closed D3386: Scrape Postfix metrics into Prometheus.
Jul 31 2024, 16:33
DorianWinty committed rOPSe8b486d9e75b: Scrape Postfix metrics into Prometheus (authored by DorianWinty).
Scrape Postfix metrics into Prometheus
Jul 31 2024, 16:33
dereckson moved T1991: Context has again been lost on /var/log/www from Backlog to Bug and issues on the Salt board.
Jul 31 2024, 16:30 · upstream, Regression, Servers, Salt
dereckson added a project to T1991: Context has again been lost on /var/log/www: upstream.
Jul 31 2024, 16:30 · upstream, Regression, Servers, Salt
dereckson added a comment to T1991: Context has again been lost on /var/log/www.

Already reported upstream: https://github.com/saltstack/salt/issues/65608

Jul 31 2024, 16:30 · upstream, Regression, Servers, Salt
dereckson added a comment to T1991: Context has again been lost on /var/log/www.
Complector
$ cd /opt/salt/nasqueron-operations
$ salt dwellers state.apply roles/webserver-core/nginx/config
[…]
----------                                                                                                                                                                                                                                    [3/295]
          ID: selinux_context_nginx_logs
    Function: selinux.fcontext_policy_present
        Name: /var/log/www
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2428, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1317, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/selinux.py", line 326, in fcontext_policy_present
                  current_state = __salt__["selinux.fcontext_get_policy"](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/selinux.py", line 507, in fcontext_get_policy
                  "filespec": parts.group(1).strip(),
              AttributeError: 'NoneType' object has no attribute 'group'
     Started: 16:25:51.413301
    Duration: 391.186 ms
     Changes:
----------
          ID: selinux_context_nginx_logs_applied
    Function: selinux.fcontext_policy_applied
        Name: /var/log/www
      Result: True
     Comment: SElinux policies are already applied for filespec "/var/log/www"
     Started: 16:25:51.804764
    Duration: 6.322 ms
     Changes:
----------
[…]
Jul 31 2024, 16:27 · upstream, Regression, Servers, Salt
DorianWinty added a comment to T1505: Automate Let's Encrypt TLS certificates management for every server.

31/07/2024 at 12h the devcentral.nasqueron.org certificate expired

Jul 31 2024, 16:24 · Servers
dereckson added a comment to T1991: Context has again been lost on /var/log/www.

Issue can be repro on Dwellers:

Jul 31 2024, 16:23 · upstream, Regression, Servers, Salt
dereckson edited Description on Regression.
Jul 31 2024, 16:22
dereckson added a project to T1991: Context has again been lost on /var/log/www: Regression.
Jul 31 2024, 16:22 · upstream, Regression, Servers, Salt
dereckson triaged T1991: Context has again been lost on /var/log/www as High priority.
Jul 31 2024, 16:21 · upstream, Regression, Servers, Salt

Jul 30 2024

dereckson accepted D3386: Scrape Postfix metrics into Prometheus.
Jul 30 2024, 21:32
dereckson assigned T1990: Export metrics for Postfix to DorianWinty.

Author seems to report issues with the exporter and uses mtail.

Jul 30 2024, 21:30 · Mail, Monitoring and reporting
DorianWinty added a revision to T1990: Export metrics for Postfix: D3386: Scrape Postfix metrics into Prometheus.
Jul 30 2024, 21:00 · Mail, Monitoring and reporting
DorianWinty requested review of D3386: Scrape Postfix metrics into Prometheus.
Jul 30 2024, 21:00
DorianWinty added a subtask for T1930: Postfix Provisioning: T1990: Export metrics for Postfix.
Jul 30 2024, 20:59 · Mail, Restricted Project, Servers
DorianWinty added a parent task for T1990: Export metrics for Postfix: T1930: Postfix Provisioning.
Jul 30 2024, 20:59 · Mail, Monitoring and reporting
DorianWinty updated the diff for D3242: Install postfix.

update depending of the hand made changes

Jul 30 2024, 17:23
DorianWinty accepted D3385: Switch from fixes to flags in node pillar.
Jul 30 2024, 16:11

Jul 29 2024

dereckson triaged T1990: Export metrics for Postfix as Normal priority.
Jul 29 2024, 20:33 · Mail, Monitoring and reporting
DorianWinty updated the diff for D3242: Install postfix.

update

Jul 29 2024, 20:02
dereckson added inline comments to D3242: Install postfix.
Jul 29 2024, 19:54
DorianWinty closed D3364: Provisioning Dovecot Config.
Jul 29 2024, 19:03
DorianWinty committed rOPSf6419f2b78d8: Provisioning Dovecot Config (authored by DorianWinty).
Provisioning Dovecot Config
Jul 29 2024, 19:03
dereckson accepted D3364: Provisioning Dovecot Config.
Jul 29 2024, 19:01
dereckson moved T1989: Merge Nasqueron infrastructure reference into ops grimoire from Backlog - On hold pending T1475 to Checks after T1475 on the Mail board.
Jul 29 2024, 18:58 · Mail, Nasqueron Operations Squad, Servers, documentation
dereckson triaged T1989: Merge Nasqueron infrastructure reference into ops grimoire as Normal priority.
Jul 29 2024, 18:58 · Mail, Nasqueron Operations Squad, Servers, documentation