Page MenuHomeDevCentral
Feed All Stories

Mar 25 2023

dereckson added a comment to T1805: Sentry can't send mail anymore.

Test mail received with D2918:

Mar 25 2023, 14:21 · Monitoring and reporting
dereckson accepted D2918: Update EXIM containers uid/gid.

Works like a charm, but by the way sentry_smtp logs every mail dialog (excepted mail content), not sure we want that to stdout (and so docker logs).

Mar 25 2023, 14:16
dereckson added a comment to D2918: Update EXIM containers uid/gid.
Complector
$ salt docker-002 state.sls roles/paas-docker/containers/exim
docker-002:
----------
          ID: /srv/exim/sentry_smtp
    Function: file.directory
      Result: True
     Comment: Directory /srv/exim/sentry_smtp updated
     Started: 14:08:15.648086
    Duration: 466.741 ms
     Changes:
              ----------
              /srv/exim/sentry_smtp:
                  ----------
                  group:
                      101
                  user:
                      101
              group:
                  101
              user:
                  101
----------
          ID: /srv/exim/sentry_smtp/spool
    Function: file.directory
      Result: True
     Comment: Directory /srv/exim/sentry_smtp/spool updated
     Started: 14:08:16.115024
    Duration: 4.355 ms
     Changes:
              ----------
              /srv/exim/sentry_smtp/spool:
                  ----------
                  group:
                      101
                  user:
                      101
              group:
                  101
              user:
                  101
----------
          ID: /srv/exim/sentry_smtp/log
    Function: file.directory
      Result: True
     Comment: Directory /srv/exim/sentry_smtp/log updated
     Started: 14:08:16.119537
    Duration: 4.736 ms
     Changes:
              ----------
              /srv/exim/sentry_smtp/log:
                  ----------
                  group:
                      101
                  user:
                      101
              group:
                  101
              user:
                  101
[…]
Mar 25 2023, 14:12
dereckson added a revision to T1805: Sentry can't send mail anymore: D2918: Update EXIM containers uid/gid.
Mar 25 2023, 14:07 · Monitoring and reporting
dereckson requested review of D2918: Update EXIM containers uid/gid.
Mar 25 2023, 14:07
dereckson requested review of D2917: Upgrade to PHP 8.2.4.
Mar 25 2023, 14:05
dereckson closed D2916: Enable sockets extension.
Mar 25 2023, 14:01
dereckson committed rDPHPCLI1d78152e54c1: Enable sockets extension (authored by dereckson).
Enable sockets extension
Mar 25 2023, 14:01
dereckson accepted D2916: Enable sockets extension.
Dwellers
$ docker build .
[…]
 => => writing image sha256:a6682dbc98cdbbd455263de28a47da8147a06d57eee249d47493a87449b97d4e
Mar 25 2023, 14:00
dereckson added a comment to T1805: Sentry can't send mail anymore.

The "could not open panic log" could also mean error also occurs for /srv/exim/sentry_smtp/log

Mar 25 2023, 13:58 · Monitoring and reporting
dereckson added a comment to T1805: Sentry can't send mail anymore.
docker-002
$ docker logs sentry_smtp
   10 LOG: MAIN
   10   exim 4.92 daemon started: pid=10, no queue runs, listening for SMTP on port 25 (IPv6 and IPv4)
   11 LOG: host_lookup_failed MAIN
   11   no host name found for IP address 172.18.3.19
   11 LOG: MAIN PANIC DIE
   11   Failed to create directory "/var/spool/exim4/input": Permission denied
   11
   11 2023-03-14 23:14:08 1pcDqK-00000B-Rm Failed to create directory "/var/spool/exim4/input": Permission denied
   11
   11 2023-03-14 23:14:08 1pcDqK-00000B-Rm Failed to create directory "/var/spool/exim4/input": Permission denied
   11
   11 exim: could not open panic log - aborting: see message(s) above
   12 LOG: host_lookup_failed MAIN
   12   no host name found for IP address 172.18.3.19
   12 LOG: MAIN PANIC DIE
   12   Failed to create directory "/var/spool/exim4/input": Permission denied
   12
   12 2023-03-14 23:15:31 1pcDrf-00000C-5i Failed to create directory "/var/spool/exim4/input": Permission denied
   12
   12 2023-03-14 23:15:31 1pcDrf-00000C-5i Failed to create directory "/var/spool/exim4/input": Permission denied
   12
   12 exim: could not open panic log - aborting: see message(s) above
   13 LOG: host_lookup_failed MAIN
   13   no host name found for IP address 172.18.3.21
   13 LOG: MAIN PANIC DIE
   13   Failed to create directory "/var/spool/exim4/input": Permission denied
   13
   13 2023-03-20 12:00:02 1peEBG-00000D-9c Failed to create directory "/var/spool/exim4/input": Permission denied
   13
   13 2023-03-20 12:00:02 1peEBG-00000D-9c Failed to create directory "/var/spool/exim4/input": Permission denied
   13
   13 exim: could not open panic log - aborting: see message(s) above
Mar 25 2023, 13:56 · Monitoring and reporting
dereckson triaged T1805: Sentry can't send mail anymore as Normal priority.
Mar 25 2023, 13:55 · Monitoring and reporting
dereckson reopened T1800: Ensure PHP sockets extension is available everywhere as "Open".
Mar 25 2023, 13:42 · Docker images, Servers, Notifications center, Keruald
dereckson reopened T1800: Ensure PHP sockets extension is available everywhere, a subtask of T1798: Upgrade php-amqplib/php-amqplib in keruald/broker, as Open.
Mar 25 2023, 13:42 · Notifications center, Keruald
dereckson updated the diff for D2916: Enable sockets extension.

Rebased against D2915.

Mar 25 2023, 13:41
dereckson closed D2915: Upgrade to PHP 8.1.17.
Mar 25 2023, 13:40
dereckson committed rDPHPCLIfdc23ec6a793: Upgrade to PHP 8.1.17 (authored by dereckson).
Upgrade to PHP 8.1.17
Mar 25 2023, 13:40
dereckson accepted D2915: Upgrade to PHP 8.1.17.
Dwellers
$ docker build .
[…]
=> writing image sha256:e7278eae680aae23a401a6548a76c5342428b01125a37f8ff81328fb4bf93c25              0.0s
Mar 25 2023, 13:39
dereckson requested review of D2916: Enable sockets extension.
Mar 25 2023, 13:38
dereckson added a revision to T1800: Ensure PHP sockets extension is available everywhere: D2916: Enable sockets extension.
Mar 25 2023, 13:38 · Docker images, Servers, Notifications center, Keruald
dereckson requested review of D2915: Upgrade to PHP 8.1.17.
Mar 25 2023, 13:26
dereckson added a comment to T1798: Upgrade php-amqplib/php-amqplib in keruald/broker.

Jenkins PHP agent doesn't have this either.

Mar 25 2023, 13:22 · Notifications center, Keruald
dereckson committed rNOTIF0a1189f54a23: Drop unused dev dependencies (authored by dereckson).
Drop unused dev dependencies
Mar 25 2023, 13:17
dereckson closed D2914: Drop unused dev dependencies.
Mar 25 2023, 13:17
dereckson accepted D2914: Drop unused dev dependencies.
Mar 25 2023, 13:17
dereckson published D2914: Drop unused dev dependencies for review.
Mar 25 2023, 13:17
dereckson closed D2913: Align GateController::getServiceName visibility.
Mar 25 2023, 12:49
dereckson closed T1804: Align access levels for GateController::getServiceName() as Resolved by committing rNOTIFc3a6214efbc5: Align GateController::getServiceName visibility.
Mar 25 2023, 12:49 · Notifications center
dereckson committed rNOTIFc3a6214efbc5: Align GateController::getServiceName visibility (authored by dereckson).
Align GateController::getServiceName visibility
Mar 25 2023, 12:49
dereckson accepted D2913: Align GateController::getServiceName visibility.
Mar 25 2023, 12:49
dereckson published D2913: Align GateController::getServiceName visibility for review.
Mar 25 2023, 12:48
dereckson added a revision to T1804: Align access levels for GateController::getServiceName(): D2913: Align GateController::getServiceName visibility.
Mar 25 2023, 12:48 · Notifications center
dereckson updated the task description for T1804: Align access levels for GateController::getServiceName().
Mar 25 2023, 12:45 · Notifications center
dereckson updated subscribers of T1804: Align access levels for GateController::getServiceName().
Mar 25 2023, 12:44 · Notifications center
dereckson moved T1804: Align access levels for GateController::getServiceName() from Backlog to Bugs on the Notifications center board.
Mar 25 2023, 12:43 · Notifications center
dereckson triaged T1804: Align access levels for GateController::getServiceName() as Unbreak Now! priority.
Mar 25 2023, 12:43 · Notifications center
dereckson added a comment to D2912: Build Apache Ant from ports on FreeBSD devserver.

Two changes for the ports build workflow:

Mar 25 2023, 12:37
dereckson committed rOPS12f75d2e50c6: Build Apache Ant from ports on FreeBSD devserver (authored by dereckson).
Build Apache Ant from ports on FreeBSD devserver
Mar 25 2023, 12:34
dereckson closed D2912: Build Apache Ant from ports on FreeBSD devserver.
Mar 25 2023, 12:34
dereckson accepted D2912: Build Apache Ant from ports on FreeBSD devserver.
Mar 25 2023, 12:34
dereckson requested review of D2912: Build Apache Ant from ports on FreeBSD devserver.
Mar 25 2023, 12:34
dereckson closed D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:24
dereckson added a reverting change for D2910: Bump OpenJDK version to 19 on devserver: rOPS9429d23a59ac: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson added a reverting change for rOPS451ece6b1803: Bump OpenJDK version to 19 on devserver: rOPS9429d23a59ac: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson committed rOPS9429d23a59ac: Revert "Bump OpenJDK version to 19 on devserver" (authored by dereckson).
Revert "Bump OpenJDK version to 19 on devserver"
Mar 25 2023, 12:23
dereckson accepted D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson updated the test plan for D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson requested review of D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson added a reverting change for rOPS451ece6b1803: Bump OpenJDK version to 19 on devserver: D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson added a reverting change for D2910: Bump OpenJDK version to 19 on devserver: D2911: Revert "Bump OpenJDK version to 19 on devserver".
Mar 25 2023, 12:23
dereckson closed D2910: Bump OpenJDK version to 19 on devserver.
Mar 25 2023, 12:15
dereckson committed rOPS451ece6b1803: Bump OpenJDK version to 19 on devserver (authored by dereckson).
Bump OpenJDK version to 19 on devserver
Mar 25 2023, 12:15
dereckson accepted D2910: Bump OpenJDK version to 19 on devserver.
Mar 25 2023, 12:15
dereckson retitled D2910: Bump OpenJDK version to 19 on devserver from Bump OpenJDK version to 19 to Bump OpenJDK version to 19 on devserver.
Mar 25 2023, 12:15
dereckson requested review of D2910: Bump OpenJDK version to 19 on devserver.
Mar 25 2023, 12:15
dereckson committed rNOTIFbc7824663d06: Fix php-amqplib deprecations (authored by dereckson).
Fix php-amqplib deprecations
Mar 25 2023, 12:05
dereckson closed D2895: Fix php-amqplib deprecations.
Mar 25 2023, 12:05
DorianWinty accepted D2895: Fix php-amqplib deprecations.
Mar 25 2023, 12:00
DorianWinty closed D2909: Declare GateController as an abstract.
Mar 25 2023, 11:59
DorianWinty committed rNOTIF57e8fae268ce: Declare GateController as an abstract (authored by DorianWinty).
Declare GateController as an abstract
Mar 25 2023, 11:59
dereckson accepted D2909: Declare GateController as an abstract.
Mar 25 2023, 11:57
dereckson published D2909: Declare GateController as an abstract for review.
Mar 25 2023, 11:57
DorianWinty closed D2706: Add missing parameters return type of each function.
Mar 25 2023, 11:37
DorianWinty committed rNOTIFe564474dc768: Add missing parameters return type of each function (authored by DorianWinty).
Add missing parameters return type of each function
Mar 25 2023, 11:37
dereckson accepted D2706: Add missing parameters return type of each function.
Mar 25 2023, 11:36
DorianWinty updated the diff for D2706: Add missing parameters return type of each function.

restore array phpDocs

Mar 25 2023, 11:34
dereckson triaged T1803: Move and migrate Ysul production services elsewhere as Normal priority.
Mar 25 2023, 11:02 · Operations sprints (Ignite Alkane Propulsion), Alkane, Servers
dereckson closed T1791: Refresh Sentry installation as Resolved.
Mar 25 2023, 09:53 · Monitoring and reporting
dereckson added a comment to T1750: Import FANTOIR database.

To configure Airflow, we can need a PostgreSQL DSN.

Mar 25 2023, 09:46 · Nasqueron Databases
dereckson added a comment to T1791: Refresh Sentry installation.

Current status

Mar 25 2023, 09:45 · Monitoring and reporting
dereckson closed T1802: Upgrade Sentry to 23.3.1 as Resolved.
docker-002
$ docker pull nasqueron/sentry
[…]
$ sentry nasqueron upgrade
[…]
Running migrations for default
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0380_backfill_monitor_env_initial... OK
  Applying sentry.0381_fix_org_slug_casing... OK
  Applying sentry.0382_add_installation_id_to_service_hook... OK
  Applying sentry.0383_mv_user_avatar... OK
  Applying sentry.0384_backfill_installation_ids... OK
  Applying sentry.0385_service_hook_hc_fk... OK
  Applying sentry.0386_backfill_monitor_env_checkins... OK
  Applying sentry.0387_add_until_escalating_column_to_groupsnooze... OK
  Applying sentry.0388_remove_unique_constraint_artifact_bundle... OK
  Applying sentry.0389_nullify_debug_id_artifact_bundle_field... OK
  Applying sentry.0390_remove_field_in_bundle_model... OK
  Applying sentry.0391_remove_field_in_bundle_model_from_db... OK
  Applying sentry.0392_add_date_uploaded_field_to_bundle... OK
  Applying sentry.0393_create_groupforecast_table... OK
  Applying sentry.0394_set_none_date_uploaded_to_date_added... OK
Creating missing DSNs
Correcting Group.num_comments counter
$ deploy-container sentry
[…]
Mar 25 2023, 09:29 · Monitoring and reporting
dereckson closed D2908: Pull nasqueron/sentry image.
Mar 25 2023, 08:49
dereckson committed rOPS2517ada8c0e0: Pull nasqueron/sentry image (authored by dereckson).
Pull nasqueron/sentry image
Mar 25 2023, 08:49
dereckson updated the diff for D2908: Pull nasqueron/sentry image.

Alphabetic order

Mar 25 2023, 08:49
dereckson accepted D2908: Pull nasqueron/sentry image.
Mar 25 2023, 08:49
dereckson requested review of D2908: Pull nasqueron/sentry image.
Mar 25 2023, 08:48
dereckson added a revision to T1791: Refresh Sentry installation: D2908: Pull nasqueron/sentry image.
Mar 25 2023, 08:48 · Monitoring and reporting

Mar 24 2023

dereckson added a comment to T1802: Upgrade Sentry to 23.3.1.
docker-002
$ docker logs sentry_snuba_api
2023-03-24 20:58:44,295 Initializing Snuba...
2023-03-24 20:58:47,701 Snuba initialization took 3.4077668339014053s
2023-03-24 20:58:48,246 Initializing Snuba...
2023-03-24 20:58:51,448 Snuba initialization took 3.2032665936276317s
*** Starting uWSGI 2.0.20 (64bit) on [Fri Mar 24 20:58:51 2023] ***
compiled with version: 8.3.1 20190311 (Red Hat 8.3.1-3) on 11 November 2021 18:13:24
os: Linux-5.14.0-162.12.1.el9_1.0.2.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 30 22:14:42 UTC 2023
nodename: b225edf80078
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /usr/src/snuba
detected binary path: /usr/local/bin/python
your memory page size is 4096 bytes
detected max file descriptor number: 1073741816
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
Python version: 3.8.13 (default, Aug 23 2022, 09:42:19)  [GCC 10.2.1 20210110]
--- Python VM already initialized ---
Python main interpreter initialized at 0x564efb872870
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145808 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
initialized 38 metrics
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 28, cores: 1)
metrics collector thread started
2023-03-24 20:58:51,868 Initializing Snuba...
WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x564efb872870 pid: 28 (default app)
Mar 24 2023, 21:01 · Monitoring and reporting
dereckson added a comment to T1802: Upgrade Sentry to 23.3.1.

ModuleNotFoundError: No module named 'snuba.settings.settings_self_hosted'

Mar 24 2023, 20:59 · Monitoring and reporting
dereckson added a comment to T1802: Upgrade Sentry to 23.3.1.
docker-002
$ docker logs sentry_zookeeper
[…]
===> User
uid=0(root) gid=0(root) groups=0(root)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/zookeeper/data is writable ...
===> Check if /var/lib/zookeeper/log is writable ...
===> Launching ...
===> Launching zookeeper ...
[2023-03-24 20:43:14,853] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2023-03-24 20:43:15,007] WARN o.e.j.s.ServletContextHandler@33f88ab{/,null,STOPPED} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2023-03-24 20:43:15,007] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2023-03-24 20:43:15,078] WARN ServletContext@o.e.j.s.ServletContextHandler@33f88ab{/,null,STARTING} has uncovered http methods for path: /* (org.eclipse.jetty.security.SecurityHandler)
Mar 24 2023, 20:48 · Monitoring and reporting
dereckson closed D2907: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:40
dereckson committed rOPSbe16147e6590: Upgrade Sentry to 23.3.1 (authored by dereckson).
Upgrade Sentry to 23.3.1
Mar 24 2023, 20:40
dereckson accepted D2907: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:40
dereckson added a revision to T1802: Upgrade Sentry to 23.3.1: D2907: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:39 · Monitoring and reporting
dereckson requested review of D2907: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:39
dereckson updated the task description for T1802: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:37 · Monitoring and reporting
dereckson claimed T1802: Upgrade Sentry to 23.3.1.
Mar 24 2023, 20:35 · Monitoring and reporting
dereckson renamed T1243: Deploy Varnish on webserver-legacy from Deploy Varnish on Ysul to Deploy Varnish on webserver-legacy.
Mar 24 2023, 01:17 · Servers
dereckson closed T720: Ysul: cannot stat /etc/login.conf: Not permitted in capability mode as Resolved.

Not present in recent FreeBSD machines, so I guess it was solved during an OS upgrade.

Mar 24 2023, 01:14 · security, Servers
dereckson lowered the priority of T448: More modern jails management on Ysul from High to Low.

We currently use jails only on devserver role for Poudriere, not anymore for hosting purpose.

Mar 24 2023, 01:11 · IPv6, Operations sprint 0, Servers
dereckson added a parent task for T761: Automate to create subdomains DNS records: T1217: Host our DNS servers.
Mar 24 2023, 01:09 · DNS, Servers
dereckson added a subtask for T1217: Host our DNS servers: T761: Automate to create subdomains DNS records.
Mar 24 2023, 01:09 · Servers
dereckson added a comment to T761: Automate to create subdomains DNS records.

I guess it would make sense to manage DNS records as a Git repository.

Mar 24 2023, 01:09 · DNS, Servers
dereckson renamed T761: Automate to create subdomains DNS records from Install a DNS server on ysul to Automate to create subdomains DNS records.
Mar 24 2023, 01:08 · DNS, Servers
dereckson added a comment to T1731: Ensure Notifications Center CI jobs run smoothly in PHP 8.1 .

phpmd and pdpend are still missing per T1002.

Mar 24 2023, 01:07 · Continous integration and delivery, Notifications center
dereckson merged T1002: Install pdepend, phpcpd, phpcs, phploc on Ysul into T1731: Ensure Notifications Center CI jobs run smoothly in PHP 8.1 .
Mar 24 2023, 01:06 · Continous integration and delivery, Notifications center
dereckson merged task T1002: Install pdepend, phpcpd, phpcs, phploc on Ysul into T1731: Ensure Notifications Center CI jobs run smoothly in PHP 8.1 .
Mar 24 2023, 01:06 · Servers
dereckson closed T1230: Low disk space on Ysul as Resolved.

Currently 79G free.

Mar 24 2023, 01:05 · Servers
dereckson closed T1031: Server outage: ysul.nasqueron.org doesn't reply to ping as Resolved.
Mar 24 2023, 01:05 · Servers
dereckson closed T1312: ZFS snapshots on devserver as Resolved by committing rOPS537a319f6313: Setup ZFS snapshots for home volumes.
Mar 24 2023, 00:58 · Servers, Operations sprints (The Dreadnought will produce new officers)