Page MenuHomeDevCentral
Feed All Stories

Jan 13 2024

dereckson updated the diff for D3273: Allow Airflow to connect to Vault.

Provision airflow.cfg

Jan 13 2024, 13:01
dereckson closed D3274: Document correct source file path for Airflow wrapper.
Jan 13 2024, 12:53
dereckson committed rOPSc2b7afbb55a9: Document correct source file path for Airflow wrapper (authored by dereckson).
Document correct source file path for Airflow wrapper
Jan 13 2024, 12:53
dereckson accepted D3274: Document correct source file path for Airflow wrapper.
Jan 13 2024, 12:53
dereckson requested review of D3274: Document correct source file path for Airflow wrapper.
Jan 13 2024, 12:52
dereckson planned changes to D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 00:49
dereckson added a comment to D3273: Allow Airflow to connect to Vault.

The tojson filter created this:

Jan 13 2024, 00:31
dereckson updated the diff for D3273: Allow Airflow to connect to Vault.

Add salt to get secret id / role id

Jan 13 2024, 00:21
dereckson updated the diff for D3273: Allow Airflow to connect to Vault.

Move dictionary build to credentials module

Jan 13 2024, 00:16
dereckson updated the summary of D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 00:03
dereckson added a revision to T1812: Deploy Airflow: D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 00:03 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 00:03

Jan 12 2024

dereckson closed D3271: Add Nasqueron Vault CA certificate to image.
Jan 12 2024, 23:23
dereckson committed rDAF07b7040c55c2: Add Nasqueron Vault CA certificate to image (authored by dereckson).
Add Nasqueron Vault CA certificate to image
Jan 12 2024, 23:23
dereckson accepted D3271: Add Nasqueron Vault CA certificate to image.
Dwellers
$ docker build -t nasqueron/airflow .
[...]
Jan 12 2024, 23:23
dereckson committed rRPRT8fc27bd6a3f9: Put MySQL queries in correct db-B cluster (authored by dereckson).
Put MySQL queries in correct db-B cluster
Jan 12 2024, 23:12
dereckson closed D3272: Put MySQL queries in correct db-B cluster.
Jan 12 2024, 23:12
dereckson accepted D3272: Put MySQL queries in correct db-B cluster.
Jan 12 2024, 23:12
dereckson requested review of D3272: Put MySQL queries in correct db-B cluster.
Jan 12 2024, 23:12
dereckson requested review of D3271: Add Nasqueron Vault CA certificate to image.
Jan 12 2024, 22:47
dereckson added a revision to T1812: Deploy Airflow: D3271: Add Nasqueron Vault CA certificate to image.
Jan 12 2024, 22:47 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson added a revision to T1812: Deploy Airflow: D3270: Create Vault policy for Airflow.
Jan 12 2024, 20:26 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3270: Create Vault policy for Airflow.
Jan 12 2024, 20:26
dereckson added a comment to T1812: Deploy Airflow.

DNS -> airflow.nasqueron.org. 172800 IN CNAME app2.nasqueron.org.

Jan 12 2024, 19:10 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson closed D3269: Use nasqueron/airflow image for Airflow command wrapper too.
Jan 12 2024, 19:02
dereckson committed rOPS76b53a39115b: Use nasqueron/airflow image for Airflow command wrapper too (authored by dereckson).
Use nasqueron/airflow image for Airflow command wrapper too
Jan 12 2024, 19:02
dereckson accepted D3269: Use nasqueron/airflow image for Airflow command wrapper too.
Dwellers
$ airflow nasqueron db upgrade
/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py:68 DeprecationWarning: `db upgrade` is deprecated. Use `db migrate` instead.
DB: postgresql+psycopg2://airflow:***@172.27.27.8/airflow
Performing upgrade to the metadata database postgresql+psycopg2://airflow:***@172.27.27.8/airflow
[2024-01-12T18:56:14.127+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2024-01-12T18:56:14.128+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2024-01-12T18:56:14.139+0000] {db.py:1615} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
INFO  [alembic.runtime.migration] Running upgrade c804e5c76e3e -> 937cbd173ca1, Add index to task_instance table
INFO  [alembic.runtime.migration] Running upgrade 937cbd173ca1 -> 788397e78828, Add custom_operator_name column
INFO  [alembic.runtime.migration] Running upgrade 788397e78828 -> 405de8318b3a, add include_deferred column to pool
INFO  [alembic.runtime.migration] Running upgrade 405de8318b3a -> 375a816bbbf4, add new field 'clear_number' to dagrun
INFO  [alembic.runtime.migration] Running upgrade 375a816bbbf4 -> f7bf2a57d0a6, Add owner_display_name to (Audit) Log table
INFO  [alembic.runtime.migration] Running upgrade f7bf2a57d0a6 -> bd5dfbe21f88, Make connection login/password TEXT
INFO  [alembic.runtime.migration] Running upgrade bd5dfbe21f88 -> 10b52ebd31f7, Add processor_subdir to ImportError.
Database migrating done!
Jan 12 2024, 19:01
dereckson requested review of D3269: Use nasqueron/airflow image for Airflow command wrapper too.
Jan 12 2024, 19:00
dereckson added a revision to T1812: Deploy Airflow: D3269: Use nasqueron/airflow image for Airflow command wrapper too.
Jan 12 2024, 19:00 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson added a comment to D3268: Upgrade to AirFlow 2.8.0 and Python 3.11.

One of the issue with the current procedure is the airflow command is still under 2.5.2:

Jan 12 2024, 18:55
dereckson added a comment to D3268: Upgrade to AirFlow 2.8.0 and Python 3.11.
Dwellers
$ deploy-containers-service airflow
[…]
----------
          ID: airflow_web
    Function: docker_container.running
      Result: True
     Comment: Replaced container 'airflow_web'. State changed from 'stopped' to 'running'. Container has a new image.
     Started: 18:45:58.009764
    Duration: 6838.744 ms
     Changes:
              ----------
              container_id:
                  ----------
                  added:
                      225f26be8d11efdfe1ca5f6fdaa0292b469411c3fd6f17448cea1ce6f2302748
                  removed:
                      - 7db02e9132e4034668ea6436f6abca659995bd7c3fb9d8e27e09ee2d1aeb7f76
              image:
                  ----------
                  new:
                      sha256:51f5e1b11188b4b10efd6a44528a13a559a937cc18cc9c6f0b1166c8dc8090b7
                  old:
                      sha256:12037a55b9dd7fb6d59308e92b619f8bff2a016ea04889b2bd464b8d8db0f3bd
              state:
                  ----------
                  new:
                      running
                  old:
                      stopped
[…]
Jan 12 2024, 18:47
dereckson closed D3268: Upgrade to AirFlow 2.8.0 and Python 3.11.
Jan 12 2024, 18:44
dereckson committed rDAFc6b6da7ceafb: Upgrade to AirFlow 2.8.0 and Python 3.11 (authored by dereckson).
Upgrade to AirFlow 2.8.0 and Python 3.11
Jan 12 2024, 18:44
dereckson accepted D3268: Upgrade to AirFlow 2.8.0 and Python 3.11.
Jan 12 2024, 18:43
dereckson requested review of D3268: Upgrade to AirFlow 2.8.0 and Python 3.11.
Jan 12 2024, 18:43
dereckson closed D3267: Configure Arcanist.
Jan 12 2024, 18:39
dereckson committed rDAF1bc4aaf65c99: Configure Arcanist (authored by dereckson).
Configure Arcanist
Jan 12 2024, 18:39
dereckson accepted D3267: Configure Arcanist.
Jan 12 2024, 18:38
dereckson requested review of D3267: Configure Arcanist.
Jan 12 2024, 18:38
dereckson closed D3266: Create custom Docker image with Sentry.
Jan 12 2024, 18:37
dereckson committed rDAF201c7874ddae: Create custom Docker image with Sentry (authored by dereckson).
Create custom Docker image with Sentry
Jan 12 2024, 18:37
dereckson accepted D3266: Create custom Docker image with Sentry.

Not yet committed code from last March, so that explains the version lag. 2.5.2 is still what's currently deployed.

Jan 12 2024, 18:37
dereckson requested review of D3266: Create custom Docker image with Sentry.
Jan 12 2024, 18:36
dereckson added a revision to T1812: Deploy Airflow: D3266: Create custom Docker image with Sentry.
Jan 12 2024, 18:36 · Product evaluation, Continous integration and delivery, Nasqueron Databases

Jan 10 2024

dereckson updated the task description for T1938: Non open-source infrastructure software policy.
Jan 10 2024, 22:30 · discussion, Servers
dereckson added a comment to T1938: Non open-source infrastructure software policy.

Added RHEL question.

Jan 10 2024, 22:23 · discussion, Servers
dereckson updated the task description for T1938: Non open-source infrastructure software policy.
Jan 10 2024, 22:23 · discussion, Servers
dereckson lowered the priority of T1939: Implement blue/green deployment or immutable artefacts for router-001 from Normal to Low.
Jan 10 2024, 22:11 · Servers, Drake network
dereckson added a comment to T1939: Implement blue/green deployment or immutable artefacts for router-001.

router-001 update would cut network connections between Ysul, WindRiver, CloudHugger and IntraNought VMs.

I think that's actually acceptable, as production services can reach web-001/db-*/ directly through their dedicated network card.

Impact:

  • IRC bots wouldn't be able to reach MySQL or Vault as they're still on Ysul
  • Services still on Ysul can't reach
  • Development servers can't reach production services, or Dwellers

Another issue is we lose connections to all those machines, as router-001 is used to route traffic to them.

Probably best to ship router-002 under FreeBSD 14 so we can prepare to minimize this impact if we need more hypervisors in the future.

Jan 10 2024, 22:10 · Servers, Drake network
dereckson updated the task description for T1939: Implement blue/green deployment or immutable artefacts for router-001.
Jan 10 2024, 22:10 · Servers, Drake network
dereckson closed T1924: Upgrade servers to FreeBSD 14 as Resolved.

router-001 is out of scope as long as T1939 is implemented

Jan 10 2024, 22:09 · Servers
dereckson triaged T1939: Implement blue/green deployment or immutable artefacts for router-001 as Normal priority.
Jan 10 2024, 22:08 · Servers, Drake network
dereckson added a comment to D3264: Bump FreeBSD version in MOTD.
Complector
$ salt -G 'os:FreeBSD' state.apply roles/core/motd
[…]
db-A-001:
----------
          ID: motd
    Function: file.managed
        Name: /etc/motd.template
      Result: True
     Comment: File /etc/motd.template updated
     Started: 21:59:23.854873
    Duration: 574.128 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,7 +1,7 @@
                       ____  ______  ___
                      /    )/      \/   \    db-A-001.nasqueron.drake
                     (     / __    _\    )
                  -   \    (/ o)  ( o)   )   IP: 172.27.27.8        OS: FreeBSD 13
                  +   \    (/ o)  ( o)   )   IP: 172.27.27.8        OS: FreeBSD 14
                       \_  (_  )   \ )  /    GW: 172.27.27.1        Cluster A | PostgreSQL
                         \  /\_/    \)_/
                          \/  //|  |\\       This server hosts databases for Nasqueron projects.
[…]
Jan 10 2024, 22:01
dereckson closed D3264: Bump FreeBSD version in MOTD.
Jan 10 2024, 21:57
dereckson committed rOPS573a4ba11ba8: Bump FreeBSD version in MOTD (authored by dereckson).
Bump FreeBSD version in MOTD
Jan 10 2024, 21:57
dereckson committed rOPS422ae85e9f5a: Use Etherpad /health endpoint (authored by dereckson).
Use Etherpad /health endpoint
Jan 10 2024, 21:56
dereckson closed D3263: Use Etherpad /health endpoint.
Jan 10 2024, 21:56
dereckson closed D3257: Add non-free-firmware repository component on Debian.
Jan 10 2024, 21:55
dereckson committed rOPS1304de63d3af: Add non-free-firmware repository component on Debian (authored by dereckson).
Add non-free-firmware repository component on Debian
Jan 10 2024, 21:55
dereckson closed D3256: Update copyright date.
Jan 10 2024, 21:54 · Docker images
dereckson committed rDOCKERWWWafe29e42862c: Update copyright date (authored by dereckson).
Update copyright date
Jan 10 2024, 21:54

Jan 8 2024

dereckson closed D3265: Disable Terrapin sensible ciphers and algorithms.
Jan 8 2024, 21:54
dereckson committed rOPS72f249f5686b: Disable Terrapin sensible ciphers and algorithms (authored by dereckson).
Disable Terrapin sensible ciphers and algorithms
Jan 8 2024, 21:54
DorianWinty accepted D3265: Disable Terrapin sensible ciphers and algorithms.
Jan 8 2024, 21:54
dereckson added a revision to T1935: OPENSSH 9.6: D3265: Disable Terrapin sensible ciphers and algorithms.
Jan 8 2024, 21:54 · security
dereckson requested review of D3265: Disable Terrapin sensible ciphers and algorithms.
Jan 8 2024, 21:54
DorianWinty closed T1935: OPENSSH 9.6 as Resolved.
Jan 8 2024, 21:13 · security
DorianWinty shifted T1935: OPENSSH 9.6 from the Restricted Space space to the S1 Nasqueron space.
Jan 8 2024, 21:13 · security
DorianWinty shifted T1935: OPENSSH 9.6 from the S1 Nasqueron space to the Restricted Space space.
Jan 8 2024, 21:11 · security
DorianWinty shifted T1935: OPENSSH 9.6 from the Restricted Space space to the S1 Nasqueron space.
Jan 8 2024, 21:11 · security
DorianWinty accepted D3256: Update copyright date.
Jan 8 2024, 20:34 · Docker images
dereckson added a comment to D3264: Bump FreeBSD version in MOTD.

Can be automated with grain osmajorrelease.

Jan 8 2024, 18:58
DorianWinty accepted D3257: Add non-free-firmware repository component on Debian.
Jan 8 2024, 18:44
DorianWinty accepted D3263: Use Etherpad /health endpoint.
Jan 8 2024, 18:43
DorianWinty accepted D3264: Bump FreeBSD version in MOTD.

seem good for me,
Is there a way to make this change automatic ?
based on the version of the os in the server ?

Jan 8 2024, 18:41

Jan 7 2024

dereckson updated the task description for T1935: OPENSSH 9.6.
Jan 7 2024, 18:05 · security
dereckson updated the task description for T1935: OPENSSH 9.6.
Jan 7 2024, 18:01 · security
dereckson added a comment to T1924: Upgrade servers to FreeBSD 14.

router-001 update would cut network connections between Ysul, WindRiver, CloudHugger and IntraNought VMs.

Jan 7 2024, 18:01 · Servers
dereckson updated the task description for T1924: Upgrade servers to FreeBSD 14.
Jan 7 2024, 17:46 · Servers
dereckson requested review of D3264: Bump FreeBSD version in MOTD.
Jan 7 2024, 16:20
dereckson added a revision to T1924: Upgrade servers to FreeBSD 14: D3264: Bump FreeBSD version in MOTD.
Jan 7 2024, 16:20 · Servers
dereckson requested review of D3263: Use Etherpad /health endpoint.
Jan 7 2024, 00:40
dereckson committed rDETHERd09fdd64691c: Merge branch 'main' into production (authored by dereckson).
Merge branch 'main' into production
Jan 7 2024, 00:31
dereckson committed rDETHER694b1fde33d7: Fix whitespace issue (authored by dereckson).
Fix whitespace issue
Jan 7 2024, 00:29
dereckson closed D3262: Fix whitespace issue.
Jan 7 2024, 00:29
dereckson accepted D3262: Fix whitespace issue.
Jan 7 2024, 00:28
dereckson requested review of D3262: Fix whitespace issue.
Jan 7 2024, 00:28
dereckson committed rDETHER04fea0750f59: Refresh installation method (authored by dereckson).
Refresh installation method
Jan 7 2024, 00:21
dereckson closed D3261: Refresh installation method.
Jan 7 2024, 00:21
dereckson closed D3260: Upgrade to Debian Bookworm and Node 18.
Jan 7 2024, 00:21
dereckson committed rDETHER9c5d27f38b23: Upgrade to Debian Bookworm and Node 18 (authored by dereckson).
Upgrade to Debian Bookworm and Node 18
Jan 7 2024, 00:21
dereckson requested review of D3261: Refresh installation method.
Jan 7 2024, 00:21
dereckson requested review of D3260: Upgrade to Debian Bookworm and Node 18.
Jan 7 2024, 00:21
dereckson accepted D3260: Upgrade to Debian Bookworm and Node 18.
Jan 7 2024, 00:21
dereckson accepted D3261: Refresh installation method.
Jan 7 2024, 00:21
dereckson planned changes to D3260: Upgrade to Debian Bookworm and Node 18.

Other changes seem needed:

Jan 7 2024, 00:21
dereckson updated the task description for T1935: OPENSSH 9.6.
Jan 7 2024, 00:21 · security
dereckson added a comment to D3260: Upgrade to Debian Bookworm and Node 18.

The Etherpad projects offers a procedure to install Etherpad with npm link. That works well if we combine D3260 and D3261.

Jan 7 2024, 00:21
dereckson requested review of D3260: Upgrade to Debian Bookworm and Node 18.
Jan 7 2024, 00:21

Jan 6 2024

dereckson created P346 python-packages-netbox.txt.
Jan 6 2024, 18:51