In D3275#49866, @dereckson wrote:The AIRFLOW__DATABASE__SQL_ALCHEMY_CONN environment variable will be removed later, allowing faster container spawn time, but that's for D3273.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed All Stories
All Stories
All Stories
Jan 13 2024
Jan 13 2024
Create Vault policy for Airflow
Policy validated during D3273 tests: read is enough.
Tests passed. Works well, but fail silently if the password contains special characters as such we can't parse the DSN as a valid URL.
Use airflow.cfg for service containers too
Correct database schemes for the PostgreSQLdatabase
dereckson committed rOPS075d7c8cb787: Allow to enter a shell for Airflow maintenance operations (authored by dereckson).
Allow to enter a shell for Airflow maintenance operations
When deployed to Dwellers, code is no-op regarding the code tested there, so looks good to me.
dereckson added a revision to T1812: Deploy Airflow: D3275: Allow to enter a shell for Airflow maintenance operations.
Prune extraneous comma. Grrmbl JSON.
uid 50000, JSON syntax
Fix typo for airflow configuration path
Restore credentials module, we don't need to inject secret_id and role_id to a dictionary anymore
Provision airflow.cfg
dereckson committed rOPSc2b7afbb55a9: Document correct source file path for Airflow wrapper (authored by dereckson).
Document correct source file path for Airflow wrapper
The tojson filter created this:
Add salt to get secret id / role id
Move dictionary build to credentials module
Jan 12 2024
Jan 12 2024
dereckson committed rDAF07b7040c55c2: Add Nasqueron Vault CA certificate to image (authored by dereckson).
Add Nasqueron Vault CA certificate to image
Dwellers
$ docker build -t nasqueron/airflow . [...]
dereckson committed rRPRT8fc27bd6a3f9: Put MySQL queries in correct db-B cluster (authored by dereckson).
Put MySQL queries in correct db-B cluster
dereckson added a revision to T1812: Deploy Airflow: D3271: Add Nasqueron Vault CA certificate to image.
DNS -> airflow.nasqueron.org. 172800 IN CNAME app2.nasqueron.org.
dereckson committed rOPS76b53a39115b: Use nasqueron/airflow image for Airflow command wrapper too (authored by dereckson).
Use nasqueron/airflow image for Airflow command wrapper too
Dwellers
$ airflow nasqueron db upgrade /home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py:68 DeprecationWarning: `db upgrade` is deprecated. Use `db migrate` instead. DB: postgresql+psycopg2://airflow:***@172.27.27.8/airflow Performing upgrade to the metadata database postgresql+psycopg2://airflow:***@172.27.27.8/airflow [2024-01-12T18:56:14.127+0000] {migration.py:213} INFO - Context impl PostgresqlImpl. [2024-01-12T18:56:14.128+0000] {migration.py:216} INFO - Will assume transactional DDL. [2024-01-12T18:56:14.139+0000] {db.py:1615} INFO - Creating tables INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table INFO [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables INFO [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table INFO [alembic.runtime.migration] Running upgrade c804e5c76e3e -> 937cbd173ca1, Add index to task_instance table INFO [alembic.runtime.migration] Running upgrade 937cbd173ca1 -> 788397e78828, Add custom_operator_name column INFO [alembic.runtime.migration] Running upgrade 788397e78828 -> 405de8318b3a, add include_deferred column to pool INFO [alembic.runtime.migration] Running upgrade 405de8318b3a -> 375a816bbbf4, add new field 'clear_number' to dagrun INFO [alembic.runtime.migration] Running upgrade 375a816bbbf4 -> f7bf2a57d0a6, Add owner_display_name to (Audit) Log table INFO [alembic.runtime.migration] Running upgrade f7bf2a57d0a6 -> bd5dfbe21f88, Make connection login/password TEXT INFO [alembic.runtime.migration] Running upgrade bd5dfbe21f88 -> 10b52ebd31f7, Add processor_subdir to ImportError. Database migrating done!
dereckson added a revision to T1812: Deploy Airflow: D3269: Use nasqueron/airflow image for Airflow command wrapper too.
One of the issue with the current procedure is the airflow command is still under 2.5.2:
Dwellers
$ deploy-containers-service airflow […] ---------- ID: airflow_web Function: docker_container.running Result: True Comment: Replaced container 'airflow_web'. State changed from 'stopped' to 'running'. Container has a new image. Started: 18:45:58.009764 Duration: 6838.744 ms Changes: ---------- container_id: ---------- added: 225f26be8d11efdfe1ca5f6fdaa0292b469411c3fd6f17448cea1ce6f2302748 removed: - 7db02e9132e4034668ea6436f6abca659995bd7c3fb9d8e27e09ee2d1aeb7f76 image: ---------- new: sha256:51f5e1b11188b4b10efd6a44528a13a559a937cc18cc9c6f0b1166c8dc8090b7 old: sha256:12037a55b9dd7fb6d59308e92b619f8bff2a016ea04889b2bd464b8d8db0f3bd state: ---------- new: running old: stopped […]
dereckson committed rDAFc6b6da7ceafb: Upgrade to AirFlow 2.8.0 and Python 3.11 (authored by dereckson).
Upgrade to AirFlow 2.8.0 and Python 3.11
Configure Arcanist
dereckson committed rDAF201c7874ddae: Create custom Docker image with Sentry (authored by dereckson).
Create custom Docker image with Sentry
Not yet committed code from last March, so that explains the version lag. 2.5.2 is still what's currently deployed.
Jan 10 2024
Jan 10 2024
Added RHEL question.
dereckson lowered the priority of T1939: Implement blue/green deployment or immutable artefacts for router-001 from Normal to Low.
dereckson added a comment to T1939: Implement blue/green deployment or immutable artefacts for router-001.
In T1924#28790, @dereckson wrote:router-001 update would cut network connections between Ysul, WindRiver, CloudHugger and IntraNought VMs.
I think that's actually acceptable, as production services can reach web-001/db-*/ directly through their dedicated network card.
Impact:
- IRC bots wouldn't be able to reach MySQL or Vault as they're still on Ysul
- Services still on Ysul can't reach
- Development servers can't reach production services, or Dwellers
Another issue is we lose connections to all those machines, as router-001 is used to route traffic to them.
Probably best to ship router-002 under FreeBSD 14 so we can prepare to minimize this impact if we need more hypervisors in the future.
dereckson updated the task description for T1939: Implement blue/green deployment or immutable artefacts for router-001.
router-001 is out of scope as long as T1939 is implemented
dereckson triaged T1939: Implement blue/green deployment or immutable artefacts for router-001 as Normal priority.
Complector
$ salt -G 'os:FreeBSD' state.apply roles/core/motd […] db-A-001: ---------- ID: motd Function: file.managed Name: /etc/motd.template Result: True Comment: File /etc/motd.template updated Started: 21:59:23.854873 Duration: 574.128 ms Changes: ---------- diff: --- +++ @@ -1,7 +1,7 @@ ____ ______ ___ / )/ \/ \ db-A-001.nasqueron.drake ( / __ _\ ) - \ (/ o) ( o) ) IP: 172.27.27.8 OS: FreeBSD 13 + \ (/ o) ( o) ) IP: 172.27.27.8 OS: FreeBSD 14 \_ (_ ) \ ) / GW: 172.27.27.1 Cluster A | PostgreSQL \ /\_/ \)_/ \/ //| |\\ This server hosts databases for Nasqueron projects. […]
Bump FreeBSD version in MOTD
Use Etherpad /health endpoint
dereckson committed rOPS1304de63d3af: Add non-free-firmware repository component on Debian (authored by dereckson).
Add non-free-firmware repository component on Debian
Update copyright date
Jan 8 2024
Jan 8 2024
dereckson committed rOPS72f249f5686b: Disable Terrapin sensible ciphers and algorithms (authored by dereckson).
Disable Terrapin sensible ciphers and algorithms
dereckson added a revision to T1935: OPENSSH 9.6: D3265: Disable Terrapin sensible ciphers and algorithms.
Can be automated with grain osmajorrelease.
seem good for me,
Is there a way to make this change automatic ?
based on the version of the os in the server ?
Jan 7 2024
Jan 7 2024
router-001 update would cut network connections between Ysul, WindRiver, CloudHugger and IntraNought VMs.
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator