Query: All Stories

	Include stories about projects I am a member of.

Fix spaces. Add kludge to remove contextvars from egg info about Salt.

Two lines after import

2.8.0 / add header / Notification singular / database_url

Rebased against D3277.

WindRiver

$ cd /tmp/fantoir
$ rm FANTOIR0423
$ fantoir-datasource fetch
FANTOIR_FILE=FANTOIR0423
FANTOIR_TABLE=fantoir_202304

Use fantoir_YYYMM format to suggest FANTOIR table name

WindRiver

$ cd /tmp/fantoir
$ rm FANTOIR0423
$ fantoir-datasource fetch
FANTOIR_FILE=FANTOIR0423
FANTOIR_TABLE=fantoir_202304

Use fantoir_YYYMM format to suggest FANTOIR table name

The crc32fast is a dependency of reqwest, it can be built in version 1.3.2 for reqwest 0.11.17.

Create workspace directory for Airflow pipelines

Dwellers

$ airflow nasqueron shell
$ airflow variables get fantoir_directory
/home/airflow/workspace/fantoir
$ ls ~/workspace
fantoir

Previous status from 2023 installation:

Allow Airflow to connect to Vault

In D3275#49866, @dereckson wrote:

The AIRFLOW__DATABASE__SQL_ALCHEMY_CONN environment variable will be removed later, allowing faster container spawn time, but that's for D3273.

Create Vault policy for Airflow

Policy validated during D3273 tests: read is enough.

Tests passed. Works well, but fail silently if the password contains special characters as such we can't parse the DSN as a valid URL.

Use airflow.cfg for service containers too

Correct database schemes for the PostgreSQLdatabase

Allow to enter a shell for Airflow maintenance operations

When deployed to Dwellers, code is no-op regarding the code tested there, so looks good to me.

Prune extraneous comma. Grrmbl JSON.

uid 50000, JSON syntax

Fix typo for airflow configuration path

Restore credentials module, we don't need to inject secret_id and role_id to a dictionary anymore

Provision airflow.cfg

Document correct source file path for Airflow wrapper

The tojson filter created this:

Add salt to get secret id / role id

Move dictionary build to credentials module

Add Nasqueron Vault CA certificate to image

Dwellers

$ docker build -t nasqueron/airflow .
[...]

Put MySQL queries in correct db-B cluster

DNS -> airflow.nasqueron.org. 172800 IN CNAME app2.nasqueron.org.

Use nasqueron/airflow image for Airflow command wrapper too

Dwellers

$ airflow nasqueron db upgrade
/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py:68 DeprecationWarning: `db upgrade` is deprecated. Use `db migrate` instead.
DB: postgresql+psycopg2://airflow:***@172.27.27.8/airflow
Performing upgrade to the metadata database postgresql+psycopg2://airflow:***@172.27.27.8/airflow
[2024-01-12T18:56:14.127+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2024-01-12T18:56:14.128+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2024-01-12T18:56:14.139+0000] {db.py:1615} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
INFO  [alembic.runtime.migration] Running upgrade c804e5c76e3e -> 937cbd173ca1, Add index to task_instance table
INFO  [alembic.runtime.migration] Running upgrade 937cbd173ca1 -> 788397e78828, Add custom_operator_name column
INFO  [alembic.runtime.migration] Running upgrade 788397e78828 -> 405de8318b3a, add include_deferred column to pool
INFO  [alembic.runtime.migration] Running upgrade 405de8318b3a -> 375a816bbbf4, add new field 'clear_number' to dagrun
INFO  [alembic.runtime.migration] Running upgrade 375a816bbbf4 -> f7bf2a57d0a6, Add owner_display_name to (Audit) Log table
INFO  [alembic.runtime.migration] Running upgrade f7bf2a57d0a6 -> bd5dfbe21f88, Make connection login/password TEXT
INFO  [alembic.runtime.migration] Running upgrade bd5dfbe21f88 -> 10b52ebd31f7, Add processor_subdir to ImportError.
Database migrating done!

One of the issue with the current procedure is the airflow command is still under 2.5.2:

Dwellers

$ deploy-containers-service airflow
[…]
----------
          ID: airflow_web
    Function: docker_container.running
      Result: True
     Comment: Replaced container 'airflow_web'. State changed from 'stopped' to 'running'. Container has a new image.
     Started: 18:45:58.009764
    Duration: 6838.744 ms
     Changes:
              ----------
              container_id:
                  ----------
                  added:
                      225f26be8d11efdfe1ca5f6fdaa0292b469411c3fd6f17448cea1ce6f2302748
                  removed:
                      - 7db02e9132e4034668ea6436f6abca659995bd7c3fb9d8e27e09ee2d1aeb7f76
              image:
                  ----------
                  new:
                      sha256:51f5e1b11188b4b10efd6a44528a13a559a937cc18cc9c6f0b1166c8dc8090b7
                  old:
                      sha256:12037a55b9dd7fb6d59308e92b619f8bff2a016ea04889b2bd464b8d8db0f3bd
              state:
                  ----------
                  new:
                      running
                  old:
                      stopped
[…]

Upgrade to AirFlow 2.8.0 and Python 3.11

Create custom Docker image with Sentry

All Stories
Use Results
Edit Query
Hide Query

Jan 14 2024

Jan 13 2024

Jan 12 2024

All StoriesUse ResultsEdit QueryHide Query

Jan 14 2024

Jan 13 2024

Jan 12 2024

All Stories
Use Results
Edit Query
Hide Query