Page MenuHomeDevCentral

Allow Airflow to connect to Vault
ClosedPublic

Authored by dereckson on Jan 13 2024, 00:03.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Nov 22, 18:15
Unknown Object (File)
Fri, Nov 22, 16:54
Unknown Object (File)
Fri, Nov 22, 12:17
Unknown Object (File)
Fri, Nov 22, 04:00
Unknown Object (File)
Fri, Nov 22, 03:38
Unknown Object (File)
Fri, Nov 22, 02:41
Unknown Object (File)
Fri, Nov 22, 02:25
Unknown Object (File)
Fri, Nov 22, 01:31
Subscribers
None

Details

Summary

Airflow configuration is moved to airflow.cfg, that will be easier to read
and maintain than environment variables, especially for Vault back-end.
Also, that allows a faster deployment time as secrets need only to be read once.

The Vault application role matches the airflow policy as defined in D3270.

Ref T1812.

Test Plan
  • redeploy Airflow containers on Dwellers
  • airflow config get-value secrets backend
  • airflow connections get postgresql_fantoir

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Passed
Unit
No Test Coverage
Branch
airflow-vault
Build Status
Buildable 5162
Build 5443: arc lint + arc unit

Event Timeline

dereckson created this revision.
dereckson edited the test plan for this revision. (Show Details)

Move dictionary build to credentials module

Add salt to get secret id / role id

The tojson filter created this:

$ env | grep SECRETS__BACKEND
AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
AIRFLOW__SECRETS__BACKEND__KWARGS=OrderedDict([('url', 'https://172.27.27.7:8200'), ('auth_type', 'approle'), ('mount_point', 'apps'), ('connections_path', 'airflow/connections'), [...])

Restore credentials module, we don't need to inject secret_id and role_id to a dictionary anymore

Fix typo for airflow configuration path

Prune extraneous comma. Grrmbl JSON.

Correct database schemes for the PostgreSQLdatabase

Use airflow.cfg for service containers too

Tests passed. Works well, but fail silently if the password contains special characters as such we can't parse the DSN as a valid URL.

This revision is now accepted and ready to land.Jan 13 2024, 14:07

The AIRFLOW__DATABASE__SQL_ALCHEMY_CONN environment variable will be removed later, allowing faster container spawn time, but that's for D3273.

To answer that comment, container spawn time is now virtually instant, as it doesn't need to open a bus with Salt so Salt can then query Vault.

This revision was automatically updated to reflect the committed changes.