Page MenuHomeDevCentral
Feed All Stories

Jan 14 2024

dereckson requested review of D3285: Check FANTOIR filename using FANTOIRMMYY format.
Jan 14 2024, 21:57
dereckson closed D3284: Mount Airflow volumes in maintenance container too.
Jan 14 2024, 21:25
dereckson committed rOPSb4dc5820ee6b: Mount Airflow volumes in maintenance container too (authored by dereckson).
Mount Airflow volumes in maintenance container too
Jan 14 2024, 21:25
dereckson accepted D3284: Mount Airflow volumes in maintenance container too.
Jan 14 2024, 21:25
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.
  • Notification gate is singular
  • Get FANTOIR connection from secret back-end
  • Inject PATH in environment so we can find fantoir-datasource in /usr/local/bin
  • Decode bytes and split lines into a list for stdout, stderr
Jan 14 2024, 21:24
dereckson added a revision to T1812: Deploy Airflow: D3284: Mount Airflow volumes in maintenance container too.
Jan 14 2024, 20:50 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson updated the summary of D3284: Mount Airflow volumes in maintenance container too.
Jan 14 2024, 20:50
dereckson requested review of D3284: Mount Airflow volumes in maintenance container too.
Jan 14 2024, 20:50
dereckson committed rDAF0ad3018bf03a: Inject fantoir-datasource in Airflow image (authored by dereckson).
Inject fantoir-datasource in Airflow image
Jan 14 2024, 19:12
dereckson closed D3283: Inject fantoir-datasource in Airflow image.
Jan 14 2024, 19:12
dereckson accepted D3283: Inject fantoir-datasource in Airflow image.

Both fantoir-datasource fetch and fantoir-datasource import work correctly in airflow nasqueron shell: rows are correctly imported into PostgreSQL database.

Jan 14 2024, 19:12
dereckson added a revision to T1812: Deploy Airflow: D3283: Inject fantoir-datasource in Airflow image.
Jan 14 2024, 19:10 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3283: Inject fantoir-datasource in Airflow image.
Jan 14 2024, 19:10
dereckson closed D3282: Provide build instructions and Docker image.
Jan 14 2024, 18:53
dereckson committed rDS77691169e3a7: Provide build instructions and Docker image (authored by dereckson).
Provide build instructions and Docker image
Jan 14 2024, 18:53
dereckson updated the diff for D3282: Provide build instructions and Docker image.

Improve Makefile

Jan 14 2024, 18:52
dereckson accepted D3282: Provide build instructions and Docker image.
Jan 14 2024, 18:49
dereckson requested review of D3282: Provide build instructions and Docker image.
Jan 14 2024, 18:48
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

Create table if it doesn't exist

Jan 14 2024, 18:41
dereckson committed rOPSde2fb22e15a3: Provision Airflow DAGs folders (authored by dereckson).
Provision Airflow DAGs folders
Jan 14 2024, 17:06
dereckson closed D3281: Provision Airflow DAGs folders.
Jan 14 2024, 17:06
dereckson accepted D3281: Provision Airflow DAGs folders.
Jan 14 2024, 17:05
dereckson added a comment to D3281: Provision Airflow DAGs folders.

Works fine. https://airflow.nasqueron.org/dags/fantoir_fetch/grid for example

Jan 14 2024, 17:05
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

TaskFlow DAGs are defined by a function, but that function needs to be called.

Jan 14 2024, 17:04
dereckson added a comment to D2754: Run fantoir-datasource as Airflow pipeline.

At the end of the TaskFlow DAG, function should be called, for example at the end of fantoir_fetch.py we need to add fantoir_fetch_dag().

Jan 14 2024, 11:59
dereckson updated the diff for D3280: Configure Salt reactor to provision Airflow pipeline for datasources.

s/dag/dags/g to be coherent with D3281

Jan 14 2024, 11:49
dereckson updated the diff for D3281: Provision Airflow DAGs folders.

Fix state through Complector test deployment

Jan 14 2024, 11:47
dereckson updated the diff for D3281: Provision Airflow DAGs folders.

Hunt insecable spaces

Jan 14 2024, 11:38
dereckson updated the diff for D3281: Provision Airflow DAGs folders.

Make directory src/

Jan 14 2024, 11:35
dereckson added a revision to T1812: Deploy Airflow: D3281: Provision Airflow DAGs folders.
Jan 14 2024, 11:34 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3281: Provision Airflow DAGs folders.
Jan 14 2024, 11:34
dereckson added a revision to T1750: Import FANTOIR database: D3280: Configure Salt reactor to provision Airflow pipeline for datasources.
Jan 14 2024, 11:06 · Nasqueron Databases
dereckson added a revision to T1942: Allow Jenkins to trigger deployment through Salt: D3280: Configure Salt reactor to provision Airflow pipeline for datasources.
Jan 14 2024, 11:06 · Continous integration and delivery, Salt
dereckson added a revision to T1812: Deploy Airflow: D3280: Configure Salt reactor to provision Airflow pipeline for datasources.
Jan 14 2024, 11:06 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3280: Configure Salt reactor to provision Airflow pipeline for datasources.
Jan 14 2024, 11:06
dereckson closed D3279: Configure reactor on Salt primary server.
Jan 14 2024, 11:05
dereckson committed rOPS9c0b161cdbc4: Configure reactor on Salt primary server (authored by dereckson).
Configure reactor on Salt primary server
Jan 14 2024, 11:05
dereckson added a comment to D3279: Configure reactor on Salt primary server.

Tested successfully on Complector, behaves like as defined in the test plan.

Jan 14 2024, 11:04
dereckson accepted D3279: Configure reactor on Salt primary server.
Jan 14 2024, 11:04
dereckson requested review of D3279: Configure reactor on Salt primary server.
Jan 14 2024, 11:04
dereckson added a revision to T1942: Allow Jenkins to trigger deployment through Salt: D3279: Configure reactor on Salt primary server.
Jan 14 2024, 11:04 · Continous integration and delivery, Salt
dereckson closed D3251: Provide TLS 1.3 only nginx configuration.
Jan 14 2024, 10:31
dereckson committed rOPSe691945ff036: Provide TLS 1.3 only nginx configuration (authored by dereckson).
Provide TLS 1.3 only nginx configuration
Jan 14 2024, 10:31
dereckson closed D3250: Update cipher suites.
Jan 14 2024, 10:30
dereckson committed rOPS61ced979c184: Update cipher suites (authored by dereckson).
Update cipher suites
Jan 14 2024, 10:30
dereckson accepted D3250: Update cipher suites.
Jan 14 2024, 10:30
dereckson accepted D3251: Provide TLS 1.3 only nginx configuration.
Jan 14 2024, 10:28
dereckson updated the task description for T1942: Allow Jenkins to trigger deployment through Salt.
Jan 14 2024, 01:54 · Continous integration and delivery, Salt
dereckson closed D3278: Deploy REST API for Salt using rest_cherrypy.
Jan 14 2024, 01:53
dereckson committed rOPS6c4328e270b9: Deploy REST API for Salt using rest_cherrypy (authored by dereckson).
Deploy REST API for Salt using rest_cherrypy
Jan 14 2024, 01:53
dereckson accepted D3278: Deploy REST API for Salt using rest_cherrypy.
Jan 14 2024, 01:52
dereckson updated the diff for D3278: Deploy REST API for Salt using rest_cherrypy.

Reorder pillar/services/table.sls

Jan 14 2024, 01:51
dereckson updated the diff for D3278: Deploy REST API for Salt using rest_cherrypy.

Declare Salt API in services table so Jenkins can know what URL to use without hardcoding it

Jan 14 2024, 01:46
dereckson updated the diff for D3278: Deploy REST API for Salt using rest_cherrypy.

+service

Jan 14 2024, 01:44
dereckson updated the diff for D3278: Deploy REST API for Salt using rest_cherrypy.

hunt-insecable-spaces sponsored by I can't use a MacBook Pro (tm)

Jan 14 2024, 01:30
dereckson updated the diff for D3278: Deploy REST API for Salt using rest_cherrypy.

Fix spaces. Add kludge to remove contextvars from egg info about Salt.

Jan 14 2024, 01:29

Jan 13 2024

dereckson added a comment to D3278: Deploy REST API for Salt using rest_cherrypy.

Current status

Jan 13 2024, 23:49
dereckson requested review of D3278: Deploy REST API for Salt using rest_cherrypy.
Jan 13 2024, 23:47
dereckson added a revision to T1942: Allow Jenkins to trigger deployment through Salt: D3278: Deploy REST API for Salt using rest_cherrypy.
Jan 13 2024, 23:47 · Continous integration and delivery, Salt
dereckson triaged T1942: Allow Jenkins to trigger deployment through Salt as Normal priority.
Jan 13 2024, 22:22 · Continous integration and delivery, Salt
dereckson added a comment to D2754: Run fantoir-datasource as Airflow pipeline.

Ready to test.

Jan 13 2024, 20:20
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

Two lines after import

Jan 13 2024, 20:20
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

2.8.0 / add header / Notification singular / database_url

Jan 13 2024, 20:17
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

Rebased against D3277.

Jan 13 2024, 20:09
dereckson added a comment to T1941: Zerofill FANTOIR_TABLE information.
WindRiver
$ cd /tmp/fantoir
$ rm FANTOIR0423
$ fantoir-datasource fetch
FANTOIR_FILE=FANTOIR0423
FANTOIR_TABLE=fantoir_202304
Jan 13 2024, 20:09 · Nasqueron Databases
dereckson closed T1941: Zerofill FANTOIR_TABLE information as Resolved by committing rDSfc13adda27ce: Use fantoir_YYYMM format to suggest FANTOIR table name.
Jan 13 2024, 20:08 · Nasqueron Databases
dereckson committed rDSfc13adda27ce: Use fantoir_YYYMM format to suggest FANTOIR table name (authored by dereckson).
Use fantoir_YYYMM format to suggest FANTOIR table name
Jan 13 2024, 20:08
dereckson closed D3277: Use fantoir_YYYMM format to suggest FANTOIR table name.
Jan 13 2024, 20:08
dereckson added a comment to D3277: Use fantoir_YYYMM format to suggest FANTOIR table name.
WindRiver
$ cd /tmp/fantoir
$ rm FANTOIR0423
$ fantoir-datasource fetch
FANTOIR_FILE=FANTOIR0423
FANTOIR_TABLE=fantoir_202304
Jan 13 2024, 20:07
dereckson accepted D3277: Use fantoir_YYYMM format to suggest FANTOIR table name.
Jan 13 2024, 20:05
dereckson requested review of D3277: Use fantoir_YYYMM format to suggest FANTOIR table name.
Jan 13 2024, 20:05
dereckson added a revision to T1941: Zerofill FANTOIR_TABLE information: D3277: Use fantoir_YYYMM format to suggest FANTOIR table name.
Jan 13 2024, 20:05 · Nasqueron Databases
dereckson updated the diff for D2754: Run fantoir-datasource as Airflow pipeline.

Use fantoir_YYYMM format to suggest FANTOIR table name

Jan 13 2024, 20:03
dereckson claimed T1941: Zerofill FANTOIR_TABLE information.
Jan 13 2024, 19:49 · Nasqueron Databases
dereckson moved T1941: Zerofill FANTOIR_TABLE information from Backlog to Working on on the Nasqueron Databases board.
Jan 13 2024, 19:49 · Nasqueron Databases
dereckson triaged T1941: Zerofill FANTOIR_TABLE information as Normal priority.
Jan 13 2024, 19:49 · Nasqueron Databases
dereckson closed T1940: Can't compile crc32fast on FreeBSD with current datasources dependencies as Resolved.

The crc32fast is a dependency of reqwest, it can be built in version 1.3.2 for reqwest 0.11.17.

Jan 13 2024, 19:25 · Regression, Nasqueron Databases
dereckson triaged T1940: Can't compile crc32fast on FreeBSD with current datasources dependencies as High priority.
Jan 13 2024, 19:19 · Regression, Nasqueron Databases
dereckson closed D3276: Create workspace directory for Airflow pipelines.
Jan 13 2024, 18:46
dereckson committed rOPSe584e2dcf7d0: Create workspace directory for Airflow pipelines (authored by dereckson).
Create workspace directory for Airflow pipelines
Jan 13 2024, 18:46
dereckson accepted D3276: Create workspace directory for Airflow pipelines.
Dwellers
$ airflow nasqueron shell
$ airflow variables get fantoir_directory
/home/airflow/workspace/fantoir
$ ls ~/workspace
fantoir
Jan 13 2024, 18:36
dereckson added inline comments to D2754: Run fantoir-datasource as Airflow pipeline.
Jan 13 2024, 18:27
dereckson updated the diff for D3276: Create workspace directory for Airflow pipelines.

+wrapper

Jan 13 2024, 18:21
dereckson added a revision to T1812: Deploy Airflow: D3276: Create workspace directory for Airflow pipelines.
Jan 13 2024, 18:21 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson requested review of D3276: Create workspace directory for Airflow pipelines.
Jan 13 2024, 18:21
dereckson moved T1751: Enrich FANTOIR database with Wikidata information from Working on to Backlog on the Nasqueron Databases board.
Jan 13 2024, 18:16 · Wikimedia, Nasqueron Databases
dereckson moved T1875: Refresh RFC entries from Working on to Backlog on the Nasqueron Databases board.
Jan 13 2024, 18:15 · Nasqueron Databases, Odderon
dereckson moved T1131: Populate language codes from Working on to Backlog on the Nasqueron Databases board.
Jan 13 2024, 18:15 · Nasqueron Databases, Odderon
dereckson moved T1812: Deploy Airflow from Backlog to Working on on the Nasqueron Databases board.
Jan 13 2024, 18:15 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson added a comment to T1812: Deploy Airflow.

Previous status from 2023 installation:

Jan 13 2024, 16:46 · Product evaluation, Continous integration and delivery, Nasqueron Databases
dereckson added inline comments to D2754: Run fantoir-datasource as Airflow pipeline.
Jan 13 2024, 14:32
dereckson committed rOPSd20a9814bb4c: Allow Airflow to connect to Vault (authored by dereckson).
Allow Airflow to connect to Vault
Jan 13 2024, 14:25
dereckson closed D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 14:25
dereckson updated the summary of D3273: Allow Airflow to connect to Vault.
Jan 13 2024, 14:24
dereckson added a comment to D3273: Allow Airflow to connect to Vault.

The AIRFLOW__DATABASE__SQL_ALCHEMY_CONN environment variable will be removed later, allowing faster container spawn time, but that's for D3273.

Jan 13 2024, 14:23
dereckson added a comment to D3270: Create Vault policy for Airflow.
Jan 13 2024, 14:22
dereckson closed D3270: Create Vault policy for Airflow.
Jan 13 2024, 14:19
dereckson committed rOPS9e46c1130e03: Create Vault policy for Airflow (authored by dereckson).
Create Vault policy for Airflow
Jan 13 2024, 14:19
dereckson updated the test plan for D3270: Create Vault policy for Airflow.
Jan 13 2024, 14:19
dereckson accepted D3270: Create Vault policy for Airflow.

Policy validated during D3273 tests: read is enough.

Jan 13 2024, 14:15