Page MenuHomeDevCentral
Create Task
Maniphest T2124

Update reports automatically on Agora
Open, WishlistPublic
Actions

Description

The reports repository contains two reports to publish and update Agora content:

Database serverReport for Agora pageCommitQuery file
AcquisitariatAI contentbb14cdd26c66tokens-language-models.sql
db-BOperations grimoire/Old content report3c94fd688d3eoperations-grimoire-older-pages.sql

It would make sense if this could be automated.

Plan is to:

  • tag the pages to help machine to identify automated report section
  • write helpers script to run the query and format the output as MediaWiki table
  • run an automated bot to run the report generator and update the relevant page

Revisions and Commits

rOPS Nasqueron Operations
Needs ReviewD3698 Deploy Rhyne-Wyse on reports role
Needs ReviewD3693 Give access to Vault to Rhyne-Wyse automated agent
Needs ReviewD3674 Schedule daily run of report generation on docker-002
Needs ReviewD3669 Deploy nasqueron-reports
Needs ReviewD3667 Give read-only access to Rhyne-Wyse to query Agora pages
D3673rOPS7c31e0895a1b Allow to generate a report against acquisitariat
rRPRT Nasqueron internal reports
D3704rRPRTa3a0169a347c Allow rhyne-wyse user to run the service too
D3697rRPRTbd6dd2bea022 Configure Vault connection with secretsmith
D3678rRPRT41ba75522dfa Update reports automatically on Agora
D3677rRPRT7157b5a1300b Extract logic from run-report script into an action module
D3671rRPRTf1822a7a1901 Allow to query acquisitariat
D3668rRPRT825ed9baf9e7 Run query and format as MediaWiki table
rDWWW Nasqueron documentation
D3705rDWWWfab53747ce7c Add secretsmith to documentation index

Related Objects

Event Timeline

dereckson triaged this task as Wishlist priority.Fri, Sep 12, 19:04
dereckson created this task.
dereckson added a comment.EditedFri, Sep 12, 19:14

Naming

On Agora, an account name is required for the automated agent / bot.
The agent will be called Rhyne-Wyse [1].

Credentials

Saved the following credentials into Vault:

ServicePurposeVault pathAccount status
AgoraEdit the wikiapps/rhyne-wyse/agora✅
db-BRun MariaDB queries against Agora databaseops/secrets/dbserver/cluster-B/users/rhyne-wyseD3667
acquisitariatRun MySQL queries against DevCentral databaseops/secrets/nasqueron/reports/acquisitariat✅

Notes

[1] Generated with the help of https://donjon.bin.sh/scifi/name/ (cyberpunk, megacorp), like usual for the bots.

dereckson claimed this task.Fri, Sep 12, 19:14
dereckson updated the task description. (Show Details)

Claiming temporarily for credentials and if time allows this evening write helpers script. Will release afterwards, so anyone can create the bot.

dereckson added a revision: D3667: Give read-only access to Rhyne-Wyse to query Agora pages.Fri, Sep 12, 19:21
dereckson added a revision: D3668: Run query and format as MediaWiki table.Sat, Sep 13, 02:06
dereckson added a commit: rRPRT825ed9baf9e7: Run query and format as MediaWiki table.Sat, Sep 13, 12:52
dereckson added a comment.Sat, Sep 13, 15:10

Deployed on WindRiver. Anyone with a vault token valid for ops/secrets/dbserver/cluster-B/users/rhyne-wyse can run there:

$ run-report agora-operations-grimoire-older-pages
dereckson added a revision: D3669: Deploy nasqueron-reports.Sat, Sep 13, 15:18
dereckson added a comment.Sat, Sep 13, 15:19

Releasing the task assignment.

Next step: create a MediaWiki client to automate the wiki publishing (based on pywikibot for example)

dereckson removed dereckson as the assignee of this task.Sat, Sep 13, 15:19
dereckson added a revision: D3671: Allow to query acquisitariat.Sat, Sep 13, 20:08
dereckson added a commit: rRPRTf1822a7a1901: Allow to query acquisitariat.Sat, Sep 13, 20:43
dereckson added a revision: D3673: Allow to generate a report against acquisitariat.Sun, Sep 14, 00:49
dereckson added a commit: rOPS7c31e0895a1b: Allow to generate a report against acquisitariat.Sun, Sep 14, 01:19
dereckson added a revision: D3674: Schedule daily run of report generation on docker-002.Sun, Sep 14, 01:41
dereckson added a revision: D3677: Extract logic from run-report script into an action module.Sun, Sep 14, 16:09
dereckson added a commit: rRPRT7157b5a1300b: Extract logic from run-report script into an action module.Sun, Sep 14, 16:20
dereckson claimed this task.EditedMon, Sep 15, 01:01

At this stage, we've an auto update working when running the helper script manually:

Screenshot From 2025-09-15 02-49-51.png (704×1 px, 1 MB)

Code is ready for anyone with a Vault token to run the bot, as hvac detects authentication methods from environment and known paths.

Next steps for full automation:

  • allow to login with proper credentials
    • Vault via rOPS: Create a permission for access to relevant credentials (see first comment of this post, acquisitariat not needed)
    • Vault manually: Create an AppRole linked to that permission
    • Code: allow to login to Vault with AppRole when available in environment or when a file exists, TO IMPLEMENT FOR BOTH TOOLS (bot needs Agora, run-report needs db-B-001).
  • deploy the bot on WindRiver
    • Vault via rOPS: credentials to query Vault (ie secret id and role id from the AppRole)
    • Salt: create rhyne-wyse user
    • Salt: create /var/db/rhyne-wyse, chown to rhyne-wyse:nasquenautes in 775, so everyone can run the bot too
    • Salt: provide /etc/periodic configuration to run the script daily

Asked ChatGPT to review D3678. An important note was raised about periodic system:

If the bot runs from periodic, environment may lack $HOME, so place config explicitly (-dir: option or set PYWIKIBOT_DIR).

dereckson added a revision: D3678: Update reports automatically on Agora.Mon, Sep 15, 01:12
dereckson attached a referenced file: F11718636: Screenshot From 2025-09-15 02-49-51.png. (Show Details)Mon, Sep 15, 01:12
dereckson added a revision: D3693: Give access to Vault to Rhyne-Wyse automated agent.Fri, Sep 19, 02:03
dereckson added a commit: rRPRT41ba75522dfa: Update reports automatically on Agora.Sat, Sep 20, 00:31
dereckson added a revision: D3697: Configure Vault connection with secretsmith.Sat, Sep 20, 00:37
dereckson added a revision: D3698: Deploy Rhyne-Wyse on reports role.Sat, Sep 20, 00:44
dereckson added a commit: rRPRTbd6dd2bea022: Configure Vault connection with secretsmith.Sat, Sep 20, 22:32
dereckson added a revision: D3704: Allow rhyne-wyse user to run the service too.Sun, Sep 21, 00:50
dereckson added a commit: rRPRTa3a0169a347c: Allow rhyne-wyse user to run the service too.Sun, Sep 21, 00:51
dereckson added a comment.Sun, Sep 21, 00:54

Just for information, log doesn't have any timestamp.

Last entries of successful run:

Logging in to agora:agora as Rhyne-Wyse
[wiki.update_report] Report devcentral-token-language-models is up to date.
[wiki.update_report] Report agora-operations-grimoire-older-pages is up to date.
dereckson added a comment.Sun, Sep 21, 01:04
In T2124#32520, @dereckson wrote:

Next steps for full automation:

  • allow to login with proper credentials
    • Vault via rOPS: Create a permission for access to relevant credentials (see first comment of this post, acquisitariat not needed)
    • Vault manually: Create an AppRole linked to that permission
    • Code: allow to login to Vault with AppRole when available in environment or when a file exists, TO IMPLEMENT FOR BOTH TOOLS (bot needs Agora, run-report needs db-B-001).
  • deploy the bot on WindRiver
    • Vault via rOPS: credentials to query Vault (ie secret id and role id from the AppRole)
    • Salt: create rhyne-wyse user
    • Salt: create /var/db/rhyne-wyse, chown to rhyne-wyse:nasquenautes in 775, so everyone can run the bot too
    • Salt: provide /etc/periodic configuration to run the script daily

Done, operations part pending review, code part committed.

Improvement from last week plan:

  • approle credentials have been deployed through Terraform and not manually
  • crontab entry with wrapper script instead of hijacking system periodic system

If the bot runs from periodic, environment may lack $HOME, so place config explicitly (-dir: option or set PYWIKIBOT_DIR).

The wrapper explicitly uses /var/run/rhyne-wyse as working directory.

That's where Pywikibot populates pywikibot-Rhyne-Wyse.lwp, throttle.ctrl and apicache/
That's also where we copy the config (symlink was ignored as file didn't belong to rhyne-wyse or root, but to builder)

dereckson added a revision: D3705: Add secretsmith to documentation index.Sun, Sep 21, 01:07
dereckson added a commit: rDWWWfab53747ce7c: Add secretsmith to documentation index.Sun, Sep 21, 01:08
dereckson moved this task from Backlog (content) to Done on the Agora board.Sun, Sep 21, 01:10
dereckson added a project: Servers.Mon, Sep 22, 21:45
dereckson moved this task from Backlog to Pending review on the Servers board.
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator