No OneTemporary
Actions

Size

37 KB

Referenced Files

None

Subscribers

None

View Options

	diff --git a/README.md b/README.md
	index bb0c12f..f0350ee 100644
	--- a/README.md
	+++ b/README.md
	@@ -1,32 +1,36 @@
	## Internal reports

	Reports about Nasqueron internal data.

	This repository can host:

	- SQL queries to get report data
	- Tools to produce reports

	### SQL queries

	Queries are organized by cluster/server name, then by service:

	- acquisitariat/ contains the queries for MySQL Docker container
	used by dev & community services like DevCentral

	### Tools

	Tools and utilities to work with reports are located in the tools/ folder:

	* [nasqueron-reports](tools/nasqueron-reports/README.md):
	allows run the MariaDB or MySQL query and format
	the result as expected, like as a MediaWiki table

	+* [secretsmith](tools/secretsmith/README.md):
	+ wrapper around the hvac library to get secrets from Vault or OpenBao,
	+ allow to authenticate with token or AppRole
	+
	### Contribute

	This repository is intended to behave as a monorepo for reporting.

	You can so add any project to generate or use a report at Nasqueron here,
	regardless of the choice of technology stack.

	Projects in tools/<name of the project> are intended to be built autonomously.
	diff --git a/tools/nasqueron-reports/conf/reports.yaml b/tools/nasqueron-reports/conf/reports.yaml
	index da9360e..3bc044d 100644
	--- a/tools/nasqueron-reports/conf/reports.yaml
	+++ b/tools/nasqueron-reports/conf/reports.yaml
	@@ -1,48 +1,57 @@
	# -------------------------------------------------------------
	# Reports configuration
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# -------------------------------------------------------------

	services:
	acquisitariat:
	connector: MySQL
	hostname: mysql
	credentials:
	driver: env
	variables:
	username: DB_USERNAME
	password: DB_PASSWORD

	db-B:
	connector: MariaDB
	hostname: db-B-001
	credentials:
	driver: vault
	secret: ops/secrets/dbserver/cluster-B/users/rhyne-wyse

	reports:
	agora-operations-grimoire-older-pages:
	path: sql/db-B/agora/operations-grimoire-older-pages.sql
	service: db-B

	format: mediawiki
	format_options:
	# Mapping between query columns and table headings
	cols:
	page_link: Article
	age: Age (days)

	devcentral-tokens-language-models:
	path: sql/acquisitariat/devcentral/tokens-language-models.sql
	service: acquisitariat

	format: mediawiki
	format_options:
	# Mapping between query columns and table headings
	cols:
	revision: Revision
	title: Commit title
	date: Date
	userName: Author
	repository: Repository
	+
	+# -------------------------------------------------------------
	+# Secretsmith configuration
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+vault:
	+ server:
	+ url: https://172.27.27.7:8200
	+ verify: /usr/local/share/certs/nasqueron-vault-ca.crt
	diff --git a/tools/nasqueron-reports/requirements.txt b/tools/nasqueron-reports/requirements.txt
	index 8f0b23b..5c32612 100644
	--- a/tools/nasqueron-reports/requirements.txt
	+++ b/tools/nasqueron-reports/requirements.txt
	@@ -1,4 +1,4 @@
	-hvac~=2.3.0
	mysql-connector-python~=9.4.0
	PyYAML~=6.0.2
	+secretsmith~=0.1.0
	sqlparse~=0.5.3
	diff --git a/tools/nasqueron-reports/setup.cfg b/tools/nasqueron-reports/setup.cfg
	index b365747..87d4d36 100644
	--- a/tools/nasqueron-reports/setup.cfg
	+++ b/tools/nasqueron-reports/setup.cfg
	@@ -1,35 +1,35 @@
	[metadata]
	name = nasqueron-reports
	version = 0.1.0
	author = Sébastien Santoro
	author_email = dereckson@espace-win.org
	description = Run MariaDB or MySQL query and format as report
	long_description = file: README.md
	long_description_content_type = text/markdown
	license = BSD-2-Clause
	url = https://devcentral.nasqueron.org/source/reports/
	project_urls =
	Bug Tracker = https://devcentral.nasqueron.org/
	classifiers =
	Programming Language :: Python :: 3
	Operating System :: OS Independent
	Environment :: Console
	Intended Audience :: Developers
	Topic :: Software Development

	[options]
	package_dir =
	= src
	packages = find:
	scripts =
	bin/run-report
	bin/sql-result-to-mediawiki-table
	python_requires = >=3.6
	install_requires =
	PyYAML>=6.0,<7.0
	- hvac>=2.3,<3.0
	mysql-connector-python>=9.4,<10.0
	+ secretsmith>=0.1.0,<1.0
	sqlparse>=0.5,<0.6

	[options.packages.find]
	where = src
	diff --git a/tools/nasqueron-reports/src/nasqueron_reports/config.py b/tools/nasqueron-reports/src/nasqueron_reports/config.py
	index 2a60534..f37ea5a 100644
	--- a/tools/nasqueron-reports/src/nasqueron_reports/config.py
	+++ b/tools/nasqueron-reports/src/nasqueron_reports/config.py
	@@ -1,114 +1,117 @@
	# -------------------------------------------------------------
	# Nasqueron Reports :: Config
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# Description: Reports configuration
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	import os

	import yaml

	from nasqueron_reports.credentials import resolve_credentials
	from nasqueron_reports.errors import NasqueronReportConfigError


	# -------------------------------------------------------------
	# Configuration paths
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	DEFAULT_CONFIG_PATHS = [
	"conf/reports.yaml",
	".reports.yaml",
	"/usr/local/etc/reports.yaml",
	"/etc/reports.yaml",
	]


	DEFAULT_SQL_PATHS = [
	".",
	"/usr/local/share/nasqueron-reports",
	"/usr/share/nasqueron-reports",
	]


	def get_config_path():
	for config_path in DEFAULT_CONFIG_PATHS:
	if os.path.exists(config_path):
	return config_path

	return None


	def resolve_sql_path(sql_path):
	for sql_directory in DEFAULT_SQL_PATHS:
	full_path = os.path.join(sql_directory, sql_path)
	if os.path.exists(full_path):
	return full_path

	return sql_path


	# -------------------------------------------------------------
	# Main configuration
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def get_config():
	config_path = get_config_path()

	if not config_path:
	raise NasqueronReportConfigError(
	"You need to create a reports.yaml config file"
	)

	with open(config_path) as fd:
	config = yaml.safe_load(fd)

	return config


	# -------------------------------------------------------------
	# Report configuration
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def inject_service_config(config, report_config):
	try:
	service_name = report_config["service"]
	except KeyError:
	raise NasqueronReportConfigError(
	f"Service parameter missing in report configuration"
	)

	try:
	report_config["service_options"] = config["services"][service_name]
	except KeyError:
	raise NasqueronReportConfigError(
	f"Service not declared in configuration: {service_name}"
	)

	if "credentials" in report_config["service_options"]:
	credentials = resolve_credentials(
	config, report_config["service_options"]["credentials"]
	)
	else:
	credentials = {}

	report_config["service_options"]["credentials"] = credentials


	-def parse_report_config(report_name):
	+def parse_report_config(report_name, extra_config=None):
	config = get_config()

	try:
	report_config = config["reports"][report_name]
	except KeyError:
	raise NasqueronReportConfigError(f"Report not found: {report_name}")

	+ if extra_config:
	+ config.update(extra_config)
	+
	inject_service_config(config, report_config)

	return report_config
	diff --git a/tools/nasqueron-reports/src/nasqueron_reports/credentials/credentials.py b/tools/nasqueron-reports/src/nasqueron_reports/credentials/credentials.py
	index c45c436..df2df3e 100644
	--- a/tools/nasqueron-reports/src/nasqueron_reports/credentials/credentials.py
	+++ b/tools/nasqueron-reports/src/nasqueron_reports/credentials/credentials.py
	@@ -1,38 +1,39 @@
	# -------------------------------------------------------------
	# Nasqueron Reports :: Credentials :: Vault
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# Description: Read credentials from Vault or OpenBao
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	import os

	from nasqueron_reports.credentials import vault
	from nasqueron_reports.errors import NasqueronReportConfigError


	# -------------------------------------------------------------
	# Credentials wiring
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	-def resolve_credentials(config):
	- if config["driver"] == "vault":
	- return vault.fetch_credentials(config["secret"])
	+def resolve_credentials(full_config, credentials_config):
	+ if credentials_config["driver"] == "vault":
	+ vault_config = full_config.get("vault", {})
	+ return vault.fetch_credentials(vault_config, credentials_config["secret"])

	- if config["driver"] == "env":
	- variables = config.get("variables", {})
	+ if credentials_config["driver"] == "env":
	+ variables = credentials_config.get("variables", {})
	return read_environment(variables)

	raise NasqueronReportConfigError("Credentials driver parameter is missing")


	# -------------------------------------------------------------
	# Environment
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def read_environment(variables):
	return {k: os.environ.get(v, "") for k, v in variables.items()}
	diff --git a/tools/nasqueron-reports/src/nasqueron_reports/credentials/vault.py b/tools/nasqueron-reports/src/nasqueron_reports/credentials/vault.py
	index 1469cde..87274e1 100644
	--- a/tools/nasqueron-reports/src/nasqueron_reports/credentials/vault.py
	+++ b/tools/nasqueron-reports/src/nasqueron_reports/credentials/vault.py
	@@ -1,31 +1,19 @@
	# -------------------------------------------------------------
	# Nasqueron Reports :: Credentials :: Vault
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# Description: Read credentials from Vault or OpenBao
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	-import hvac
	+from secretsmith.vault.client import from_config as client_from_config
	+from secretsmith.vault.secrets import read_secret
	+from secretsmith.vault.utils import split_path


	-VAULT_CA_CERTIFICATE = "/usr/local/share/certs/nasqueron-vault-ca.crt"
	+def fetch_credentials(vault_config, full_secret_path):
	+ vault_client = client_from_config(vault_config)

	-
	-def fetch_credentials(secret_path):
	- vault_client = hvac.Client(
	- verify=VAULT_CA_CERTIFICATE,
	- )
	-
	- tokens = secret_path.split("/")
	- secret_mount = tokens[0]
	- secret_path = "/".join(tokens[1:])
	-
	- secret = vault_client.secrets.kv.read_secret_version(
	- mount_point=secret_mount,
	- path=secret_path,
	- raise_on_deleted_version=True,
	- )
	-
	- return secret["data"]["data"]
	+ mount_point, secret_path = split_path(full_secret_path)
	+ return read_secret(vault_client, mount_point, secret_path)
	diff --git a/tools/rhyne-wyse/conf/rhyne-wyse.yaml b/tools/rhyne-wyse/conf/rhyne-wyse.yaml
	index 8707ca6..d739dad 100644
	--- a/tools/rhyne-wyse/conf/rhyne-wyse.yaml
	+++ b/tools/rhyne-wyse/conf/rhyne-wyse.yaml
	@@ -1,22 +1,25 @@
	wiki:
	credentials:
	driver: vault
	- secret: agora
	+ vault_credentials: /usr/local/etc/secrets/rhyne-wyse.yaml
	+
	+ mount_point: apps
	+ secret_path: rhyne-wyse/agora

	reports:
	- report: devcentral-token-language-models
	tool: fetch
	tool_options:
	url: https://docker-002.nasqueron.org/reports/devcentral-tokens-language-models.txt
	page: AI content
	tweaks:
	- compute-hash-ignoring-date

	- report: agora-operations-grimoire-older-pages
	tool: nasqueron-reports
	tool_options:
	vault_credentials: /usr/local/etc/secrets/rhyne-wyse.yaml
	page: Operations grimoire/Old content report
	tweaks:
	- compute-hash-first-column
	- update-at-least-monthly
	diff --git a/tools/rhyne-wyse/requirements.txt b/tools/rhyne-wyse/requirements.txt
	index 69ea400..39918df 100644
	--- a/tools/rhyne-wyse/requirements.txt
	+++ b/tools/rhyne-wyse/requirements.txt
	@@ -1,4 +1,4 @@
	-hvac~=2.3.0
	nasqueron-reports~=0.1.0
	pywikibot~=10.4.0
	PyYAML~=6.0.2
	+secretsmith~=0.1.0
	diff --git a/tools/rhyne-wyse/setup.cfg b/tools/rhyne-wyse/setup.cfg
	index 280780b..5519ec9 100644
	--- a/tools/rhyne-wyse/setup.cfg
	+++ b/tools/rhyne-wyse/setup.cfg
	@@ -1,34 +1,34 @@
	[metadata]
	name = rhyne-wyse
	version = 0.1.0
	author = Sébastien Santoro
	author_email = dereckson@espace-win.org
	description = Automated agent to publish reports
	long_description = file: README.md
	long_description_content_type = text/markdown
	license = BSD-2-Clause
	url = https://devcentral.nasqueron.org/source/reports/
	project_urls =
	Bug Tracker = https://devcentral.nasqueron.org/
	classifiers =
	Programming Language :: Python :: 3
	Operating System :: OS Independent
	Environment :: Console
	Intended Audience :: Developers
	Topic :: Software Development

	[options]
	package_dir =
	= src
	packages = find:
	scripts =
	bin/update-agora-reports
	python_requires = >=3.6
	install_requires =
	PyYAML>=6.0,<7.0
	- hvac>=2.3,<3.0
	nasqueron-reports>=0.1.0,<1.0
	pywikibot>=10.4.0,<11.0
	+ secretsmith>=0.1.0,<1.0

	[options.packages.find]
	where = src
	diff --git a/tools/rhyne-wyse/src/rhyne_wyse/client.py b/tools/rhyne-wyse/src/rhyne_wyse/client.py
	index 5890103..bacaf08 100644
	--- a/tools/rhyne-wyse/src/rhyne_wyse/client.py
	+++ b/tools/rhyne-wyse/src/rhyne_wyse/client.py
	@@ -1,48 +1,47 @@
	# -------------------------------------------------------------
	# Rhyne-Wyse :: Client
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# Description: MediaWiki client
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	from typing import Dict

	import pywikibot
	from pywikibot.login import ClientLoginManager

	from rhyne_wyse.credentials import vault


	# -------------------------------------------------------------
	# Wiki authentication
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def fetch_credentials(credentials_config: Dict) -> Dict:
	try:
	driver = credentials_config["driver"]
	except KeyError:
	raise ValueError("Missing config key: wiki.credentials.driver")

	if driver == "vault":
	- client = vault.connect_to_vault()
	- return vault.read_app_secret(client, credentials_config["secret"])
	+ return vault.read_app_secret(credentials_config)

	raise ValueError(f"Unknown credentials driver: {driver}")


	def connect_to_site(config: Dict):
	site = pywikibot.Site()

	credentials = fetch_credentials(config["credentials"])
	manager = ClientLoginManager(
	site=site,
	user=credentials["username"],
	password=credentials["password"],
	)
	manager.login()
	site.login()

	return site
	diff --git a/tools/rhyne-wyse/src/rhyne_wyse/credentials/vault.py b/tools/rhyne-wyse/src/rhyne_wyse/credentials/vault.py
	index 1fc6133..6b9c886 100644
	--- a/tools/rhyne-wyse/src/rhyne_wyse/credentials/vault.py
	+++ b/tools/rhyne-wyse/src/rhyne_wyse/credentials/vault.py
	@@ -1,37 +1,27 @@
	#!/usr/bin/env python3

	# -------------------------------------------------------------
	# Rhyne-Wyse :: Credentials :: Vault
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# Description: Fetch credentials from Vault
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	from typing import Dict

	-import hvac
	+import secretsmith
	+from secretsmith.vault.secrets import read_secret


	-VAULT_CA_CERTIFICATE = "/usr/local/share/certs/nasqueron-vault-ca.crt"
	+def read_app_secret(config: Dict[str, str]) -> Dict[str, str]:
	+ config_path = config.get("vault_credentials", None)

	+ try:
	+ vault_client = secretsmith.login(config_path)
	+ except PermissionError:
	+ # Allow running the bot under a user account too
	+ vault_client = secretsmith.login()

	-def connect_to_vault():
	- return hvac.Client(
	- verify=VAULT_CA_CERTIFICATE,
	- )
	-
	-
	-def read_secret(
	- vault_client, mount_point: str, prefix: str, key: str
	-) -> Dict[str, str]:
	- secret = vault_client.secrets.kv.read_secret_version(
	- mount_point=mount_point,
	- path=prefix + "/" + key,
	- )
	- return secret["data"]["data"]
	-
	-
	-def read_app_secret(vault_client, key: str) -> Dict[str, str]:
	- return read_secret(vault_client, "apps", "rhyne-wyse", key)
	+ return read_secret(vault_client, config["mount_point"], config["secret_path"])
	diff --git a/tools/rhyne-wyse/src/rhyne_wyse/tasks/reports.py b/tools/rhyne-wyse/src/rhyne_wyse/tasks/reports.py
	index 91c214a..9329ff5 100644
	--- a/tools/rhyne-wyse/src/rhyne_wyse/tasks/reports.py
	+++ b/tools/rhyne-wyse/src/rhyne_wyse/tasks/reports.py
	@@ -1,73 +1,102 @@
	# -------------------------------------------------------------
	# Rhyne-Wyse :: Tasks :: Reports
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	# Project: Nasqueron
	# License: BSD-2-Clause
	# -------------------------------------------------------------


	from typing import Dict

	import requests
	+import yaml

	from nasqueron_reports.actions.reports import generate_report
	from nasqueron_reports.config import parse_report_config

	from rhyne_wyse.wiki.page import get_page_age
	from rhyne_wyse.utils.hashes import *


	# -------------------------------------------------------------
	# Main tasks
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def prepare_report(report_options: Dict) -> Report:
	if report_options["tool"] == "nasqueron-reports":
	- report_config = parse_report_config(report_options["report"])
	- return generate_report(report_config)
	+ return generate_nasqueron_report(report_options)
	elif report_options["tool"] == "fetch":
	return fetch_report(report_options)

	raise ValueError("Unknown report tool: " + report_options["tool"])


	def needs_report_update(site, page_title, report: Report, tweaks: List) -> bool:
	to_update = False
	report_hash = ""

	# Do not eagerly return True, as we need to update the hash either

	if "update-at-least-monthly" in tweaks:
	age = get_page_age(site, page_title)
	if age > 30:
	to_update = True

	if "compute-hash-ignoring-date" in tweaks:
	report_hash = compute_hash_ignoring_date(report)
	elif "compute-hash-first-column" in tweaks:
	report_hash = compute_hash_from_first_column(report)

	if report_hash is not None:
	current_hash = read_hash_from_datastore(page_title)
	if current_hash != report_hash:
	to_update = True
	write_hash_to_datastore(page_title, report_hash)

	return to_update


	+# -------------------------------------------------------------
	+# Call Nasqueron Reports to generate a report
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+
	+def parse_nasqueron_report_config(report_options):
	+ tool_options = report_options.get("tool_options", {})
	+ vault_credentials = tool_options.get("vault_credentials", None)
	+
	+ if vault_credentials is not None:
	+ try:
	+ with open(vault_credentials) as fd:
	+ return {
	+ "vault": yaml.safe_load(fd),
	+ }
	+ except PermissionError:
	+ # Allow running the bot under a user account too
	+ pass
	+
	+ return {}
	+
	+
	+def generate_nasqueron_report(report_options):
	+ extra_config = parse_nasqueron_report_config(report_options)
	+ report_config = parse_report_config(report_options["report"], extra_config)
	+
	+ return generate_report(report_config)
	+
	+
	# -------------------------------------------------------------
	# Fetch an already generated report from a specific URL
	#
	# For reports configured with `tool: fetch`
	# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	def fetch_report(report_options) -> Report:
	url = report_options["tool_options"]["url"]

	response = requests.get(url)
	response.raise_for_status()

	return Report(None, response.text)
	diff --git a/tools/secretsmith/README.md b/tools/secretsmith/README.md
	new file mode 100644
	index 0000000..65c6a3d
	--- /dev/null
	+++ b/tools/secretsmith/README.md
	@@ -0,0 +1,120 @@
	+## secretsmith
	+
	+The secretsmith Python package allows connecting to Vault or OpenBao,
	+with support for several authentication methods, including using a token
	+or AppRole.
	+
	+It also provides a simple wrapper to query secrets from a kv2 store.
	+
	+This is a high-level wrapper around [hvac](https://python-hvac.org/).
	+
	+At Nasqueron, we use this package to avoid writing boilerplate code in each
	+application that needs to interact with Vault or OpenBao to:
	+ - read a configuration file to determine login parameters
	+ - query a simple password from kv2 store from a path
	+
	+When more and more applications need to interact with Vault or OpenBao,
	+and use the same authentication methods, the same patterns to query secrets,
	+to maintain this wrapper high-level library becomes useful.
	+
	+### Login
	+
	+Secretsmith uses the `hvac` library to connect to Vault or OpenBao.
	+
	+If nothing is specified, it will try to connect to Vault using the environment
	+variables `VAULT_ADDR` and `VAULT_TOKEN`, or reading a token file at the
	+default path. Especially convenient during the development workflow.
	+
	+When it's ready to be deployed, write a configuration file explaining how to
	+connect to Vault or OpenBao.
	+
	+#### How to use in code?
	+
	+Call secretsmith.login() with the path to the configuration file:
	+
	+```python
	+import secretsmith
	+
	+VAULT_CONFIG_PATH = '/path/to/config.yaml'
	+
	+vault_client = secretsmith.login(config_path=VAULT_CONFIG_PATH)
	+```
	+
	+Then, you can use the client as a hvac library Vault client.
	+
	+We provide helper methods for common tasks, but you can also directly use hvac.
	+
	+#### Configuration file
	+
	+Secretsmith uses a YAML configuration file to determine the login parameters:
	+
	+```
	+vault:
	+ server:
	+ url: https://127.0.0.1:8200
	+ auth:
	+ token: hvs.000000000000000000000000
	+```
	+
	+When using AppRole, the configuration file will look like:
	+
	+```
	+vault:
	+ server:
	+ url: https://127.0.0.1:8200
	+ verify: /path/to/ca.pem
	+ auth:
	+ method: approle
	+ role_id: e5a7b66e-5d08-da9c-7075-71984634b882
	+ secret_id: 841771dc-11c9-bbc7-bcac-6a3945a69cd9
	+```
	+
	+The format is based on the Vault execution module for SaltStack.
	+
	+The following parameters are supported:
	+ - `server` - a block to specify the Vault or OpenBao server parameters
	+ - `url` - the URL
	+ - `verify` - the path to a CA certificate to verify the server's certificate
	+ - `namespace` - the namespace to use (by default, will follow environment)
	+ - `auth` - a block to specify the authentication method and parameters
	+ - `method` - what authentication backend to use, by default 'token'
	+
	+Additional parameters are supported in the `auth` block depending
	+on the authentication method.
	+
	+When the method is `token`:
	+ - `token` - the token to use
	+ - `token_file` - alternatively, the path to a file containing the token
	+
	+When the method is `approle`:
	+ - `role_id` - the AppRole role ID (required)
	+ - `secret_id` - the AppRole secret ID (optional)
	+
	+### Querying secrets
	+
	+For kv2, we also provide helper methods for more common use cases.
	+
	+If you store a password in the password field of the 'secret/app/db' path:
	+
	+```python
	+import secretsmith
	+from secretsmith.vault import secrets
	+
	+vault_client = secretsmith.login()
	+password = secrets.get_password(vault_client, "secret", "app/db")
	+```
	+
	+To get the full k/v store at the 'secret/app/db' path:
	+
	+```python
	+secret = secrets.read_secret(vault_client, "secret", "app/db")
	+```
	+
	+If you also store custom metadata, you can use:
	+
	+```python
	+secret, metadata = secrets.read_secret_with_custom_metadata(vault_client, "secret", "app/db")
	+```
	+
	+In all those examples, you need to replace "secret" by your kv2 mount point.
	+The "secret" mount point is the default one if you didn't configure Vault.
	diff --git a/tools/secretsmith/pyproject.toml b/tools/secretsmith/pyproject.toml
	new file mode 100644
	index 0000000..fd2f351
	--- /dev/null
	+++ b/tools/secretsmith/pyproject.toml
	@@ -0,0 +1,14 @@
	+# -------------------------------------------------------------
	+# secretsmith
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+[build-system]
	+requires = [
	+ "setuptools>=42",
	+ "wheel"
	+]
	+
	+build-backend = "setuptools.build_meta"
	diff --git a/tools/rhyne-wyse/setup.cfg b/tools/secretsmith/setup.cfg
	similarity index 70%
	copy from tools/rhyne-wyse/setup.cfg
	copy to tools/secretsmith/setup.cfg
	index 280780b..ee33aca 100644
	--- a/tools/rhyne-wyse/setup.cfg
	+++ b/tools/secretsmith/setup.cfg
	@@ -1,34 +1,31 @@
	[metadata]
	-name = rhyne-wyse
	+name = secretsmith
	version = 0.1.0
	author = Sébastien Santoro
	author_email = dereckson@espace-win.org
	-description = Automated agent to publish reports
	+description = Connect to Vault with a configuration file
	long_description = file: README.md
	long_description_content_type = text/markdown
	license = BSD-2-Clause
	url = https://devcentral.nasqueron.org/source/reports/
	project_urls =
	Bug Tracker = https://devcentral.nasqueron.org/
	+ Documentation = https://docs.nasqueron.org/secretsmith/
	+ Source Code = https://devcentral.nasqueron.org/source/reports/browse/main/tools/secretsmith/
	classifiers =
	Programming Language :: Python :: 3
	Operating System :: OS Independent
	- Environment :: Console
	Intended Audience :: Developers
	- Topic :: Software Development
	+ Topic :: Security

	[options]
	package_dir =
	= src
	packages = find:
	-scripts =
	- bin/update-agora-reports
	python_requires = >=3.6
	install_requires =
	PyYAML>=6.0,<7.0
	hvac>=2.3,<3.0
	- nasqueron-reports>=0.1.0,<1.0
	- pywikibot>=10.4.0,<11.0

	[options.packages.find]
	where = src
	diff --git a/tools/secretsmith/src/secretsmith/__init__.py b/tools/secretsmith/src/secretsmith/__init__.py
	new file mode 100644
	index 0000000..6dceb04
	--- /dev/null
	+++ b/tools/secretsmith/src/secretsmith/__init__.py
	@@ -0,0 +1,9 @@
	+# -------------------------------------------------------------
	+# Secretsmith
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: Trivial work, not eligible to copyright
	+# -------------------------------------------------------------
	+
	+
	+from .vault.client import login
	diff --git a/tools/secretsmith/src/secretsmith/vault/client.py b/tools/secretsmith/src/secretsmith/vault/client.py
	new file mode 100644
	index 0000000..f80670e
	--- /dev/null
	+++ b/tools/secretsmith/src/secretsmith/vault/client.py
	@@ -0,0 +1,81 @@
	+# -------------------------------------------------------------
	+# Secretsmith :: Vault :: client
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+
	+import os
	+from typing import Dict
	+
	+from hvac import Client
	+
	+from secretsmith.vault.config import load_config
	+
	+
	+# -------------------------------------------------------------
	+# General login
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+
	+def login(config_path: str \| None = None) -> Client:
	+ if config_path is None:
	+ config = {}
	+ else:
	+ config = load_config(config_path).get("vault", {})
	+
	+ return from_config(config)
	+
	+
	+def from_config(config: Dict) -> Client:
	+ config_server = config.get("server", {})
	+ url = config_server.get("url", None)
	+ verify = config_server.get("verify", None)
	+ namespace = resolve_namespace(config_server)
	+
	+ config_auth = config.get("auth", {})
	+ auth_method = config_auth.get("method", "token")
	+ if auth_method == "token":
	+ token = resolve_token(config_auth)
	+ else:
	+ token = None
	+
	+ client = Client(url=url, token=token, verify=verify, namespace=namespace)
	+
	+ if auth_method == "approle":
	+ login_with_approle(client, config_auth)
	+ elif auth_method != "token":
	+ raise ValueError(f"Unknown auth method: {auth_method}")
	+
	+ return client
	+
	+
	+def resolve_token(config_auth):
	+ if "tokenfile" in config_auth:
	+ with open(config_auth["tokenfile"]) as fd:
	+ return fd.read().strip()
	+
	+ return config_auth.get("token", None)
	+
	+
	+def resolve_namespace(config: dict) -> str \| None:
	+ try:
	+ return config["namespace"]
	+ except KeyError:
	+ return os.environ.get("VAULT_NAMESPACE", None)
	+
	+
	+# -------------------------------------------------------------
	+# Additional authentication backends
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+
	+def login_with_approle(client: Client, config_auth: dict):
	+ if "role_id" not in config_auth:
	+ raise ValueError("Missing role_id in auth configuration")
	+
	+ client.auth.approle.login(
	+ role_id=config_auth["role_id"],
	+ secret_id=config_auth.get("secret_id", None),
	+ )
	diff --git a/tools/secretsmith/src/secretsmith/vault/config.py b/tools/secretsmith/src/secretsmith/vault/config.py
	new file mode 100644
	index 0000000..ad3a89f
	--- /dev/null
	+++ b/tools/secretsmith/src/secretsmith/vault/config.py
	@@ -0,0 +1,14 @@
	+# -------------------------------------------------------------
	+# Secretsmith :: Vault :: configuration
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+
	+import yaml
	+
	+
	+def load_config(path: str) -> dict:
	+ with open(path) as fd:
	+ return yaml.safe_load(fd)
	diff --git a/tools/secretsmith/src/secretsmith/vault/secrets.py b/tools/secretsmith/src/secretsmith/vault/secrets.py
	new file mode 100644
	index 0000000..7f8e249
	--- /dev/null
	+++ b/tools/secretsmith/src/secretsmith/vault/secrets.py
	@@ -0,0 +1,79 @@
	+# -------------------------------------------------------------
	+# Secretsmith :: Vault :: KV secrets engine - version 2
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+
	+from typing import Any, Dict, Tuple
	+
	+from hvac import Client
	+
	+
	+# -------------------------------------------------------------
	+# Fetch secret from kv engine
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+
	+def read_secret(client: Client, mount_point: str, secret_path: str) -> Dict[str, str]:
	+ secret = client.secrets.kv.read_secret_version(
	+ mount_point=mount_point,
	+ path=secret_path,
	+ raise_on_deleted_version=True,
	+ )
	+ return secret["data"]["data"]
	+
	+
	+def read_secret_with_metadata(
	+ client: Client, mount_point: str, secret_path: str
	+) -> Tuple[dict[str, str], dict[str, Any]]:
	+ """
	+ Read a secret and return the data and the metadata dictionaries.
	+ """
	+ secret = client.secrets.kv.read_secret_version(
	+ mount_point=mount_point,
	+ path=secret_path,
	+ raise_on_deleted_version=True,
	+ )
	+
	+ return secret["data"]["data"], secret["data"]["metadata"]
	+
	+
	+def read_secret_with_custom_metadata(
	+ client: Client, mount_point: str, secret_path: str
	+) -> Tuple[dict[str, str], dict[str, Any]]:
	+ """
	+ Read a secret and return the data and the metadata dictionaries.
	+
	+ The custom metadata keys are directly merged into the metadata dictionary.
	+ """
	+ data, metadata = read_secret_with_metadata(client, mount_point, secret_path)
	+
	+ if "custom_metadata" in metadata:
	+ metadata.update(metadata["custom_metadata"])
	+ del metadata["custom_metadata"]
	+
	+ return data, metadata
	+
	+
	+# -------------------------------------------------------------
	+# Helpers to select common fields
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+
	+
	+def get_username(client: Client, mount_point: str, secret_path: str) -> str:
	+ return get_field(client, mount_point, secret_path, "username")
	+
	+
	+def get_password(client: Client, mount_point: str, secret_path: str) -> str:
	+ return get_field(client, mount_point, secret_path, "password")
	+
	+
	+def get_field(client: Client, mount_point: str, secret_path: str, field: str) -> str:
	+ secret = read_secret(client, mount_point, secret_path)
	+
	+ try:
	+ return secret[field]
	+ except KeyError:
	+ raise ValueError(f"Missing {field} field in {mount_point}/{secret_path}")
	diff --git a/tools/secretsmith/src/secretsmith/vault/utils.py b/tools/secretsmith/src/secretsmith/vault/utils.py
	new file mode 100644
	index 0000000..170f620
	--- /dev/null
	+++ b/tools/secretsmith/src/secretsmith/vault/utils.py
	@@ -0,0 +1,22 @@
	+# -------------------------------------------------------------
	+# Secretsmith :: Vault :: Utilities
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+
	+from typing import Tuple
	+
	+
	+def split_path(full_path: str) -> Tuple[str, str]:
	+ """
	+ Split a full path into mount point and secret path,
	+ assuming the first part of the full path is the mount point.
	+ """
	+ tokens = full_path.split("/")
	+
	+ mount_point = tokens[0]
	+ secret_path = "/".join(tokens[1:])
	+
	+ return mount_point, secret_path
	diff --git a/tools/secretsmith/tests/vault/test_secrets.py b/tools/secretsmith/tests/vault/test_secrets.py
	new file mode 100644
	index 0000000..e796e26
	--- /dev/null
	+++ b/tools/secretsmith/tests/vault/test_secrets.py
	@@ -0,0 +1,98 @@
	+# -------------------------------------------------------------
	+# Secretsmith :: Vault :: KV secrets engine - version 2
	+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	+# Project: Nasqueron
	+# License: BSD-2-Clause
	+# -------------------------------------------------------------
	+
	+import unittest
	+from unittest.mock import MagicMock
	+
	+from secretsmith.vault.secrets import *
	+
	+
	+class TestReadSecret(unittest.TestCase):
	+ def setUp(self):
	+ self.mock_client = MagicMock()
	+
	+ secret_mock = MagicMock(return_value=self.mock_kv2_secret())
	+ self.mock_client.secrets.kv.read_secret_version = secret_mock
	+
	+ @staticmethod
	+ def mock_kv2_secret():
	+ return {
	+ "data": {
	+ "data": {
	+ "username": "someuser",
	+ "password": "somepass",
	+ },
	+ "metadata": {
	+ "created_time": "2021-01-01T00:00:00.000000Z",
	+ "deletion_time": "",
	+ "destroyed": False,
	+ "version": 1,
	+ "custom_metadata": {"owner": "someone"},
	+ },
	+ }
	+ }
	+
	+ def test_read_secret(self):
	+ result = read_secret(self.mock_client, "test_mount", "test_path")
	+
	+ expected = {"username": "someuser", "password": "somepass"}
	+ self.assertEqual(expected, result)
	+
	+ def test_read_secret_empty_data(self):
	+ self.mock_client.secrets.kv.read_secret_version.return_value = {
	+ "data": {"data": {}}
	+ }
	+
	+ result = read_secret(self.mock_client, "test_mount", "empty_data_path")
	+
	+ self.assertEqual({}, result)
	+
	+ def test_read_secret_with_metadata_(self):
	+ result_data, result_metadata = read_secret_with_metadata(
	+ self.mock_client, "test_mount", "test_path"
	+ )
	+ expected_data = {"username": "someuser", "password": "somepass"}
	+ expected_metadata = {
	+ "created_time": "2021-01-01T00:00:00.000000Z",
	+ "deletion_time": "",
	+ "destroyed": False,
	+ "version": 1,
	+ "custom_metadata": {"owner": "someone"},
	+ }
	+ self.assertEqual(expected_data, result_data)
	+ self.assertEqual(expected_metadata, result_metadata)
	+
	+ def test_read_secret_with_custom_metadata(self):
	+ result_data, result_metadata = read_secret_with_custom_metadata(
	+ self.mock_client, "test_mount", "test_path"
	+ )
	+ expected_data = {"username": "someuser", "password": "somepass"}
	+ expected_metadata = {
	+ "created_time": "2021-01-01T00:00:00.000000Z",
	+ "deletion_time": "",
	+ "destroyed": False,
	+ "version": 1,
	+ "owner": "someone",
	+ }
	+ self.assertEqual(expected_data, result_data)
	+ self.assertEqual(expected_metadata, result_metadata)
	+
	+ def test_get_username(self):
	+ result = get_username(self.mock_client, "test_mount", "test_path")
	+ self.assertEqual("someuser", result)
	+
	+ def test_get_password(self):
	+ result = get_password(self.mock_client, "test_mount", "test_path")
	+ self.assertEqual("somepass", result)
	+
	+ def test_get_field(self):
	+ result = get_field(self.mock_client, "test_mount", "test_path", "username")
	+ self.assertEqual("someuser", result)
	+
	+
	+if __name__ == "__main__":
	+ unittest.main()

File Metadata

Mime Type: text/x-diff
Expires: Sat, Oct 11, 22:24 (1 d, 10 h)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 3064173
Default Alt Text: (37 KB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions