Page MenuHomeDevCentral
Feed All Stories

Jul 26 2024

dereckson accepted D3376: Scrape RabbitMQ metrics into Prometheus.
Jul 26 2024, 23:05
dereckson accepted D3374: Expose RabbitMQ metrics on port 15692.

Still to deploy.

Jul 26 2024, 23:04
dereckson added a comment to T650: Deploy PCP on Docker engines.

T651 has a Grafana ready if we wish to retest this on Dwellers, green light.

Jul 26 2024, 19:39 · Monitoring and reporting, Servers
dereckson added a comment to T651: Deploy Grafana.

Deployed at https://grafana.nasqueron.org/

Jul 26 2024, 19:38 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
dereckson updated the summary of D3377: Deploy Grafana.
Jul 26 2024, 19:38
dereckson added a revision to T651: Deploy Grafana: D3377: Deploy Grafana.
Jul 26 2024, 19:38 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
dereckson requested review of D3377: Deploy Grafana.
Jul 26 2024, 19:37
dereckson added a revision to T1633: Collect metrics from RabbitMQ: D3376: Scrape RabbitMQ metrics into Prometheus.
Jul 26 2024, 19:34 · Operations sprints (Consolidate them all), Servers
dereckson requested review of D3376: Scrape RabbitMQ metrics into Prometheus.
Jul 26 2024, 19:34
dereckson added a comment to T1496: Deploy StatsD and Graphite.

With a transformation, we can multiply the field by 10 to solve that issue: https://grafana.nasqueron.org/d/fdsy3oogbchs0f/graphite-quux-sandbox?orgId=1&tab=transform&from=1722018733917&to=1722022333917

Jul 26 2024, 19:32 · Operations sprints (Ignite Alkane Propulsion), Nasqueron Docker deployment squad, Monitoring and reporting
dereckson added a comment to T1496: Deploy StatsD and Graphite.

The graphite Docker image provides both pieces of software.

Jul 26 2024, 19:24 · Operations sprints (Ignite Alkane Propulsion), Nasqueron Docker deployment squad, Monitoring and reporting
dereckson renamed T1496: Deploy StatsD and Graphite from Deploy graphite to Deploy StatsD and Graphite.
Jul 26 2024, 19:13 · Operations sprints (Ignite Alkane Propulsion), Nasqueron Docker deployment squad, Monitoring and reporting
dereckson merged task T1495: Deploy StatsD into T1496: Deploy StatsD and Graphite.
Jul 26 2024, 19:13 · Operations sprints (Ignite Alkane Propulsion), Nasqueron Docker deployment squad, Monitoring and reporting
dereckson merged T1495: Deploy StatsD into T1496: Deploy StatsD and Graphite.
Jul 26 2024, 19:13 · Operations sprints (Ignite Alkane Propulsion), Nasqueron Docker deployment squad, Monitoring and reporting

Jul 25 2024

dereckson added a comment to T1505: Automate Let's Encrypt TLS certificates management for every server.

rOPS1e9a54c10365 has worked like a charm on WindRiver to generate grafana.nasqueron.org through DNS.

Jul 25 2024, 20:43 · Servers
dereckson triaged T1505: Automate Let's Encrypt TLS certificates management for every server as Normal priority.
Jul 25 2024, 20:42 · Servers
dereckson committed rOPS1e9a54c10365: Deploy Certbot everywhere (authored by dereckson).
Deploy Certbot everywhere
Jul 25 2024, 20:41
dereckson closed D3248: Deploy Certbot everywhere.
Jul 25 2024, 20:41
dereckson updated the diff for D3248: Deploy Certbot everywhere.

Rebased. Use /usr/local/etc/periodic. Clean certbot_dir.

Jul 25 2024, 20:41
DorianWinty updated the diff for D3242: Install postfix.

add hostname

Jul 25 2024, 18:56
dereckson added a comment to T651: Deploy Grafana.

DNS: grafana. CNAME www-dev.nasqueron.org

Jul 25 2024, 18:49 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
dereckson added a comment to T651: Deploy Grafana.

Deployment can be using sqlite3 as long as it's still performant
as we want our monitoring tools to be resiliant.

Jul 25 2024, 18:49 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
DorianWinty updated the diff for D3364: Provisioning Dovecot Config.

follow comments

Jul 25 2024, 18:46
dereckson added a comment to T650: Deploy PCP on Docker engines.

Probably a good part of roles/core/monitoring when grains["os_family"] == "RedHat". Eglide has "Debian" for that grain, but not sure if we've enough RAM there.

Jul 25 2024, 18:17 · Monitoring and reporting, Servers
DorianWinty updated the diff for D3364: Provisioning Dovecot Config.

chmod of files and folder + encryption

Jul 25 2024, 18:14
dereckson added a comment to T652: Install PCP on Dwellers.

Just for reference, this was a test deployment. This is not currently installed on Dwellers, and needs to be in Salt as part of T650.

Jul 25 2024, 18:13 · Servers
dereckson renamed T650: Deploy PCP on Docker engines from Give access to Dwellers key statistics to Deploy PCP on Docker engines.
Jul 25 2024, 18:12 · Monitoring and reporting, Servers
dereckson raised the priority of T651: Deploy Grafana from Low to Normal.
Jul 25 2024, 18:12 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
dereckson claimed T651: Deploy Grafana.

This task has been created in 2016 to publish metrics from PCP (Performance Co-Pilot) on RHEL-like servers, especially our Docker engines.

Jul 25 2024, 18:12 · Monitoring and reporting, Operations sprints (Ignite Alkane Propulsion), Servers
dereckson added a comment to T1623: Deploy Prometheus to gain observability.

RabbitMQ exporters have been added to NetBox under the tag observability -> https://netbox.nasqueron.org/ipam/services/?tag=observability 🔒

Jul 25 2024, 18:04 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson added parent tasks for T1987: Dovecot Metrics: T1931: Dovecot Provisioning, T1623: Deploy Prometheus to gain observability.
Jul 25 2024, 18:03 · Restricted Project, Mail
dereckson added a subtask for T1623: Deploy Prometheus to gain observability: T1987: Dovecot Metrics.
Jul 25 2024, 18:03 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson added a subtask for T1931: Dovecot Provisioning: T1987: Dovecot Metrics.
Jul 25 2024, 18:03 · Mail, Restricted Project, Servers
dereckson added inline comments to D3364: Provisioning Dovecot Config.
Jul 25 2024, 17:47
dereckson requested changes to D3364: Provisioning Dovecot Config.
Jul 25 2024, 17:45
dereckson added a comment to T1932: ViMbAdmin Provisioning.

Next: memcached

Jul 25 2024, 17:39 · Mail, Restricted Project, Servers
dereckson closed D3373: Enable rabbitmq_prometheus plugin.
Jul 25 2024, 17:37
dereckson committed rDRABBITMQ0b92cff35baa: Enable rabbitmq_prometheus plugin (authored by dereckson).
Enable rabbitmq_prometheus plugin
Jul 25 2024, 17:37
dereckson accepted D3373: Enable rabbitmq_prometheus plugin.
Jul 25 2024, 17:36
dereckson added a comment to T1633: Collect metrics from RabbitMQ.

Pending container redeployment with D3374, we can reach metrics set in D3373 with socat:

Jul 25 2024, 17:36 · Operations sprints (Consolidate them all), Servers
DorianWinty published D3364: Provisioning Dovecot Config for review.
Jul 25 2024, 17:09

Jul 24 2024

DorianWinty added a project to T1987: Dovecot Metrics: Restricted Project.
Jul 24 2024, 21:02 · Restricted Project, Mail
DorianWinty triaged T1987: Dovecot Metrics as Normal priority.
Jul 24 2024, 21:01 · Restricted Project, Mail
dereckson added a parent task for T1986: Upgrade Debian version for docker-nginx-php-fpm image: Unknown Object (Maniphest Task).
Jul 24 2024, 19:55 · Docker images
dereckson triaged T1986: Upgrade Debian version for docker-nginx-php-fpm image as High priority.
Jul 24 2024, 19:54 · Docker images
dereckson added a subtask for T1950: Deploy PHP 8.3: Unknown Object (Maniphest Task).
Jul 24 2024, 19:36 · Servers, PHP 8.x support
DorianWinty closed D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:49
DorianWinty committed rOPSfc0d46d845df: Configure pg_HBA for dovecot user (authored by DorianWinty).
Configure pg_HBA for dovecot user
Jul 24 2024, 17:49
dereckson updated the summary of D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:47
dereckson accepted D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:45
DorianWinty added a revision to T1931: Dovecot Provisioning: D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:27 · Mail, Restricted Project, Servers
DorianWinty updated the summary of D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:27
DorianWinty retitled D3375: Configure pg_HBA for dovecot user from configure pg_HBA for dovecot user to Configure pg_HBA for dovecot user.
Jul 24 2024, 17:26
DorianWinty requested review of D3375: Configure pg_HBA for dovecot user.
Jul 24 2024, 17:26
DorianWinty committed rOPS5903a7ce83ce: Upgrade certbot to Python 3.11 (authored by DorianWinty).
Upgrade certbot to Python 3.11
Jul 24 2024, 15:56
DorianWinty closed D3366: Upgrade certbot to Python 3.11.
Jul 24 2024, 15:56
dereckson updated the task description for T1983: Enable telemetry on Vault.
Jul 24 2024, 00:00 · Vault, Monitoring and reporting

Jul 23 2024

dereckson triaged T1983: Enable telemetry on Vault as Low priority.
Jul 23 2024, 23:57 · Vault, Monitoring and reporting
dereckson requested review of D3374: Expose RabbitMQ metrics on port 15692.
Jul 23 2024, 23:34
dereckson added a revision to T1633: Collect metrics from RabbitMQ: D3374: Expose RabbitMQ metrics on port 15692.
Jul 23 2024, 23:34 · Operations sprints (Consolidate them all), Servers
dereckson updated the task description for T1633: Collect metrics from RabbitMQ.
Jul 23 2024, 23:27 · Operations sprints (Consolidate them all), Servers
dereckson updated the diff for D3373: Enable rabbitmq_prometheus plugin.

+port

Jul 23 2024, 23:25
dereckson updated the summary of D3373: Enable rabbitmq_prometheus plugin.
Jul 23 2024, 23:24
dereckson added a revision to T1633: Collect metrics from RabbitMQ: D3373: Enable rabbitmq_prometheus plugin.
Jul 23 2024, 23:24 · Operations sprints (Consolidate them all), Servers
dereckson requested review of D3373: Enable rabbitmq_prometheus plugin.
Jul 23 2024, 23:15
dereckson added a revision to T1623: Deploy Prometheus to gain observability: D3373: Enable rabbitmq_prometheus plugin.
Jul 23 2024, 23:15 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson committed rOPSd98f98eb5e5d: Collect Docker metrics with Prometheus (authored by dereckson).
Collect Docker metrics with Prometheus
Jul 23 2024, 23:08
dereckson closed D3372: Collect Docker metrics with Prometheus.
Jul 23 2024, 23:08
dereckson added a comment to D3372: Collect Docker metrics with Prometheus.

Deployed on both Docker engines, but docker-002 is still to be restarted, probably good idea to sync that with dnf update

Jul 23 2024, 23:08
dereckson accepted D3372: Collect Docker metrics with Prometheus.
Jul 23 2024, 23:07
dereckson added a revision to T1623: Deploy Prometheus to gain observability: D3372: Collect Docker metrics with Prometheus.
Jul 23 2024, 23:05 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson requested review of D3372: Collect Docker metrics with Prometheus.
Jul 23 2024, 23:05
dereckson added a reverting change for D2609: Install Python 3.9 on CentOS/Rocky 8.5 machines: D3371: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Jul 23 2024, 22:41
dereckson requested review of D3371: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Jul 23 2024, 22:41
dereckson added a reverting change for rOPSe509029a4712: Install Python 3.9 on CentOS/Rocky 8.5 machines: D3371: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Jul 23 2024, 22:41
dereckson added a revision to T1982: Upgrade from Python 3.9 to Python 3.11+: D3371: Revert "Install Python 3.9 on CentOS/Rocky 8.5 machines".
Jul 23 2024, 22:41 · Servers
dereckson updated the task description for T1982: Upgrade from Python 3.9 to Python 3.11+.
Jul 23 2024, 22:40 · Servers
dereckson added a task to D3366: Upgrade certbot to Python 3.11: T1982: Upgrade from Python 3.9 to Python 3.11+.
Jul 23 2024, 22:38
dereckson added a revision to T1982: Upgrade from Python 3.9 to Python 3.11+: D3366: Upgrade certbot to Python 3.11.
Jul 23 2024, 22:38 · Servers
dereckson added a task to D3368: Bump default versions to build ports: T1982: Upgrade from Python 3.9 to Python 3.11+.
Jul 23 2024, 22:38
dereckson added a revision to T1982: Upgrade from Python 3.9 to Python 3.11+: D3368: Bump default versions to build ports.
Jul 23 2024, 22:38 · Servers
dereckson triaged T1982: Upgrade from Python 3.9 to Python 3.11+ as Normal priority.
Jul 23 2024, 22:37 · Servers
dereckson added a revision to T1623: Deploy Prometheus to gain observability: D3370: Deploy Prometheus on WindRiver.
Jul 23 2024, 22:28 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson added a comment to T1931: Dovecot Provisioning.

Also, we need to declare Dovecot ports at https://netbox.nasqueron.org/virtualization/virtual-machines/10/ services table (on the public IP)

Jul 23 2024, 21:36 · Mail, Restricted Project, Servers
dereckson removed a parent task for T1981: Upgrade to FreeBSD 14.1: T1980: ZFS collector doesn't work everywhere.
Jul 23 2024, 21:15 · Servers
dereckson removed a subtask for T1980: ZFS collector doesn't work everywhere: T1981: Upgrade to FreeBSD 14.1.
Jul 23 2024, 21:15 · Monitoring and reporting, Servers
dereckson closed T1980: ZFS collector doesn't work everywhere as Resolved.

I suspect the version 1.6.1 (currently in packages) is compatible with FreeBSD 13 while the version 1.8.2 is compatible with FreeBSD 14.

Jul 23 2024, 21:15 · Monitoring and reporting, Servers
dereckson moved T1978: Document monitoring checks from Backlog to Checks on the Monitoring and reporting board.
Jul 23 2024, 20:58 · Salt, Monitoring and reporting
dereckson moved T1945: Deploy a simple Nagios or Naemon to have a reference implementation from Backlog to Checks on the Monitoring and reporting board.
Jul 23 2024, 20:58 · Monitoring and reporting
dereckson moved T1623: Deploy Prometheus to gain observability from Backlog to Prometheus on the Monitoring and reporting board.
Jul 23 2024, 20:57 · Monitoring and reporting, Operations sprints (Consolidate them all), Servers
dereckson moved T1980: ZFS collector doesn't work everywhere from Backlog to Prometheus on the Monitoring and reporting board.
Jul 23 2024, 20:57 · Monitoring and reporting, Servers
dereckson moved T1392: Evaluate Prometheus from Backlog to Prometheus on the Monitoring and reporting board.
Jul 23 2024, 20:57 · Monitoring and reporting, Product evaluation
dereckson added a subtask for T1981: Upgrade to FreeBSD 14.1: T1972: Update WindRiver to FreeBSD 14.1.
Jul 23 2024, 20:57 · Servers
dereckson added a parent task for T1972: Update WindRiver to FreeBSD 14.1: T1981: Upgrade to FreeBSD 14.1.
Jul 23 2024, 20:57 · Servers
dereckson triaged T1981: Upgrade to FreeBSD 14.1 as High priority.
Jul 23 2024, 20:57 · Servers
dereckson triaged T1980: ZFS collector doesn't work everywhere as Low priority.
Jul 23 2024, 20:55 · Monitoring and reporting, Servers
dereckson committed rOPSa1b74616cbad: Deploy Prometheus Node Exporter (authored by dereckson).
Deploy Prometheus Node Exporter
Jul 23 2024, 20:10
dereckson closed D3369: Deploy Prometheus Node Exporter.
Jul 23 2024, 20:10
dereckson accepted D3369: Deploy Prometheus Node Exporter.
Jul 23 2024, 20:09
dereckson updated the diff for D3369: Deploy Prometheus Node Exporter.

Align names

Jul 23 2024, 19:58