Maniphest T1623

Deploy Prometheus to gain observability
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dereckson
	Sep 22 2020, 02:02

Tags

Referenced Files

None

Subscribers

Description

We want the service mesh to be observable.

Kubernetes, Calico (the network data plane for our service mesh), Kuma can send metrics to Prometheus:

Plan is to:

deploy Prometheus on Kubernetes cluster
enable Kuma metrics policy
send Calico metrics
send Kubernetes metrics
publish Grafana dashboards

Revisions and Commits

rDRABBITMQ RabbitMQ Docker image
	D3373	rDRABBITMQ0b92cff35baa Enable rabbitmq_prometheus plugin
rOPS Nasqueron Operations
	D3381	rOPS3dec771810b4 Configure Docker metrics service in firewalld
	D3370	rOPSf2455785f16d Deploy Prometheus on WindRiver
	D3372	rOPSd98f98eb5e5d Collect Docker metrics with Prometheus
	D3369	rOPSa1b74616cbad Deploy Prometheus Node Exporter

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T1621 Prepare a more flexible containers platform
		Open		None	T1633 Collect metrics from RabbitMQ
		Resolved		dereckson	T1392 Evaluate Prometheus
		Resolved		dereckson	T1623 Deploy Prometheus to gain observability
		Open		None	T1983 Enable telemetry on Vault
		Resolved		DorianWinty	T1987 Dovecot Metrics
		Open		DorianWinty	T1990 Export metrics for Postfix

Event Timeline

dereckson created this task.Sep 22 2020, 02:02

dereckson added a parent task: T1621: Prepare a more flexible containers platform.

dereckson mentioned this in T1624: Install an Elastic stack for nasqueron-infra forest.Sep 22 2020, 02:05

dereckson mentioned this in T1625: Deploy a service mesh.Sep 22 2020, 02:10

dereckson mentioned this in T1633: Collect metrics from RabbitMQ.Sep 30 2020, 01:28

dereckson added a parent task: T1633: Collect metrics from RabbitMQ.

dereckson added a parent task: T1392: Evaluate Prometheus.Sep 30 2020, 01:31

dereckson added a project: Monitoring and reporting.Mar 2 2022, 23:46

dereckson mentioned this in T1809: Propagate containers-related events.Mar 28 2023, 23:19

dereckson moved this task from Backlog to Not for this sprint on the Operations sprints (Consolidate them all) board.May 18 2023, 11:49

dereckson mentioned this in D3158: Restore metrics collector in management plugin.May 30 2023, 19:24

dereckson mentioned this in rDRABBITMQ41cfea3ddaac: Restore metrics collector in management plugin.May 30 2023, 19:34

Independently of the 2020 plan for a service mesh, we're going to deploy Prometheus right now to gain observability on currently deployed services.

I've did some experiments this week-end on WindRiver, it's straightforward enough to deploy to be a big win.

dereckson added a revision: D3369: Deploy Prometheus Node Exporter.Jul 23 2024, 18:30

dereckson added a commit: rOPSa1b74616cbad: Deploy Prometheus Node Exporter.Jul 23 2024, 20:10

dereckson moved this task from Backlog to Prometheus on the Monitoring and reporting board.Jul 23 2024, 20:57

dereckson added a revision: D3370: Deploy Prometheus on WindRiver.Jul 23 2024, 22:28

dereckson added a revision: D3372: Collect Docker metrics with Prometheus.Jul 23 2024, 23:05

dereckson added a commit: rOPSd98f98eb5e5d: Collect Docker metrics with Prometheus.Jul 23 2024, 23:08

dereckson added a revision: D3373: Enable rabbitmq_prometheus plugin.Jul 23 2024, 23:15

dereckson added a commit: rDRABBITMQ0b92cff35baa: Enable rabbitmq_prometheus plugin.Jul 25 2024, 17:37

RabbitMQ exporters have been added to NetBox under the tag observability -> https://netbox.nasqueron.org/ipam/services/?tag=observability 🔒

dereckson mentioned this in T651: Deploy Grafana.Jul 25 2024, 18:12

dereckson added a commit: rOPSf2455785f16d: Deploy Prometheus on WindRiver.Jul 27 2024, 12:10

dereckson added a revision: D3381: Configure Docker metrics service in firewalld.Jul 27 2024, 17:07

dereckson added a commit: rOPS3dec771810b4: Configure Docker metrics service in firewalld.Jul 27 2024, 17:08

Prometheus is available, regardless of the initial goal to offer a service mesh on Kubernetes.

It correctly scrapes data since July and Grafana can connect to it without any issue.

It's documented at https://agora.nasqueron.org/Operations_grimoire/Prometheus.

dereckson closed this task as Resolved.Aug 4 2024, 16:34

DorianWinty closed subtask T1987: Dovecot Metrics as Resolved.Aug 8 2024, 19:11