Paths

Table of Contentst

Differential D2793

Provision RabbitMQ configuration
ClosedPublic
Actions

Authored by dereckson on Feb 10 2023, 19:38.

Tags

None

Referenced Files

	Unknown Object (File)
	Mon, Sep 22, 11:32

	Unknown Object (File)
	Mon, Sep 22, 08:26

	Unknown Object (File)
	Sun, Sep 21, 10:20

	Unknown Object (File)
	Sun, Sep 21, 10:20

	Unknown Object (File)
	Fri, Sep 19, 09:20

	Unknown Object (File)
	Fri, Sep 19, 05:54

	Unknown Object (File)
	Thu, Sep 18, 13:51

	Unknown Object (File)
	Wed, Sep 17, 06:33

Subscribers

None

Details

Reviewers

Maniphest Tasks

T752: Salt configuration for White Rabbit

Commits

rOPS79c7da764e62: Provision RabbitMQ configuration

Summary

RabbitMQ clusters can be configured in pillar/saas/rabbitmq.sls

HTTP API requests through the management plugin are fired
to ensure each cluster is configured as expected.

Vault configuration
The salt-primary node policy has read access to the credentials defined
in rabbitmq_clusters pillar, so it can run the saas-rabbitmq role.

Policies are now templates, and the node policy reads the rendered template
version from /srv/policies instead of Salt roles/ folder.

Ref T752.

Test Plan

Provision white-rabbit cluster

Diff Detail

Repository

rOPS Nasqueron Operations

Lint

Lint Errors

Severity	Location	Code	Message
Error	_modules/rabbitmq_api.py:63	E501	flake8 E501
Advice	_modules/rabbitmq_api.py:221	F821	flake8 F821

Unit

No Test Coverage

Branch

rabbitmq-content

Build Status

Buildable 4431
Build 4698: arc lint + arc unit

Event Timeline

dereckson requested review of this revision.Feb 10 2023, 19:38

dereckson created this revision.

Harbormaster completed remote builds in B4423: Diff 7089.Feb 10 2023, 19:38

Remove  

Harbormaster completed remote builds in B4424: Diff 7090.Feb 10 2023, 19:41

Here the output of salt-call --local state.show_sls roles/saas-rabbitmq:

P311 roles/saas-rabbitmq with D2793

1	local:
2	rabbitmq_cluster_white-rabbit_user_notifications:
3	__env__: base
4	__sls__: roles/saas-rabbitmq.server.content
5	rabbitmq:
6	- name: notifications
7	- cluster: white-rabbit
8	- credential: ops/secrets/nasqueron.notifications.broker
9	- user_present
10	- order: 10000
11	rabbitmq_cluster_white-rabbit_user_wearg:
12	__env__: base
13	__sls__: roles/saas-rabbitmq.server.content
14	rabbitmq:
15	- name: wearg
16	- cluster: white-rabbit
17	- credential: apps/viperserv/broker
18	- user_present
19	- order: 10001
20	rabbitmq_cluster_white-rabbit_user_notifications-ysul:
21	__env__: base
22	__sls__: roles/saas-rabbitmq.server.content
23	rabbitmq:
24	- name: notifications-ysul
25	- cluster: white-rabbit
26	- credential: ops/secrets/nasqueron/notifications/notifications-cli/ysul
27	- user_present
28	- order: 10002
29	rabbitmq_cluster_white-rabbit_user_notifications-windriver:
30	__env__: base
31	__sls__: roles/saas-rabbitmq.server.content
32	rabbitmq:
33	- name: notifications-windriver
34	- cluster: white-rabbit
35	- credential: ops/secrets/nasqueron/notifications/notifications-cli/windriver
36	- user_present
37	- order: 10003
38	rabbitmq_cluster_white-rabbit_vhost_dev:
39	__env__: base
40	__sls__: roles/saas-rabbitmq.server.content
41	rabbitmq:
42	- name: dev
43	- cluster: white-rabbit
44	- description: Nasqueron dev services
45	- vhost_present
46	- order: 10004
47	rabbitmq_cluster_white-rabbit_vhost_dev_exchange_notifications:
48	__env__: base
49	__sls__: roles/saas-rabbitmq.server.content
50	rabbitmq:
51	- name: notifications
52	- cluster: white-rabbit
53	- vhost: dev
54	- type: topic
55	- exchange_present
56	- order: 10005
57	rabbitmq_cluster_white-rabbit_vhost_dev_queue_wearg-notifications:
58	__env__: base
59	__sls__: roles/saas-rabbitmq.server.content
60	rabbitmq:
61	- name: wearg-notifications
62	- cluster: white-rabbit
63	- vhost: dev
64	- queue_present
65	- order: 10006
66	rabbitmq_cluster_white-rabbit_vhost_dev_binding_1:
67	__env__: base
68	__sls__: roles/saas-rabbitmq.server.content
69	rabbitmq:
70	- queue: wearg-notifications
71	- cluster: white-rabbit
72	- vhost: dev
73	- exchange: notifications
74	- routing_key: '#'
75	- queue_binding
76	- order: 10007
77	rabbitmq_cluster_white-rabbit_vhost_dev_permissions_user_notifications:
78	__env__: base
79	__sls__: roles/saas-rabbitmq.server.content
80	rabbitmq:
81	- cluster: white-rabbit
82	- vhost: dev
83	- user: notifications
84	- permissions:
85	configure: ''
86	read: ''
87	write: ^notifications$
88	- user_permissions
89	- order: 10008
90	rabbitmq_cluster_white-rabbit_vhost_dev_permissions_user_wearg:
91	__env__: base
92	__sls__: roles/saas-rabbitmq.server.content
93	rabbitmq:
94	- cluster: white-rabbit
95	- vhost: dev
96	- user: wearg
97	- permissions:
98	configure: ''
99	read: ^wearg\-notifications$
100	write: ''
101	- user_permissions
102	- order: 10009
103	rabbitmq_cluster_white-rabbit_vhost_dev_permissions_user_notifications-ysul:
104	__env__: base
105	__sls__: roles/saas-rabbitmq.server.content
106	rabbitmq:
107	- cluster: white-rabbit
108	- vhost: dev
109	- user: notifications-ysul
110	- permissions:
111	configure: ^amq\.gen.*$
112	read: ^(amq\.gen.*\|notifications)$
113	write: ^amq\.gen.*$
114	- user_permissions
115	- order: 10010
116	rabbitmq_cluster_white-rabbit_vhost_dev_permissions_user_notifications-windriver:
117	__env__: base
118	__sls__: roles/saas-rabbitmq.server.content
119	rabbitmq:
120	- cluster: white-rabbit
121	- vhost: dev
122	- user: notifications-windriver
123	- permissions:
124	configure: ^amq\.gen.*$
125	read: ^(amq\.gen.*\|notifications)$
126	write: ^amq\.gen.*$
127	- user_permissions
128	- order: 10011

dereckson mentioned this in D2794: WIP: Provision RabbitMQ configuration.Feb 14 2023, 12:48

Switch to Python syntax for roles/saas-rabbitmq/server/content.sls as we've too many for loops,
and as hunting Jinja issues like '#' routing key is parsed as None isn't fun.

It's more easy to read the loops in configure_cluster function than in the previous Jinja/YAML format.

Provide a module and states to query Management API. The module could be generic enough to commit
upstream to Salt. The states could be, but as they're already the rabbitmqctl ones, could be confusing.
In both cases, we've to handle our credential for rabbitmq.user_present from Vault separetely than the upstream verion.

Tested: vhost, exchange, queue

Still to do: user, binding, permission

Harbormaster completed remote builds in B4428: Diff 7095.Feb 14 2023, 19:35

dereckson planned changes to this revision.Feb 14 2023, 19:37

dereckson added inline comments.

roles/saas-rabbitmq/server/content.sls
146	We can directly use this dictionary.

Use directly privilege in configure_user_permission

Harbormaster completed remote builds in B4429: Diff 7096.Feb 14 2023, 19:40

dereckson edited the summary of this revision. (Show Details)Feb 14 2023, 22:33

User tested. Still to do: binding, permission

Harbormaster completed remote builds in B4431: Diff 7098.Feb 14 2023, 23:43

F821

Harbormaster completed remote builds in B4432: Diff 7099.Feb 14 2023, 23:44

Added the mission execution module methods (untested code): list_bindings queue_bind exchange_bind update_permissions. TODO: test, write state module methods for those.

Harbormaster completed remote builds in B4433: Diff 7100.Feb 15 2023, 00:20

Final version, idempotence checked, see P312.

Harbormaster completed remote builds in B4451: Diff 7130.Feb 24 2023, 19:08

Fixed F841 and F541

Harbormaster completed remote builds in B4452: Diff 7131.Feb 24 2023, 19:11

dereckson retitled this revision from WIP: Provision RabbitMQ configuration to Provision RabbitMQ configuration.Feb 24 2023, 19:12

DorianWinty accepted this revision.Feb 27 2023, 21:22

This revision is now accepted and ready to land.Feb 27 2023, 21:22

Change on operations/_tests/modules/test_rabbitmq.py

Declare notifications exchange as durable

This revision is now accepted and ready to land.Mar 2 2023, 16:25

Harbormaster completed remote builds in B4469: Diff 7198.Mar 2 2023, 16:25

Wearg queue disappeared on reboot

pillar/saas/rabbitmq.sls
74	Should be durable too.

declare wearg-notifications as durable

This revision is now accepted and ready to land.Mar 31 2023, 21:06

Harbormaster completed remote builds in B4664: Diff 7543.Mar 31 2023, 21:06

Last tweaks, e.g. read durable parameters and adjust permissions like currently deployed

Harbormaster completed remote builds in B4665: Diff 7544.Mar 31 2023, 21:21

This revision was landed with ongoing or failed builds.Mar 31 2023, 21:23

Closed by commit rOPS79c7da764e62: Provision RabbitMQ configuration (authored by dereckson). · Explain Why

This revision was automatically updated to reflect the committed changes.

dereckson added a commit: rOPS79c7da764e62: Provision RabbitMQ configuration.

dereckson mentioned this in D2761: Provision Notifications center integration environment.Apr 1 2023, 18:37

Revision Contents
Changeset List

Path

Size

1 line

_modules/

rabbitmq_api.py

233 lines

_states/

342 lines

pillar/

credentials/

2 lines

saas/

105 lines

3 lines

roles/

saas-rabbitmq/

9 lines

server/

141 lines

12 lines

8 lines

vault/

policies/

files/

salt-primary.hcl

15 lines

1 line

14 lines

Diff 7098

.arclint

Loading...

_modules/rabbitmq.py

Loading...

_modules/rabbitmq_api.py

Loading...

_states/rabbitmq.py

Loading...

pillar/credentials/vault.sls

Loading...

pillar/saas/rabbitmq.sls

Loading...

pillar/top.sls

Loading...

roles/saas-rabbitmq/init.sls

Loading...

roles/saas-rabbitmq/server/content.sls

Loading...

roles/saas-rabbitmq/server/init.sls

Loading...

roles/saas-rabbitmq/server/software.sls

Loading...

roles/vault/policies/files/salt-primary.hcl

Loading...

roles/vault/policies/init.sls

Loading...

services.sls

Loading...