Change Details

As we migrate credentials to Vault, we can leverage them to configure automatically the RabbitMQ instance used by the Notifications center. We can prepare a generic saas/rabbitmq Salt role, with config about instances, users, queues, exchanges in pillar, so we can in the future add other RabbitMQ servers/clusters. **Plan** For the white-rabbit cluster, the following steps are needed: - create administrator user from ops/secrets/nasqueron.white-rabbit.root, this one should be used by Salt (see bootstrap below) - create vhosts -- / -- dev - create exchanges -- notifications (dev vhost, topic) -- github-events (dev vhost, direct) - create queues -- wearg-notifications (dev vhost, bound to notifications exchange, routing key `#`) - create users -- notifications: ops/secrets/nasqueron.notifications.broker -- wearg: apps/viperserv/broker -- notifications-ysul: (not yet in Vault) -- notifications-windriver: (not yet in Vault) -- personal accounts for ops members who manage the cluster, for purge, etc. _____ **Bootstrap the RabbitMQ account for Salt** 1. In paas-docker role, for RabbitMQ container, create the account root, ensure admin user is removed: ```name=Script to run on Docker instance,lang=bash set -e docker exec -it <instance name> rabbitmqctl add_user <username> <password> docker exec -it <instance name> rabbitmqctl set_user_tags <username> administrator # Gives access to /, /dev (where do we have the list of vhosts?) docker exec -it <instance name> rabbitmqctl delete_user administrator ; true touch /srv/rabbitmq/<instance name>/.root_account_created ``` Pitfalls: * add_user returns 70 if the user exists -> not acceptable, exits through `set -e` (delete/recreate would also be acceptable) * delete_user returns 67 if the user doesn't exist -> acceptable, so we can continue execution * don't use `set -x` to avoid credentials in log ```name=Salt state,lang=yaml rabbitmq_{{ instance }}_administrator_account: cmd.script: - ... - creates: /srv/rabbitmq/{{ instance }}/.root_account_created ``` **Option A:** 2. On salt-primary role, save credentials in /usr/local/etc/salt/master.d/rabbitmq.conf (chmod 400) 3. Everything else can happens in a saas/rabbitmq role, with connections using saved configuration **Option B:** 2. On salt-primary role, save instance credential path (e.g. ops/secrets/nasqueron.white-rabbit.root) in /usr/local/etc/salt/master.d/rabbitmq.conf 3. In saas/rabbitmq role, requires credentials from Vault each time Option B is perfect for Nasqueron internal use, option A could be better if we maintain a RabbitMQ plugin.

As we migrate credentials to Vault, we can leverage them to configure automatically the RabbitMQ instance used by the Notifications center. We can prepare a generic saas/rabbitmq Salt role, with config about instances, users, queues, exchanges in pillar, so we can in the future add other RabbitMQ servers/clusters. **Plan** For the white-rabbit cluster, the following steps are needed: - create administrator user from ops/secrets/nasqueron.white-rabbit.root, this one should be used by Salt (see bootstrap below) - create vhosts -- / -- dev - create exchanges -- notifications (dev vhost, topic) -- github-events (dev vhost, direct) - create queues -- wearg-notifications (dev vhost, bound to notifications exchange, routing key `#`) - create users -- notifications: ops/secrets/nasqueron.notifications.broker -- wearg: apps/viperserv/broker -- notifications-ysul: (not yet in Vault) -- notifications-windriver: (not yet in Vault) -- personal accounts for ops members who manage the cluster, for purge, etc. _____ **Bootstrap the RabbitMQ account for Salt** 1. In paas-docker role, for RabbitMQ container, create the account root, ensure admin user is removed: ```name=Script to run on Docker instance (Jinja template),lang=bash set -e docker exec -it {{ instance name }} rabbitmqctl add_user {{ username }} {{ password }} docker exec -it {{ instance name }} rabbitmqctl set_user_tags {{ username }} administrator {% for vhost in vhosts %} docker exec -it {{ instance name }} rabbitmqctl set_permissions --vhost {{ vhost }} {{ username }} .* .* .* {% endfor %} docker exec -it {{ instance name }} rabbitmqctl delete_user administrator ; true touch /srv/rabbitmq/{{ instance name }}/.root_account_created ``` Pitfalls: * add_user returns 70 if the user exists -> not acceptable, exits through `set -e` (delete/recreate would also be acceptable) * delete_user returns 67 if the user doesn't exist -> acceptable, so we can continue execution * don't use `set -x` to avoid credentials in log ```name=Salt state,lang=yaml rabbitmq_{{ instance }}_administrator_account: cmd.script: - ... - creates: /srv/rabbitmq/{{ instance }}/.root_account_created ``` **Option A:** 2. On salt-primary role, save credentials in /usr/local/etc/salt/master.d/rabbitmq.conf (chmod 400) 3. Everything else can happens in a saas/rabbitmq role, with connections using saved configuration **Option B:** 2. On salt-primary role, save instance credential path (e.g. ops/secrets/nasqueron.white-rabbit.root) in /usr/local/etc/salt/master.d/rabbitmq.conf 3. In saas/rabbitmq role, requires credentials from Vault each time Option B is perfect for Nasqueron internal use, option A could be better if we maintain a RabbitMQ plugin. General pitfalls: * When a new vhost is created, the administrator root account needs access on it.

As we migrate credentials to Vault, we can leverage them to configure automatically the RabbitMQ instance used by the Notifications center. We can prepare a generic saas/rabbitmq Salt role, with config about instances, users, queues, exchanges in pillar, so we can in the future add other RabbitMQ servers/clusters. **Plan** For the white-rabbit cluster, the following steps are needed: - create administrator user from ops/secrets/nasqueron.white-rabbit.root, this one should be used by Salt (see bootstrap below) - create vhosts -- / -- dev - create exchanges -- notifications (dev vhost, topic) -- github-events (dev vhost, direct) - create queues -- wearg-notifications (dev vhost, bound to notifications exchange, routing key `#`) - create users -- notifications: ops/secrets/nasqueron.notifications.broker -- wearg: apps/viperserv/broker -- notifications-ysul: (not yet in Vault) -- notifications-windriver: (not yet in Vault) -- personal accounts for ops members who manage the cluster, for purge, etc. _____ **Bootstrap the RabbitMQ account for Salt** 1. In paas-docker role, for RabbitMQ container, create the account root, ensure admin user is removed: ```name=Script to run on Docker instance (Jinja template),lang=bash set -e docker exec -it <{{ instance name> }} rabbitmqctl add_user <{{ username> < }} {{ password> }} docker exec -it <{{ instance name> }} rabbitmqctl set_user_tags <{{ username> }} administrator # Gives access to /, /dev (where do we have the list of{% for vhost in vhosts?) %} docker exec -it <{{ instance name> }} rabbitmqctl delete_user administrator ; trueset_permissions --vhost {{ vhost }} {{ username }} .* .* .* touch /srv/rabbitmq/<instance name>{% endfor %} docker exec -it {{ instance name }} rabbitmqctl delete_user administrator ; true touch /srv/rabbitmq/{{ instance name }}/.root_account_created ``` Pitfalls: * add_user returns 70 if the user exists -> not acceptable, exits through `set -e` (delete/recreate would also be acceptable) * delete_user returns 67 if the user doesn't exist -> acceptable, so we can continue execution * don't use `set -x` to avoid credentials in log ```name=Salt state,lang=yaml rabbitmq_{{ instance }}_administrator_account: cmd.script: - ... - creates: /srv/rabbitmq/{{ instance }}/.root_account_created ``` **Option A:** 2. On salt-primary role, save credentials in /usr/local/etc/salt/master.d/rabbitmq.conf (chmod 400) 3. Everything else can happens in a saas/rabbitmq role, with connections using saved configuration **Option B:** 2. On salt-primary role, save instance credential path (e.g. ops/secrets/nasqueron.white-rabbit.root) in /usr/local/etc/salt/master.d/rabbitmq.conf 3. In saas/rabbitmq role, requires credentials from Vault each time Option B is perfect for Nasqueron internal use, option A could be better if we maintain a RabbitMQ plugin. General pitfalls: * When a new vhost is created, the administrator root account needs access on it.