HomeDevCentral

Adopt Salt Tower to merge pillar files

Description

Adopt Salt Tower to merge pillar files

Summary:
Pillar represents structured data and is intended to be easy to iterate.

For several entries, a pillar dictionary is subdivised by node. If it's rather
straightforward to use, this situation has some drawbacks:

  • it doesn't follow the principle of less surprise: Salt documents the pillar to be node-specific: "targeted data to be securely sent only to the relevant minion"
  • it asks to add programmation constructs in templates or modules to browse data, for example through the node.filter_by_name added in 1609054d, or through expressions like pillar["key" + grains["id"]].
  • a pillar can be loooooong, e.g. for docker_containers

This change refactors those entries into several pillar files.
The dictionaries are merged at the end by Salt Tower, an external pillar
for Salt to declare pillar files by grain, node or pillar value.

Why Salt Tower?

In the Puppet world, hiera is a solution for this configuration problem,
allowing to define configuration at several level, globally, by group, by node.

While SaltStack offers hiera compatibility, it doesn't seem well documented
or maintained. T1758 offers to use PillarStack to create a pillar stack
by node/role/grain. This solves most of our needs, excepted:

  • there is no top-file, so the list of the stack files should be in the external pillar configuration
  • even if the external pillar configuration is a pillar is a symlink to a file in this repository, paths must be relative to the stack configuration directory (ie to /usr/local/etc/master.d/) or absolute

A well-maintained and active external pillar is Salt Tower, which allows
to define a "tower" top file, with references to the relevant files.
It also allows to target by node or grain. This solution is adopted here.

Salt Tower in action

One most relevant section of the pillar is the PaaS Docker configuration:
instead of 597 lines of data about everything about Docker, minus some values
defined instead in pillar/saas, the data is now organized by Docker node,
and furthermore, is split under several files, to separate service definitions.

That creates a more legible and easier to maintain configuration, at pillar,
states or custom module level.

If the dictionaries are perfectly merged, the lists are concatenated.
As such, a function is added to the paas_docker execution module
to return docker_images pillar as a set.

The nodes pillar value is out of scope of this change, as plan is to generate
it from NetBox API.

Fixes T1768.

Test Plan:
salt docker-002 pillar.get docker_containers
salt dwellers pillar.get docker_containers

Same with docker_images, docker_networks.

Run roles/paas-docker and ensure it's no-op.

Reviewers: dereckson

Reviewed By: dereckson

Maniphest Tasks: T1768

Differential Revision: https://devcentral.nasqueron.org/D2862

Details

Provenance
derecksonAuthored on Mar 8 2023, 11:24
derecksonPushed on Mar 9 2023, 20:48
Reviewer
dereckson
Differential Revision
D2862: Adopt Salt Tower to merge pillar files
Parents
rOPSad12ca153247: Deploy Salt Tower on salt-primary role
Branches
Unknown
Tags
Unknown
Tasks
T1768: Simplify paas-docker pillar with PillarStack