Pillar represents structured data and is intended to be easy to iterate.
For several entries, a pillar dictionary is subdivised by node. If it's rather
straightforward to use, this situation has some drawbacks:
- it doesn't follow the principle of less surprise: Salt documents the pillar to be node-specific: "targeted data to be securely sent only to the relevant minion"
- it asks to add programmation constructs in templates or modules to browse data, for example through the node.filter_by_name added in 1609054d, or through expressions like pillar["key" + grains["id"]].
- a pillar can be loooooong, e.g. for docker_containers
This change refactors those entries into several pillar files.
The dictionaries are merged at the end by Salt Tower, an external pillar
for Salt to declare pillar files by grain, node or pillar value.
Why Salt Tower?
In the Puppet world, hiera is a solution for this configuration problem,
allowing to define configuration at several level, globally, by group, by node.
While SaltStack offers hiera compatibility, it doesn't seem well documented
or maintained. T1758 offers to use PillarStack to create a pillar stack
by node/role/grain. This solves most of our needs, excepted:
- there is no top-file, so the list of the stack files should be in the external pillar configuration
- even if the external pillar configuration is a pillar is a symlink to a file in this repository, paths must be relative to the stack configuration directory (ie to /usr/local/etc/master.d/) or absolute
A well-maintained and active external pillar is Salt Tower, which allows
to define a "tower" top file, with references to the relevant files.
It also allows to target by node or grain. This solution is adopted here.
Salt Tower in action
One most relevant section of the pillar is the PaaS Docker configuration:
instead of 597 lines of data about everything about Docker, minus some values
defined instead in pillar/saas, the data is now organized by Docker node,
and furthermore, is split under several files, to separate service definitions.
That creates a more legible and easier to maintain configuration, at pillar,
states or custom module level.
If the dictionaries are perfectly merged, the lists are concatenated.
As such, a function is added to the paas_docker execution module
to return docker_images pillar as a set.
The nodes pillar value is out of scope of this change, as plan is to generate
it from NetBox API.
Fixes T1768.