Details

Reviewers

DorianWinty
yousra

Maniphest Tasks

T1762: Deploy NetBox

Summary

This change is a proof of concept we can use NetBox
to generate information for node pillar

Usage: utils/netbox/pillarize.py <node name>

Additional credits:
cidr_to_netmask method by Trenton McKinney
See https://stackoverflow.com/a/43904598/1930997

Ref T1762

Test Plan

Compare pillarize output with current pillar/nodes/nodes.sls content

Diff Detail

Repository

rOPS Nasqueron Operations

Lint

Lint Passed

Severity	Location	Code	Message
Advice	utils/netbox/pillarize.py:131	F821	flake8 F821

Unit

No Test Coverage

Branch

netbox-pillar

Build Status

Buildable 4419
Build 4686: arc lint + arc unit

Event Timeline

dereckson requested review of this revision.Feb 8 2023, 20:08

dereckson created this revision.

Harbormaster completed remote builds in B4419: Diff 7084.Feb 8 2023, 20:08

dereckson retitled this revision from Generate a pillar from NetBox information to IWP: Generate a pillar from NetBox information.Feb 8 2023, 20:08

dereckson retitled this revision from IWP: Generate a pillar from NetBox information to WIP: Generate a pillar from NetBox information.

dereckson edited the summary of this revision. (Show Details)

s/RuntimeException/RuntimeError

Harbormaster completed remote builds in B4420: Diff 7085.Feb 8 2023, 20:09

Example run

utils/netbox/pillarize.py router-001

etc_hosts:
- 172.27.27.1 router-001 router-001.nasqueron.drake
- 172.27.27.4 dwellers dwellers.nasqueron.drake
- 172.27.27.5 docker-002 docker-002.nasqueron.drake
- 172.27.27.6 docker-001 docker-001.nasqueron.org
- 172.27.27.7 complector complector.nasqueron.drake
- 172.27.27.8 db-a-001 db-a-001.nasqueron.drake
- 172.27.27.28 cloudhugger cloudhugger.nasqueron.drake
- 172.27.27.33 ysul ysul.nasqueron.drake
- 172.27.27.35 windriver windriver.nasqueron.drake
node:
  forest: nasqueron-infra
  hostname: router-001.nasqueron.drake
  network:
    interfaces:
      intranought:
        device: vmx1
        ipv4:
          address: 172.27.27.1
          netmask: 255.255.255.240
      public:
        device: vmx0
        ipv4:
          address: 51.255.124.8
          gateway: 91.121.86.254
          netmask: 255.255.255.252
  roles:
  - router

dereckson mentioned this in D2826: Update api-datasources service URL.Mar 3 2023, 17:19

We also need pillar/core/network Drake content

Support for IPv6 and for dedicated baremetal instances. Rebased.

Harbormaster completed remote builds in B4789: Diff 7734.Apr 16 2023, 01:23

dereckson mentioned this in T1859: Represent IPv6 gateway in NetBox.May 8 2023, 00:41

Blocked by T1859 change.

Rebased. Use default_gateways and default_gateways_virt fields. Add support for IPv6.

Harbormaster completed remote builds in B4884: Diff 7896.May 13 2023, 16:24

black

Harbormaster completed remote builds in B4885: Diff 7897.May 13 2023, 16:25

What's missing compared to https://devcentral.nasqueron.org/source/operations/browse/main/pillar/nodes/nodes.sls?

IPv6 flags at network level
- ipv6_native`
- ipv6_tunnel
Flags at interface level
- ipv4_ovh_failover: could be derived from a new specific tag "IPFO" applied to failover IPs, and the already existing tag "OVH"
- skip_interface_configuration: used only on Eglide, currently out of scope of NetBox, but if we use this, we'll need to add Eglide too -> custom field with the same name

We then need to carefully compare roles and information, and that's ready to switch from nodes["router-001"] to node directly.

To check if GRE configuration don't try to read IPs from the nodes. If so, we can output a specific pillar entry specific for GRE tunnels, especially as we've all the needed information in the GRE interfaces.

We don't need any flag ipv6_ovh_fix, as we used instead a module method: network_utils.can_directly_be_discovered

Next: ipv6_dhcp_duid flag at interface level

Also gateways are now a multi-value field to allow IPv4 + IPv6

OK, but for hervil, we need to figure how to output 255.255.255.255 as netmask

Harbormaster completed remote builds in B5059: Diff 8221.Jun 29 2023, 22:25

OK, but for hervil, we need to figure how to output 255.255.255.255 as netmask

Harbormaster completed remote builds in B5060: Diff 8222.Jun 29 2023, 22:26

Read IPFO / OVH tags to get netmask right

Harbormaster completed remote builds in B5061: Diff 8224.Jun 29 2023, 23:33

Only require IPFO tag, READ OVH from Tenant

Harbormaster completed remote builds in B5062: Diff 8225.Jun 29 2023, 23:35

Also missing: "ipv6_dhcp_duid" flag for windriver

dereckson mentioned this in P381 Node entry for router-002 .Mon, Feb 9, 20:05

yousra mentioned this in T2203: CARP setup (IP allocation, vhid, advskew / priorité (master / backup), shared secret).Thu, Feb 19, 13:19

Fix duplicate primary IP in aliases

Harbormaster completed remote builds in B6343: Diff 10290.Fri, Feb 20, 09:56

Fix OVH + IPv6 /64 => netmask gateway /56

Harbormaster completed remote builds in B6344: Diff 10291.Fri, Feb 20, 11:10

Avoid to include nodes.sls on the review

Harbormaster completed remote builds in B6345: Diff 10292.Fri, Feb 20, 11:43

Add functions to get the CARP configurations in NetBox

Harbormaster completed remote builds in B6349: Diff 10297.Sun, Feb 22, 15:48

yousra mentioned this in D3974: Synchronize nodes.sls with NetBox pillar.Sun, Feb 22, 20:03

yousra added a reviewer: yousra.Mon, Feb 23, 08:47

I removed the priority and name entries from the generated pillar, as they are not used by Salt in our current setup.
Keeping only the fields actually consumed by the states makes the pillar cleaner and avoids maintaining redundant data.

Harbormaster completed remote builds in B6355: Diff 10303.Mon, Feb 23, 13:58

Remove an incorrect character from the code (star that shouldn't be there).

Harbormaster completed remote builds in B6356: Diff 10304.Mon, Feb 23, 14:43

Fix : when an IPv6 address belongs to the Scaleway tenant, add the
ipv6_dhcp_duid flag in the pillar

Harbormaster completed remote builds in B6357: Diff 10305.Mon, Feb 23, 15:02

yousra accepted this revision.Wed, Feb 25, 13:50

This comment was removed by yousra.

This revision is now accepted and ready to land.Wed, Feb 25, 13:50

We need to use OrderedDict from the collections module because the pillar generation script must preserve the order of the information. This ensures that the generated pillar file remains consistent and human-readable. Maintaining a stable order also makes it easier to review changes, compare versions, and debug configurations.

This revision now requires changes to proceed.Wed, Feb 25, 14:02

Updated some functions to preserve a consistent and deterministic
display order by using OrderedDict.

Harbormaster completed remote builds in B6393: Diff 10345.Mon, Mar 2, 23:49

yousra accepted this revision.Tue, Mar 3, 19:55

This revision is now accepted and ready to land.Tue, Mar 3, 19:55

WIP: Generate a pillar from NetBox information
AcceptedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 7084

utils/netbox/pillarize.py

WIP: Generate a pillar from NetBox informationAcceptedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 7084

utils/netbox/pillarize.py

WIP: Generate a pillar from NetBox information
AcceptedPublic
Actions

Revision Contents
Changeset List