Page MenuHomeDevCentral
Differential D4033

Attach OVH VIP to the CARP PRIMARY MAC
Needs ReviewPublic
Actions

Authored by yousra on Tue, Mar 31, 14:17.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Apr 28, 17:04
Unknown Object (File)
Tue, Apr 28, 17:04
Unknown Object (File)
Tue, Apr 28, 17:03
Unknown Object (File)
Tue, Apr 28, 14:43
Unknown Object (File)
Tue, Apr 28, 13:56
Unknown Object (File)
Mon, Apr 27, 14:23
Unknown Object (File)
Mon, Apr 27, 05:31
Unknown Object (File)
Sun, Apr 26, 19:40
View All Files
Subscribers
dereckson
Duranzed

Details

Reviewers
dereckson
Duranzed
Maniphest Tasks
T2276: Automate CARP VIP MAC reassignment using devd and OVH API
Summary

Add a Python script to automatically manage OVH failover IP (VIP) assignment based on CARP state changes.
The script runs on router nodes and ensures the VIP is attached to the MAC of the ACTIVE node only.

This script also send a event to Salt master for T2319.

We use here an absolute shebang (/usr/local/bin/python3) because devd runs with a limited environment and does not include /usr/local/bin in its PATH. This is an exception to the usual #!/usr/bin/env python3 pattern, which cannot work in this context.

Ref T2276

Test Plan
  • Triggered failover between routers
  • Confirmed that the VIP is correctly assigned to the CARP MASTER MAC in OVH
  • Confirmed that it is removed from the previous MAC
  • Verified that everything works as expected

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Skipped
Unit
No Test Coverage
Branch
arcpatch-D4033
Build Status
Buildable 6675
Build 6963: arc lint + arc unit

Event Timeline

yousra requested review of this revision.Tue, Mar 31, 14:17
yousra created this revision.
Harbormaster completed remote builds in B6548: Diff 10540.Tue, Mar 31, 14:17
Duranzed accepted this revision.Tue, Mar 31, 14:18
This revision is now accepted and ready to land.Tue, Mar 31, 14:18
yousra mentioned this in T2276: Automate CARP VIP MAC reassignment using devd and OVH API.Tue, Mar 31, 14:28
yousra updated this revision to Diff 10542.Tue, Mar 31, 20:12

Fix the path of the script : /usr/local/scripts/carp/carp-ovh.py

Harbormaster completed remote builds in B6550: Diff 10542.Tue, Mar 31, 20:12
dereckson requested changes to this revision.Tue, Mar 31, 21:16
dereckson mentioned this in T2292: Create syslog configuration for CARP.

If we're going to flood /var/log/messages with carp debug information, perhaps should we create a separate log topic, but that can be another change. I've created T2292.

roles/router/carp/files/carp-ovh-switch.sh
1 ↗(On Diff #10542)

This file can be skipped as we can call directly the Python script:

#   -------------------------------------------------------------
#   Application entry point
#   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


def run(interface, state):
    # here the logic part (lines 149-188)
    pass


if __name__ == "__main__":
    argc = len(sys.argv)

    if argc < 3:
        print(f"Usage: {sys.argv[0]} <subsystem> <state>", file=sys.stderr)
        sys.exit(1)

    try:
        interface = subsystem.split("@", 1)[1]
    except IndexError:
        print(f"Subsystem doesn't contain @", file=sys.stderr)
        sys.exit(2)

    run(interface, sys.argv[2])
roles/router/carp/files/carp-ovh.py
1 ↗(On Diff #10542)

For executables, best practice is to avoid to hardcode the interpreter path.

25 ↗(On Diff #10542)

Follow https://github.com/nasqueron/snippets/blob/main/python/command.py

We need to organize the script in three parts:

(1) the config
(2) the functions and helper functions applying the changes
(3) the entry point with if __name__ == "__main__"

168 ↗(On Diff #10542)

At the last iteration of the loop, delay is at 16 seconds.

We'll have waited 31 seconds + API response time.

If we abort here, we break the router as the MAC address doesn't go anywhere.

We need to retry *a lot* more time.

Don't break until 1 month.
We're already breaking with the exception if there is any issue with OVH request.

Also we need to find a way to warn for failures when we reach 128 seconds.
That's T771 job normally.

roles/router/carp/init.sls
44

There are folders for executables: /usr/local/bin or /usr/local/libexec

Executable don't have the extension

45

https://docs.saltproject.io/en/3006/ref/states/all/salt.states.file.html

Can clearly be 755 directly

This revision now requires changes to proceed.Tue, Mar 31, 21:17
yousra updated this revision to Diff 10548.Thu, Apr 2, 22:34

Many changes:

  • Improved script structure by separating configuration, helper functions, main function and application entry point.
  • Improved retry logic using a while true loop with exponential backoff instead of stopping early. The script now keeps retrying until the deletion is actually confirmed, avoiding inconsistent states where previously, the script could attempt to add the VIP while the deletion from the previous MAC was still in progress. This could make the add operation being rejected, leaving the VIP temporarily not attached to any MAC. So the script now waits and retries until the deletion is fully completed before attempting the addition.
  • Added a warn when we reach 128 seconds and VIP still not removed or added.
  • Set correct permissions (not 0 in front and string (like '0644') for Salt (more like 644), only on Ansible) for the script.
  • Executable files do not require extensions on Unix systems, as execution is determined by the shebang.
Harbormaster completed remote builds in B6556: Diff 10548.Thu, Apr 2, 22:34
yousra mentioned this in D4034: Add debug scripts for Vault, OVH, and VIP assignment.Mon, Apr 6, 15:55
yousra updated this revision to Diff 10570.EditedMon, Apr 6, 16:14

When testing the script manually, it’s useful to see the logs directly in the terminal, while still sending them to system logs like sudo python3 carp-ovh-failover 2@vmx1 MASTER

Harbormaster completed remote builds in B6577: Diff 10570.Mon, Apr 6, 16:14
dereckson requested changes to this revision.Sat, Apr 18, 16:35

March 31 review comments seem to still need action - see https://devcentral.nasqueron.org/D4033#63080

roles/router/carp/files/carp-ovh-failover
1 ↗(On Diff #10570)

#!/usr/bin/env python3

This revision now requires changes to proceed.Sat, Apr 18, 16:35
yousra updated this revision to Diff 10661.Sat, Apr 18, 17:01

!/usr/local/bin/python3 uses a fixed Python path, while !/usr/bin/env python3 finds Python dynamically from the system’s PATH,
making it more portable (Python can be installed elsewhere depending on the OS)

Harbormaster completed remote builds in B6637: Diff 10661.Sat, Apr 18, 17:01
yousra updated this revision to Diff 10668.EditedSat, Apr 18, 20:03
  • use config dictionary directly instead of intermediate variables
  • rename script to .py in repository
  • add in config.yaml, the path of the secret
Harbormaster completed remote builds in B6641: Diff 10668.Sat, Apr 18, 20:03
yousra updated this revision to Diff 10671.Sat, Apr 18, 22:00

The CARP OVH failover script remains in the folder libexec as it is an internal executable triggered by devd (UNIX conventions).

The configuration file yaml has been moved to /usr/local/etc/carp/config.yaml, as files yaml should not be stored
in libexec but in etc according to UNIX conventions.

To do that we use the map.jinja to retrieve directory paths (dirs).

Harbormaster completed remote builds in B6644: Diff 10671.Sat, Apr 18, 22:00
yousra updated this revision to Diff 10684.Sun, Apr 19, 16:19

Actually, this config CARP will work only in FreeBSD machine because DEVD works only in FreeBSD, so it is not necessary to use dir from map.jinja,
we can put manually the path directory for the scrit carp-ovh-failover and config.yaml.

Harbormaster completed remote builds in B6649: Diff 10684.Sun, Apr 19, 16:19
yousra updated this revision to Diff 10696.Sun, Apr 19, 19:44

Use an absolute shebang (/usr/local/bin/python3) because devd runs with a limited environment and does not include /usr/local/bin in its PATH.
This is an exception to the usual #!/usr/bin/env python3 pattern, which cannot work in this context.

Harbormaster completed remote builds in B6658: Diff 10696.Sun, Apr 19, 19:44
yousra edited the summary of this revision. (Show Details)Sun, Apr 19, 19:46
yousra edited the summary of this revision. (Show Details)Mon, Apr 20, 11:20
yousra edited the test plan for this revision. (Show Details)
yousra mentioned this in T2319: Automate GRE tunnel failover on CARP PRIMARY router.Sun, Apr 26, 13:52
yousra updated this revision to Diff 10717.Sun, Apr 26, 16:50

For T2319, I added a function to send event to Salt master.

Harbormaster completed remote builds in B6674: Diff 10717.Sun, Apr 26, 16:50
yousra edited the summary of this revision. (Show Details)Sun, Apr 26, 16:51
yousra updated this revision to Diff 10718.Tue, Apr 28, 08:02

Modified send_salt_event to use the full salt-call path :

The event was sent correctly when running the script manually, but not when triggered by devd.
Using the absolute path fixes this because devd does not use the same environment variables as a normal shell.

Harbormaster completed remote builds in B6675: Diff 10718.Tue, Apr 28, 08:02
yousra updated this revision to Diff 10719.Wed, Apr 29, 12:33

Fix the tag of the event to network/carp/primary

Harbormaster completed remote builds in B6676: Diff 10719.Wed, Apr 29, 12:33
yousra retitled this revision from Attach OVH VIP to the CARP MASTER MAC to Attach OVH VIP to the CARP PRIMARY MAC.Wed, Apr 29, 12:41

Revision Contents
Changeset List

PathSize
roles/
router/
carp/
files/
268 lines
27 lines
12 lines

Diff 10718

View Options

roles/router/carp/files/carp-ovh-failover.py

Loading...
View Options

roles/router/carp/files/config.yaml

Loading...
View Options

roles/router/carp/init.sls

Loading...
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator