Add a Python script to automatically manage OVH failover IP (VIP) assignment based on CARP state changes.
The script runs on router nodes and ensures the VIP is attached to the MAC of the MASTER node only.
Ref T2276
Details
Details
- Reviewers
dereckson Duranzed - Maniphest Tasks
- T2276: Automate CARP VIP MAC reassignment using devd and OVH API
- Triggered failover between routers
- Confirmed that the VIP is correctly assigned to the MASTER MAC in OVH
- Confirmed that it is removed from the previous MAC
- Verified that everything works as expected
Diff Detail
Diff Detail
- Repository
- rOPS Nasqueron Operations
- Lint
Lint Skipped - Unit
No Test Coverage - Branch
- script-ovh-carp
- Build Status
Buildable 6577 Build 6861: arc lint + arc unit
Event Timeline
Comment Actions
If we're going to flood /var/log/messages with carp debug information, perhaps should we create a separate log topic, but that can be another change. I've created T2292.
| roles/router/carp/files/carp-ovh-switch.sh | ||
|---|---|---|
| 1 ↗ | (On Diff #10542) | This file can be skipped as we can call directly the Python script: # ------------------------------------------------------------- # Application entry point # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - def run(interface, state): # here the logic part (lines 149-188) pass if __name__ == "__main__": argc = len(sys.argv) if argc < 3: print(f"Usage: {sys.argv[0]} <subsystem> <state>", file=sys.stderr) sys.exit(1) try: interface = subsystem.split("@", 1)[1] except IndexError: print(f"Subsystem doesn't contain @", file=sys.stderr) sys.exit(2) run(interface, sys.argv[2]) |
| roles/router/carp/files/carp-ovh.py | ||
| 1 ↗ | (On Diff #10542) | For executables, best practice is to avoid to hardcode the interpreter path. |
| 25 ↗ | (On Diff #10542) | Follow https://github.com/nasqueron/snippets/blob/main/python/command.py We need to organize the script in three parts: (1) the config |
| 168 ↗ | (On Diff #10542) | At the last iteration of the loop, delay is at 16 seconds. We'll have waited 31 seconds + API response time. If we abort here, we break the router as the MAC address doesn't go anywhere. We need to retry *a lot* more time. Don't break until 1 month. Also we need to find a way to warn for failures when we reach 128 seconds. |
| roles/router/carp/init.sls | ||
| 44 | There are folders for executables: /usr/local/bin or /usr/local/libexec Executable don't have the extension | |
| 45 | https://docs.saltproject.io/en/3006/ref/states/all/salt.states.file.html Can clearly be 755 directly | |
Comment Actions
Many changes:
- Improved script structure by separating configuration, helper functions, main function and application entry point.
- Improved retry logic using a while true loop with exponential backoff instead of stopping early. The script now keeps retrying until the deletion is actually confirmed, avoiding inconsistent states where previously, the script could attempt to add the VIP while the deletion from the previous MAC was still in progress. This could make the add operation being rejected, leaving the VIP temporarily not attached to any MAC. So the script now waits and retries until the deletion is fully completed before attempting the addition.
- Added a warn when we reach 128 seconds and VIP still not removed or added.
- Moved hardcoded configuration (service, VIP, MAC addresses, Vault config) into a YAML file (method PyYAML) https://www.freecodecamp.org/news/how-to-work-with-yaml-in-python-a-guide-with-examples/
- According to https://man.freebsd.org/cgi/man.cgi?query=hier&sektion=7, /usr/libexec contains utilities executed by programs, making it the appropriate location for this script.
- Set correct permissions (not 0 in front and string (like '0644') for Salt (more like 644), only on Ansible) for the script.
- Adapt the script on how should an executable UNIX be : https://github.com/nasqueron/snippets/blob/main/python/command.py
- Executable files do not require extensions on Unix systems, as execution is determined by the shebang.
Comment Actions
When testing the script manually, it’s useful to see the logs directly in the terminal, while still sending them to system logs like sudo python3 carp-ovh-failover 2@vmx1 MASTER