Page MenuHomeDevCentral
Create Task
Maniphest T2319

Automate GRE tunnel failover on CARP PRIMARY router
Open, NormalPublic
Actions

Description

The goal of this task is to implement an automated failover mechanism that ensures GRE tunnels always point to the current CARP PRIMARY.

When a router transitions to the ACTIVE state, a devd-triggered script implemented in D4033 will emit a Salt event. This event is received by the Salt master, which reacts using a reactor to trigger GRE tunnel reconfiguration on Ysul and Windriver.

The reconfiguration process will:

  1. Remove the existing GRE tunnel
  2. Recreate a new tunnel toward the new ACTIVE router
  3. Reload IPsec

This approach ensures that tunnel configuration dynamically follows CARP state changes, avoiding manual intervention and reducing downtime during failover events.

Steps :

  • 1. Send a test Salt event from a router to validate event emission
sudo salt-call event.send 'test/carp' '{"router": "router-003"}'
  • 2. Verify that the event is correctly received on the Salt master event bus
salt-run state.event pretty=True
  • 3. Integrate the event emission into the devd-triggered script (D4033)
  • 4. Configure a test Salt reactor to listen for the carp/master event and trigger an action
  • 5. Implement a script to handle GRE tunnel reconfiguration ( D4098)
  • 6. Trigger the reconfiguration from the reactor upon event reception (D4098)
  • 7. Test the full failover scenario (CARP switch) and validate tunnel recreation

References :

https://docs.saltproject.io/en/3007/ref/modules/all/salt.modules.event.html
https://mpolinowski.github.io/docs/DevOps/Salt/2020-06-20--salt-reactor-events/2020-06-20/

Revisions and Commits

rOPS Nasqueron Operations
AcceptedD4098 Add Salt reactor for GRE tunnel creation on CARP failover

Related Objects

Event Timeline

Duranzed triaged this task as Normal priority.Wed, Apr 22, 12:26
Duranzed created this task.
yousra moved this task from Backlog to Working on on the Secure HA tunnels board.Fri, Apr 24, 07:30
yousra mentioned this in T2302: CARP + GRE + OSPF conflict.Fri, Apr 24, 08:20
yousra added projects: Drake network, Salt, Python.Fri, Apr 24, 08:26
yousra added a project: Monitoring and reporting.
yousra updated the task description. (Show Details)Sun, Apr 26, 13:52
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Sun, Apr 26, 14:00
yousra updated the task description. (Show Details)Sun, Apr 26, 14:03
yousra added a comment.EditedSun, Apr 26, 14:19

Test to validate Salt event emission and reception on the master :

yousra@complector /opt/salt/nasqueron-operations]$ salt-run state.event pretty=True

salt/auth	{
    "_stamp": "2026-04-26T14:27:48.132991",
    "act": "accept",
    "id": "router-003",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmiiNW666p/EZ3lVD1cCi\nCYsrF+V4v4eiZG+5fKXVJAoknyrcQ7ED6LhAagR1aqWI9KkEVALKM8TMB7qT827A\n/c/+YnQauVADmdjOkEs7DIaJa5RQZgx4qEmqHPryJUGCkS8AfxF8H6YoM8CUyTuR\na2JrLrMerXqHyFC/ZNt9yIujXhZowIRTR7WQCd6OGShnUH/uKi8TSc8yjwfqGUdM\nQJ0HJjjVNaZ8XfNox4ml6Q1acMCZtTOH5/LHstqrIp3Y87g8bLtnk0jJbp8Hbfbj\nwDhhXZYt2W9eEdCdEMhgPr4campuuE9DrMV3UfRieXPSrRb3gqCpyLy8EXQDZtX7\nJQIDAQAB\n-----END PUBLIC KEY-----",
    "result": true
}
minion/refresh/router-003	{
    "Minion data cache refresh": "router-003",
    "_stamp": "2026-04-26T14:27:49.020788"
}
test/carp	{
    "_stamp": "2026-04-26T14:27:49.150709",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20260426142749137759",
        "__pub_pid": 6144,
        "__pub_tgt": "salt-call",
        "router": "router-003"
    },
    "id": "router-003",
    "tag": "test/carp",
    "ts": 1777213669
}
salt/job/20260426142749162569/ret/router-003	{
    "_stamp": "2026-04-26T14:27:49.164093",
    "arg": [
        "test/carp",
        "{\"router\": \"router-003\"}"
    ],
    "cmd": "_return",
    "fun": "event.send",
    "fun_args": [
        "test/carp",
        "{\"router\": \"router-003\"}"
    ],
    "id": "router-003",
    "jid": "20260426142749162569",
    "retcode": 0,
    "return": true,
    "tgt": "router-003",
    "tgt_type": "glob",
    "ts": 1777213669
}

[yousra@router-003 ~]$ sudo salt-call event.send 'test/carp' '{"router": "router-003"}'
local:

True
yousra updated the task description. (Show Details)Sun, Apr 26, 14:21
yousra updated the task description. (Show Details)Sun, Apr 26, 14:34
yousra mentioned this in D4033: Attach OVH VIP to the CARP PRIMARY MAC.Sun, Apr 26, 16:50
yousra updated the task description. (Show Details)Sun, Apr 26, 16:52
yousra updated the task description. (Show Details)Sun, Apr 26, 17:24
yousra added a comment.EditedSun, Apr 26, 18:19

I created and tested a Salt reactor that listens for the carp/master event sent by the routers. For now, the reactor only runs a test command on Ysul and Windriver to confirm that the event is correctly received and that the master can trigger actions on those hosts.

The next step will be to replace this test command with the real GRE tunnel reconfiguration script

[yousra@router-003 /usr/local/libexec/carp]$ sudo /usr/local/libexec/carp/carp-ovh-failover 2@vmx1 MASTER

Detected MAC on vmx1: 00:50:56:09:98:fc
Checking current state...
Checking IPs for MAC 00:50:56:09:98:fc
OVH returned: ['51.68.252.230', '178.32.70.111']
VIP is already on correct MAC -> nothing to do
Sending Salt event: carp/master {"router": "router-003"}
local:
    True

On /usr/local/etc/salt/master.d/carp-master-reactor.conf

reactor:
  - 'carp/master':
    - /srv/reactor/gre.sls

On /srv/reactor/gre.sls

test_reactor:
  local.cmd.run:
    - tgt: 'ysul,windriver'
    - tgt_type: list
    - arg:
      - 'echo OK_CARP_MASTER_{{ data["data"]["router"] }} >> /tmp/test-reactor.log'

Result : the action was successfully executed

[yousra@windriver /tmp]$ cat test-reactor.log 
OK_CARP_MASTER_router-003
OK_CARP_MASTER_router-003
OK_CARP_MASTER_router-003
[yousra@ysul /tmp]$ cat test-reactor.log 
OK_CARP_MASTER_router-003
OK_CARP_MASTER_router-003
OK_CARP_MASTER_router-003

@dereckson I am not sure about the paths, could you check please ?

yousra updated the task description. (Show Details)Sun, Apr 26, 19:02
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Sun, Apr 26, 19:05
Duranzed added a comment.Mon, Apr 27, 12:37

When creating a GRE tunnel to the alias IP of ysul as an endpoint the tunnel is unpingable however when creating the GRE tunnel using the public IP of ysul, GRE tunnel responds well to ping.
I suspect that the problem might come from using an alias IP as GRE endpoint that might cause this as it suggest encapsulation/decapsulation issues

	inet 163.172.49.16 netmask 0xffffff00 broadcast 163.172.49.255
	inet 212.83.187.132 netmask 0xffffffff broadcast 212.83.187.132
Duranzed mentioned this in T2322: Create log file for GRE tunnel creation script.Mon, Apr 27, 14:59
Duranzed added a comment.Mon, Apr 27, 15:10

We noticed that Windriver is unable to ping the public IP addresses of router-002 and router-003. However, GRE tunnel creation is successful, and tunnel connectivity works with router-002, although pinging its public IP is still unsuccessful.

yousra updated the task description. (Show Details)Tue, Apr 28, 07:45
yousra updated the task description. (Show Details)Tue, Apr 28, 08:11
yousra updated the task description. (Show Details)
yousra renamed this task from Automate GRE tunnel failover on CARP master switch to Automate GRE tunnel failover on CARP PRIMARY router.Wed, Apr 29, 12:18
yousra claimed this task.
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Wed, Apr 29, 12:34
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Wed, Apr 29, 12:36
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Wed, Apr 29, 12:39
yousra added a revision: D4098: Add Salt reactor for GRE tunnel creation on CARP failover.Wed, Apr 29, 17:55
yousra updated the task description. (Show Details)
yousra updated the task description. (Show Details)Sun, May 3, 16:46
yousra added a comment.Sun, May 3, 16:48

Now, regardless of whether router-002 or router-003 is the primary router, the tunnel with Windriver works perfectly. However, there is still an issue with Ysul: the tunnel works when router-003 is primary, but it does not work when router-002 is primary.

yousra updated the task description. (Show Details)Sun, May 3, 16:49
yousra added a comment.Sun, May 3, 16:52

@dereckson While testing and modifying the GRE tunnel, it took so long on Ysul that SSH stopped responding, and we lost access to the machine.

yousra updated the task description. (Show Details)Mon, May 4, 11:53
yousra mentioned this in T2303: Installation and configuration of FRRouting.Mon, May 4, 14:01
yousra moved this task from Working on to Done on the Secure HA tunnels board.Tue, May 5, 18:20
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator