Page MenuHomeDevCentral

Implement blue/green deployment or immutable artefacts for router-001
Open, NormalPublic

Description

As the router allows every service to communicate with each other, we want to optimise the procedure to minimise as much as possible the downtime.

Immutable artefact

Each router VM is deployed once, not upgraded. When we want to update:

  • Spawn a new VM
  • Validate it
  • Move the IP and tunnels on it
  • Destroy the previous one

Blue/green deployment

Maintain two VMs, one "blue", one "green".

We start with current router-001 as blue.
Active router is blue.

We create a new VM router-002 as green.
Active router is still blue.

Each machines is connected both to green and blue:

  • if blue/green has a dedicated IP, each machine needs to have routes to both, and we update the route for 172.27.27.0/24
  • we can also have a router virtual IP and the active server got it, the other release it

Maintenance occurs on green.
When maintenance is done, we promote green as active, ie we update the routes or green takes the IP.
Active router is green.

Next maintenance will occur on blue this time.

Event Timeline

dereckson triaged this task as Normal priority.Jan 10 2024, 22:08
dereckson created this task.

router-001 update would cut network connections between Ysul, WindRiver, CloudHugger and IntraNought VMs.

I think that's actually acceptable, as production services can reach web-001/db-*/ directly through their dedicated network card.

Impact:

  • IRC bots wouldn't be able to reach MySQL or Vault as they're still on Ysul
  • Services still on Ysul can't reach
  • Development servers can't reach production services, or Dwellers

Another issue is we lose connections to all those machines, as router-001 is used to route traffic to them.

Probably best to ship router-002 under FreeBSD 14 so we can prepare to minimize this impact if we need more hypervisors in the future.

dereckson lowered the priority of this task from Normal to Low.Jan 10 2024, 22:11
dereckson raised the priority of this task from Low to Normal.Jul 27 2024, 20:38

Increasing priority as FreeBSD 13.2 is now EOL for one month (2024-06-30).

Create router-002 under FreeBSD 14.1 and see how it behaves with Salt and NetBox-based tunnel information would be nice:

  • that would give a testing ground to deploy correctly without risk roles/core/network
  • we'll be able to see how we can switch traffic to this new router progressively