Page MenuHomeDevCentral

Deploy Anubis for DevCentral
Needs ReviewPublic

Authored by ptdradmin on Fri, Feb 6, 16:32.

Details

Reviewers
dereckson
Maniphest Tasks
T2193: Investigate Anubis
Summary

Integrate Anubis as a WAF/Reverse Proxy for DevCentral to secure traffic
and challenge AI scrapers.

Ref T2193.

Test Plan

Initial proof of concept verified on Dwellers.

Tests for production:

  • Anubis service starts and populates socket
  • Sockets permissions are correct
  • It's possible to connect to the socket and get challenge, then site
  • Bot policies correctly filter and challenge traffic via curl tests
  • nginx redirects correctly to Anubis instead of the site directly

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Skipped
Unit
No Test Coverage
Branch
master
Build Status
Buildable 6238
Build 6522: arc lint + arc unit

Event Timeline

ptdradmin created this revision.
This comment was removed by ptdradmin.
dereckson requested changes to this revision.Fri, Feb 6, 17:32
dereckson added inline comments.
pillar/paas/docker/docker-002/main.sls
312

It's under docker_images, so make sense only if we deploy Anubis as a Docker image.

(In that case, could be complicated to get the socket path)


If the goal is to give a new configuration for Anubis deployed outside Docker, this pillar is fine, but we need to put it at the same level than docker_images (so one less tab)

313

Format is:

docker_images:
  <service name>:
    <container name>:

The container name devcentral is already used, so I guess that would be something like anubis_devcentral

(but only if we use Docker)

roles/paas-docker/anubis.sls
1 ↗(On Diff #10126)

Should be put in a roles/paas-docker/anubis/ directory.

We can use:

  • one file, so this would be roles/paas-docker/anubis/init.sls
  • different files living in roles/paas-docker/anubis (to split software/services/config)

I see we're at +- 40 lines so one file is good

11 ↗(On Diff #10126)

That's the name of the pillar entry, so in that case, in pillar/paas/docker/docker-002/main.sls we should have something like:

anubis_instances:

devcentral:
  socket: /run/anubis/devcentral.sock
  policies_file: /usr/local/etc/anubis/devcentral.yaml
roles/paas-docker/anubis/files/env.j2
1 ↗(On Diff #10126)

Ansible convention is to use .j2 extension for Jinja2 templates.

In our Salt repository, I see two conventions:

  • to directly use the filename for config, text
  • to append .jinja for scripts, to bypass linter (e.g. mw.sh.jinja to avoid shellcheck to lint mw.sh and reports {{ is an errror)

Here it would be safe to use something like "instance.env" as filename.

7 ↗(On Diff #10126)

So we have a target key in the anubis_instances pillar.

At this stage I imagine something like:

anubis_instances:
  devcentral:
    socket: /run/anubis/devcentral.sock
    policies_file: /usr/local/etc/anubis/devcentral.yaml
    target: http://localhost:31080

Or:

anubis_instances:
  devcentral:
    socket: /run/anubis/devcentral.sock
    policies_file: /usr/local/etc/anubis/devcentral.yaml
    target:
      type: docker # we ignore it for now, but in the future if we've a no Docker we can add target logic
      service: phabricator
      container: devcentral

The target would then be http://localhost:{{ docker_containers[config[service]][config[container]]["app_port"] }}

And we automatically grab app_port from docker_containers pillar.

10 ↗(On Diff #10126)

Same logic than app_port grabbing, but host instead

15 ↗(On Diff #10126)

We need to provide logic to provision those two credentials into Vault.

Something like in fix_anubis_devcentral.sh, but with a write in Vault to run before provisioning this.

18 ↗(On Diff #10126)

Can we try with a UNIX socket? Like in /run/anubis/metrics/?

(That's perhaps that one we didn't succeed to query)

If we can do that, that will avoid to maintain a new ports table for the Anubis metrics.

roles/paas-docker/anubis/files/policies.yaml.j2
11 ↗(On Diff #10126)

No, no, no, we're especially heavily attacked on the files :/

14 ↗(On Diff #10126)

Same, they query A LOT the source code files in Diffusion.

I guess the intent of the LLM is to get "public access", but our software isn't intended to be downloaded from DevCentral directly so we're good there

17 ↗(On Diff #10126)

We're in two scenarii:

So that rule to allow all traffic isn't useful neither

20 ↗(On Diff #10126)

Hmmmm, those aren't really annoying, the ones that are annoying are the stealth ones, the ones that masquerade themselves as legit browser traffic

roles/paas-docker/nginx/files/vhosts/phabricator.conf
47
scripts/fix_anubis_devcentral.sh
2

Hmmm

Actually, that one conflicts with everything else.

For example lines 19 to 34 offers to do the job we already do with the env file above.

What we would need is perhaps to document or script the keys generation part, the two openssl rand -base64 32, and write it to Vault afterwards (vault kv write).

This revision now requires changes to proceed.Fri, Feb 6, 17:32
dereckson edited the test plan for this revision. (Show Details)
dereckson awarded a token.
ptdradmin edited the test plan for this revision. (Show Details)

Summary: Refactored Anubis deployment according to review feedback.

  • Moved to roles/paas-docker/anubis/init.sls
  • Fixed pillar indentation and schema
  • Added dynamic port detection from docker_containers
  • Switched metrics to UNIX socket
  • Simplified provisioning script for Vault only
  • Cleaned up bot policies

.

  • Refactor Anubis deployment based on review feedback. Highlights: fixed pillar indentation, moved to init.sls, added dynamic port detection, and configured metrics socket.
  1. Updating D3908: Deploy Anubis for DevCentral #
  2. Enter a brief description of the changes included in this update.
  3. The first line is used as subject, next lines as comment. #
  4. If you intended to create a new revision, use:
  5. $ arc diff --create

:wq

^O
^X