Refresh Sentry installation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dereckson
	Mar 7 2023, 17:47

Description

Sentry was installed in 2016 as Docker containers.

Notifications Center uses it to report exceptions, and there are plans for:

Zed: report exceptions (old code with Raven lib proof of concept to modernize)
Auth Grove: same configuration than Notifications Center (but see T1771 moratoire on Auth Grove dev)
Wearg / Dæghrefn: check for new errorInfo content (see T1790)

It's especially important for Wearg, Dæghrefn and Notifications Center as humans don't have access to the errors when it's automatically processed.

We've some Gb of data of repeating errors from Notifications Center in pgsql, a db schema not compatible with 9.1.2 version (the stop before switching to CalVer). Interest of those are dubious, as Dorian updated the code to modern Laravel, and as I've hunted and solved Notifications Center / Phabricator main issues.

Meanwhile, Sentry codebase evolved. The old image seems to be stucked to 9.1.2 3 years ago

Instead of upgrading, I suggest to reinstall it from scratch, especially as we don't have any used history or applications reporting.

Then setup reporting for the applications needing this service the most.

Revisions and Commits

rDSENTRY Docker image for Sentry
	D2901	rDSENTRYae68f9289508 Build a Sentry + hvac Docker image
rDPGSENTRY PostgreSQL Docker image for Sentry
	D2867	rDPGSENTRYc7e5a06f8159 Configure client authentication
	D2866	rDPGSENTRY956f458e66d3 Use preferred exec form for CMD
	D2864	rDPGSENTRY1d59efe54d37 Provide PostgreSQL and wal2json Docker image
rOPS Nasqueron Operations
	D2945	rOPSdf873afbe1d2 Trace every request for Notifications Center to Sentry
	D2944	rOPSd7e6cddec8b7 Provide SENTRY_DSN for Notifications Center
	D2908	rOPS2517ada8c0e0 Pull nasqueron/sentry image
	D2888	rOPS249e7f0fe3e8 Deploy Relay container for Sentry
	D2885	rOPS3674f0694509 Upgrade Sentry to 23.2.0
	D2886	rOPS992f97a678f1 Connect PostgreSQL to Sentry network
	D2882	rOPSf67aaac6854f Deploy Symbolicator container for Sentry
	D2881	rOPS9693fb5f8d66 Deploy Snuba containers for Sentry
	D2875	rOPS0751ed08f600 Prune geoipupdate container after use
	D2874	rOPSe542d69a0933 Fetch GeoIP databases
	D2873	rOPSaa82d0a5723c Deploy ClickHouse container for Sentry
	D2872	rOPS93d77ac8df79 Fix ZooKeeper data volume path
	D2871	rOPS0ce2c4dad220 Provide Kafka and ZooKeeper instances for Sentry
	D2865	rOPS573cff29c5b5 Switch to specialized Sentry PostgreSQL and wal2json Docker image
	D2861	rOPSe26f76557ce5 Configure network and health check for Redis
	D2859	rOPSec1f0fce0832 Deploy memcached container for Sentry
	D2858	rOPS1e938695e69e Use 172.18.3.0/24 as Docker network for Sentry containers
	D2857	rOPSb104a9d10f59 Update Exim MTA container configuration
rNOTIF Notifications center
	D2935	rNOTIF22b5718666cd Switch Sentry implementation to sentry-laravel

Related Objects
Search...

Status	Assigned	Task
Resolved	dereckson	T1791 Refresh Sentry installation
Open	None	T1790 Trace eggdrop errors
Resolved	dereckson	T1805 Sentry can't send mail anymore
Resolved	dereckson	T1806 Integrate Sentry and GitHub

Event Timeline

dereckson triaged this task as Normal priority.Mar 7 2023, 17:47

dereckson created this task.

dereckson renamed this task from Refresh Sentry installation to CalVer modern releases to Refresh Sentry installation.Mar 7 2023, 17:51

dereckson added a subtask: T1790: Trace eggdrop errors.

dereckson updated the task description. (Show Details)

dereckson added a revision: D2857: Update Exim MTA container configuration.Mar 7 2023, 22:06

dereckson added a commit: rOPSb104a9d10f59: Update Exim MTA container configuration.Mar 7 2023, 22:11

dereckson added a revision: D2858: Use 172.18.3.0/24 as Docker network for Sentry containers.Mar 7 2023, 22:46

dereckson added a commit: rOPS1e938695e69e: Use 172.18.3.0/24 as Docker network for Sentry containers.Mar 7 2023, 22:54

dereckson added a revision: D2859: Deploy memcached container for Sentry.Mar 8 2023, 00:31

dereckson added a revision: D2860: Deploy memcached container for Sentry.Mar 8 2023, 00:49

dereckson added a commit: rOPSec1f0fce0832: Deploy memcached container for Sentry.Mar 8 2023, 00:54

dereckson removed a revision: D2860: Deploy memcached container for Sentry.Mar 8 2023, 00:54

dereckson added a revision: D2861: Configure network and health check for Redis.Mar 8 2023, 01:00

dereckson added a commit: rOPSe26f76557ce5: Configure network and health check for Redis.Mar 8 2023, 01:05

PostgreSQL for Sentry requires a custom extension, wal2json.

Created new repository rDPGSENTRY to host nasqueron/postgres-sentry Docker image.

dereckson added a revision: D2864: Provide PostgreSQL and wal2json Docker image.Mar 8 2023, 23:32

dereckson added a commit: rDPGSENTRY1d59efe54d37: Provide PostgreSQL and wal2json Docker image.Mar 8 2023, 23:33

dereckson added a revision: D2865: Switch to specialized Sentry PostgreSQL and wal2json Docker image.Mar 8 2023, 23:47

dereckson added a revision: D2866: Use preferred exec form for CMD.Mar 8 2023, 23:59

dereckson added a commit: rDPGSENTRY956f458e66d3: Use preferred exec form for CMD.Mar 9 2023, 00:02

dereckson added a revision: D2867: Configure client authentication.Mar 9 2023, 01:51

dereckson added a commit: rOPS573cff29c5b5: Switch to specialized Sentry PostgreSQL and wal2json Docker image.Mar 9 2023, 01:57

dereckson added a commit: rDPGSENTRYc7e5a06f8159: Configure client authentication.

dereckson added a revision: D2871: Provide Kafka and ZooKeeper instances for Sentry.Mar 9 2023, 23:14

dereckson added a commit: rOPS0ce2c4dad220: Provide Kafka and ZooKeeper instances for Sentry.Mar 9 2023, 23:19

dereckson added a revision: D2872: Fix ZooKeeper data volume path.Mar 9 2023, 23:43

dereckson added a commit: rOPS93d77ac8df79: Fix ZooKeeper data volume path.Mar 9 2023, 23:43

Current status

Deployed a PostgreSQL specific image with wal2json
Added memcached, Kafka, Zookeeper on the mix
Reported upstream leftover discovered during the work (https://github.com/getsentry/self-hosted/pull/2020)

Next steps

Deploy Sentry new container
Import the logic in getsentry/self-hosted install/ in rOPS (Kafka topics and wal2json parts done)

dereckson added a revision: D2873: Deploy ClickHouse container for Sentry.Mar 10 2023, 01:19

dereckson added a commit: rOPSaa82d0a5723c: Deploy ClickHouse container for Sentry.Mar 10 2023, 01:36

dereckson added a revision: D2874: Fetch GeoIP databases.Mar 10 2023, 20:06

dereckson added a commit: rOPSe542d69a0933: Fetch GeoIP databases.Mar 10 2023, 20:11

dereckson added a revision: D2875: Prune geoipupdate container after use.Mar 11 2023, 00:48

dereckson added a commit: rOPS0751ed08f600: Prune geoipupdate container after use.Mar 11 2023, 00:48

dereckson added a revision: D2881: Deploy Snuba containers for Sentry.Mar 12 2023, 01:42

dereckson added a commit: rOPS9693fb5f8d66: Deploy Snuba containers for Sentry.Mar 12 2023, 10:32

dereckson added a revision: D2882: Deploy Symbolicator container for Sentry.Mar 12 2023, 22:54

dereckson added a commit: rOPSf67aaac6854f: Deploy Symbolicator container for Sentry.Mar 12 2023, 23:41

dereckson added a revision: D2885: Upgrade Sentry to 23.2.0.Mar 13 2023, 18:22

dereckson added a revision: D2886: Connect PostgreSQL to Sentry network.Mar 14 2023, 17:32

dereckson added a commit: rOPS992f97a678f1: Connect PostgreSQL to Sentry network.Mar 14 2023, 17:33

dereckson added a commit: rOPS3674f0694509: Upgrade Sentry to 23.2.0.Mar 14 2023, 18:55

dereckson added a revision: D2888: Deploy Relay container for Sentry.Mar 14 2023, 19:59

dereckson added a commit: rOPS249e7f0fe3e8: Deploy Relay container for Sentry.Mar 14 2023, 23:23

A test exception has been sent from Notifications Center to Sentry successfully:
https://sentry.nasqueron.org/organizations/nasqueron/issues/1

Without login, it can be shown at https://sentry.nasqueron.org/share/issue/24e8af88fdf04636a37aeffccf78c4eb/

T1796 has been created from Sentry UI, so the Sentry/Phabricator integration works.
Alken-Orin account description has been accordingly updated.

dereckson added a revision: D2901: Build a Sentry + hvac Docker image.Mar 16 2023, 19:06

dereckson added a commit: rDSENTRYae68f9289508: Build a Sentry + hvac Docker image.Mar 16 2023, 19:07

dereckson added a revision: D2908: Pull nasqueron/sentry image.Mar 25 2023, 08:48

dereckson added a commit: rOPS2517ada8c0e0: Pull nasqueron/sentry image.Mar 25 2023, 08:49

Current status

Front-end seems to work fine and receive payloads from Notifications Center.

T1802 shows it can be upgraded.

Still to check and configure

What about CDC? According https://github.com/getsentry/cdc we should have a cdc process somewhere. Work has been prepared at https://github.com/getsentry/self-hosted/commit/8dc84600c5ceb7063813d18657011fa53a659d3f for PostgreSQL, but cdc process doesn't run.
- It needs replication access, self-hosted used a blind trust, we require regular authentication, so container must have PostgreSQL credentials too in CDC_POSTGRES_DSN_TEMPLATE environment variable. Reference cdc docker-compose uses postgres://postgres:password@postgres:5432/{database}. We should add to credentials mod a postgres_get_dsn(hostname="localhost", database=None) returning postgres://...:...@sentry_db:5432/ and append {database} (hence the "template" variable name).