Page MenuHomeDevCentral
Differential D2881

Deploy Snuba containers for Sentry
ClosedPublic
Actions

Authored by dereckson on Mar 12 2023, 01:42.
Tags
None
Referenced Files
F3762043: D2881.id7340.diff
Thu, Nov 21, 06:39
Unknown Object (File)
Wed, Nov 20, 09:21
Unknown Object (File)
Tue, Nov 19, 18:25
Unknown Object (File)
Tue, Nov 19, 12:56
Unknown Object (File)
Tue, Nov 19, 05:52
Unknown Object (File)
Tue, Nov 19, 05:05
Unknown Object (File)
Mon, Nov 18, 18:37
Unknown Object (File)
Mon, Nov 18, 18:37
View All Files
Subscribers
None

Details

Reviewers
dereckson
Maniphest Tasks
T1791: Refresh Sentry installation
Commits
rOPS9693fb5f8d66: Deploy Snuba containers for Sentry
Summary

Snuba is a front-end on the top of ClickHouse used by Sentry
to do OLAP analysis.

Each container uses the same image, with a different launch command.

Reference: https://blog.sentry.io/2019/05/16/introducing-snuba-sentrys-new-search-infrastructure/

Ref T1791

Test Plan

Deploy on docker-002

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dereckson requested review of this revision.Mar 12 2023, 01:42
dereckson created this revision.
Harbormaster completed remote builds in B4544: Diff 7338.Mar 12 2023, 01:42
dereckson updated this revision to Diff 7339.Mar 12 2023, 01:46

Remove port bindings

Harbormaster completed remote builds in B4545: Diff 7339.Mar 12 2023, 01:46
dereckson added a comment.Mar 12 2023, 01:53
docker_002
$ snuba_containers="sentry_snuba_api sentry_snuba_consumer sentry_snuba_outcomes_consumer sentry_snuba_sessions_consumer sentry_snuba_transactions_consumer sentry_snuba_replays_consumer sentry_snuba_replacer sentry_snuba_subscription_consumer_events sentry_snuba_subscription_consumer_sessions sentry_snuba_subscription_consumer_transactions"
$ docker stats $snuba_containers
CONTAINER ID   NAME                                              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
d7fb9bf79329   sentry_snuba_api                                  0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B     0
0ca62382910e   sentry_snuba_consumer                             0.45%     92.93MiB / 31.09GiB   0.29%     21.1kB / 10.9kB   0B / 0B     11
b01ba3b46457   sentry_snuba_outcomes_consumer                    0.10%     92.27MiB / 31.09GiB   0.29%     8.08kB / 3.76kB   0B / 0B     5
30fcc4b9374d   sentry_snuba_sessions_consumer                    0.14%     92.26MiB / 31.09GiB   0.29%     10.2kB / 5.66kB   0B / 0B     5
fdabe0103fcf   sentry_snuba_transactions_consumer                0.18%     92.23MiB / 31.09GiB   0.29%     10.2kB / 5.14kB   0B / 0B     8
d67d7260ab3e   sentry_snuba_replays_consumer                     0.11%     92.5MiB / 31.09GiB    0.29%     7.46kB / 3.4kB    0B / 0B     5
7b5920f801d8   sentry_snuba_replacer                             0.10%     92.45MiB / 31.09GiB   0.29%     7.75kB / 3.56kB   0B / 0B     5
8a16768ac8a2   sentry_snuba_subscription_consumer_events         0.28%     93.58MiB / 31.09GiB   0.29%     11.7kB / 5.81kB   0B / 0B     9
0f043aee7ce5   sentry_snuba_subscription_consumer_sessions       0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B     0
1686aea3ef0c   sentry_snuba_subscription_consumer_transactions   0.24%     93.42MiB / 31.09GiB   0.29%     10.3kB / 5.14kB   0B / 0B     9

All starts, excepted sentry_snuba_api and sentry_snuba_subscription_consumer_sessions

For API, command isn't the good one:

docker-002
$ docker logs sentry_snuba_api
Usage: snuba api [OPTIONS]
Try 'snuba api --help' for help.

Error: No such option: --http Did you mean --help?

For the second one, there is a Python error:

docker-002
$ docker logs sentry_snuba_subscription_consumer_sessions
2023-03-12 01:45:24,745 Initializing Snuba...
2023-03-12 01:45:28,878 Snuba initialization took 4.134511881042272s
2023-03-12 01:45:29,411 Initializing Snuba...
2023-03-12 01:45:33,449 Snuba initialization took 4.038913033902645s
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 33, in <module>
    sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/subscriptions_scheduler_executor.py", line 110, in subscriptions_scheduler_executor
    assert result_topic_spec is not None
AssertionError

For the first issue, we should switch to the default command of the image?

The second issue is reported upstream at https://github.com/getsentry/self-hosted/issues/2022

dereckson planned changes to this revision.Mar 12 2023, 02:08

Not sure the network parameter works, as container _consumer wasn't connected:

$ docker logs sentry_snuba_consumer
[…]
2023-03-12 02:00:43,381 Error callback from librdKafka -193, _RESOLVE, sentry_kafka:9092/bootstrap: Failed to resolve 'sentry_kafka:9092': Name or service not known (after 15ms in state CONNECT, 15 identical error(s) suppressed)
[…]
$ docker network connect sentry sentry_snuba_consumer
$ docker network connect sentry sentry_snuba_consumer
Error response from daemon: endpoint with name sentry_snuba_consumer already exists in network sentry

Once connected, a topic isn't available on Kafka:

2023-03-12 02:04:50,376 Caught exception, shutting down...
Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 175, in run
  self._run_once()
File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 206, in _run_once
  self.__message = self.__consumer.poll(timeout=1.0)
File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 409, in poll
  raise ConsumerError(str(error))

arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: events: Broker: Unknown topic or partition"}
2023-03-12 02:04:50,378 Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7ff583545430>...
2023-03-12 02:04:50,380 Processor terminated
Traceback (most recent call last):

File "/usr/local/bin/snuba", line 33, in <module>
  sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
  return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
  rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
  return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
  return __callback(*args, **kwargs)
File "/usr/src/snuba/snuba/cli/consumer.py", line 191, in consumer
  consumer.run()
File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 175, in run
  self._run_once()
File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 206, in _run_once
  self.__message = self.__consumer.poll(timeout=1.0)
File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 409, in poll
  raise ConsumerError(str(error))

arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: events: Broker: Unknown topic or partition"}

dereckson updated this revision to Diff 7340.Mar 12 2023, 10:01

Nightly image

Harbormaster completed remote builds in B4546: Diff 7340.Mar 12 2023, 10:01
dereckson updated this revision to Diff 7341.Mar 12 2023, 10:06

Use full snuba command with docker exec

Harbormaster completed remote builds in B4547: Diff 7341.Mar 12 2023, 10:06
dereckson accepted this revision.Mar 12 2023, 10:19

All containers are up:

docker-002
$ docker stats $snuba_containers
CONTAINER ID   NAME                                              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
e5bee69ba1cc   sentry_snuba_api                                  0.04%     93.53MiB / 31.09GiB   0.29%     455kB / 673kB     0B / 0B     4
d3a9d3c1e8c0   sentry_snuba_consumer                             0.32%     90.76MiB / 31.09GiB   0.29%     120kB / 87.9kB    0B / 0B     11
48f8b5467858   sentry_snuba_outcomes_consumer                    0.21%     90.32MiB / 31.09GiB   0.28%     114kB / 82.5kB    0B / 0B     5
88c31fd33610   sentry_snuba_sessions_consumer                    0.21%     91.03MiB / 31.09GiB   0.29%     102kB / 73.5kB    0B / 0B     8
4a5fc314a3e2   sentry_snuba_transactions_consumer                0.29%     90.54MiB / 31.09GiB   0.28%     84.9kB / 62.6kB   0B / 0B     8
d06b2b8cafc4   sentry_snuba_replays_consumer                     0.29%     90.63MiB / 31.09GiB   0.28%     110kB / 81.1kB    0B / 0B     9
e131fd35f0bc   sentry_snuba_replacer                             0.18%     90.56MiB / 31.09GiB   0.28%     110kB / 79.6kB    0B / 0B     5
88ea7ed40cbc   sentry_snuba_subscription_consumer_events         0.29%     91.02MiB / 31.09GiB   0.29%     95.8kB / 71.7kB   0B / 0B     9
a2ec19816388   sentry_snuba_subscription_consumer_sessions       0.28%     91MiB / 31.09GiB      0.29%     93.9kB / 70kB     0B / 0B     9
035afa585d20   sentry_snuba_subscription_consumer_transactions   0.30%     91.02MiB / 31.09GiB   0.29%     90.5kB / 68kB     0B / 0B     9

Logs from indivual containers look good, for example:

docker-002
$ docker logs sentry_snuba_subscription_consumer_events
2023-03-12 10:05:36,800 Initializing Snuba...
2023-03-12 10:05:41,105 Snuba initialization took 4.306813777890056s
2023-03-12 10:05:41,667 Initializing Snuba...
2023-03-12 10:05:45,943 Snuba initialization took 4.277813965920359s
2023-03-12 10:05:48,972 New partitions assigned: {Partition(topic=Topic(name='snuba-commit-log'), index=0): 0}

The Kafka topics have correctly been created during bootstrap operation:

sentry_kafka container on docker-002
$ kafka-topics --list --bootstrap-server localhost:9092
__consumer_offsets
cdc
event-replacements
events
events-subscription-results
generic-events
generic-metrics-counters-subscription-results
generic-metrics-distributions-subscription-results
generic-metrics-sets-subscription-results
ingest-attachments
ingest-events
ingest-replay-events
ingest-replay-recordings
ingest-sessions
ingest-transactions
metrics-subscription-results
outcomes
processed-profiles
profiles-call-tree
scheduled-subscriptions-events
scheduled-subscriptions-generic-metrics-counters
scheduled-subscriptions-generic-metrics-distributions
scheduled-subscriptions-generic-metrics-sets
scheduled-subscriptions-metrics
scheduled-subscriptions-sessions
scheduled-subscriptions-transactions
sessions-subscription-results
snuba-attribution
snuba-commit-log
snuba-dead-letter-generic-events
snuba-dead-letter-generic-metrics
snuba-dead-letter-metrics
snuba-dead-letter-replays
snuba-dead-letter-sessions
snuba-generic-events-commit-log
snuba-generic-metrics
snuba-generic-metrics-counters-commit-log
snuba-generic-metrics-distributions-commit-log
snuba-generic-metrics-sets-commit-log
snuba-metrics
snuba-metrics-commit-log
snuba-queries
snuba-sessions-commit-log
snuba-transactions-commit-log
transactions
transactions-subscription-results
This revision is now accepted and ready to land.Mar 12 2023, 10:19
dereckson edited the summary of this revision. (Show Details)Mar 12 2023, 10:21
dereckson updated this revision to Diff 7342.Mar 12 2023, 10:30

Reconsolidate snuba and snuba_api: that will be less confusing to maintain if snuba is only snuba.

Harbormaster completed remote builds in B4548: Diff 7342.Mar 12 2023, 10:30
dereckson added a comment.Mar 12 2023, 10:31

So we decided to run each time bootstrap/migrate to ensure we're up-to-date.

That could instead be a part of the sentry wrapper.

Here a no-op bootstrap and migrations process expected output:

----------
          ID: Boostrap and migrate Snuba - sentry_snuba_api
    Function: cmd.run
        Name: docker exec sentry_snuba_api snuba bootstrap --no-migrate --force
docker exec sentry_snuba_api snuba migrations migrate --force

      Result: True
     Comment: Command "docker exec sentry_snuba_api snuba bootstrap --no-migrate --force
              docker exec sentry_snuba_api snuba migrations migrate --force
              " run
     Started: 10:29:47.982962
    Duration: 7871.078 ms
     Changes:
              ----------
              pid:
                  1954120
              retcode:
                  0
              stderr:
                  2023-03-12 10:29:48,534 Initializing Snuba...
                  2023-03-12 10:29:51,754 Snuba initialization took 3.2210601889528334s
                  2023-03-12 10:29:51,755 Attempting to connect to Kafka (attempt 0)...
                  2023-03-12 10:29:51,765 Connected to Kafka on attempt 0
                  2023-03-12 10:29:51,766 Creating Kafka topics...
                  2023-03-12 10:29:52,462 Initializing Snuba...
                  2023-03-12 10:29:55,685 Snuba initialization took 3.224485707934946s
              stdout:
                  Finished running migrations
dereckson edited the summary of this revision. (Show Details)Mar 12 2023, 10:32
Closed by commit rOPS9693fb5f8d66: Deploy Snuba containers for Sentry (authored by dereckson). · Explain WhyMar 12 2023, 10:32
This revision was automatically updated to reflect the committed changes.
dereckson added a commit: rOPS9693fb5f8d66: Deploy Snuba containers for Sentry.
dereckson mentioned this in D2885: Upgrade Sentry to 23.2.0.Mar 13 2023, 18:22
dereckson mentioned this in rOPS3674f0694509: Upgrade Sentry to 23.2.0.Mar 14 2023, 18:55

Revision Contents
Changeset List

PathSize
pillar/
paas/
docker/
docker-002/
71 lines
roles/
paas-docker/
containers/
53 lines

Diff 7343

View Options

pillar/paas/docker/docker-002/sentry.sls

Loading...
View Options

roles/paas-docker/containers/snuba.sls

Loading...
Nasqueron DevCentral · If it had been much bigger the moon would have had a core of ice. · Powered by Phabricator