First, that needs better monitoring and backup.

So to avoid queue to be clobbered again: while sleep 30; do clear-video-queue; done &

Backlog has been cleared, but I let this task open, as there is some actionable to follow-up, like write a postmortem.

To detect easily this situation in the future we can use this nagios check:
https://github.com/wanelo/nagios-checks/blob/master/check_sidekiq_queue

There are big ffmpeg video rendering tasks currently:

I restarted Sidekiq, it's currently processing new jobs. I'm watching the queues evolution, processed count increases, enqueued is currently stable.

We need docker-compose to process.

Credentials are stored in K103 and documented at https://agora.nasqueron.org/Operations_grimoire/External_services

In https://social.nasqueron.org/@Kumkum/398092 that works too.

Looks good to me now — https://social.nasqueron.org/@Dereckson/398055

It's easier than that: in most recent Mastodon version, the process runs under a Mastodon user, so it now needs to own the /mastodon/public directory.

I switched the SELinux configuration to Permissive, so we'll get logs, and we'll be apply to apply sand_virt_t SELinux policy to the directories to manage by the container.

Could be a SELinux issue, as the same time in the main logs:

Issue was the size kept was older logs.

Bad timing, https://github.com/tootsuite/mastodon/issues/791