Page MenuHomeDevCentral

Provision an utility to kill stuck ffmpeg jobs on Mastodon
ClosedPublic

Authored by dereckson on Dec 8 2018, 18:17.
Tags
None
Referenced Files
F34177967: D2037.diff
Fri, Jun 26, 01:56
Unknown Object (File)
Sun, Jun 21, 08:57
Unknown Object (File)
Fri, Jun 19, 12:39
Unknown Object (File)
Thu, Jun 18, 08:13
Unknown Object (File)
Wed, Jun 17, 19:20
Unknown Object (File)
Wed, Jun 17, 13:48
Unknown Object (File)
Tue, Jun 16, 23:49
Unknown Object (File)
Tue, Jun 16, 20:28
Subscribers
None

Details

Summary

Mastodon queue can be stuck by ffmpeg jobs.

If a job uses more than 1 minute CPU time, it's reasonnable
to consider we can skip it.

Meanwhile, regular and legitimate ffmpeg jobs are useful to extract
thumbnails for video and complete in some seconds.

This change provisions an utility to kill ffmpeg jobs
to Sidekiq containers if they spend more than one minute CPU time.

Ref T1492.

Test Plan

Script has been tested with succes on the Nasqueron instance.

Unit tests are provided with a data sample taken from production.

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dereckson created this revision.
dereckson added inline comments.
roles/paas-docker/containers/files/mastodon/clear-video-queue.py
80

One kill command is enough.

Decrease a little bit the threshold, I've seen several times ffmpeg processes becoming quasi-zombies at 0:38 CPU time

Shadow, fix test for 30 seconds

dereckson marked an inline comment as done.
This revision is now accepted and ready to land.Dec 9 2018, 00:24
This revision was automatically updated to reflect the committed changes.