Page MenuHomeDevCentral

Provision an utility to kill stuck ffmpeg jobs on Mastodon
ClosedPublic

Authored by dereckson on Dec 8 2018, 18:17.
Tags
None
Referenced Files
F2746314: D2037.id5150.diff
Fri, Mar 29, 02:12
F2746246: D2037.id5147.diff
Fri, Mar 29, 01:58
Unknown Object (File)
Thu, Mar 28, 10:05
Unknown Object (File)
Thu, Mar 28, 04:42
Unknown Object (File)
Wed, Mar 27, 03:18
Unknown Object (File)
Wed, Mar 27, 02:43
Unknown Object (File)
Mon, Mar 25, 00:09
Unknown Object (File)
Sat, Mar 23, 15:29
Subscribers
None

Details

Summary

Mastodon queue can be stuck by ffmpeg jobs.

If a job uses more than 1 minute CPU time, it's reasonnable
to consider we can skip it.

Meanwhile, regular and legitimate ffmpeg jobs are useful to extract
thumbnails for video and complete in some seconds.

This change provisions an utility to kill ffmpeg jobs
to Sidekiq containers if they spend more than one minute CPU time.

Ref T1492.

Test Plan

Script has been tested with succes on the Nasqueron instance.

Unit tests are provided with a data sample taken from production.

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Passed
Unit
No Test Coverage
Branch
T1492 (branched from master)
Build Status
Buildable 3183
Build 3432: arc lint + arc unit

Event Timeline

dereckson created this revision.
dereckson added inline comments.
roles/paas-docker/containers/files/mastodon/clear-video-queue.py
80

One kill command is enough.

Decrease a little bit the threshold, I've seen several times ffmpeg processes becoming quasi-zombies at 0:38 CPU time

Shadow, fix test for 30 seconds

dereckson marked an inline comment as done.
This revision is now accepted and ready to land.Dec 9 2018, 00:24
This revision was automatically updated to reflect the committed changes.