Page MenuHomeDevCentral

Provision an utility to kill stuck ffmpeg jobs on Mastodon
ClosedPublic

Authored by dereckson on Dec 8 2018, 18:17.
Tags
None
Referenced Files
F11149846: D2037.id5151.diff
Sun, Aug 17, 08:57
Unknown Object (File)
Sat, Aug 16, 04:02
Unknown Object (File)
Fri, Aug 15, 01:14
Unknown Object (File)
Wed, Aug 13, 17:05
Unknown Object (File)
Tue, Aug 12, 12:15
Unknown Object (File)
Mon, Aug 11, 17:13
Unknown Object (File)
Mon, Aug 11, 08:27
Unknown Object (File)
Sat, Aug 9, 19:35
Subscribers
None

Details

Summary

Mastodon queue can be stuck by ffmpeg jobs.

If a job uses more than 1 minute CPU time, it's reasonnable
to consider we can skip it.

Meanwhile, regular and legitimate ffmpeg jobs are useful to extract
thumbnails for video and complete in some seconds.

This change provisions an utility to kill ffmpeg jobs
to Sidekiq containers if they spend more than one minute CPU time.

Ref T1492.

Test Plan

Script has been tested with succes on the Nasqueron instance.

Unit tests are provided with a data sample taken from production.

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dereckson created this revision.
dereckson added inline comments.
roles/paas-docker/containers/files/mastodon/clear-video-queue.py
80

One kill command is enough.

Decrease a little bit the threshold, I've seen several times ffmpeg processes becoming quasi-zombies at 0:38 CPU time

Shadow, fix test for 30 seconds

dereckson marked an inline comment as done.
This revision is now accepted and ready to land.Dec 9 2018, 00:24
This revision was automatically updated to reflect the committed changes.