Page MenuHomeDevCentral

Provision an utility to kill stuck ffmpeg jobs on Mastodon
ClosedPublic

Authored by dereckson on Dec 8 2018, 18:17.

Details

Summary

Mastodon queue can be stuck by ffmpeg jobs.

If a job uses more than 1 minute CPU time, it's reasonnable
to consider we can skip it.

Meanwhile, regular and legitimate ffmpeg jobs are useful to extract
thumbnails for video and complete in some seconds.

This change provisions an utility to kill ffmpeg jobs
to Sidekiq containers if they spend more than one minute CPU time.

Ref T1492.

Test Plan

Script has been tested with succes on the Nasqueron instance.

Unit tests are provided with a data sample taken from production.

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

dereckson requested review of this revision.Dec 8 2018, 18:17
dereckson created this revision.
dereckson updated this revision to Diff 5146.Dec 8 2018, 18:24

Fix shadow use issues

dereckson planned changes to this revision.Dec 8 2018, 18:40
dereckson added inline comments.
roles/paas-docker/containers/files/mastodon/clear-video-queue.py
80

One kill command is enough.

dereckson updated this revision to Diff 5147.Dec 8 2018, 19:57

Use MAX_TIME

dereckson updated this revision to Diff 5148.Dec 8 2018, 19:58

Decrease a little bit the threshold, I've seen several times ffmpeg processes becoming quasi-zombies at 0:38 CPU time

dereckson updated this revision to Diff 5149.Dec 9 2018, 00:02

Shadow, fix test for 30 seconds

dereckson accepted this revision.Dec 9 2018, 00:24
dereckson marked an inline comment as done.
This revision is now accepted and ready to land.Dec 9 2018, 00:24
dereckson updated this revision to Diff 5150.Dec 9 2018, 00:24

Improve title

dereckson updated this revision to Diff 5151.Dec 9 2018, 00:26

Whiteline

This revision was automatically updated to reflect the committed changes.