Page MenuHomeDevCentral

Monitor container disk usage
Open, HighPublic

Description

Phabricator container for DevCentral successfully filed 10G of logs and then reported no space left on device. Congratulations to nginx logs storage in containers!

Each container receives a 10 Gb volume, so regardless of the storage available on lvm data volume, a container can't store more. This volume is mounted as / and can receive modified files. Those files will be lost at container deletion, as not stored in a volume.

To mitigate this situation, store the logs elsewhere than in container (ELK stack) and do a rotation is cool,
but we can detect container issues by inspecting /dev/dm-<number> disk use.

docker-001
$ df -h | grep /dev/dm
/dev/dm-6                 10G  557M  9.5G   6% /var/lib/docker/devicemapper/mnt/4b7102c3b1cdc1779406c9101f418c797503c487cbee2a3a5e3af074ecfb1826
/dev/dm-7                 10G  1.5G  8.6G  15% /var/lib/docker/devicemapper/mnt/b30d7193550efe6f29ec6928d2b487ed3600523f12edc6533aef865183a370f8
/dev/dm-11                10G  530M  9.5G   6% /var/lib/docker/devicemapper/mnt/3648ff6f389a7e7b571412d47e380f30709cd184e81440f0961529396d497059
/dev/dm-12                10G  244M  9.8G   3% /var/lib/docker/devicemapper/mnt/610a6535d794c86b0ae90b1909cff94c380490962de027322dd10cc1a53f6043
/dev/dm-13                10G  1.8G  8.3G  18% /var/lib/docker/devicemapper/mnt/fc46bb8a14037a8c5f6d36bdc6e0760421ecc3a798b3d828bfb2279ff9dc8265
/dev/dm-15                10G  1.3G  8.8G  13% /var/lib/docker/devicemapper/mnt/b152adca2d875b1ad77eb8404c901481680afef70862548cb472893983e9b709
/dev/dm-16                10G  2.6G  7.5G  26% /var/lib/docker/devicemapper/mnt/a688b8417ee2365a51b7729fd2ff43c640135fa70a71638dd8033d4f5837ddad
/dev/dm-8                 10G  1.8G  8.3G  18% /var/lib/docker/devicemapper/mnt/bf5b8435921f74e078c1092329ac3d5f86881592a38c0a7bf7cefc2cc7013fbe
/dev/dm-17                10G  1.8G  8.3G  18% /var/lib/docker/devicemapper/mnt/e1edde345b34af3b08b51ac441d82dfa08a55130174330b600dfd762845bbfb3
/dev/dm-18                10G  474M  9.6G   5% /var/lib/docker/devicemapper/mnt/ce2409c068496e513d028c9e819034f30159a34fab561ba37346efac439252f1
/dev/dm-19                10G  220M  9.8G   3% /var/lib/docker/devicemapper/mnt/e08be82b3761e20468190cc1ceda5bb3f42915d032096481af7c18a761892e16
/dev/dm-20                10G  842M  9.2G   9% /var/lib/docker/devicemapper/mnt/6ff749a2a78d66b5e328cb765acd71c0f469fe8ddb4b1e7f91e665b3483895f7
/dev/dm-22                10G  857M  9.2G   9% /var/lib/docker/devicemapper/mnt/dda98db700fd4ac1fd41ef51ea8cc146496e81e4bbc64d82300186d92e82e25d
/dev/dm-21                10G  178M  9.9G   2% /var/lib/docker/devicemapper/mnt/f4dfa09e1bd7d9aba18ac053e63500b1e8abd1e07e0f03e9084208f4d3d1c8fb
/dev/dm-23                10G   48M   10G   1% /var/lib/docker/devicemapper/mnt/60ae8cef5d766367c238246efa9e301115331a5c3a0114dab64f4a0d56cd75ad
/dev/dm-24                10G  1.3G  8.8G  13% /var/lib/docker/devicemapper/mnt/f16929120565ceadca42867e68ad5b78a25af8892dd7b6e12056027557fb3603
/dev/dm-25                10G  1.5G  8.6G  15% /var/lib/docker/devicemapper/mnt/be80e1bef385f10a5c59fa93e483c4f65d5a4bec169002f7628d1255355fb496
/dev/dm-26                10G  1.1G  9.0G  11% /var/lib/docker/devicemapper/mnt/85c8f898d9c6ecdf1185b4eaec985817dc1ccc1ac9302251c5ec3e5351be7f36
/dev/dm-27                10G  970M  9.1G  10% /var/lib/docker/devicemapper/mnt/0a87712057de2a5f155e9c70d52fea7b6e915fed33f4e33a8b88ae595bb6ac9e
/dev/dm-28                10G   86M   10G   1% /var/lib/docker/devicemapper/mnt/d20db29bfeeaad79d6f6a381a2cfe01ce05de358e9dd94041351ed70b3c3e0f9
/dev/dm-29                10G   72M   10G   1% /var/lib/docker/devicemapper/mnt/8b567d181861cc030ac427cb622424bb5199ef5a694aa1af578be0f2bcbd6fce
/dev/dm-31                10G  1.1G  9.0G  11% /var/lib/docker/devicemapper/mnt/52fd9fb2f04982fc84f742daef066e32de205b68422c0c0c4e62724a942f791f
/dev/dm-33                10G  1.5G  8.6G  15% /var/lib/docker/devicemapper/mnt/722bcc4b53736f55d4df7dcacc7b4c1e94e0752ed70067bb2b21c507c488db8a
/dev/dm-35                10G  992M  9.1G  10% /var/lib/docker/devicemapper/mnt/13bf0b21d85e04299f790afd92e1a192f0558b326f3334729dd2b3dd868a227d
/dev/dm-34                10G  3.4G  6.6G  34% /var/lib/docker/devicemapper/mnt/54014380f8463b1643d969111aaf229449b2cec35b15d8a5eb9302d8cfbb871c
/dev/dm-30                10G  130M  9.9G   2% /var/lib/docker/devicemapper/mnt/c2c4dc9c138b0a6f911181268d2f639f8282bef14ef58270afc5d0150a3dfe19
/dev/dm-14                10G  1.8G  8.3G  18% /var/lib/docker/devicemapper/mnt/7855e6f69db8b602df9bdd57cf4c7f6b67f435bb1944e3d2852858c4ec77c490
/dev/dm-9                 10G  501M  9.6G   5% /var/lib/docker/devicemapper/mnt/832e045ccb0132e12fc21c8e7143d9c817ca75c745b34360efb4f3714e735869
/dev/dm-10                10G  501M  9.6G   5% /var/lib/docker/devicemapper/mnt/94e415b4685517ec37ada7e57660a31ac5807813642bfc6368a834101c8f07b2

Event Timeline

dereckson triaged this task as High priority.Wed, Sep 14, 18:23
dereckson created this task.