Phabricator container for DevCentral successfully filed 10G of logs and then reported no space left on device. Congratulations to nginx logs storage in containers!
Each container receives a 10 Gb volume, so regardless of the storage available on lvm data volume, a container can't store more. This volume is mounted as / and can receive modified files. Those files will be lost at container deletion, as not stored in a volume.
To mitigate this situation, store the logs elsewhere than in container (ELK stack) and do a rotation is cool,
but we can detect container issues by inspecting /dev/dm-<number> disk use.
docker-001
$ df -h | grep /dev/dm /dev/dm-6 10G 557M 9.5G 6% /var/lib/docker/devicemapper/mnt/4b7102c3b1cdc1779406c9101f418c797503c487cbee2a3a5e3af074ecfb1826 /dev/dm-7 10G 1.5G 8.6G 15% /var/lib/docker/devicemapper/mnt/b30d7193550efe6f29ec6928d2b487ed3600523f12edc6533aef865183a370f8 /dev/dm-11 10G 530M 9.5G 6% /var/lib/docker/devicemapper/mnt/3648ff6f389a7e7b571412d47e380f30709cd184e81440f0961529396d497059 /dev/dm-12 10G 244M 9.8G 3% /var/lib/docker/devicemapper/mnt/610a6535d794c86b0ae90b1909cff94c380490962de027322dd10cc1a53f6043 /dev/dm-13 10G 1.8G 8.3G 18% /var/lib/docker/devicemapper/mnt/fc46bb8a14037a8c5f6d36bdc6e0760421ecc3a798b3d828bfb2279ff9dc8265 /dev/dm-15 10G 1.3G 8.8G 13% /var/lib/docker/devicemapper/mnt/b152adca2d875b1ad77eb8404c901481680afef70862548cb472893983e9b709 /dev/dm-16 10G 2.6G 7.5G 26% /var/lib/docker/devicemapper/mnt/a688b8417ee2365a51b7729fd2ff43c640135fa70a71638dd8033d4f5837ddad /dev/dm-8 10G 1.8G 8.3G 18% /var/lib/docker/devicemapper/mnt/bf5b8435921f74e078c1092329ac3d5f86881592a38c0a7bf7cefc2cc7013fbe /dev/dm-17 10G 1.8G 8.3G 18% /var/lib/docker/devicemapper/mnt/e1edde345b34af3b08b51ac441d82dfa08a55130174330b600dfd762845bbfb3 /dev/dm-18 10G 474M 9.6G 5% /var/lib/docker/devicemapper/mnt/ce2409c068496e513d028c9e819034f30159a34fab561ba37346efac439252f1 /dev/dm-19 10G 220M 9.8G 3% /var/lib/docker/devicemapper/mnt/e08be82b3761e20468190cc1ceda5bb3f42915d032096481af7c18a761892e16 /dev/dm-20 10G 842M 9.2G 9% /var/lib/docker/devicemapper/mnt/6ff749a2a78d66b5e328cb765acd71c0f469fe8ddb4b1e7f91e665b3483895f7 /dev/dm-22 10G 857M 9.2G 9% /var/lib/docker/devicemapper/mnt/dda98db700fd4ac1fd41ef51ea8cc146496e81e4bbc64d82300186d92e82e25d /dev/dm-21 10G 178M 9.9G 2% /var/lib/docker/devicemapper/mnt/f4dfa09e1bd7d9aba18ac053e63500b1e8abd1e07e0f03e9084208f4d3d1c8fb /dev/dm-23 10G 48M 10G 1% /var/lib/docker/devicemapper/mnt/60ae8cef5d766367c238246efa9e301115331a5c3a0114dab64f4a0d56cd75ad /dev/dm-24 10G 1.3G 8.8G 13% /var/lib/docker/devicemapper/mnt/f16929120565ceadca42867e68ad5b78a25af8892dd7b6e12056027557fb3603 /dev/dm-25 10G 1.5G 8.6G 15% /var/lib/docker/devicemapper/mnt/be80e1bef385f10a5c59fa93e483c4f65d5a4bec169002f7628d1255355fb496 /dev/dm-26 10G 1.1G 9.0G 11% /var/lib/docker/devicemapper/mnt/85c8f898d9c6ecdf1185b4eaec985817dc1ccc1ac9302251c5ec3e5351be7f36 /dev/dm-27 10G 970M 9.1G 10% /var/lib/docker/devicemapper/mnt/0a87712057de2a5f155e9c70d52fea7b6e915fed33f4e33a8b88ae595bb6ac9e /dev/dm-28 10G 86M 10G 1% /var/lib/docker/devicemapper/mnt/d20db29bfeeaad79d6f6a381a2cfe01ce05de358e9dd94041351ed70b3c3e0f9 /dev/dm-29 10G 72M 10G 1% /var/lib/docker/devicemapper/mnt/8b567d181861cc030ac427cb622424bb5199ef5a694aa1af578be0f2bcbd6fce /dev/dm-31 10G 1.1G 9.0G 11% /var/lib/docker/devicemapper/mnt/52fd9fb2f04982fc84f742daef066e32de205b68422c0c0c4e62724a942f791f /dev/dm-33 10G 1.5G 8.6G 15% /var/lib/docker/devicemapper/mnt/722bcc4b53736f55d4df7dcacc7b4c1e94e0752ed70067bb2b21c507c488db8a /dev/dm-35 10G 992M 9.1G 10% /var/lib/docker/devicemapper/mnt/13bf0b21d85e04299f790afd92e1a192f0558b326f3334729dd2b3dd868a227d /dev/dm-34 10G 3.4G 6.6G 34% /var/lib/docker/devicemapper/mnt/54014380f8463b1643d969111aaf229449b2cec35b15d8a5eb9302d8cfbb871c /dev/dm-30 10G 130M 9.9G 2% /var/lib/docker/devicemapper/mnt/c2c4dc9c138b0a6f911181268d2f639f8282bef14ef58270afc5d0150a3dfe19 /dev/dm-14 10G 1.8G 8.3G 18% /var/lib/docker/devicemapper/mnt/7855e6f69db8b602df9bdd57cf4c7f6b67f435bb1944e3d2852858c4ec77c490 /dev/dm-9 10G 501M 9.6G 5% /var/lib/docker/devicemapper/mnt/832e045ccb0132e12fc21c8e7143d9c817ca75c745b34360efb4f3714e735869 /dev/dm-10 10G 501M 9.6G 5% /var/lib/docker/devicemapper/mnt/94e415b4685517ec37ada7e57660a31ac5807813642bfc6368a834101c8f07b2