Page MenuHomeDevCentral

Low space disk on Dwellers
Closed, ResolvedPublic

Description

We had a smaller / as storage for Docker containers use another partition, but it seems Mastodon logs filled this partition.

/dev/mapper/centos_dwellers-root 50G 50G 20K 100% /

Event Timeline

dereckson lowered the priority of this task from Unbreak Now! to High.Apr 13 2017, 21:48
Dwellers
$ journalctl --vacuum-size=500M
Vacuuming done, freed 3.5G of archived journals on disk.

Hypervisor side

Dreadnought
$ vmkfstools -c 200G air-bubble.vmdk
Create: 100% done.                     
$ vim-cmd vmsvc/getallvms

4      Dwellers    [datastore1] legacy/Dwellers/Dwellers.vmx   centos64Guest     vmx-08

$ vim-cmd vmsvc/device.diskaddexisting 4 /vmfs/volumes/59ded2a4-67d32157-eb6
c-002590534f50/legacy/Dwellers/air-bubble.vmdk 0 2

Dwellers journal

Oct 23 19:27:46 dwellers.nasqueron.org kernel: vmw_pvscsi: msg type: 0x0 - MSG RING: 1/0 (5)
Oct 23 19:27:46 dwellers.nasqueron.org kernel: vmw_pvscsi: msg: device added at scsi0:2:0
Oct 23 19:27:46 dwellers.nasqueron.org kernel: scsi 2:0:2:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] 419430400 512-byte logical blocks: (214 GB/200 GiB)
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] Write Protect is off
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] Mode Sense: 61 00 00 00
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] Cache data unavailable
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] Assuming drive cache: write through
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: Attached scsi generic sg3 type 0
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sdc: unknown partition table
Oct 23 19:27:46 dwellers.nasqueron.org kernel: sd 2:0:2:0: [sdc] Attached SCSI disk

Server side

Dwellers
$ fdisk /dev/sdc
[ create a new GPT table, and a partition ]
Calling ioctl() to re-read partition table.
Syncing disks.
$ pvcreate /dev/sdc1
  Physical volume "/dev/sdc1" successfully created.
$ vgscan
  Reading volume groups from cache.
  Found volume group "centos_dwellers" using metadata type lvm2
  Found volume group "mastodon" using metadata type lvm2
$ vgextend centos_dwellers /dev/sdc1
  Volume group "centos_dwellers" successfully extended
$ xfs_growfs /
eta-data=/dev/mapper/centos_dwellers-root isize=256    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13107200 to 52427776
$ df -h | head
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/centos_dwellers-root    200G   50G  151G  25% /