No space left on device but can't figure out why or where

[justwondering 14]

I have a Raspberry Pi 4 running Emby Server that has been working great up until now. It's running on a 32 GB micro SD card, and all the media is on external drives. I've set the per-library options for all my libraries (I'm pretty sure) to save images and metadata to the external drives, so there shouldn't be much on the internal. It started as a clean install of Raspbian with Samba and Emby and nothing else. Suddenly it's unavailable, so I SSH in and check status:

user@Server:/mnt $ sudo service emby-server status
● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2022-01-28 22:21:29 GMT; 5s ago
  Process: 936 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=ABRT)
 Main PID: 936 (code=killed, signal=ABRT)

Jan 28 22:21:29 Server emby-server[936]: Unhandled exception. System.IO.IOException: No space left on device
Jan 28 22:21:29 Server emby-server[936]:    at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
Jan 28 22:21:29 Server emby-server[936]:    at System.IO.FileStream.FlushWriteBuffer()
Jan 28 22:21:29 Server emby-server[936]:    at System.IO.FileStream.FlushInternalBuffer()
Jan 28 22:21:29 Server emby-server[936]:    at System.IO.FileStream.Flush(Boolean flushToDisk)
Jan 28 22:21:29 Server emby-server[936]:    at Emby.Server.Implementations.Logging.FileLogger.Dispose()
Jan 28 22:21:29 Server emby-server[936]:    at Emby.Server.Implementations.Logging.SimpleLogManager.Dispose()
Jan 28 22:21:29 Server emby-server[936]:    at EmbyServer.Program.Main(String[] args)
Jan 28 22:21:29 Server systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
Jan 28 22:21:29 Server systemd[1]: emby-server.service: Failed with result 'signal'.

OK, there's no space left. I check df and get confirmation: /dev/root is 100% used:

user@Server:/mnt $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   29G     0 100% /
devtmpfs        805M     0  805M   0% /dev
tmpfs           934M     0  934M   0% /dev/shm
tmpfs           934M   11M  924M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           934M     0  934M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   50M  203M  20% /boot
/dev/sdd1       3.6T  2.5T 1010G  72% /mnt/storage1
/dev/sdc1       3.6T  2.5T 1004G  72% /mnt/storage2
/dev/sdb1       9.1T  5.0T  3.7T  58% /mnt/media2
/dev/sda1       9.1T  5.0T  3.7T  58% /mnt/media1
tmpfs           187M     0  187M   0% /run/user/1000

If it was Emby metadata filling it up, that would be in /var/lib/emby right? So I check that:

user@Server:/var/lib/emby $ ls -lh
total 172K
drwxr-xr-x 13 emby emby 4.0K Jan 25 20:19 cache
drwxr-xr-x  4 emby emby 4.0K May 12  2021 config
drwxr-xr-x  5 emby emby 4.0K Nov 30 11:39 data
drwxr-xr-x  2 emby emby 144K Jan 28 22:21 logs
drwxr-xr-x  5 emby emby 4.0K May 12  2021 metadata
drwxr-xr-x  3 emby emby 4.0K May 11  2021 plugins
drwxr-xr-x  3 emby emby 4.0K May 11  2021 root

It doesn't seem like there's anything anything taking up space in there. I try using du to see what's taking up space:

user@Server:/ $ sudo du -sh *
0	bin
50M	boot
0	dev
3.9M	etc
48K	home
0	lib
16K	lost+found
4.0K	media
15T	mnt
240M	opt
du: cannot access 'proc/1036/task/1036/fd/4': No such file or directory
du: cannot access 'proc/1036/task/1036/fdinfo/4': No such file or directory
du: cannot access 'proc/1036/fd/3': No such file or directory
du: cannot access 'proc/1036/fdinfo/3': No such file or directory
0	proc
24K	root
11M	run
0	sbin
4.0K	srv
0	sys
32K	tmp
1.3G	usr
1.5G	var

Nothing is showing as large enough to fill the SD card! Except mnt at 15 TB, but that's obviously my external drives. There should be 20+ GB of free space on the SD card. But there's not?

I'm not great at Linux so this is totally baffling to me. How can I figure out what's taking up space so I can free it? I could move to a larger SD card, but without figuring out the problem it could just fill up again. I'm not even sure if this is an Emby issue but I would appreciate any help.

[justwondering 14]

Solved it! So... nevermind!

In case anyone else runs into this, it turned out there were large files in the mount directory (on the system SD card) for one of my external drives. With the external drive mounted, that directory was completely hidden.

I have a cron job that runs rsync every night to back up one media drive (media1) to another (media2). This has worked flawlessly for a long time, but apparently a couple of nights ago (perhaps due to a power outage) the primary media drive mounted but the backup drive did not. Rsync executed anyway, and backed up /mnt/media1 to /mnt/media2. But since media2 was not mounted, /mnt/media2 was just a directory on the system volume on the SD card. Rsync wrote as much as it could to that directory and filled up the internal disk.

The next time I rebooted the server, /mnt/media2 mounted as normal. So it was now mounted over the /mnt/media2 directory where 29 GB of backup was written. In order to see it, I had to unmount the external. Once I did that it showed up and I could delete it. Now my SD card is 89% empty and everything is fully functional again.

[Luke 42078]

Thanks for the feedback.