User lost admin access after running out of space

[XcOM9876 9]

• Host: Microsoft Hyper-V v9
• Client: Ubuntu 20.04 server headless
• CPU: Xeon E5 4 cores
• Ram: 4GB boot, 32gb max permitted dynamically allocated
• Storage: 60GB root drive, 32TB media drives
• Emby Version: 4.6.2.0 (After updating from 4.5.x.x)

So for some reason I ran out of storage on my server's root OS drive causing it to have a hissy fit and lock my users out, the service would fail to start with a dump warning, after doing some checks I identified that I had ran out of storage on my root drive, digging around it appeared that the transcode folder was using 18gb and filled up the root drive, I ended up clearing out the transcode folder located in /var/lib/emby/ which then allowed me to reload the server, but I was missing all admin rights on all my users on the server.

As I was 1 release behind on the Ubuntu version I opted to update the server to the latest version but none of my users had admin rights still, I assumed that if no users were set to be admin there was supposed to be a fallback but this didn't seem to kick in for me, After this I decided to modify the XML files for all users to check and none of them had IsAdministrator set so I set it for them all but this didn't make any difference, I then opted to delete the users folder from /var/lib/emby/users (renamed to .old) and restarted the deamon as a last ditch attempt before posting here for help after I spent the best part of 2 hours scouring the forum following advise from people for similar issues.

The deleting of the user folder seemed to restore my admin rights, I am unclear what caused all this but I have it back now, I have retained all my logs but they are huge and seen sanitizing before uploading, praise Hyper-V letting me mess about with snapshots.

Long term I am going to create a new vDisk and attach it to use for transcodes

Making this post partly for others if they come across a similar issue and also to see if anyone has any insight as to what caused this?

Edited June 12, 2021 by XcOM9876
Typo

[XcOM9876 9]

Update: I think I might have more serious issues, I created a new drive which I attached, I granted the emby user read-write access to the new drive/folder/mount I created in /media/transocde, all of a sudden as soon as I set the transcode to use that folder it broke Emby server entirely, so I rebooted, same, set file permissions to all users full rights, no difference, had to restore a snapshot and then apply my fix again.

Fun times, does appear to be working once again though, just seems a little slow navigating around and some context menus seem a bit buggy

[Luke 42077]

Hi @XcOM9876, when the drive holding the database runs out of space, this will cause all database operations to fail, including authenticating. It's something we plan to look at making the server more resilient towards being able to handle. Apologies for the disruption.

[XcOM9876 9]

Hi @Luke, Thanks for the response, no need for apologies, I should have been taking better care of my servers, I am looking at implementing some monitoring as for some reason I can't seem to expand the hard drive or attach a second drive.

Happy to supply any of the logs I took just before and after the crash if it helps development teams to understand why the transcode folder wasn't clearing itself when no longer required.