Exception "Read-only file system"

[dext 0]

Hi,

I had Emby working perfectly for a couple of months, but then after reboot I'm getting this exceptions pasted at the end of the post.
I'm couple of hours into this problem, I can't find anything relevant on the Internet. I have triple checked all permissions I could think off. All mount points are for sure rw, not ro. I've even tried to recreate PROGRAM_DATA directory - no change. The exception doesn't give me any info where the problem actually is, so I'm lost at this point. Could someone please give my some direction what to try next?

I'm using Arch Linux and natively installed emby-server package (no docker).

Jan 04 08:15:27 vaduz systemd[1]: Started Emby brings together your videos, music, photos, and live television..
Jan 04 08:15:27 vaduz emby-server[24993]: Error GlobalExceptionHandler: UnhandledException
Jan 04 08:15:27 vaduz emby-server[24993]:         System.AggregateException: System.AggregateException: One or more errors occurred. (Read-only file system)
Jan 04 08:15:27 vaduz emby-server[24993]:          ---> System.IO.IOException: Read-only file system
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.IO.FileSystem.CreateDirectory(String fullPath)
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.IO.Directory.CreateDirectory(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:            at Emby.Server.Implementations.Logging.FileLogger..ctor(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:            at Emby.Server.Implementations.Logging.SimpleLogManager.ReloadLogger(LogSeverity severity, CancellationToken cancellationToken)
Jan 04 08:15:27 vaduz emby-server[24993]:            --- End of inner exception stack trace ---
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
Jan 04 08:15:27 vaduz emby-server[24993]:            at EmbyServer.Program.Main(String[] args)
Jan 04 08:15:27 vaduz emby-server[24993]:         Source: System.Private.CoreLib
Jan 04 08:15:27 vaduz emby-server[24993]:         TargetSite: Void ThrowAggregateException(System.Collections.Generic.List`1[System.Exception])
Jan 04 08:15:27 vaduz emby-server[24993]:         InnerException: System.IO.IOException: Read-only file system
Jan 04 08:15:27 vaduz emby-server[24993]:         Source: System.IO.FileSystem
Jan 04 08:15:27 vaduz emby-server[24993]:         TargetSite: Void CreateDirectory(System.String)
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.IO.FileSystem.CreateDirectory(String fullPath)
Jan 04 08:15:27 vaduz emby-server[24993]:            at System.IO.Directory.CreateDirectory(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:            at Emby.Server.Implementations.Logging.FileLogger..ctor(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:            at Emby.Server.Implementations.Logging.SimpleLogManager.ReloadLogger(LogSeverity severity, CancellationToken cancellationToken)
Jan 04 08:15:27 vaduz emby-server[24993]:         
Jan 04 08:15:27 vaduz emby-server[24993]: Unhandled exception. System.AggregateException: One or more errors occurred. (Read-only file system)
Jan 04 08:15:27 vaduz emby-server[24993]:  ---> System.IO.IOException: Read-only file system
Jan 04 08:15:27 vaduz emby-server[24993]:    at System.IO.FileSystem.CreateDirectory(String fullPath)
Jan 04 08:15:27 vaduz emby-server[24993]:    at System.IO.Directory.CreateDirectory(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:    at Emby.Server.Implementations.Logging.FileLogger..ctor(String path)
Jan 04 08:15:27 vaduz emby-server[24993]:    at Emby.Server.Implementations.Logging.SimpleLogManager.ReloadLogger(LogSeverity severity, CancellationToken cancellationToken)
Jan 04 08:15:27 vaduz emby-server[24993]:    --- End of inner exception stack trace ---
Jan 04 08:15:27 vaduz emby-server[24993]:    at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
Jan 04 08:15:27 vaduz emby-server[24993]:    at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
Jan 04 08:15:27 vaduz emby-server[24993]:    at EmbyServer.Program.Main(String[] args)
Jan 04 08:15:28 vaduz systemd[1]: emby-server.service: Main process exited, code=dumped, status=6/ABRT
Jan 04 08:15:28 vaduz systemd[1]: emby-server.service: Failed with result 'core-dump'.

 

[Q-Droid 989]

What user:group owns /var/lib/emby? Does it match the Emby server runtime user?

Was a new log created in /var/lib/emby/logs? If so please attach it.

Was your initial setup following this info? https://wiki.archlinux.org/index.php/Emby

 

[dext 0]

Thanks for the response.

I did follow Arch Linux wiki, with some changes:

• I don't use "/var/lib/emby". Instead I set "PROGRAM_DATA=/srv/emby" in /etc/conf.d/emby-server
• Instead of "media" group, I used "emby"
   

/srv/emby/logs have only old logs.

This is a server, which I never touch. Only thing that happened, was system update + reboot. Which for Arch Linux could mean a lot, but I didn't do anything with config or file permissions.

I see in logs last time Emby worked was 26 of October 2020. Maybe Emby had some major release?

ps aux | grep emby # quick, before crash
emby       26188  7.6  0.5 3054372 41784 ?       Ssl  14:16   0:00 dotnet /usr/lib/emby-server/EmbyServer.dll -programdata /srv/emby -ffdetect /usr/bin/ffdetect-emby -ffmpeg /usr/bin/ffmpeg-emby -ffprobe /usr/bin/ffprobe-emby -restartexitcode 3

cat /etc/conf.d/emby-server | grep PROGRAM_DATA
PROGRAM_DATA=/srv/emby

ll /srv/ | grep emby                                                          
drwxr-xr-x 12 emby         emby          4096 Jan  4 08:11 emby/

# just to be sure, again:
chmod u+rwX,g+rwX,o+rX -R /srv/emby
chown emby:emby -R /srv/emby

# this works:
su - emby
touch /srv/emby/test123

 

[Q-Droid 989]

Was /srv/emby explicitly allowed in ReadWritePaths before in systemd for the service? Since you're deviating from default is it possibly needed now? Perhaps an Arch update changed the behavior... 

I always think of something after I post. 

Is the emby user home dir also /srv/emby?

Edited January 4, 2021 by Q-Droid

[dext 0]

Thank you very much for help. You were right, it was Systemd's sandboxing. I didn't have it configured at all, and looks like it started to be enforced with some update.

I hate Systemd with all my heart from the first time I've seen it, and this hatred grows with every such case, and interaction with their useless documentation. But that's another topic