Server crashing when setting up and won't restart

[RasmusHetoft 0]

Hi there,

First off, thanks for a great service (when it works!). I'm having some issues getting everything set up properly, however. The server seems to crash when I'm importing my Movies and TV Shows, perhaps when combined with having an M3U tuner specified (TVHeadend – both using the plugin and native M3U tuner support). I've tried both with the stable and beta versions (as in the logs), and the problem remains.

Attached here are the logs - both from when the initial crash happened, and the logs from every time I now try to restart the service (it crashes before I can access the GUI). 

Service status after the intial crash reads as:

 emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Thu 2021-08-12 23:38:05 CEST; 8h ago
  Process: 10622 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=ABRT)
 Main PID: 10622 (code=killed, signal=ABRT)

aug 12 22:50:29 osmc emby-server[10622]: Info App: Entry point completed: Emby.Server.Sync.SyncManagerEntryPoint. Duration: 0.0417534 seconds
aug 12 22:50:29 osmc emby-server[10622]: Info App: Starting entry point Emby.Server.Sync.SyncNotificationEntryPoint
aug 12 22:50:29 osmc emby-server[10622]: Info HttpClient: GET https://www.mb3admin.com/admin/service/EmbyPackages.json
aug 12 22:50:29 osmc emby-server[10622]: Info App: Entry point completed: Emby.Server.Sync.SyncNotificationEntryPoint. Duration: 0.0269649 seconds
aug 12 22:50:29 osmc emby-server[10622]: Info App: Starting entry point EmbyServer.Windows.LoopUtilEntryPoint
aug 12 22:50:29 osmc emby-server[10622]: Info App: Entry point completed: EmbyServer.Windows.LoopUtilEntryPoint. Duration: 0.0015149 seconds
aug 12 22:50:29 osmc emby-server[10622]: Info App: All entry points have started
aug 12 23:38:05 osmc emby-server[10622]: Stack overflow.
aug 12 23:38:05 osmc systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
aug 12 23:38:05 osmc systemd[1]: emby-server.service: Failed with result 'signal'.

And service status after each service restart of reboot looks like this:

emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2021-08-13 08:01:56 CEST; 7min ago
  Process: 19522 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=ABRT)
 Main PID: 19522 (code=killed, signal=ABRT)

aug 13 08:01:55 osmc emby-server[19522]: Info App: Starting entry point Emby.Server.Implementations.EntryPoints.AutomaticRestartEntryPoint
aug 13 08:01:55 osmc emby-server[19522]: Info App: Entry point completed: Emby.Server.Implementations.EntryPoints.AutomaticRestartEntryPoint. Duration: 0.0021425 sec
aug 13 08:01:55 osmc emby-server[19522]: Info App: Starting entry point Emby.Server.Implementations.EntryPoints.KeepServerAwake
aug 13 08:01:55 osmc emby-server[19522]: Info App: Entry point completed: Emby.Server.Implementations.EntryPoints.KeepServerAwake. Duration: 0.0009087 seconds
aug 13 08:01:55 osmc emby-server[19522]: Info App: Starting entry point Emby.Server.Implementations.EntryPoints.LibraryChangedNotifier
aug 13 08:01:56 osmc emby-server[19522]: Info App: Entry point completed: Emby.Server.Implementations.EntryPoints.LibraryChangedNotifier. Duration: 0.0283994 seconds
aug 13 08:01:56 osmc emby-server[19522]: Info App: Starting entry point Emby.Server.Implementations.EntryPoints.ServerEventNotifier
aug 13 08:01:56 osmc emby-server[19522]: Stack overflow.
aug 13 08:01:56 osmc systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
aug 13 08:01:56 osmc systemd[1]: emby-server.service: Failed with result 'signal'.

Please see the attached logs for more information.

Some details about my setup:
Hardware: Vero 4K, running OSMC (Debian Stretch)
Server: Emby 4.6.4.0 or 4.7.0.7 Beta (Same thing happens)
Architecture: ARMHF / ARMv7 (tried with ARM64, but that wouldn't install).

My own initial thoughts of what might cause the crash:
1) Not properly supported hardware (the Vero 4K is made for running OSMC (Kodi) and AFAIK not a 'clean' version of ARMHF Debian
2) The server simply being overloaded when handling everything at once (but wouldn't it then be able to restart?)
3) Some random issues with the TVHeadend handling (why I tried both the plugin and through the native M3U support)
4) Some issue with FFMpeg (I see some FFMpeg log errors, and some channels won't play)

Thanks so much in advance for helping me solve this!

embyserver_initial-crash.txt embyserver_startup-crash.txt

[RasmusHetoft 0]

Okay, here's an update. I tried uninstalling and reinstalling setting everything up again. Then, as I added my Movies library during the initial setup guide, it happened again. Guess that rules out crashing because of TVHeadend then... Emby server status reads:

emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2021-08-13 11:32:20 CEST; 59s ago
  Process: 3225 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=ABRT)
 Main PID: 3225 (code=killed, signal=ABRT)

aug 13 11:31:34 osmc emby-server[3225]: Info App: SyncRepository Initialize write lock taken
aug 13 11:31:34 osmc emby-server[3225]: Info App: Entry point completed: Emby.Server.Sync.SyncManagerEntryPoint. Duration: 0.0281016 seconds
aug 13 11:31:34 osmc emby-server[3225]: Info App: Starting entry point Emby.Server.Sync.SyncNotificationEntryPoint
aug 13 11:31:34 osmc emby-server[3225]: Info App: Entry point completed: Emby.Server.Sync.SyncNotificationEntryPoint. Duration: 0.0109702 seconds
aug 13 11:31:34 osmc emby-server[3225]: Info App: Starting entry point EmbyServer.Windows.LoopUtilEntryPoint
aug 13 11:31:34 osmc emby-server[3225]: Info App: Entry point completed: EmbyServer.Windows.LoopUtilEntryPoint. Duration: 0.0006454 seconds
aug 13 11:31:34 osmc emby-server[3225]: Info App: All entry points have started
aug 13 11:32:20 osmc emby-server[3225]: Stack overflow.
aug 13 11:32:20 osmc systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
aug 13 11:32:20 osmc systemd[1]: emby-server.service: Failed with result 'signal'.

New log file attached here, along with the hardware detection file. 

So maybe this has got something to do with adding the media..?

Update #1: Aaaand, the same thing also happens when trying to add my Music folder as a library after a while... Hmm...

Update #2: It seems to be related to scanning of my libraries, as the server runs fine if I delete library.db and restart. Not sure if it's some specific files causing the issue (though it happens both when scanning my Music, TV Shows and Movies folders), or if it's a 'general' issue, possibly related to the server being overloaded and failing while scanning...

embyserver.txt hardware_detection-63764451097.txt

Edited August 13, 2021 by RasmusHetoft

[RasmusHetoft 0]

So, I did some more research and tried a lot of different things, like disabling DLNA, tried localising corrupted files, etc.

Through all that, I found out that what caused the crashes were having "Real-time monitoring" enabled in my libraries. As soon as I turned that off, it just works – no more crashes. 

However, I want to be able to use Real time monitoring on my libraries, so how would I do that? I understand that the process uses inotify. My drive is a local USB mounted NTFS 8TB hard drive, mounted through fstab (so not a network drive). Shouldn't that work..? @Luke

If for whatever reason NTFS doesn't work with inotify, perhaps it would be an idea to write it in the description on the library settings, or maybe just having it disabled by default (at least on NTFS drives). I can't imagine being the only one who has experienced this. 

[Luke 42078]

Thanks for the update. We'll be updating the .net core runtime to a newer version in the near future and that may help with this.

 

[RasmusHetoft 0]

11 minutes ago, Luke said:

Thanks for the update. We'll be updating the .net core runtime to a newer version in the near future and that may help with this.

 

Okay, good to hear @Luke.

In the mean time, is there something else I can try to make real-time monitoring work – or could it have something to do with permissions? My fstab mount looks like this; would that be correct?

UUID=CA40D30940D2FB61 /mnt/media ntfs-3g defaults,nofail,rw,users,umask=000 0 0

 

[RasmusHetoft 0]

@Luke So, I experimented some more, and I was actually able to get Real-Time Monitoring to work! Not by enabling it in Emby, however (I became certain that RTM was what caused my crashes), but by getting and configuring Autoscan (https://github.com/Cloudbox/autoscan). It works wonders, so I'm satisfied for now. 

That also means, however, that inotify actually does work with my locally USB mounted NTFS (ntfs-3g) drive, as Autoscan is also based on that. So, the error maybe lies somewhere in the way that Emby communicates with inotify, and perhaps this is possible to fix? Or maybe as you say, the newer .net core runtime will work with this. Not having to run Autoscan would of course be preferable. 

[Luke 42078]

Emby doesn't actually use Inotify directly, it uses the .net core file monitoring api's, and those in turn use INotify. So that's an extra layer in between. Thanks for the info though. When we update we'll be getting over a year's worth of improvements to the runtime so let's re-evaluate then. Thanks.