[Solved][Armbian] Random crashing, status=6 or 11 at times, even when not streaming

[Zoxfee 1]

I've updated to Chromecast w/ GoogleTV and installed the Emby app & unlocked.

The server itself runs on a Cubox-i4pro running Armbian. Setting up was straight forward but I am having difficulty diagnosing why I get random crashes

At first I had an issue with Opensubtitles where a long response time or a "timeout" would result in a crash, so I disabled auto download and only request subs when I need them

Then every once in while while streaming the stream would freeze and check that emby-server crashed. Nothing of note stood out in the logs but an exit code "status=11" from systemctl. Restart, started stream again, crashed again but much later on. Again nothing of note besides the status 11.

Today I am not even streaming but just organizing my TV shows (scraper failed to combine seasons so I am manually identifying them) and it crashed. This time with status=6. Attached is the logs from this crash

serv@CuboxServ:~$ systemctl status emby-server
● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2021-03-12 08:42:40 EST; 15min ago
  Process: 5639 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=ABRT)
 Main PID: 5639 (code=killed, signal=ABRT)

Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: SyncRepository Initialize taking write lock
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: SyncRepository Initialize write lock taken
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: Entry point completed: Emby.Server.Sync.SyncManagerEntryPoint. Duration: 0.0461303 seconds
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: Starting entry point Emby.Server.Sync.SyncNotificationEntryPoint
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: Entry point completed: Emby.Server.Sync.SyncNotificationEntryPoint. Duration: 0.0183993 seconds
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: Starting entry point EmbyServer.Windows.LoopUtilEntryPoint
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: Entry point completed: EmbyServer.Windows.LoopUtilEntryPoint. Duration: 0.000899 seconds
Mar 11 13:39:04 CuboxServ emby-server[5639]: Info App: All entry points have started
Mar 12 08:42:40 CuboxServ systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
Mar 12 08:42:40 CuboxServ systemd[1]: emby-server.service: Failed with result 'signal'.

 

Is there something I am missing from the logs? Could it be resource allocation that I can remedy with putting emby-server to higher priority?

embyserver.txt hardware_detection-63751066745.txt

Edited April 12, 2021 by Zoxfee
Solved tag

[Zoxfee 1]

Another crash today, only when opening the "Movies" library from the webpage

This time status=11, segment fault?

serv@CuboxServ:~$ systemctl status emby-server
● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Mon 2021-03-15 13:16:08 EDT; 13min ago
  Process: 5934 ExecStart=/opt/emby-server/bin/emby-server (code=killed, signal=SEGV)
 Main PID: 5934 (code=killed, signal=SEGV)

Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: SyncRepository Initialize taking write lock
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: SyncRepository Initialize write lock taken
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: Entry point completed: Emby.Server.Sync.SyncManagerEntryPoint. Duration: 0.3872092 seconds
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: Starting entry point Emby.Server.Sync.SyncNotificationEntryPoint
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: Entry point completed: Emby.Server.Sync.SyncNotificationEntryPoint. Duration: 0.0183883 seconds
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: Starting entry point EmbyServer.Windows.LoopUtilEntryPoint
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: Entry point completed: EmbyServer.Windows.LoopUtilEntryPoint. Duration: 0.0013744 seconds
Mar 15 09:36:13 CuboxServ emby-server[5934]: Info App: All entry points have started
Mar 15 13:16:08 CuboxServ systemd[1]: emby-server.service: Main process exited, code=killed, status=11/SEGV
Mar 15 13:16:08 CuboxServ systemd[1]: emby-server.service: Failed with result 'signal'.

Attached is the most recent log

Only thing that I can think is that there were a bunch of GET calls without waiting for a RESPONSE from the webpage?

embyserver.txt

[Zoxfee 1]

After some digging I noticed the /opt/emby-server directory was not owned by emby so I have chown -R that

Double-checked every other folder emby needs and it all has the right ownership

Now no more random crashing, just the typical crash-while-streaming

 

Recently learned there is a debug info switch so I checked that, did a stream test and after a little while this error shows up and causes a program shutdown (not a crash, no signal sent to systemctl)

2021-03-16 10:09:50.676 Error GlobalExceptionHandler: UnhandledException
        *** Error Report ***
        Version: 4.5.4.0
        Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_armhf.deb
        Operating system: Linux version 5.10.21-imx6 (root@hirsute) (arm-linux-gnueabihf-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GN
        Framework: .NET Core 3.1.8
        OS/Process: arm/arm
        Runtime: opt/emby-server/system/System.Private.CoreLib.dll
        Processor count: 4
        Data path: /var/lib/emby
        Application path: /opt/emby-server/system
        System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
           at System.Threading.ThreadPoolWorkQueue.Dequeue(ThreadPoolWorkQueueThreadLocals tl, Boolean& missedSteal)
           at System.Threading.ThreadPoolWorkQueue.Dispatch()
           at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
        Source: System.Private.CoreLib
        TargetSite: System.Object Dequeue(System.Threading.ThreadPoolWorkQueueThreadLocals, Boolean ByRef)

2021-03-16 10:09:50.845 Info Server: Stopping HttpListener...

 

 

Checked that bin folder and noticed the apps are from Aug 2017??

Edit: disregarding the 2017 files, even the files from the armhf.deb are showing as 2017, oddly enough

 

Edited March 16, 2021 by Zoxfee

[Zoxfee 1]

Disabled real time monitoring on all libraries as it was suggested in another thread

Once again this evening had a crash of a different sort

Mar 16 20:25:58 CuboxServ emby-server[5209]: Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corru
Mar 16 20:25:58 CuboxServ emby-server[5209]:    at System.Threading.ThreadPoolWorkQueue.Dequeue(System.Threading.ThreadPoolWorkQueueThreadLocals, Boolean ByRef)
Mar 16 20:25:58 CuboxServ emby-server[5209]:    at System.Threading.ThreadPoolWorkQueue.Dispatch()
Mar 16 20:25:58 CuboxServ emby-server[5209]:    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Mar 16 20:25:58 CuboxServ systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
Mar 16 20:25:58 CuboxServ systemd[1]: emby-server.service: Failed with result 'signal'.

Mentions corruption but was able to restart the file fine. Note the "System.Threading.ThreadPoolWork%%()" is the same in all the errors I am getting

[Luke 37164]

hi @Zoxfee are you still having an issue with this?

[Zoxfee 1]

@Luke Yes, I haven't responded back in while as I've tried a few things but still get similar results

• Updated to the beta, reset any permissions to allow emby to use /var/lib/emby & /opt/emby-server dirs. No change, still crashing
• Modified the launch command to use the Armbian supplied ffmpeg (/usr/bin/[ffmpeg & ffprobe] vs the $APP_DIR, ffdetect is not provided by Armbian). Had snappier library scrolling & what seemed to be file loading. However then began crashes with 11/SEGV and no unhandled log file created
• When emby began scanning music the Armbian supplied ffprobe began to fail (status 139) which killed emby-server. Since then have switched back to $APP_DIR/ffprobe and while library & file loading is slower I have not had any issues with scanning files

This has been the last modification I have done to the setup. Emby-server while not serving a file has not had any more issues since switching back the ffprobe. However when serving a file after some time a crash will occur with 11/SEGV as the reason in systemctl & no unhandled log file created. Crash can occur a few minutes into a file or after a number of files played in the playlist.

Attached is the latest log with crash which was caused by playing a file to the web client (similar results when playing to a local Chromecast with Emby installed but I am away from home during testing). This time I did get a unhandled log before the server crashed and, strangely, the systemctl reports it as exited successfully, as if it was sent an exit command vs a crash signal

serv@CuboxServ:~$ systemctl status emby-server
● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2021-03-23 09:39:19 EDT; 5min ago
  Process: 1410 ExecStart=/opt/emby-server/bin/emby-server (code=exited, status=0/SUCCESS)
 Main PID: 1410 (code=exited, status=0/SUCCESS)

Mar 22 19:12:57 CuboxServ emby-server[1410]: Info App: Entry point completed: EmbyServer.Windows.LoopUtilEntryPoint. Duration: 0.000751 seconds
Mar 22 19:12:57 CuboxServ emby-server[1410]: Info App: All entry points have started
Mar 23 09:39:18 CuboxServ emby-server[1410]: UnhandledException
Mar 23 09:39:18 CuboxServ emby-server[1410]: System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
Mar 23 09:39:18 CuboxServ emby-server[1410]:    at System.Threading.ThreadPoolWorkQueue.Dequeue(ThreadPoolWorkQueueThreadLocals tl, Boolean& missedSteal)
Mar 23 09:39:18 CuboxServ emby-server[1410]:    at System.Threading.ThreadPoolWorkQueue.Dispatch()
Mar 23 09:39:18 CuboxServ emby-server[1410]:    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Mar 23 09:39:18 CuboxServ emby-server[1410]: Source: System.Private.CoreLib
Mar 23 09:39:18 CuboxServ emby-server[1410]: TargetSite: System.Object Dequeue(System.Threading.ThreadPoolWorkQueueThreadLocals, Boolean ByRef)
Mar 23 09:39:19 CuboxServ systemd[1]: emby-server.service: Succeeded.

 

embyserver.txt

Edited March 23, 2021 by Zoxfee
Missed a word

[Zoxfee 1]

Another crash, no unhandled log created but systemctl reports:

Mar 26 11:53:02 CuboxServ emby-server[25615]: Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Mar 26 11:53:02 CuboxServ emby-server[25615]:    at System.Threading.ThreadPoolWorkQueue.Dequeue(System.Threading.ThreadPoolWorkQueueThreadLocals, Boolean ByRef)
Mar 26 11:53:02 CuboxServ emby-server[25615]:    at System.Threading.ThreadPoolWorkQueue.Dispatch()
Mar 26 11:53:02 CuboxServ emby-server[25615]:    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Mar 26 11:53:02 CuboxServ systemd[1]: emby-server.service: Main process exited, code=killed, status=6/ABRT
Mar 26 11:53:02 CuboxServ systemd[1]: emby-server.service: Failed with result 'signal'.

Each library is a folder of symlinks to each media file located in another dir. The dir that hosts the content is an unorganized combination of all formats, so a folder with symlinks allows me to keep it organized for Music/TV/Movies seperately.

The library folder & symlinks are all owned by use emby just like the /opt/emby & /var/lib/emby dirs. The source dir is not..

Would this raise an issue? Most of these crashes occur after nearly an hour of playing content so Emby is proven to have read access to these files but I cannot find more info as to what exactly is this "read or write protected memory" it is trying to access.

 

Scrubbing through stackoverflow it seems to be an issue related to .NET framework or it's call to x86 vs. other architectures

[Luke 37164]

As a test, can you try without using symlinks?

[Zoxfee 1]

I'll give it a shot

Could I just make multiple libraries of the same source folder? Like make one of TV, Movies, and Music but all have the same messy source folder for each?

Want to know before I try to scan a few hundred movies from a folder of a few thousand files..

[Luke 37164]

You can't use the same folder in multiple libraries of varying content types, but is that needed to try and reproduce this?

[Zoxfee 1]

No not necessarily, I can make a test dir right now

I was only asking to avoid that. Also because I do not know how to organize my source dir to separate the content cleanly (it is a sink for all forms of downloads/backups into that folder)

Will update this post after testing non-symlink playback

2 hour + streaming to web client and not a single crash when going direct to source vs using symlink

Is there any option I have to use maybe hard-links vs symlinks? SInce I cannot make multiple libraries of the same folder..

Edited March 28, 2021 by Zoxfee
promised update

[Zoxfee 1]

Further testing, I have made another test but this time with a symlink w/ 777 permissions and another 2+ hours working

I'll try to keep this up for a week and see if there are any errors down the road

Alas, after some time I began to get 11/SEGV crashes again. No unhandled log mentions, no permission exception lines in the log.

This may be caused by a different problem but to be sure I am streaming directly from source again to see if I get any SEGV signals. If I do then it is a deep problem I am dealing with.

Edited March 29, 2021 by Zoxfee
new info

[Zoxfee 1]

After much back and forth I decided to rebuild the media folders and have Emby not use any symlinks

The result is no crashing and much snappier loading of libraries on clients.

Playback is still an issue but that is more than likely a hardware limitation (stutters but no crashes like before)

 

Conclusion: Symlinks may appear to work but are not reliable

[Luke 37164]

Thanks for the feedback !