Periodic connection failures requiring service restart

[brownbat 0]

I have Emby running on a DS213j. Works well most of the time, but every few days will timeout any attempts to connect to the server. Same result on web, Roku, or Android. The timeouts are persistent -- ie, just waiting an hour or day and trying again doesn't resolve the issue. Alternate Roku apps (Beta, Blue Neon) do not work either.

 

Restarting the Diskstation restores my ability to connect. Even just stopping and starting the Emby service from the DSM Package Center also works (the package is listed by DSM as "running" during these errors).

 

I've seen a few similar posts, but not convinced any are the exact same issue... but let me know if this is a known issue.

 

I just rebooted the NAS again, but happy to troubleshoot or gather more info next time the issue crops up. 

 

Thanks for your time, just let me know if there's any way I can help.

 

 

embyserver.txt

embyserver-63686227303.txt

embyserver-63686321195.txt

embyserver-63686364625.txt

hardware_detection-63686364671.txt

[Luke 38850]

Hi, these logs are very brief, i'm guessing they are from the restart after the problem occurred, right? Do you have the previous one? thanks.

[brownbat 0]

Unfortunately no older logs...  is it flushing the logs on restart by design?

 

I can't figure out how to access the existing logs when it's in the running-but-nonresponsive state.

 

(Possibly I just need to ssh in...)

 

Server is still running today. It's been hard to test options when it is so intermittent, but I'll post and not restart next time I catch it hanging again.

Edited February 22, 2019 by brownbat

[Luke 38850]

Let us know how you get on. Thanks.

[solabc16 379]

Hello @@brownbat

 

If/when your system hangs again, can you run the Send Logs utility and post the reference here.

 

https://github.com/MediaBrowser/Wiki/wiki/Synology-:-How-to-Send-us-Support-Logs

 

Best

- James

[brownbat 0]

Still humming fine a week later. Maybe this was a self-resolving network issue or was fixed with the update.

 

If I hit any hangs again I will run the Send Logs utility, thanks.

[Luke 38850]

Thanks for the feedback !

[brownbat 0]

@@solabc16 -

 

System was all good for a while, but the issue came back again. Sendlogs filename: 

sendlogs_NAS_synology_armada370_213j_20190318T111005UTC.tgz 

I had some issues this weekend too, but I didn't focus on the hang while I was troubleshooting collections. 

 

After the collections issue resolved, I copied over one newly ripped season (22 episodes) of a television show to the NAS. (I had seasons 1-3 of Buffy working in Emby, just added season 4).

 

When I woke up I checked the status, but couldn't access the Emby server. Accessing the NAS through Synology's web interface or ssh worked fine (but maybe a little sluggish at first?). The server did not add any metadata for the new season as it usually does, so it may have been offline all night.

 

Thanks in advance for any recommendations, happy to debug further.

[solabc16 379]

Hello @@brownbat

 

The problem with Emby Server not being available, is due to a failure of the runtime, for example I am seeing this in your logs...

------------------------------ 2019-03-17 21:40:31 UTC ------------------------------

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.ThreadPool.NotifyWorkItemComplete () <0x00033>
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x0007c] in <06b73b509b314fc8a9db8d5f73b3c223>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <06b73b509b314fc8a9db8d5f73b3c223>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0001e] in <06b73b509b314fc8a9db8d5f73b3c223>:0
/proc/self/maps:

...which results in a SIGSEGV.

 

I can improve the automatic recovery, which I will do for the next stable release. It will take a little longer to get to the bottom of what is causing it in this instance.

 

I noticed some partial and incomplete logs, are you periodically deleting the logs from Emby Server's var/logs directory?

 

Best

- James

[brownbat 0]

Thanks, that's really interesting, let me know if I can help debug on my end.

 

> I noticed some partial and incomplete logs, are you periodically deleting the logs from Emby Server's var/logs directory?

 

Weird, shouldn't be. This response was the first time I set up SSH, just used it to run sendlogs.

 

I'll run AV to make sure nothing is trashing logs on the system for antiforensics. Longshot, but best I can come up with.

 

Maybe the hardware is just failing, scary thought.

[solabc16 379]

Hello @@brownbat

 

Thanks, will do.

 

I have included the automatic recovery improvement in the latest beta, which has just gone out - so this will be in the next stable.

 

Let me know either way if you find anything causing log deletion, I'll look into it further based on your response.

 

Best

- James

[brownbat 0]

Synology's AV options weren't... fantastic. Even so, my leading theory is that I moved some log files in an early debugging attempt.

 

I apparently made some local copies of my logs on the 16th, before I even tried SSH/SCP, but, comically, I forget how...

 

It's possible I moved some logs at that time when I only intended to copy them.

 

Below are the file names and modify dates I have on my server and locally. This is just FYSA, but feel free to reach out if either:

 

( a ) the server logs look different than what you got, or,

( b ) if any of those local files look interesting and you'd like me to submit them separately.

SERVER
20190323-19:19  start-stop-status_20190324.log
20190323-16:46  embyserver.txt
20190323-05:28  embyserver-63688915739.txt
20190322-16:16  start-stop-status_20190322.log
20190322-06:13  embyserver-63688832017.txt
20190321-13:26  start-stop-status_20190321.log
20190321-02:17  embyserver-63688731434.txt
20190320-02:44  embyserver-63688655698.txt


LOCAL
20190316-09:57      998  certsync.log
20190316-09:57      184  drmsetup.log
20190316-09:21   75,384  embyserver-63688314173.txt
20190316-09:44   48,624  embyserver-63688316284.txt
20190316-11:54  259,618  embyserver.txt
20190316-09:57  146,019  embysvr.stderr.txt
20190316-09:59  133,018  embysvr.stdout.txt
20190316-09:58    6,487  ESdaemon.log
20190316-09:24   93,577  hardware_detection-63688314266.txt
20190316-09:58   93,577  hardware_detection-63688316337.txt
20190316-11:52   36,927  start-stop-status_20190316.log

Note: Server is currently crashed again unfortunately, restarting. Haven't been watching anything lately, just copying over additional content today. Can't wait for improved recovery!

 

Thanks for all your help.

Edited March 24, 2019 by brownbat

[solabc16 379]

Thanks for the update and information above @@brownbat, I'll make sure to follow this up once the next stable is released, to see where we're at.

 

Best

- James