Jump to content

Emby Server Hangs on DS1813

childsy

By childsy
in Synology

 Share
Go to solution Solved by solabc16,

Recommended Posts

shad0h

shad0h   8

Posted
Posted

Hi James, 

 

Yes I guess my library is large - I have about 2200 movies, and about 300 TV shows (maybe 13K episodes).
 

What interests me is what has changed between the old manual installation and the package installation.

Prior to making the change, I was running flawlessly on both the stable and then also the beta (up to 3.1.82).

 

Since migrating to the package, I am getting this "hanging" issue when performing just about any task on the original 3.0.6020, and then also the same issues after migrating back to the beta on every version up to 3.1.85

Each of the installs of the package, I have dropped back to a clean install - removing all the old instance - so there should be no remnants of previous installations impacting.

On this latest install of the beta package, I have moved my metadata folder to another location to see if there was an issue with capacity etc in the original directory. 

This doesn't seem to have made any difference.

 

I currently have both DLNA fully disabled and library monitoring set to off. While just watching a show through Kodi now, the server disconnected and reconnected many times, so it really isn't well, though with both these disabled, I am not seeing a web page error regarding "Too Many Files Open" at all. I can also consistently get the application to freeze when doing a couple of identifies of a tv program, or running a full scan (by about 60%), so I guess that process of seeking / updating metadata seems to be bringing it to it's knees, and that is why I suspected open files.

In any case, here are some recent logs - it has been hanging a lot, so there should be more than a few remnants of issues to locate.

 

At this point my system is unstable enough I have no issues blowing it away in the name of diagnosis, so please let me know if you need me to make any changes, or share anything else - 1812+ 3GB Ram

 

server-63605763149.txt

server-63605770373.txt

start-stop-status_20160731.log

start-stop-status_20160801.log

start-stop-status_20160802.log

unhandled_703418b7-fb7d-4a7a-8471-8504caa80739.txt

post-125325-0-29973900-1470138820_thumb.png

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted

Hi @@shad0h

 

Can you follow the steps here to send across a full set of logs, which should be easier for you and gives us some more information: https://github.com/MediaBrowser/Wiki/wiki/Synology-:-How-to-Send-us-Support-Logs

 

Also, appreciate if you can run the steps above to get a list of open files, next time your system becomes unstable.

 

From the logs above, the unhandled exception 'Unable to Open Database File' was caused by 'Too Many Open Files', so the root cause appears the same.

 

The most notable change is support for Mono 4.4SR0, but I'll leave this particular point to @@Luke to comment further.

 

- James

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted (edited)

 

Hi James, 
 
Getting a different result to your output.

 

 

 

Hi @shad0h

 

One moment, I can see the problem, will update instruction.

 

- James

Edited by solabc16
Link to comment
Share on other sites
shad0h

shad0h   8

Posted
Posted

Hi James, it was when I ran that - in the hung state of presenting error regarding Too Many Open Files.

 

I have just restarted it following running the log script (they should be with you now), and here is the result of that command again.

However, at this time the system is working - not hung yet - and roughly the same outcome.

 

SSH Output 2.txt

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted

Hi James, it was when I ran that - in the hung state of presenting error regarding Too Many Open Files.

 

I have just restarted it following running the log script (they should be with you now), and here is the result of that command again.

However, at this time the system is working - not hung yet - and roughly the same outcome.

 

Thanks for the logs, I've just updated the instruction above, so if you try that it should now work as expected.

 

- James

Link to comment
Share on other sites
andcbii

andcbii   12

Posted
Posted

@@solabc16 To answer some of your questions from this thread...

 

Yes I installed using the DSM package installer. My library is about 550 movies and 80 TV shows (maybe 1000 episodes). I'll try running the command you posted later today. I would be will to try a beta package. 

Link to comment
Share on other sites
shad0h

shad0h   8

Posted
Posted

Hi James, 

 

Here is the first rerun of the command. I have not received the error yet, but the application does appear to have hung again from a web management point of view.

SSH Output 4.txt

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted

Hi James, 

 

Here is the first rerun of the command. I have not received the error yet, but the application does appear to have hung again from a web management point of view.

 

Hi @shad0h

 

Thanks, that's useful. If we top and tail the output, we can see we have 1,024 FDs, which is what we expected; but we can now see what they are.

 

Device Files   : 2

Regular Files : 8

Socket           : 20

Named Pipe   : 994

 

The standout is course the number of named pipe FDs.

 

- James

  • Like 1
Link to comment
Share on other sites
mrbasic1

mrbasic1   5

Posted
Posted (edited)

Yes, I'm having the hang condition when doing a media scan on my 415+ with the latest 3.1.86.0 beta. Log indicates to many open files, however I was able to resolve a lot of UN-scaned media by using multi-select and selecting the UN-identified items per page then doing a refresh for those in a group (no hang doing that). Then setting the schedule for media scan to weekly on Sunday. That stopped the automatic media scan and inevitable hang condition at least till Sunday. Also disabled real time monitoring so it seems stable now and am able to stream movies without a hang thus far. Other issues are the Identify feature of metadata manager provides a list of identified titles but none can be selected to resolve mismatches. I can get by this way till the issues are fixed.

Edited by Mike DeGeer
Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted

Evening

 

I've just pushed 3.1.86.0-3-beta to the beta channel. This isn't a fix for the issue being discussed here, but provides an additional utility to allow the test community to provide some more information to us.

 

If you are on the beta channel, please can you run the following next time you experience a 'Too Many Open Files' issue and post the output here:-

SSH to your Synology NAS and login as 'admin'.
Change directory : cd /var/packages/EmbyServer/scripts/utilities/sbin
Run the command  : sudo ./lsof +E -n -u embysvr

Example

 

57a114a97baee_run_lsof2.png

 

- James

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted (edited)

Morning

 

We are of course looking at this issue, I just wanted to take a moment to share my analysis so far with you.

 

The logs have proved useful and the behaviour appears to be consistent across the users that have provided this data.

 

In itself, this is a good thing, as although the end result is a failure of the process, there is only one very specific issue at play here.

 

So here's the walkthrough, using 3.1.88.0-1-beta and the Web UI to playback a movie:-

* Any invocation of ffmpeg/ffprobe (e.g. library scan/chapter image extraction) yields similiar results.

 

Before Playback

 

>> This is Emby's 'resting state', this single pipe exists for the duration of the process.

mono    21751 embysvr   10r  FIFO      0,8       0t0 25509099 pipe 21751,mono,11w
mono    21751 embysvr   11w  FIFO      0,8       0t0 25509099 pipe 21751,mono,10r

During Playback

 

>> We see stdin and stderr streams created for the ffmpeg process, 0r and 2w respectively.

>> Stream 40r is reading ffmpeg's stderr output.
>> Stream 38w is writing to ffmpeg's stdin input.

>> Stream 35r (appears) to be mirroring 38w, perhaps for the purpose of logging (@@Luke ?)

mono    21751 embysvr   10r  FIFO      0,8        0t0 25509099 pipe 21751,mono,11w
mono    21751 embysvr   11w  FIFO      0,8        0t0 25509099 pipe 21751,mono,10r
mono    21751 embysvr   35r  FIFO      0,8        0t0 25516118 pipe 21751,mono,38w 22431,ffmpeg,0r
mono    21751 embysvr   38w  FIFO      0,8        0t0 25516118 pipe 21751,mono,35r 22431,ffmpeg,0r
mono    21751 embysvr   40r  FIFO      0,8        0t0 25516119 pipe 22431,ffmpeg,2w
ffmpeg  22431 embysvr    0r  FIFO      0,8        0t0 25516118 pipe 21751,mono,35r 21751,mono,38w
ffmpeg  22431 embysvr    2w  FIFO      0,8        0t0 25516119 pipe 21751,mono,40r

After Playback

 

>> Immediately after playback the ffmpeg streams (0r and 2w) are cleaned up, along with 40r which was reading 2w.

>> The stream (38w) writing to ffmpeg's stdin and the stream logging(?) that input (35r) persist)

mono    21751 embysvr   10r  FIFO      0,8       0t0 25509099 pipe 21751,mono,11w
mono    21751 embysvr   11w  FIFO      0,8       0t0 25509099 pipe 21751,mono,10r
mono    21751 embysvr   35r  FIFO      0,8       0t0 25516118 pipe 21751,mono,38w
mono    21751 embysvr   38w  FIFO      0,8       0t0 25516118 pipe 21751,mono,35r

10 Minutes After Playback

 

>> The stream that was writing to ffmpeg's stdin (38w) is cleaned up, likely by the GC.

>> Stream 35r remains, seemingly indefinitely under all tests to far.

mono    21751 embysvr   10r  FIFO      0,8       0t0 25509099 pipe 21751,mono,11w
mono    21751 embysvr   11w  FIFO      0,8       0t0 25509099 pipe 21751,mono,10r
mono    21751 embysvr   35r  FIFO      0,8       0t0 25516118 pipe

The rate of cleanup of the orphaned streams that are writing to ffmpeg's stdin and the leaked stream that is logging(?) that input, both contribute to the eventual 'too many open fiies' exception.

 

- James

Edited by solabc16
Link to comment
Share on other sites
Luke Emby Team

Luke   37049

Posted
Posted

i changed the probing process to not use stdin anymore so maybe that will help the library scan. but for playback it's something we have to have so i can't do that.

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted (edited)

Evening

 

Ok, great, will run through the same scenarios as soon as the next beta is published.

 

It nothing else, it gives us time to troubleshoot, whilst getting the platform back working.

 

Which source file is the FFmpeg part of the playback in? I'll put a minimal test harness together so I can debug the runtime and get to the bottom of it.

 

- James

Edited by solabc16
Link to comment
Share on other sites
mrbasic1

mrbasic1   5

Posted
Posted

Hi,

 

Confirmed as I was able to run a full library scan after the 3.1.88.0-1 beta update on the DS415+ NO HANG. Also added some media 3 movies and a few TV episodes and scanned again without any hangs. Looking better and works for me at least. Thanks for fixing the Identify function.

 

Regards,

Mike D.

Link to comment
Share on other sites
solabc16

solabc16   379

Posted
Posted (edited)

Evening

 

Thanks for the update, if your system is still running after the scan (i.e. you haven't restarted it), can you send the output of:-

SSH to your Synology NAS and login as 'admin'.
Change directory : cd /var/packages/EmbyServer/scripts/utilities/sbin
Run the command : sudo ./lsof +E -n -u embysvr

- James

Edited by solabc16
Link to comment
Share on other sites
mrbasic1

mrbasic1   5

Posted
Posted

I did a third full library scan while streaming a movie (King Kong 2005) to a roku 3, then did the SSH commands again after the scan finished. No Hang. Again attached the ssh file.

 

Regards,

Mike D.

SSH_EMBY_3_1_88_0_beta_2.txt

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...