Library shenanagins with latest emby versions

[isamudaison 12]

So I've 'resolved' my emby hanging by removing very large paths that couldn't be resolved, but now I've got a problem with mono-sgen pegging my CPU.

 

Attached is a log, and it's obvious there's something very wrong with the library monitoring (notice the massive 'type change detected' entries). 

 

Here's the process that's hogging all my CPU:

 

/usr/bin/mono-sgen --optimize=all /usr/lib/emby-server/bin/MediaBrowser/Server/Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh

 

 

emby_debug.txt.tar.gz

[Luke 42077]

Do you see the same behavior if you disable the realtime monitor?

[isamudaison 12]

Do you see the same behavior if you disable the realtime monitor?

I don't seem to have that option under 'library->advanced' (or anywhere, really...)

[mastrmind11 722]

I don't seem to have that option under 'library->advanced' (or anywhere, really...)

it's per library, so click on each and disable it.

[isamudaison 12]

I see some oddness with realtime monitoring... I copied a file over to one of the library locations and pulled up a system monitor to see what happens.  After a minute or so, the 'mono-sgen' process kicks up CPU usage quite a bit, and then it started to settle down, and then it kicked up even more, and started to settle down, and then it kicked up even more... it did this for a few minutes, but then finally settled back down to zero.  I could see this happening and never recovering as being a real possibility.

[isamudaison 12]

it's per library, so click on each and disable it.

That makes sense, I've disabled them all and set the media library scan to hourly, I'll keep an eye on it.

[isamudaison 12]

No bueno... I've added back my home video category and as it was scraping, the mono process spiked CPU and emby hung... I'm guessing there needs to be a gdb dump to figure this out?

[Luke 42077]

Yes please, thanks.

[isamudaison 12]

Well this is fun... every time I attempt to attach to the emby process, gdb throws an error and then emby hangs.  Ever seen this error from gdb?

 

"../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory"?

[isamudaison 12]

Here's as much as I could get (emby is in the hung state with gdb attached, I performed a bt full (see attached)

 

I think the real problem is there are over 2000 threads running...

 

*Edit: added 'full' thread dump

gdb.txt

Edited February 23, 2017 by isamudaison

[isamudaison 12]

Possibly related:

https://bugzilla.xamarin.com/show_bug.cgi?id=42665

Seems like maybe I'll have to wait until Mono releases a new stable (I'm assuming Emby has done some tweaking to their mono usage in the past couple versions to add all this 'speed' ?)

[isamudaison 12]

Wow, The mono peoples' ears must've been burning   They've just pushed 4.8 to stable.  I've upgraded and will monitor any improvements.

[Luke 42077]

Wow, wasn't expecting that just yet. Thanks for the info.

[isamudaison 12]

Just a quick update: so far, so good.  No apparent freezing/crashing, and I've also enabled real-time monitoring.  I'll give it another day before I call it 'solved'

 

Until then I'd highly recommend updating to the new mono to anyone having library scanning issues

[Luke 42077]

Very interesting, thanks for the info. For us we're not likely to put 4.8 into our stable branch yet but we'll probably start looking at it in beta soon. Thanks.

[isamudaison 12]

Let's call this solved... It's been Rock solid after the upgrade.