[HANG] Couldn't Create thread

[puithove 209]

I've been seeing quite a bit of this lately.  Every few days, the server just goes unresponsive.  None of the clients will be able to load any media.  When you try to open the web client, the browser just spins forever.

 

Emby server log shows this just repeating over and over:

Couldn't create thread
System.ExecutionEngineException
  at (wrapper managed-to-native) System.Threading.ThreadPool:RequestWorkerThread ()
  at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () <0x419c2fb0 + 0x00047> in <filename unknown>:0 
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x419c3190 + 0x0010f> in <filename unknown>:0 
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x419c30c0 + 0x0000b> in <filename unknown>:0 

Seems to normally happen at night so when we get up in the morning and want to look at the TV, it's down, but has happened at other times through the day too, so probably just coincidence (or maybe exacerbated by maintenance tasks that run at night).

 

Seems like a thread leak perhaps?

 

Full server log since midnight attached.

[hurricanehrndz 149]

Is this on arch?

[puithove 209]

Is this on arch?

 

 

Probably should have included that.  Yes, this is Archlinux.

[hurricanehrndz 149]

Can you please grep the full run command

 

Sent from my D6603 using Tapatalk

[puithove 209]

Sure - and I believe it should be in the logs I posed as well:

/usr/bin/mono-sgen /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe

I can adjust that if needed - I'm currently using the scripts packaged by Alucryd from the Arch community repo.

[hurricanehrndz 149]

Sure - and I believe it should be in the logs I posed as well:

/usr/bin/mono-sgen /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe

I can adjust that if needed - I'm currently using the scripts packaged by Alucryd from the Arch community repo.

Are you using any mono env variables that adjust how many threads per cpu are allowed? `MONO_THREADS_PER_CPU`

[puithove 209]

Actually, yes.

MONO_THREADS_PER_CPU=500
MONO_GC_PARAMS=nursery-size=512m

[hurricanehrndz 149]

Try playing with the mono_threads_per_cpu?

 

Do you know if this error started to occur around the same time mono got updated? Your running version 4.2.2 currently we have only tested up to version 4.2.1.102.

[puithove 209]

The 4.2.2 update was very recent.  I think it was happening before that even.

 

Hmm... maybe bumping it up would be good... but that actually seems to be a "minimum" that it creates when it starts.  Seems like it'll create more if needed?  I dunno.  I'll bump it up and see what happens.

 

I'll reply back after a while here with an update.

 

Oh - and thanks for the assist!  

Edited February 1, 2016 by puithove

[puithove 209]

So changing the mono threads didn't fix it.  What seems to have finally fixed it was turning off the real-time monitoring for library changes.  I saw a couple inotify errors in the logs, and thought it'd be worth trying.  Since turning that off, things are quite stable.  Now using a scheduled scan, and app notifications to trigger scans.