Scan/Refresh/Download failures causing memory exception errors and server crashes almost daily

[Dignow 2]

Ok, This is my first time reporting a problem so bear with me if I miss something.  I am following the 'how to report a problem' thread so hopefully I've got it all covered.

 

The issue concerns MB Server.  I'm using version 3.0.5050.37565.  My physical 'server' setup is a bit odd. The machine is a Windows 7 (64bit) box, Intel i7, 3.4GHz processor, 8GB RAM.  The media files are spread over 16 hard drives in Addonics mini towers that are connected to the PC via eSATA to USB. I was using an HP WHS until about three weeks ago when the server up and died on me so now I have to live with this setup until a new server can be arranged.  For the time being it seems to be working.

 

I'm not sure exactly what happens other than to say at least once a day (if not two or three times) during a library scan, a people refresh, a trailer download, etc a out of memory exception error is thrown. Unfortunately, this time I moused over the server icon in my tray before getting a screen grab of the error boxes so I don't have the exact error message but there were five or six message boxes open which seems to correlate to the the six error message on my dashboard.  This is generally nothing more than a nuisance as all that happens is the outlying images don't update - that time - and I have to restart the server.  That gets to be a pain as it takes the server software sometimes as long as 15-20 minutes to reinitialize and populate my different collections (movie folder, TV folder, etc).  

 

Anyways, screenshots and log files are here:  https://www.dropbox.com/sh/g0zpz3ep0czox7y/-TyCJvLg_J.  I wasn't sure which of the log files to include so I've uploaded everything over the last 24 hours to that dropbox folder.  If this is just the growing pains of beta software I completely understand.  If there is something I can do on my end please point me in the right direction and I'll take care of it.  If you need more info just let me know what you need and I'll throw it in the DB folder asap.  

 

 

 

 

[ebr 14904]

Interesting.  Looks like the server is crashing attempting to write a log message of an http request.

[Dignow 2]

Three more crashes in the last 12 hours.  I've been adding new log files to the DB folder as they are available.  Just posting it all as I'm not sure what's relevant/important and what's not.

[Dignow 2]

Alright, Theme Songs uninstalled and server restarting.  Its after midnight over here so I'll check in the morning and let you know if I've had any more crashes.

[ebr 14904]

For kicks, try removing the theme songs plug-in.  Nothing really to tell me that is the problem, it is just the only plug-in I see that might be making http requests and that will at least eliminate it.

[Dignow 2]

Crashed again overnight.  I've started a new DB folder to segregate the events by day.  New logs/screen caps are here: https://www.dropbox.com/sh/76fbqy97wqla7wo/AaTZrtEo1P

[ebr 14904]

This all looks like a file system issue of some sort.  Does the user running the server have full access to your media directories.  It looks like we don't have the ability to write but you have the save option enabled.

[Dignow 2]

Ok, I've confirmed that the user has 'full control' over all drives - there were three that were not set correctly so fingers crossed that was the issue.  As far as memory usage, I'll have to check - I really have never looked.  

 

EDIT - just dropped a screen grab of the resource monitor showing the current MBS memory usage in the last DB folder (link above).

Edited November 5, 2013 by Dignow

[Dignow 2]

Just added new logs - another out of memory exception error... makes five for today I think.

[Luke 37022]

We have an action item to reduce the memory foot print and it will be in an upcoming server release (not the next one). Workarounds in the meantime are turning off chapter + people scheduled tasks completely, and cover art. possibly trailers too. but do all that one at a time to find out what helps or you'll never really know.

[Luke 37022]

what kind of memory usage do you generally see from the server?

[Dignow 2]

Ok, will do.  I'll start turning plugins off one at a time and post up any improvements.  In the meantime I'll keep adding logs and screen caps to daily folders in the event that something turns up that's useful.  New DB link:  https://www.dropbox.com/sh/zor4jrvsu02algg/7UkDHE3Cux.  Thanks for checking into this.

[ebr 14904]

I have a hard time believing this is truly a memory consumption issue.  I mean, I know that is the error that is happening, but I think it is related to some odd situation as opposed to the amount of memory the app consumes normally.  So I think we do need to figure out what is going on here.

[Dignow 2]

Well it is still happening.  I've uninstalled the trailers plugin and turned off image extractions.  Neither seemed to have any effect.  I've now turned off people refresh.  If another crash occurs I'll turn off the cover art plugin.  After that I'm all out of things to do except to turn off scanning my collection - which I assume would cause any new media added not to show up with the clients.  Speaking of, I have noticed that during the crash/restart cycle that my "Movies" tab (and only my movies tab) always displays zero content.  If I go to that tab and sort by actor/direct/year/etc the movies are all there and can be eventually viewed and played but the root still says zero.  Thought there might be some correlation so I'm mentioning it.  New logs and images are still being added and put into folders by day.

[Dignow 2]

OK, I have now turned off everything but the library scan - and have NOT had a crash in the last 18 hours.  A new record.  The last thing to get turned off was the CoverArt plugin which I was very sad to see go.  I was thinking I might try re-installing the other plugins and reactivating the other scans one at a time to see if CoverArt was the sole culprit.  Still posting logs and screen grabs to the DB folder.

[crashkelly 62]

You probably know this already, but if you uninstall all of the plug-ins and put them back one at a time, leaving enough time between each plug-in installation to be "confident" that things are working correctly, server is not crashing and memory usage is normal, when you put the plug-in in that causes the issue, assuming it is a plug-in, it will be way easier to sort out what is going on and the interactions the new plug-in is having with the system and the other plug-ins.

 

I would suggest taking a copy of the running log file just before you reinstall a plug-in so there will be lots of information available for comparison.

 

As I said, you probably know all that already, but I just thought I would share

 

Cheers and good luck

[ebr 14904]

OK, I have now turned off everything but the library scan - and have NOT had a crash in the last 18 hours.  A new record.  The last thing to get turned off was the CoverArt plugin which I was very sad to see go.  I was thinking I might try re-installing the other plugins and reactivating the other scans one at a time to see if CoverArt was the sole culprit.  Still posting logs and screen grabs to the DB folder.

 

CA isn't really involved in library scans, nor does it make http requests so this is surprising. 

[Dignow 2]

Well if you thought that was surprising you're going to love this.  I went three days with no crashes, turned CA back on and it crashed within 4 hours.  Here's the best breakdown of what I did that I can detail for you:

1. On 9 Nov all task triggers and plugins (with the exception of Genre Cleaner and Rotten Tomato Reviews) were turned off.

2. On 10 November AM I manually ran a library scan - no problem.

3. On 10 November PM I manually ran a people refresh - no problem.

4. On 11 November PM I ran an image extraction by task trigger - no problem

5. Late PM 11 Nov I updated the server to version 3.0.5061.25570, installed the Restart Plugin, then re-installed CoverArt.

6. Early AM 12 Nov out of memory errors started hitting - even though there were no other scans that had been manually set to run or set by trigger.

 

I've uploaded all the logs I have to the DB folder and am now uninstalling CA again.

[ebr 14904]

I think what we have here is the fact that CA is as large as it is, it is the straw that breaks the camel's back.  I just scanned the log and CA was not involved in the scan that caused the last error.

 

Your logs are huge due to the number of unauthorized access exceptions that are occurring.

 

Do you have the logging window turned on on your server?  If so, try turning that off as I have a hunch that is the issue.

 

Then, we need to get those errors corrected.  Does the server user have proper rights to your media folders?  If not and you don't want it to, then turn off the option to save metadata within them.

[Dignow 2]

Ok, you'll have to forgive me but I'm not exactly sure what the "logging window" is on the server nor how to turn it off.  The only other option I have on permissions of these hard drives is to allow "Everyone" full control.  All other entities shown already have full control.  I'll give letting 'everyone' have full control a shot and see what happens there. If that's not successful I can certainly tell the server to stop saving metadata to the media folders but won't that eliminate a good chunk of the functionality of the software or does that data not have to be stored locally?

[ebr 14904]

If you want to save it locally, then lets concentrate on getting the permissions fixed.

 

If you don't know what the log window is, then you probably don't have it on.  It looks like this:

 

 

And is turned on with the right mouse menu on the system tray:

 

[Dignow 2]

I don't have developer tools turned on so I'm assuming that's why I don't have the log window functionality in my system tray icon. I'm assuming since you're saying to turn it off I should leave things as they are.

 

 

I've been as liberal with my security permissions as I know how to be (see the DB folder for typical permissions).  Is there something more advanced that I can do to give MBS the permissions it needs to write to my drives?

 

New logs and images are in the DB folder as usual.

[ebr 14904]

Are they network locations?  Have you tried this?

[Dignow 2]

They are not network locations - they are physically connected to the PC the MBS software is on.  They're in Addonics mini-towers with a eSATA port multiplier that is then connected to the PC via eSATA-to-USB adapters.  I thought the Addonics boxes would JBOD the drives (like the WHS used to do) but they all appear individually.  They are all shared by the MBS PC on my home network.  I've made the registry edit to see if that does the trick.  So far I've left CA uninstalled to keep the server up and running.  New logs are uploaded.

[Dignow 2]

HI again, I know its been a while but I thought I'd post an update to this as something a bit bizarre was going on.  With the Cover Art plugin disabled all of the memory exception errors stopped.  So, I left CA uninstalled.  However, I was still having issues with MB3 downloading the supporting art files to media folders (backdrops, folder images, disc images, logos, etc.).  Once the MBS upgrade appeared that let me download those images directly through the web interface I found out that I was still having issues writing to the individual movie/TV folders through the software.  The only way to add the images was to do so manually.  

 

Then I upgraded to MBS 3.0.5115.35703 and without doing anything other than upgrading MBS that problem resolved itself.  All of the missing images started to populate and I could load new media without having to download all of the miscellaneous images myself.  This was all fine and good until I had the idea that this version of MBS solved whatever issue I was having that caused the access errors and that Cover Art may work now.  Re-installed CA, turned it on, loved it, immediately crashed MBS.  Uninstalled CA, went back to MBS being stable, but now I have the same issue with MBS not allowing me to download images through the web interface and it not downloading them automatically either.  Back to manually adding in all of the auxiliary images.  

 

I've reset my auto update level to Beta now in hopes of the new Beta releases possibly fixing the automatic download issues again.  I'm working under the assumption that Cover Art has altered some MBS setting somewhere that for some reason cuts the software off from the drives.  In the event that this proves incorrect and the update to a Beta version doesn't 'reset' this switch I'll have to go back to the idea that there's some access issue local to my machine.  I've done all that I know how to do to allow access to the drives so that the images can be added but in all honesty I've not done much more than go through the advanced sharing functions and allowed for the broadest possible access on all the drives.  If there is something more specific that needs to be done it is beyond my experience and I'll need someone to point me to a set of directions on what to do.

 

So, good news, MBS has been completely stable for me while Cover Art isn't installed.  Bad news, Cover Art can't be installed without crashing my MBS.  Also, after attempting to do so MBS seems to stop downloading media image files.  Any thoughts from the folks who know what they're doing would be greatly appreciated.