Out of Memory exception

[Angelblue05 4130]

I'm posting on behalf of my brother, @@saitoh183

 

I keep seeing the following error in the MBS log. I can't figure out what could cause this error or what it means. I keep seeing it when I'm watching something via Kodi. It will randomly pop up, but it seems to originate from MBS. If someone would be able to help me, I'd greatly appreciate it. It's been plaguing his server for quite a while. I've attached his server log, if more information is needed.

 

Media Browser server log: It repeats itself randomly (in the server log, it appears 12 times).

2014-12-08 00:52:52.7415 Error - DtoUtils: ServiceBase<TRequest>::Service Exception
	Exception of type 'System.OutOfMemoryException' was thrown.
	System.OutOfMemoryException
	   at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
	   at System.Text.StringBuilder.Append(Char value, Int32 repeatCount)
	   at System.Text.StringBuilder.Append(Char value)
	   at System.IO.StringWriter.Write(Char value)
	   at ServiceStack.Text.Json.JsonUtils.WriteString(TextWriter writer, String value)
	   at ServiceStack.Text.Json.JsonTypeSerializer.WriteObjectString(TextWriter writer, Object value)
	   at ServiceStack.Text.Common.WriteType`2.WriteProperties(TextWriter writer, Object value)
	   at ServiceStack.Text.Common.WriteListsOfElements`2.WriteGenericArrayMultiDimension(TextWriter writer, Array array, Int32 rank, Int32[] indices)
	   at ServiceStack.Text.Common.WriteListsOfElements`2.WriteGenericArray(TextWriter writer, Array array)
	   at ServiceStack.Text.Common.WriteListsOfElements`2.WriteArray(TextWriter writer, Object oArrayValue)
	   at ServiceStack.Text.Common.WriteType`2.WriteProperties(TextWriter writer, Object value)
	   at ServiceStack.Text.JsonSerializer.SerializeToString(Object value, Type type)
	   at ServiceStack.Text.JsonSerializer.SerializeToString[T](T value)
	   at ServiceStack.Serialization.JsonDataContractSerializer.SerializeToString[T](T obj)
	   at ServiceStack.Host.ContentTypes.SerializeToString(IRequest req, Object response)
	   at ServiceStack.RequestExtensions.ToOptimizedResult[T](IRequest request, T dto)
	   at MediaBrowser.Server.Implementations.HttpServer.HttpResultFactory.GetOptimizedResult[T](IRequest requestContext, T result, IDictionary`2 responseHeaders)
	   at MediaBrowser.Api.BaseApiService.ToOptimizedResult[T](T result)
	   at MediaBrowser.Api.UserLibrary.ItemsService.<Get>d__0.MoveNext()
	
2014-12-08 00:52:52.7415 Error - HttpServer: Error processing request for /mediabrowser/Users/f6c51e6fd330e59d16f4cc52886702e1/items?ParentId=4c836f8411cc39bf311d5267feb5c58f&IncludeItemTypes=Movie,Series,Episode,MusicArtist,MusicVideo,Video&Fields=ParentId,Overview&Recursive=true&CollapseBoxSetItems=false&format=json
	Exception of type 'System.OutOfMemoryException' was thrown.
	ServiceStack.HttpError
	No Stack Trace Available

2014-12-08 00:52:52.7505 Debug - HttpServer: HTTP Response 500 to 192.168.13.101. Response time: 19560.1666 ms.    Url: http://192.168.13.198:8096/mediabrowser/Users/f6c51e6fd330e59d16f4cc52886702e1/items?ParentId=4c836f8411cc39bf311d5267feb5c58f&IncludeItemTypes=Movie,Series,Episode,MusicArtist,MusicVideo,Video&Fields=ParentId,Overview&Recursive=true&CollapseBoxSetItems=false&format=json

Kodi/XBMB3C log: I don't believe Kodi is the problem, but it's always better to have more information....

00:04:10 T:16372  NOTICE: HTTP response error: 500 OutOfMemoryException
00:08:11 T:16372  NOTICE: Previous line repeats 1 times.
00:08:11 T:16372   ERROR: Exception in thread Thread-8:
                                            Traceback (most recent call last):
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\threading.py", line 808, in __bootstrap_inner
                                                self.run()
                                              File "C:\Users\Angel\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\ArtworkLoader.py", line 59, in run
                                                self.run_internal()
                                              File "C:\Users\Angel\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\ArtworkLoader.py", line 76, in run_internal
                                                self.updateArtLinks()
                                              File "C:\Users\Angel\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\ArtworkLoader.py", line 308, in updateArtLinks
                                                result01 = self.updateCollectionArtLinks()
                                              File "C:\Users\Angel\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\ArtworkLoader.py", line 403, in updateCollectionArtLinks
                                                collectionResult = json.loads(jsonData)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\__init__.py", line 338, in loads
                                                return _default_decoder.decode(s)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\decoder.py", line 365, in decode
                                                obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\decoder.py", line 383, in raw_decode
                                                raise ValueError("No JSON object could be decoded")
                                            ValueError: No JSON object could be decoded

Full MBS log: https://www.dropbox.com/s/9digdfoqqkhx26w/Saitoh_serverlog_12-08-2014.txt?dl=0

Edited December 8, 2014 by Angelblue05

[ebr 14913]

That query is asking for every single item (other than individual songs) in his collection - without any limit.  I can see how that could be a problem trying to build a json response for that.

 

@@xnappo - how can that be happening?

[xnappo 1593]

Yep - we do that to build the database of all of the backgrounds.  I have started to see the out of memory error as well with later versions of the server.  We could be the database piecemeal, or we could not pre-load some of it.

 

@ @@im85288 FYI - we started off not pre-loading anything, then @ added pre-loading movies, then someone (im85288 or I, I don't remember) expanded it.

 

I will note that the new speed-up code is going to need big queries to build the in-memory database too.

 

xnappo

Edited December 8, 2014 by xnappo

[xnappo 1593]

@@Angelblue05 - you can turn off 'Enable Background Image (Requires Restart)' in the Services tab to stop this huge query.  This will, however, turn off rotating backdrop art.

 

xnappo

[ebr 14913]

That doesn't seem like it will ever be a viable solution (building an entire in-memory copy of the library).  Why do you need this?

[saitoh183 137]

@@Angelblue05 - you can turn off 'Enable Background Image (Requires Restart)' in the Services tab to stop this huge query.  This will, however, turn off rotating backdrop art.

 

xnappo

 

I turned it off and restarted...now i will just wait to see if the error returns?...do you need logs?

[xnappo 1593]

Well, actually I don't think it is needed for what it is there for right now.  @ was concerned with hitting the server every time the user changed selections to get the list of background art for rotation.  

 

However - it is going to be needed for the speedup idea where we background load the majority of the information (people, images, plot etc) and then only need to do small requests and processing at run time.  This is a 3x speed-up for us.  We may end up only doing it for movies I guess.

[xnappo 1593]

I turned it off and restarted...now i will just wait to see if the error returns?...do you need logs?

Nope, just see if it stops.  

[saitoh183 137]

Nope, just see if it stops.  

 

Ok...just restarted and nothing yet...and the Error popped up pretty quick after start up.

 

 

That query is asking for every single item (other than individual songs) in his collection - without any limit.  I can see how that could be a problem trying to build a json response for that.

 

So its my music that would have been causing the issue?

[xnappo 1593]

Ok...just restarted and nothing yet...and the Error popped up pretty quick after start up.

 

 

 

So its my music that would have been causing the issue?

 

It isn't the music - it would be everything else.

 

How many TV episodes do you think you have?

[saitoh183 137]

 This will, however, turn off rotating backdrop art.

 

xnappo

 

What does this do exactly...i ask cuz i use titan and my backgrounds still rotate on the menu screen. Does this only have a ffected when you have more than one background for a TVshow/movie...it wont rotate those when you highlight them?

 

 

It isn't the music - it would be everything else.

 

How many TV episodes do you think you have?

 

870 Movies

20356 episodes

983 songs

 

 

 

Still no error on screen

Edited December 8, 2014 by saitoh183

[xnappo 1593]

Uh, yeah, those 20000 episodes will do the trick nicely.

 

I will publish a beta with a change tonight to only pre-load series/season info.

 

2 years worth of TV if you watched it 24 hours a day.. Wow!

 

xnappo

Edited December 8, 2014 by xnappo

[saitoh183 137]

Uh, yeah, those 20000 episodes will do the trick nicely.

 

I will publish a beta with a change tonight to only pre-load series/season info.

 

2 years worth of TV if you watched it 24 hours a day.. Wow!

 

xnappo

 

Its my pride and joy

 

I guess i am considered a big library?..lol

[xnappo 1593]

I think it is shows pushing it over the top though, so I will back that out.

[xnappo 1593]

Guys,

 

Please try 0.9.731 in the beta repo. 

 

xnappo

[Angelblue05 4130]

Once updated, should we try to re-enable the background image service?

[xnappo 1593]

Yes.

[Angelblue05 4130]

The error still happens. As soon as I restarted XBMC, it was there.

 

Afterwards, I disabled the background image and restarted XBMC. I get this error going in and out of menus:

18:52:54 T:11400  NOTICE: Cache_Data_Manager: Loading Cached File
18:52:54 T:13024   ERROR: Exception in thread Thread-4:
                                            Traceback (most recent call last):
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\threading.py", line 808, in __bootstrap_inner
                                                self.run()
                                              File "C:\Users\Corruption\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\NextUpItems.py", line 42, in run
                                                self.updateNextUp()
                                              File "C:\Users\Corruption\AppData\Roaming\XBMC\addons\plugin.video.xbmb3c\resources\lib\NextUpItems.py", line 119, in updateNextUp
                                                seriesResult = json.loads(seriesJsonData)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\__init__.py", line 338, in loads
                                                return _default_decoder.decode(s)
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\decoder.py", line 365, in decode
                                                obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                                              File "C:\Program Files (x86)\XBMC\system\python\Lib\json\decoder.py", line 383, in raw_decode
                                                raise ValueError("No JSON object could be decoded")
                                            ValueError: No JSON object could be decoded
18:52:54 T:10644  NOTICE: Cache_Data_Manager: CacheManagerThread Started
18:52:54 T:11400  NOTICE: Cache_Data_Manager: Returning Cached Result

Btw, if I had known it was Kodi Mediabrowser, I would have started the thread in the appropriate section of the forums...sorry about that   If you'd like xnappo, I can open up a new thread in the Kodi section for easier follow up?

Edited December 9, 2014 by Angelblue05

[xnappo 1593]

Okay, please post a server log. Without the episodes the data size should have been fine.

[xnappo 1593]

Wait - that is a different error.  I thought we were talking about the out of memory error?

 

This one looks like what happens when the server wants to be restarted for an update.

[Angelblue05 4130]

Sorry, I should have explained better. The outofmemory exception error still happens when the background image service is enabled. It happened as soon as I've restarted XBMC.

 

However, when I decided to disable the service again (awaiting further instruction), I started getting this new error, which I didn't experience before updating to 731. That's why I posted it here (just wanted to make you aware of it, in case it's something you've recently modified or if it's something I can resolve on my side, like deleting the cache database). But it seems to have corrected itself, luckily.

 

 

@@xnappo

 

MBS log after I updated to 731 and got the outofmemory error (sorry it's almost 100MB): https://www.dropbox.com/s/7lx0nme869jt0x2/server-63553603501.txt?dl=0

The error happened around 8:07pm, you can search for HTTP Response 500 (we live in different timezones)

 

Edit: I forgot to mention that I do not see the Json error in XBMC anymore (still appears in log). However, I still see the failed to connect. It does seem to only occur once after I restart XBMC.... I haven't seen the error yet, while watching content.  It just happened again and this time I even had the Background image service disabled. I will keep testing and let you know if I notice anything else.

Edited December 9, 2014 by Angelblue05

[xnappo 1593]

Okay thanks.  @@ebr @@Luke - I don't see anything unreasonable with the queries in that log.

 

xnappo

[xnappo 1593]

The JSON/connect error is different from the memory error.  Please let's discuss that issue here:

http://mediabrowser.tv/community/index.php?/topic/15094-connection-error-when-server-updates/

 

It is very confusing otherwise - they are two different things. Disabling the background rotating art was ONLY for the out of memory error.  However now I no longer am sure the big query was the memory problem - the out of memory error still occurred with the update to remove the huge number of episodes, so now I think it is either a server error, or a meta-data problem.

 

Also, if you don't mind, please don't edit your posts and expect me to read it, I read it as soon as you hit the submit button as it goes to my email, and will not always notice edits.  Just post another reply please.

 

xnappo

Edited December 9, 2014 by xnappo

[ebr 14913]

Okay thanks.  @@ebr @@Luke - I don't see anything unreasonable with the queries in that log.

 

xnappo

 

Well, that's going to be completely dependent on the size of the person's library and their server environment.  You are asking for every single movie that exists.  That doesn't seem like a scale-able solution to me.  Someone trying to run the server on, say, a NAS with limited resources could easily have a problem with that.  Not to mention trying to run across a remote connection of some sort.

 

If you absolutely must pre-load everything like that, I suggest you break it into chunks of 500 or so items.

[xnappo 1593]

We have been doing that since day 1, and people with bigger collections than this have been running for a long time.  If large queries are going to fail, the server should have a limit itself.  800 movies is nothing compared to some of our users.

 

I will try to break it up, but I suspect it will still happen.

 

@ / @@im85288 FYI

 

@@Angelblue05 - when this happens:

   1. Are you still able to browse 'All Movies' normally?

   2. In task manager, how much memory is MediaBrowser Server taking when this error occurs?

 

xnappo

Edited December 9, 2014 by xnappo