Library keeps vanishing from all devices

[moviemadnessman 13]

Hello.  I've used emby for quite some time, and I've typically been able to get it to work as needed.  However, I've been having a similar issue to those that have been posted with the library vanishing for users.  However, even after reading the other threads, the issue seems to persist, and yet, I don't know what is causing it.

 

If I reboot my server, it runs all my scheduled tasks, and after it gets to about 20% or so of the library scan, I can see my library on all devices.  However, at some point in time after not using the server (but leaving it on and running), if I try to access my library, either through my Roku3, the emby app on my galaxy s5, or even locally from the same machine running the server, I find that I am greeted with some form of nothing (on the computer and phone, it is a screen with next up, upcoming, and two other options, on my roku it is search-settings-change server-switch user).  The only way to regain access to the library is to manually run another library scan.

 

I thought I could combat it by running a scan more frequently, and so I changed it to every 6 hours, then to every 4 hours.  However, even with it running quite a bit, I still frequently lose my library until I manually tell the server to do a library scan.  I am unsure what is going on, as I have not added new channels or plugins, and it seems odd that even the machine running the server can't access the files without a new scan.  As a side note, my scan takes about 45 minutes on average ... it has always been a long time scan for me, but seeing how some people have a 10-15 minute scan time, I'm unsure why mine takes so long, since I don't have it extracting images or anything during the scan.  But that's not the issue.

 

I will be happy to run any tests that I can to help get this issue resolved, as it has only recently started (past 2 months or so), and while I've put up with it, I've decided I'd rather try and get it working like it used to.

 

Here is the log of server activities from 1-25-2016 which will show everything following a forced reboot of the server, and by the end of the night I had lost it again.  I had to zip it because the log file was quite large.

 

Server: 3.0.5818.0 stable (had also tried a previous stable, as well as several betas, all had the same result).  This most recent stable is also preferable because, although it has issues referenced in another thread, the Reports at least sometimes populates in this version.

 

I have 3 plugins as well, which have been in place for some time: Autoboxsets, Restart Server (currently not set to run on its own as I thought it may have been a culprit somehow), and Rokumetadata.

 

 

server-63589279146.zip

[Happy2Play 9782]

Looks like all your libraries are by drive letter, does setting Path Substitutions make a difference?

 

Since you have an original install Mediabrowser-Server your system configs show a lot of old configs, I know could be a lot of work but have you considered a clean install?

[Ian-Highlander 147]

This is the exact same problem I and a few others had and it turned out to be the  Autoboxsets plugin causing it (or something to do with boxsets at least as it never really got identified), there's a thread somewhere on here about it. Remove the Autoboxsets plugin and delete the boxsets from the directory and see if it stays stable, mine hasn't done it again since after months of trying to identify the problem.

 

Check if your cpu on your server is being pegged whilst apparently doing nothing, that's also a symptom of the issue.

 

Edit: thread here http://emby.media/community/index.php?/topic/28131-server-constantly-crashing-or-hanging/

Edited January 27, 2016 by Ian-Highlander

[moviemadnessman 13]

This is the exact same problem I and a few others had and it turned out to be the  Autoboxsets plugin causing it (or something to do with boxsets at least as it never really got identified), there's a thread somewhere on here about it. Remove the Autoboxsets plugin and delete the boxsets from the directory and see if it stays stable, mine hasn't done it again since after months of trying to identify the problem.

 

Check if your cpu on your server is being pegged whilst apparently doing nothing, that's also a symptom of the issue.

 

Edit: thread here http://emby.media/community/index.php?/topic/28131-server-constantly-crashing-or-hanging/

Sad, and I hope it isn't that plugin, as it has been very helpful (I have 410 collections thanks to it).  If it is that plugin, then it had to have been caused by a relatively recent update in either the server or the plugin itself, as I have had it installed almost as long as I have had Emby installed, which would be at least a year, but anything is possible.  I haven't noticed inactivity CPU spikes (related to the server), but I will keep an eye out for them.

 

Looks like all your libraries are by drive letter, does setting Path Substitutions make a difference?

 

Since you have an original install Mediabrowser-Server your system configs show a lot of old configs, I know could be a lot of work but have you considered a clean install?

Path substitution doesn't seem to make a difference either way.  At this point, sometimes I can't even access my library from the machine running Emby, which you would think it could do no matter what, since there's no network or anything else needed at that point.

 

I have not tried a clean install as it has not been an issue until recently, however, I would do one if needed ... all my metadata and images are saved in the folders with the videos themselves already, so in theory rebuilding the library shouldn't take all that long.  My only real concern (as I'm not at supporter level and thus don't have access to the server backup plugin) is what steps would I need to take so that all the people I have given server access to (users) would not need to be manually re-added/keep their settings intact (and, if possible, keep my own server settings intact)?  Would I just keep the /appdata/roaming/media-browser/config folder and replace it when I have a new install, or are additional folders needed to be saved?  Obviously a fresh install would be done to clear out old conflicting settings so copying all the folders wouldn't be a good idea, but is that the only folder I would need to backup and copy over, and would it work the way I want it to in terms of users?

[Ian-Highlander 147]

Sad, and I hope it isn't that plugin, as it has been very helpful (I have 410 collections thanks to it).  If it is that plugin, then it had to have been caused by a relatively recent update in either the server or the plugin itself, as I have had it installed almost as long as I have had Emby installed, which would be at least a year, but anything is possible.  I haven't noticed inactivity CPU spikes (related to the server), but I will keep an eye out for them.

I hear ya. I was a bit gutted but collections as a whole isn't a function we use very often here so we're living without it. I spent a LONG time trying to diagnose the issue with various threads on here about it I participated in too until I did a clean install and the only plugin I installed was the Autoboxsets one and no other non-standard things (expect legacy images in the config) and boom it started happening again. I eventually relented, removed Autoboxsets and deleted the box sets it had created and the problem never came back. I tested the theory by reinstalling and running it and within a couple of days, the problem was back. It's been removed since. @@ebr did a fair bit of testing with it and I and a couple of others did a few tests for him but the problem was never got to the bottom of unfortunately so it's stayed off my system. I'd rather have it stable and live without collections.

 

You might be lucky and it's not that on yours, best thing would be to back up the collections folder, then uninstall the plugin and delete the contents and see if your system stabilises like mine did. If it doesn't resolve the issue, it's something else.

Edited January 27, 2016 by Ian-Highlander

[ebr 16185]

Just as a point of clarification, I'm pretty sure it isn't the plug-in per se but, rather, the box sets themselves that cause the problem.  So, if you remove the plug-in only to go re-create a whole bunch of sets manually, that may just bring it right back.  Although, if you do that, please let us know because it will help diagnose.

[Ian-Highlander 147]

Just as a point of clarification, I'm pretty sure it isn't the plug-in per se but, rather, the box sets themselves that cause the problem.  So, if you remove the plug-in only to go re-create a whole bunch of sets manually, that may just bring it right back.  Although, if you do that, please let us know because it will help diagnose.

 

Yeah, sorry, I may not have been clear enough about that, I did say "or something to do with boxsets at least" and that it was never got to the bottom of, but a valid point none the less. I still have two test boxsets in mine that I did manually during our testing that have had zero effect on the system at all and it's still stable, I never got round to adding more in slowly to see if/when it went wrong.

 

This is the exact same problem I and a few others had and it turned out to be the  Autoboxsets plugin causing it (or something to do with boxsets at least as it never really got identified), there's a thread somewhere on here about it. Remove the Autoboxsets plugin and delete the boxsets from the directory and see if it stays stable, mine hasn't done it again since after months of trying to identify the problem.

 

Check if your cpu on your server is being pegged whilst apparently doing nothing, that's also a symptom of the issue.

 

Edit: thread here http://emby.media/community/index.php?/topic/28131-server-constantly-crashing-or-hanging/

[ebr 16185]

No worries.  I just wanted to be sure someone didn't just remove the plug-in and hand-create 100 box sets only to see the problem return.

 

Although, if someone wants to do that to confirm if the problem returns, that would be great...

[moviemadnessman 13]

No worries.  I just wanted to be sure someone didn't just remove the plug-in and hand-create 100 box sets only to see the problem return.

 

Although, if someone wants to do that to confirm if the problem returns, that would be great...

What I ended up doing was exiting the server, deleting all my collections (410, all created by the plugin), then restarting the server, letting it run its scans, then forcing a refresh of the plugin.

 

It recreated 389 box sets (unsure where the 21 others went ... but also don't remember what they were, so oh well).  System seems to be functioning a little better, but it is still early.  I did notice when I deleted the collections though that most of them had the [MB Autoboxset] tag, where as only a few (and all of them now) did not have that tag.  So, it is potentially something to keep an eye on...which I will.

 

One other thing, though ... I realize that it goes by the collection name on tmdb, but might it be possible to also include the collection ID in the folder name?  My reasoning is that, while most collections have a unique name or have some qualifier (like original series, new series, etc), I've found at least one that doesn't (The Ring):

 

https://www.themoviedb.org/collection/14563-the-ring-collection[this is the american film series]

https://www.themoviedb.org/collection/93369-the-ring-collection[this is the original japanese collection]

 

Since both are called "The Ring Collection", the plugin only makes 1 folder (for the american ones, since I guess it encountered it first), and won't create one for the japanese films since it can't create another folder called "The Ring Collection".  Maybe making the folders be called "%collection.name %tmdvid" could be implimented to prevent any chance of collisions.  Plus, as an added benefit, users could still create their own collection called "The Ring Collection" and put all 7 films into it if they wanted.

 

This is, however, the only collision I have noticed so far, and I've emailed tmdb staff several times about changing it (since they are locked), but no response.  Just thoughts.

[moviemadnessman 13]

So to follow up: since deleting all the collections and going from there, the issue has returned, but far less frequently.  I still do not know what is causing it, however, it does not seem to be happening nearly as often as it was (still on the current stable 3.0.5818.0 release).  Still unsure what is causing it, though ... it is strange, whenever I try to connect from say my Roku, it will only show me the 4 options, but if I go to the server computer and go to my main management window, it will show that the roku (and my user account on it) have both been logged in and authenticated, however, nothing shows up in the roku from time to time.

 

Again, it isn't as frequent as it was, but it would still be nice to know what is causing it at all, since the server can recognize when my device(s) connect, but is not always sharing the library like it should be.

[Ian-Highlander 147]

Your previous post said you deleted the collections then re-ran the plugin to re-create them. Do you still have the plugin installed and running? If so, have you actually tried it with the collections deleted and the plugin uninstalled for a while (and not re-creating collections at all for a while)?

 

If you have then ignore me completely, but the way your posts are worded you didn't remove the plugin just the collections and then re-created them. That doesn't solve the problem if it is the same issue (it still might not be). If it is the collections issue, the only way to solve it that seems reliable is to completely remove collections. Certainly something worth testing if you haven't yet.

[ebr 16185]

His issue may have nothing to do with the collections and, instead, just be the server responding too slowly to requests the first time after a start or library scan.

[moviemadnessman 13]

Your previous post said you deleted the collections then re-ran the plugin to re-create them. Do you still have the plugin installed and running? If so, have you actually tried it with the collections deleted and the plugin uninstalled for a while (and not re-creating collections at all for a while)?

 

If you have then ignore me completely, but the way your posts are worded you didn't remove the plugin just the collections and then re-created them. That doesn't solve the problem if it is the same issue (it still might not be). If it is the collections issue, the only way to solve it that seems reliable is to completely remove collections. Certainly something worth testing if you haven't yet.

I just removed all the collections, then restarted the server.  I left the plugin intact, though.  However, at some point between initially installing it many months ago and now, the way it named collection folders changed.  By removing all of the old ones and letting it rebuild (something that was suggested in another thread), I had hoped it would resolve the issue.  And, while part of the issue is back, there has been noticeable improvement in the functionality overall.  So while it is possible that the plugin is still somehow related, it does not appear to be solely based on collection alone.

 

His issue may have nothing to do with the collections and, instead, just be the server responding too slowly to requests the first time after a start or library scan.

As noted above, it may still be related, however, after removing all the collections and letting the plugin rebuild them all (or most) with the new folder naming, performance has increased. However, it is still unclear why the server reports that my device connects, and the user is authenticated, but the library (movies, collection, tv shows, etc) do not show up [this problem, again, has not been happening as regularly as it was before I had the autobox recreate all the collections, but it is still happening from time to time. Sometimes it will show the library if I choose to change server/change user a few times, and sometimes it requires me to run a library scan on the server, but at least now the server is recognizing that devices are connecting, even if it isn't always wanting to share with them like it should.  And yes, this even happens on the server computer itself if I try to use the web interface].

 

Problem is I am unsure what condition are happening to cause my server to have a "brain fart" from time to time, and the logs are so long I don't even really know what to begin to look for in terms of where the library is dropping out.  I feel like it could be related to the long library scans (my average is around 50 minutes) and the size of my server library, but I don't know.

 

 

 

Also, @@ebr, in regards to your post in another recent thread and in regards to your plugin:

1. As mentioned earlier in this thread, it would be nice if collections could have the collection id in the folder name somewhere (it might not look pretty, but it would allow collections with the same name to exist the like 2 ring collections listed above).

2.  In regards to the other thread, I also second the suggestion of preventing the plugin from removing movies from collections once they have been put there either automatically or manually, however, I would prefer this to be a user controlled option as opposed to a set thing.  That way, people who want the collections to adjust as themoviedb adjusts can have that happen, and people who just want it to build collections but not remove films if something happens to change them online can have the plugin function that way for them.  I mean, the only user options for the plugin right now are how many movies are needed to create the collection in the first place, and to manually rescan for new collections and changes.

3.  I know I suggested this at some point, but it would also be a cool feature (if it can be implemented) to have the box sets create a shadow file for missing movies in a collection, in a similar way that you can set emby to show missing episodes and it will even show the descriptions and everything else for it (obviously not able to play them since they are missing).  This could even be another user defined option, as maybe some people wouldn't want to see what movies they are missing, or if a collection online contains a movie that has not been released yet and won't be for a few years.  Just an idea ... again, I don't know if it would be possible to have it grab that information, but I know I personally have a few collections that I thought were complete, until I checked and found that there was some sequel or prequel or something that I missed.

[speechles 2055]

... it is strange, whenever I try to connect from say my Roku, it will only show me the 4 options, but if I go to the server computer and go to my main management window, it will show that the roku (and my user account on it) have both been logged in and authenticated, however, nothing shows up in the roku from time to time.

I know why this occurs. It happens usually when the homescreen doesn't get any user views. The only view row it can show is the "Options" row. This usually occurs when you are scanning the library on a slowish server. You can use the blue neon roku app and see why it is happening. In it there is more error detection. That user view error would be shown like below.

createDialog("User Views Error!", "Failed To Get User Views (invalid).", "OK", true)

 

Then in the metadata fetcher part, there are a few more.

createDialog("User Profile Error!", "Failed To Get All User Profiles.", "OK", true)

createDialog("User Profile Error!", "Failed To Get User Profile.", "OK", true)

createDialog("Parse Response Error!", "Error getting folder items.", "OK", true)

With these, you should be able to see exactly what is causing it. After these occur, the client may crash, act strange, etc. But you will be able to know why when these pop up. They should never pop up ideally. I've only seen the user views error once since I've added the dialogs months ago. Try this and see if it helps you.

Edited February 2, 2016 by speechles

[moviemadnessman 13]

I do use the blue on my roku (sorry emby creators, I just like the color scheme, as well as the extra info like play count and when it was played, and the times being my time zone and not gmt).

 

But, the issue happens on other devices too, sometimes when no tasks are running, and even using the web client on the server computer itself sometimes shows that the browser and user (in this case, me) as authenticated and logged in, and yet nothing shows up.  And then sometimes it just acts like it should all around, so i really don't know.

[Ian-Highlander 147]

His issue may have nothing to do with the collections and, instead, just be the server responding too slowly to requests the first time after a start or library scan.

Agreed which is why I said if he's tried removing it completely already to ignore me 

[ebr 16185]

But, the issue happens on other devices too, sometimes when no tasks are running, and even using the web client on the server computer itself sometimes shows that the browser and user (in this case, me) as authenticated and logged in, and yet nothing shows up.  And then sometimes it just acts like it should all around, so i really don't know.

 

It may be that you are encountering the same performance issue where initial queries against the server will take a very long time (and sometimes time out).  This usually happens with queries for things like resume items or latest items but can also happen on the actual request for the library.

 

What you can try to do to diagnose, is, when this happens, wait about two minutes and then try to re-start the same app you had the problem with.  Does it perform normally then?

[moviemadnessman 13]

It may be that you are encountering the same performance issue where initial queries against the server will take a very long time (and sometimes time out).  This usually happens with queries for things like resume items or latest items but can also happen on the actual request for the library.

 

What you can try to do to diagnose, is, when this happens, wait about two minutes and then try to re-start the same app you had the problem with.  Does it perform normally then?

It seems to some of the time, but then some of the time it still does nothing until I force a rescan of the library.

 

I have noticed that if I get the loading issue with my roku, then use the emby app on my phone, sometimes I will see everything on my phone...at which point logging off/on or changing server on the roku brings up the library.  But sometimes, even when I use the phone app, nothing loads, which is when I'm forced to force a rescan.  And even having the library rescan itself every 4 hours does not eliminate the issue, as I've had the library "vanish" when the last scan was run just an hour previous (again, though, since redoing the collections it does not seem to be happening like this as often, but often enough that I wish it would stop, lol).

[moviemadnessman 13]

What you can try to do to diagnose, is, when this happens, wait about two minutes and then try to re-start the same app you had the problem with.  Does it perform normally then?

As a follow up ... I would say that probably 80% of the times that emby has had difficulties connecting to my server (any device), waiting a few minutes then either changing server, user, or restarting the app seems to connect.  The other 20%, I still need to run a scan.  Sooooo....what is this telling us, since the only change to the server has been upgrading to a newer version (and clearing the collections, which now that the [] tags are gone seems to also help).