v 3.3.0 update and library issue ?

[d00zah 128]

Ok your log had something in it with a blank user agent so i just assumed it was kodi. So whatever that was, if you use it again then it might cause the problem again.

 

Again, no. I've been reviewing the trailers channel daily since the purge & it hasn't repeated... it's been 3-4 days.

[d00zah 128]

So you are running 3.3.0 and this only happened the one time?

 

Correct.

[Luke 37116]

Again, no. I've been reviewing the trailers channel daily since the purge & it hasn't repeated... it's been 3-4 days.

 

I understand. You still have something in your server log connecting to your emby server that is not an emby app.

[d00zah 128]

I understand. You still have something in your server log connecting to your emby server that is not an emby app.

 

My mistake. I just went back & looked at when things went bad... it was while looking at the Podcasts channel (I missed it in the list).

I have used that quite a bit since the purge, as well, although it's a recent addition & I tend to forget it's installed until I see it on the homepage.

[Luke 37116]

Using it from which app?

[d00zah 128]

Using it from which app?

 

FireFox.

Edited February 28, 2018 by d00zah

[Luke 37116]

In incognito mode? That's the only reason I can think of that it would not send a user agent.

[d00zah 128]

In incognito mode? That's the only reason I can think of that it would not send a user agent.

No, sir. I see the user agent string reported normally until just before the purge begins, where it shows up blank...

 

http://localhost:8096/emby/Items/3125ff32eac4ec6da78d9496b981f3d8/Images/Primary?maxHeight=325&maxWidth=216&tag=90599ba0a307fd0bc2848b1358be0183&quality=90. UserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

2018-02-24 22:08:05.790 Info HttpServer: HTTP Response 304 to ::1. Time: 25ms. http://localhost:8096/emby/Items/cb8306e406bbfd792843cc42ad554ae2/Images/Primary?maxHeight=325&maxWidth=216&tag=81da0692e8dfcad6907cd0fc7e54d250&quality=90

2018-02-24 22:08:05.799 Info HttpServer: HTTP Response 304 to ::1. Time: 34ms. http://localhost:8096/emby/Items/6b5d98352e495d26ac597b432f973aa7/Images/Primary?maxHeight=325&maxWidth=216&tag=3f5dc4af4f63730f3b26958b9b2a3235&quality=90

2018-02-24 22:08:05.806 Info HttpServer: HTTP Response 304 to ::1. Time: 41ms. http://localhost:8096/emby/Items/8ea4dc1ae3f522d4943fdcabbfb07468/Images/Primary?maxHeight=325&maxWidth=216&tag=060d4c28915baf5e0cf35727cd6ab571&quality=90

2018-02-24 22:08:05.811 Info HttpServer: HTTP Response 304 to ::1. Time: 47ms. http://localhost:8096/emby/Items/9218693c7538c4a17e60b2de8bab5a25/Images/Primary?maxHeight=325&maxWidth=216&tag=706124bf0841eae3e34662825f3feb56&quality=90

2018-02-24 22:08:05.825 Info HttpServer: HTTP Response 304 to ::1. Time: 41ms. http://localhost:8096/emby/Items/3125ff32eac4ec6da78d9496b981f3d8/Images/Primary?maxHeight=325&maxWidth=216&tag=90599ba0a307fd0bc2848b1358be0183&quality=90

2018-02-24 22:08:07.969 Info HttpServer: HTTP GET http://[sRVLANIP]:8096/emby/Channels/c5e7eb27cf7d2b500e9eac8e08b771bb/Items?userid=fb41946c15e5c66587161934d6eedba1&fields=Overview,SortName,DisplayMediaType,Path,DisplayPreferencesId,DateCreated,Taglines,Metascore,MediaStreams,ParentId,DateLastMediaAdded,MediaSources,SpecialEpisodeNumbers&format=Json. UserAgent:

2018-02-24 22:08:08.021 Info HttpClient: GET

http://feeds.soundcloud.com/users/soundcloud:users:174770374/sounds.rss

 

Edited February 28, 2018 by d00zah

[Luke 37116]

No, sir. I see the user agent string reported normally until just before the purge begins, where it shows up blank...

 

http://localhost:8096/emby/Items/3125ff32eac4ec6da78d9496b981f3d8/Images/Primary?maxHeight=325&maxWidth=216&tag=90599ba0a307fd0bc2848b1358be0183&quality=90. UserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

2018-02-24 22:08:05.790 Info HttpServer: HTTP Response 304 to ::1. Time: 25ms. http://localhost:8096/emby/Items/cb8306e406bbfd792843cc42ad554ae2/Images/Primary?maxHeight=325&maxWidth=216&tag=81da0692e8dfcad6907cd0fc7e54d250&quality=90

2018-02-24 22:08:05.799 Info HttpServer: HTTP Response 304 to ::1. Time: 34ms. http://localhost:8096/emby/Items/6b5d98352e495d26ac597b432f973aa7/Images/Primary?maxHeight=325&maxWidth=216&tag=3f5dc4af4f63730f3b26958b9b2a3235&quality=90

2018-02-24 22:08:05.806 Info HttpServer: HTTP Response 304 to ::1. Time: 41ms. http://localhost:8096/emby/Items/8ea4dc1ae3f522d4943fdcabbfb07468/Images/Primary?maxHeight=325&maxWidth=216&tag=060d4c28915baf5e0cf35727cd6ab571&quality=90

2018-02-24 22:08:05.811 Info HttpServer: HTTP Response 304 to ::1. Time: 47ms. http://localhost:8096/emby/Items/9218693c7538c4a17e60b2de8bab5a25/Images/Primary?maxHeight=325&maxWidth=216&tag=706124bf0841eae3e34662825f3feb56&quality=90

2018-02-24 22:08:05.825 Info HttpServer: HTTP Response 304 to ::1. Time: 41ms. http://localhost:8096/emby/Items/3125ff32eac4ec6da78d9496b981f3d8/Images/Primary?maxHeight=325&maxWidth=216&tag=90599ba0a307fd0bc2848b1358be0183&quality=90

2018-02-24 22:08:07.969 Info HttpServer: HTTP GET http://[sRVLANIP]:8096/emby/Channels/c5e7eb27cf7d2b500e9eac8e08b771bb/Items?userid=fb41946c15e5c66587161934d6eedba1&fields=Overview,SortName,DisplayMediaType,Path,DisplayPreferencesId,DateCreated,Taglines,Metascore,MediaStreams,ParentId,DateLastMediaAdded,MediaSources,SpecialEpisodeNumbers&format=Json. UserAgent:

2018-02-24 22:08:08.021 Info HttpClient: GET

 

But what's odd there is that the web app doesn't put format=json on the string. Instead it uses the Accept header to indicate that it needs json output. There's also nothing in the web app about Metascore, which you can see is also part of that url.

[d00zah 128]

But what's odd there is that the web app doesn't put format=json on the string. Instead it uses the Accept header to indicate that it needs json output. There's also nothing in the web app about Metascore, which you can see is also part of that url.

 

Looks like the last thing EMC accessed before shutting down was refreshing the homescreen...

 

2018-02-24 22:07:57.4408,Debug,,unwatched list for Movies loaded with 20 items.,22,Worker thread for Common

2018-02-24 22:08:07.9688,Info,,Sending Http Get to http://192.168.10.2:8096/emby/Channels/c5e7eb27cf7d2b500e9eac8e08b771bb/Items?userid=fb41946c15e5c66587161934d6eedba1&fields=Overview,,SortName,,DisplayMediaType,,Path,,DisplayPreferencesId,,DateCreated,,Taglines,,Metascore,,MediaStreams,,ParentId,,DateLastMediaAdded,,MediaSources,,SpecialEpisodeNumbers&format=Json,27,Worker thread for Common

2018-02-24 22:08:09.1818,Debug,,Keep alive call,, IsCurretntlyVisible=True,1,Application

2018-02-24 22:08:10.1438,Info,,Received web socket message: LibraryChanged,10,

2018-02-24 22:08:10.1438,Debug,,Library Changed...,10,

2018-02-24 22:08:10.1448,Debug,,Folders Added to: 0 Items Added: 0 Items Removed: 0 Items Updated: 1,10,

2018-02-24 22:08:10.1448,Debug,,Changed Item 2134b3030aa8c0fa39948cfd1c284b23 is not loaded,10,

2018-02-24 22:08:21.2068,Info,,Received web socket message: LibraryChanged,10,

2018-02-24 22:08:21.2068,Debug,,Library Changed...,10,

2018-02-24 22:08:21.2068,Debug,,Folders Added to: 0 Items Added: 0 Items Removed: 3 Items Updated: 1,10,

2018-02-24 22:08:21.2078,Debug,,Folder with changes is: Podcasts,10,

2018-02-24 22:08:21.2088,Debug,,Changed Item 2134b3030aa8c0fa39948cfd1c284b23 is not loaded,10,

 

end of log

Edited February 28, 2018 by d00zah

[Luke 37116]

Ok, so if you browse podcasts again from emc then this might happen again. this is resolved in the beta.

[d00zah 128]

Ok, so if you browse podcasts again from emc then this might happen again. this is resolved in the beta.

 

I'm in there right now, but it's showing 'No Items'... building cache? Will report back if something bad happens.

[Luke 37116]

Well the server log will be showing all the deletions if that's what's happening

[d00zah 128]

Well, here's something interesting... EMC finally loaded the list of podcasts... which includes virtually EVERYTHING from my library... MOVIES, Videos, Music...  will post logs when I get some time. Company...

[d00zah 128]

Well the server log will be showing all the deletions if that's what's happening

 

So, there was no mass deletion, but EMC is REALLY confused...

 

This is the 'Podcasts' view... the list not only contains podcasts, but everything from all my libraries...

 

 

    Thinned down from 16MB+

 

 

Edited March 1, 2018 by d00zah

[d00zah 128]

Also, after reviewing the EMC log leading up to the metadata purge..

 

- Watched a movie (external player MPC-BE, direct play)

- Movie ended & returned to movie detail screen

- Backed out to EMC home screen, where all the recent additions stuff refreshed

- When it hit the recent podcasts, the server went crazy

 

This  sequence has been repeated regularly since, without any negative results... except for the 'Podcasts' view result above.

[moviefan 184]

FYI, I don't use Podcasts so whatever is happening is not specific to that plugin.

 

I do use EMC but that computer was shut down at the time this occurred.

[Ice 14]

My problem is a bit different to what has been described....

 

I haven't lost my entire library, only the folders with files that don't have standard naming conventions, for example WSBK races I have recorded were named Round 1 Race 1 Phillip Island, Round 1 Race 2 Phillip Island, they are located in a WSBK directory with subdirectories related to the year eg 2018, 2017. I have changed some of the file names to a more Emby friendly names eg 18x01 Phillip Island, after doing a Metadata update and a rescan of the library it will find the files that I have renamed. Before I updated to 3.3.0 (running 3.3.0.3 currently) Emby didn't really like my naming convention but had no problems displaying the files. Luckily for me I don't have 1000's of files to rename only a few 100, which if I have to I will do over time, either that or roll back to the previous version

Edited March 1, 2018 by Ice

[d00zah 128]

I still think it was a one-time thing & this latest event is just an new, obscure bug, but I was responding to Like's line of questioning.

[moviefan 184]

I question the one-time thing given other people's descriptions.

 

I've only experienced it one time but I also downgraded afterwards to .27.

 

Library is still rebuilding.

 

I don't see any deleting in my log.

 

Can you paste an example line where you see this happening so I can search in my logs?

[d00zah 128]

I question the one-time thing given other people's descriptions.

 

I've only experienced it one time but I also downgraded afterwards to .27.

 

Library is still rebuilding.

 

I don't see any deleting in my log.

 

Can you paste an example line where you see this happening so I can search in my logs?

 

My rescan took ~24hrs. I had recently changed my libraries to save metadata with media, but that had only been for new items. The forced rescan finished the job. Still, we'd like to choose when these things happen.

 

With Luke urging an update to the beta build, I decided to gamble on staying w/ v3.3.0.0. My thinking was, that if the metadata dir got nuked again, this time I was only risking chapter images... I'm assuming.

 

3-4 days later, even after poking at things at Luke's urging, it still hasn't repeated. If nothing else, it's a data point.

 

You can see the deletes in my log here:

 

https://emby.media/community/index.php?/topic/56434-v-330-update-and-library-issue/&do=findComment&comment=549013

 

starts deleting the metadata at ~22:08:13.

[moviefan 184]

That's not what happened in my log.  Don't see any of that at all.

 

I've been saving media into the folders for years but that didn't help the situation.  It's not the data that's gone.  It's just the DB that doesn't contain the info anymore.

 

So the chapter images are not the main time consuming task for the rebuild I do not believe.  But I could be wrong.  I am not a developer.

[d00zah 128]

That's not what happened in my log.  Don't see any of that at all.

 

I've been saving media into the folders for years but that didn't help the situation.  It's not the data that's gone.  It's just the DB that doesn't contain the info anymore.

 

So the chapter images are not the main time consuming task for the rebuild I do not believe.  But I could be wrong.  I am not a developer.

 

Nor am I, so my description is from a lay perspective. I don't doubt that the db was purged, as well, but my assumption is that the lookup & collection of all that metadata was the time consuming part. Didn't check the library.db, but did check the \metadata & \cache folders after my library contents disappeared & many GBs of data was gone. Anyway, it's back & so far, it's staying that way.

[needanewjob 0]

What you could try to do is move the program data folder to the new location based on what the new package is expecting. Then run a library scan on server startup.

Just for the sake of followup, I actually tried that before you posted and got:

 

Error: near line 11: database disk image is malformed

 

I assume something went wrong in either the upgrade, removal, or copy-over and screwed the sqlite db so I ended up just having to purge.  3 year run through however many versions and DB upgrades on an awesome community supported platform, I really have no complaints.  Losing the watched status really sucks, but I guess that's what keyboard shortcuts and a rainy afternoon are for.

 

Thanks for responding and thanks for the effort you (and everyone else) puts into this!

 

 

One related aside:  I'm sure the answer is "yes, but...", but has any thought been given to allowing a user-selectable DB so those of us with mysql/psql can administer the DB more easily than sqlite?

Edited March 1, 2018 by needanewjob

[Luke 37116]

Just for the sake of followup, I actually tried that before you posted and got:

 

Error: near line 11: database disk image is malformed

 

I assume something went wrong in either the upgrade, removal, or copy-over and screwed the sqlite db so I ended up just having to purge.  3 year run through however many versions and DB upgrades on an awesome community supported platform, I really have no complaints.  Losing the watched status really sucks, but I guess that's what keyboard shortcuts and a rainy afternoon are for.

 

Thanks for responding and thanks for the effort you (and everyone else) puts into this!

 

 

One related aside:  I'm sure the answer is "yes, but...", but has any thought been given to allowing a user-selectable DB so those of us with mysql/psql can administer the DB more easily than sqlite?

 

Is it possible you tried to do this with the server running? if yes and if you still have the files at the old location, try copying them again with the server shut down.