Jump to content

98% of my Movie Metadata wiped, most of library being re-added as if it were new. Any idea what happened??


BlazedMonkey

Recommended Posts

BlazedMonkey

 

 

While it has annoyed the absolute shit out of me, lesson has been learned, let go of old methods and use the .nfo files even if the mess they make of my folders does my head in lol relying on the db alone is a bad idea.

I use .nfo's stored in the movie folder, and I still had this issue. I had to completely rebuild 98% of my movie metadata and collections.

Link to comment
Share on other sites

BlazedMonkey

Pretty sure affected users are users that do not store metadata or images with media.  And previous times this happened, the same set of users had this issue.

 

The purge of item then re-adding causes all user that don't save with media to loose all metadata for the item to be deleted from the database and the corresponding database id linked image folder to be deleted as that id no longer exists.  Then the server starts over adding item with new id and new folder.

 

Ideally all users that do not save with media need to ensure the have a backup of the Emby programdata folder to easily restore databases and linked \metadata\library\xx\xxxxxxxxxxxxxxxxxxxxxxxxx databaseid items.

 

I regularly replace files, both manually and through automation. Even your mention of "previous times this happened" tells me that perhaps you should be doing a little more testing before pushing out updates.  Maybe part of your checklist before pushing out that update should be to delete and re-add a movie, instead of having a chunk of your users sit on their thumbs for 2 days while their entire libraries rebuild for no reason. 

Link to comment
Share on other sites

Pretty sure affected users are users that do not store metadata or images with media.  And previous times this happened, the same set of users had this issue.

 

You mean the issue of leaving the default settings in place? If that's the issue, I think there's another issue. :wacko:

Link to comment
Share on other sites

Happy2Play

I regularly replace files, both manually and through automation. Even your mention of "previous times this happened" tells me that perhaps you should be doing a little more testing before pushing out updates.  Maybe part of your checklist before pushing out that update should be to delete and re-add a movie, instead of having a chunk of your users sit on their thumbs for 2 days while their entire libraries rebuild for no reason. 

But no testers have had these issues that only select users are having.  We can only test so many variables.  I have 6 test systems that are setup by issues other users have reported and have yet to see the issue that is currently being reported.  Still don't know why it has happened to the three users so far.

 

One factor may be how old of a install users may have.  But I am only guessing.

Link to comment
Share on other sites

Happy2Play

You mean the issue of leaving the default settings in place? If that's the issue, I think there's another issue. :wacko:

It really is a guessing game as it does not affect all user.  But all we can do is test and try and find the underlying issue.

Link to comment
Share on other sites

Ideally all users that do not save with media need to ensure the have a backup of the Emby programdata folder to easily restore databases and linked \metadata\library\xx\xxxxxxxxxxxxxxxxxxxxxxxxx databaseid items.

 

OK so what I'm hearing - just to be super duper clear - is that I can fix this issue, if it affects me, with a full backup of the programdata folder?

(The whole thing - not just the metadata folder?)

Link to comment
Share on other sites

AmericanCrisis

It really is a guessing game as it does not affect all user.  But all we can do is test and try and find the underlying issue.

 

It doesn't have anything to do with .nfo files. Why was only one on my media folders affected on my server? The folders are constructed the same exact way. There are no differences in their settings on the servers and how they are constructed in my pool.

 

The only difference that I could possibly think between these media folders are:

 

In my UHD movie folder (this was the folder that was broken... thumbnails were deleted), every movie folder has two files inside of the folder. For example:

 

  • Title (Year) - 4k.mkv
  • Title (Year) - 1080p.mkv

In my HD movie folder (this folder was unaffected), some (not all) movies have two files inside the folder. For example:

 

  • Title (Year).mkv
  • Title (Year).mp4

The .mp4 file is simply a re-encoded version of the .mkv (original) file. The resolution is the same.

 

But... since only a few folders inside the HD movie folder have two files it doesn't make sense that it would be the problem.

Link to comment
Share on other sites

Happy2Play

OK so what I'm hearing - just to be super duper clear - is that I can fix this issue, if it affects me, with a full backup of the programdata folder?

(The whole thing - not just the metadata folder?)

Yes as everything within that folder contains your setup.  Or you can step a folder back to Emby-Server as that will contain the system folder which is the server version currently running that corresponds with the programdata.  I only know Windows setups but believe other OSes have a similar programdata setup.

 

So having that backup one could delete the affected Emby-Server folder and restore previous version.

Link to comment
Share on other sites

jordy

It doesn't have anything to do with .nfo files. Why was only one on my media folders affected on my server? The folders are constructed the same exact way. There are no differences in their settings on the servers and how they are constructed in my pool.

 

The only difference that I could possibly think between these media folders are:

 

In my UHD movie folder (this was the folder that was broken... thumbnails were deleted), every movie folder has two files inside of the folder. For example:

 

  • Title (Year) - 4k.mkv
  • Title (Year) - 1080p.mkv

In my HD movie folder (this folder was unaffected), some (not all) movies have two files inside the folder. For example:

 

  • Title (Year).mkv
  • Title (Year).mp4

The .mp4 file is simply a re-encoded version of the .mkv (original) file. The resolution is the same.

 

But... since only a few folders inside the HD movie folder have two files it doesn't make sense that it would be the problem.

could I suggest that you enclose  the "- 4k" and "- 1080p" inside square brackets [4k] / [1080p]. Anything inside [ ] is disregarded by the metadata search function thus providing faster and more accurate recognition of your movies should problems like this arise.

Link to comment
Share on other sites

Happy2Play

It doesn't have anything to do with .nfo files. Why was only one on my media folders affected on my server? The folders are constructed the same exact way. There are no differences in their settings on the servers and how they are constructed in my pool.

 

The only difference that I could possibly think between these media folders are:

 

In my UHD movie folder (this was the folder that was broken... thumbnails were deleted), every movie folder has two files inside of the folder. For example:

 

  • Title (Year) - 4k.mkv
  • Title (Year) - 1080p.mkv

In my HD movie folder (this folder was unaffected), some (not all) movies have two files inside the folder. For example:

 

  • Title (Year).mkv
  • Title (Year).mp4

The .mp4 file is simply a re-encoded version of the .mkv (original) file. The resolution is the same.

 

But... since only a few folders inside the HD movie folder have two files it doesn't make sense that it would be the problem.

Yes the few people that are having this issue, it is not the exact same content type purge/re-add.  So there is still the issue of figuring out why select users with select configuration are being affected.

 

What do you mean by thumbnails were delete?  Were they with the media?

If images are not with media the \metadata\library\xx\xxxxxxxxxxxxxxxxxxxxxx database id folder is deleted.   When the item is removed from the database those images are deleted as the linked ID is now does not exist.

Edited by Happy2Play
Link to comment
Share on other sites

BlazedMonkey

But no testers have had these issues that only select users are having.  We can only test so many variables.  I have 6 test systems that are setup by issues other users have reported and have yet to see the issue that is currently being reported.  Still don't know why it has happened to the three users so far.

 

One factor may be how old of a install users may have.  But I am only guessing.

What you do mean by "how old of an install"? I've been running Emby as long as this account has been active, and have been running all updates the entire time.

 

 

I'm still curious why this issue with Emby would create the issues with the Featurettes in my Plex install like I mentioned earlier.

Link to comment
Share on other sites

Happy2Play

What you do mean by "how old of an install"? I've been running Emby as long as this account has been active, and have been running all updates the entire time.

 

 

I'm still curious why this issue with Emby would create the issues with the Featurettes in my Plex install like I mentioned earlier.

There are issues that carry over from each update, there are patches in place to try and prevent forcing users to rebuild.   A build that could be 2 years old can see different issues that someone that installs today.  But my Media-Browser install that has updated from the beginning has only had to rebuilt once when there was a major database change.

 

As for featurettes I could only guess since I have never used Plex.  But it could be do to Emby writing nfo files for Extras now, but as I said it is only a guess.

Link to comment
Share on other sites

Yes as everything within that folder contains your setup.  Or you can step a folder back to Emby-Server as that will contain the system folder which is the server version currently running that corresponds with the programdata.  I only know Windows setups but believe other OSes have a similar programdata setup.

 

So having that backup one could delete the affected Emby-Server folder and restore previous version.

 

So if I wanted to keep the 4.4.1 update and also restore metadata and artwork, could I do that with just the programdata folder?

Link to comment
Share on other sites

Happy2Play

So if I wanted to keep the 4.4.1 update and also restore metadata and artwork, could I do that with just the programdata folder?

Assuming that the defect is not in 4.4.1.0 itself, yes.

Link to comment
Share on other sites

nicko84

Sorry that doesn't work for you. Try this instead (just a test).

 

Go to C:\Users\Your UserName\AppData\Roaming\Emby-Server\programdata\data\collections\ and open any folder that is labeled Collection Name [boxset]. If you find a file named collections.xml, open it in Notepad add a space to the 1st line, remove the space, then save the file (with the same name obviously). Now open your app again and check to see if that particular collection is there.

 

If it isn't there, I have no idea how to help you. If it is, one solution would be to go through and modify all collection.xml files. It's not quick, but if you only have a couple dozen collections it'll work.

 

As for your date added problem, I'm afraid I can't help you with that either. I use NFO files and when the server removes a movie from the database it just reads the NFO file when it scans it in again (although the scan takes for-fing-ever).

 

Let's just say something went seriously wrong with this server update, at least for some users. BTW, you can turn off automatic server updates in Settings.

Hello,

 

yes it works for each Collection, thanks !

 

So the entire issue of Collections, added date ... deals with Emby Server program itself and not with datas.

 

I hope Luke will find a way to restore ours library, i've no backup !

Link to comment
Share on other sites

khodges747

Hello,

 

yes it works for each Collection, thanks !

 

So the entire issue of Collections, added date ... deals with Emby Server program itself and not with datas.

 

I hope Luke will find a way to restore ours library, i've no backup !

@@nicko84

Glad that helped.

Link to comment
Share on other sites

horstepipe

I'm really wondering about a missing statement / info by the chiefs, are they so much busy or desperated because of the baconfeet drop?

Edited by horstepipe
Link to comment
Share on other sites

 or desperated because of the baconfeet drop?

 

Hi.  They have nothing to do with us.

 

We will get a blog entry out soon.

 

Thanks.

Link to comment
Share on other sites

Happy2Play

Hello,

 

yes it works for each Collection, thanks !

 

So the entire issue of Collections, added date ... deals with Emby Server program itself and not with datas.

 

I hope Luke will find a way to restore ours library, i've no backup !

There is no way to recover if items are removed and re-added to the database.  Only a backup can possible resolve the issue created.

Link to comment
Share on other sites

nicko84

There is no way to recover if items are removed and re-added to the database.  Only a backup can possible resolve the issue created.

I know and it's very bad...i've planned au database backup.

 

I'd like to change added date for an entire group of files and not one by one in order to sort movies manually...an idea ?

Link to comment
Share on other sites

moviefan

I did not have this issue thankfully.  I don't trust the DB for any sort of long term preserving of my data due to past issues.

 

I am curious though - for the people who are saying they're using local .nfos and this is still an issue - did Emby somehow overwrite the data in the files?  Or is it just not properly reading it?

  • Like 1
Link to comment
Share on other sites

Happy2Play

I know and it's very bad...i've planned au database backup.

 

I'd like to change added date for an entire group of files and not one by one in order to sort movies manually...an idea ?

Only way I can think of would be in the database itself.

 

Also what is the "Date added behavior for new content:" set to?  Dashboard-Library Advanced tab

Link to comment
Share on other sites

khodges747

I did not have this issue thankfully.  I don't trust the DB for any sort of long term preserving of my data due to past issues.

 

I am curious though - for the people who are saying they're using local .nfos and this is still an issue - did Emby somehow overwrite the data in the files?  Or is it just not properly reading it?

In my case all the NFOs were read back into the database, preserving original date added and even my custom metadata. Only HTML code in the Overview section was stripped out, but nothing else. (I use HTML code to highlight certain things, but the devs don't allow it anymore because it's an attack vector).

However, the Collections, which rely on XML files rather than NFO, were not read by the server until I did a manual scan for "missing metadata".

In short, I lost nothing but a lot of time, presumably thanks to NFO files present with my media.

  • Like 2
Link to comment
Share on other sites

nicko84

Only way I can think of would be in the database itself.

 

Also what is the "Date added behavior for new content:" set to?  Dashboard-Library Advanced tab

"Use added date in the library" : should i use "creation date of file" ?

Link to comment
Share on other sites

nicko84

In my case all the NFOs were read back into the database, preserving original date added and even my custom metadata. Only HTML code in the Overview section was stripped out, but nothing else. (I use HTML code to highlight certain things, but the devs don't allow it anymore because it's an attack vector).

However, the Collections, which rely on XML files rather than NFO, were not read by the server until I did a manual scan for "missing metadata".

In short, I lost nothing but a lot of time, presumably thanks to NFO files present with my media.

I've activated nfo and i will see (and a databse backup too !)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...