98% of my Movie Metadata wiped, most of library being re-added as if it were new. Any idea what happened??

[Happy2Play 8281]

"Use added date in the library" : should i use "creation date of file" ?

"Created date of file" is not as noticeable as "Use added date in the library", but it is all personal preference.

 

But writing nfo files allows a date to be maintained.

[nicko84 37]

"Created date of file" is not as noticeable as "Use added date in the library", but it is all personal preference.

 

But writing nfo files allows a date to be maintained.

Thanks for your answers, i hope it will be better in the future !

[cptlores 23]

Why it happens to some and not all will/could be a mystery as no ones setups is the same and everyone updates history is exactly the same.

 

Personally I expect a possible rebuild with every major release, but devs try their best for that not to happen.

 

I am still surprised there are no Announcements for this version yet.  As there is a change in Sortname that could provide different result in a new vs old install.

 

That is wrong on so many levels I hardly know where to start. But the obvious one is that Emby is now commercial software. And once they choose to go that route, it now comes with certain expectations when it comes to product quality. And "expect a possible rebuild with every major release" is certainly not one of those..

[khodges747 55]

That is wrong on so many levels I hardly know where to start. But the obvious one is that Emby is now commercial software. And once they choose to go that route, it now comes with certain expectations when it comes to product quality. And "expect a possible rebuild with every major release" is certainly not one of those..

Well said. Thanks.

[Brendon 101]

There's an update and i'm scared, any of you guys that had the same problem wanna go first   

[pwhodges 1528]

There's an update and i'm scared, any of you guys that had the same problem wanna go first     

 

There are ways to backup first, you know.

 

Paul

Edited April 8, 2020 by pwhodges

[Brendon 101]

There are ways to backup first, you know.

 

Paul

 

I know this, i was having a laugh, or did you miss that?

[Happy2Play 8281]

This is all do to the dev trying to add features or changes but still allow old installs to still function without forcing the change, and all new installs fully reflect then new ways.  Everyone is affected differently.

 

In the end RELEASE notes/Announcements need to PRECEDE these Official updates.  But there is usually nothing about how new releases could affect legacy installs.

 

So in reality old installs that have created libraries in different version are being affected by this change in 4.4.1.0.  That is why some users are only seeing this happen to one library.

[mprasil 14]

So I think I have the same issue. I'm using the official docker image and I've just restored the stuff that's normally mounted under "/config/" in the container from ~2 weeks old backup. Upon start (using the "latest" image) my library looks okay (minus the recently added movies, but that's expected) but soon after the library scan starts. It takes long time and breaks all of my custom edited items, etc..

 

I've looked into logs and I can see some interesting bits:

...SNIP...
2020-04-08 20:51:46.926 Info App: Application version: 4.4.2.0
...SNIP...
2020-04-08 20:52:09.685 Info App: Emby.Kodi.SyncQueue: Creating DB Repository...
..SNIP...
2020-04-08 20:52:52.151 Info App: Removing item from database, Type: Folder, Name: Some movie, Path: /data/movies/Some movie, Id: 39444
2020-04-08 20:52:52.156 Info App: Deleting path /config/metadata/library/39/3909fc59177ffc59177b95b3ff4db0c
2020-04-08 20:52:52.158 Info App: Deleting path /config/metadata/library/3e/3e4d9b8abbf37b0d950e9b8abbba7193
2020-04-08 20:52:52.259 Info App: Removing item from database, Type: Folder, Name: Other movie, Path: /data/movies/Other movie, Id: 39445
2020-04-08 20:52:52.260 Info App: Deleting path /config/metadata/library/09/0921fa1876cbb016ec55c76cbb09d7c2
2020-04-08 20:52:52.271 Info App: Deleting path /config/metadata/library/98/9811fa9c0cfaa0f97f261a11fa9c8324
..SNIP..
2020-04-08 20:54:49.906 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: /bin/ffprobe -i file:"/data/movies/Some movie/Some movie.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
..SNIP..
2020-04-08 20:54:50.335 Info App: MovieDbProvider: Finding id for item: Some movie
..SNIP..

It looks like the "Removing item from database" action is done for most if not all of my movies. And those are then re-added couple minutes later during the same scan? Now because about 30% of my library is miss-identified this remove/re-add cycle breaks about 30% of my library metadata.

 

I can restore the backup again and try with different version but currently even the latest one seems to be broken.

 

For the record I do not have nfo and metadata files with my media because EMBY only has read-only access there.

 

I haven't found a clear answer as for which version should contain the fix for this bug or what are the circumstances to trigger this, but I'm willing to try couple things on the backup DB if anyone can provide some pointers.
 

[Happy2Play 8281]

So I think I have the same issue. I'm using the official docker image and I've just restored the stuff that's normally mounted under "/config/" in the container from ~2 weeks old backup. Upon start (using the "latest" image) my library looks okay (minus the recently added movies, but that's expected) but soon after the library scan starts. It takes long time and breaks all of my custom edited items, etc..

 

I've looked into logs and I can see some interesting bits:

...SNIP...
2020-04-08 20:51:46.926 Info App: Application version: 4.4.2.0
...SNIP...
2020-04-08 20:52:09.685 Info App: Emby.Kodi.SyncQueue: Creating DB Repository...
..SNIP...
2020-04-08 20:52:52.151 Info App: Removing item from database, Type: Folder, Name: Some movie, Path: /data/movies/Some movie, Id: 39444
2020-04-08 20:52:52.156 Info App: Deleting path /config/metadata/library/39/3909fc59177ffc59177b95b3ff4db0c
2020-04-08 20:52:52.158 Info App: Deleting path /config/metadata/library/3e/3e4d9b8abbf37b0d950e9b8abbba7193
2020-04-08 20:52:52.259 Info App: Removing item from database, Type: Folder, Name: Other movie, Path: /data/movies/Other movie, Id: 39445
2020-04-08 20:52:52.260 Info App: Deleting path /config/metadata/library/09/0921fa1876cbb016ec55c76cbb09d7c2
2020-04-08 20:52:52.271 Info App: Deleting path /config/metadata/library/98/9811fa9c0cfaa0f97f261a11fa9c8324
..SNIP..
2020-04-08 20:54:49.906 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: /bin/ffprobe -i file:"/data/movies/Some movie/Some movie.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
..SNIP..
2020-04-08 20:54:50.335 Info App: MovieDbProvider: Finding id for item: Some movie
..SNIP..

It looks like the "Removing item from database" action is done for most if not all of my movies. And those are then re-added couple minutes later during the same scan? Now because about 30% of my library is miss-identified this remove/re-add cycle breaks about 30% of my library metadata.

 

I can restore the backup again and try with different version but currently even the latest one seems to be broken.

 

For the record I do not have nfo and metadata files with my media because EMBY only has read-only access there.

 

I haven't found a clear answer as for which version should contain the fix for this bug or what are the circumstances to trigger this, but I'm willing to try couple things on the backup DB if anyone can provide some pointers.

 

 

These switches were introduce back in v4 release and were adjusted between updates so some setups can and will have mismatched setting.

 

In your system.xml do you know you're <CollapseVideoFolders> setting before and after the update?  Believe it is in your /config/config folder

 

Same for <CollapseSingleItemFolders> in your options.xml for each library or affected libraries?  /config/root/default/each library

 

Changes in these are destructive and the library has to be rebuilt.

Edited April 8, 2020 by Happy2Play

[mprasil 14]

Thanks for update. The setup looks like this right now:

# fgrep CollapseVideoFolders config/system.xml 
  <CollapseVideoFolders>false</CollapseVideoFolders>

# fgrep CollapseSingleItemFolders root/default/Movies/options.xml
# (no result)

This is the same for backups and after running the latest Emby version. Does that answer your question? What version can I run safely right now without breaking my metadata?

[Happy2Play 8281]

Thanks for update. The setup looks like this right now:

# fgrep CollapseVideoFolders config/system.xml 
  <CollapseVideoFolders>false</CollapseVideoFolders>

# fgrep CollapseSingleItemFolders root/default/Movies/options.xml
# (no result)

This is the same for backups and after running the latest Emby version. Does that answer your question? What version can I run safely right now without breaking my metadata?

No because we would need to know if these settings are changing between server version A and B.

 

Have you looked in the options.xml?  I only know Windows and everyone of my libraries have that setting.

 

I would assume a backup of any version before 4.4.1.0 would work.

[Brendon 101]

Hi.  Did you update a previous version manually?  If you do it manually, then that option is re-enabled, I believe.

 

just to let you guys know after yesterdays update emby enabled auto-updates again even though i had it turned off, just noticed it now.

Edited April 9, 2020 by Brendon

[mprasil 14]

No because we would need to know if these settings are changing between server version A and B.

 

Have you looked in the options.xml?  I only know Windows and everyone of my libraries have that setting.

 

I would assume a backup of any version before 4.4.1.0 would work.

I've looked 9 weeks back, CollapseVideoFolders was always "false" and my Movies library settings never had CollapseSingleItemFolders setting. Is there more info to look at somewhere regarding this?

 

I tried to restore back to 4.4.0.40 and that still broke the library on first scan shortly after start. I'll try to do 4.4.0.30.

 

Edit: Tried 4.4.0.30 and that did scan the library, but it didn't break the metadata.

Edited April 9, 2020 by mprasil

[mprasil 14]

Edit: Tried 4.4.0.30 and that did scan the library, but it didn't break the metadata.

4.4.0.30 is beta version, so I figured I'll try to upgrade to 4.4.0.40 after as that is the last version before 4.4.1.0 that I can see. Just stopped the container, changed the image tag, started it again and soon after start:

...SNIP...
2020-04-09 08:58:41.080 Info App: Removing item from database, Type: Folder, Name...
...SNIP...

So whatever is causing this was introduced between 4.4.0.30 and 4.4.0.40.

 

Any ideas where to go from here? I can set the image version to 4.4.0.30 as a workaround for now, but I don't want to run weeks old version for too long.

[mprasil 14]

I've did few more experiments trying to follow the beta upgrade path:

• Started with 4.4.0.30 - everything is okay, can't reproduce the problem
• Upgraded to 4.5.0.1 - everything still okay, triggering library scan manually, etc.. can't reproduce the problem
• Upgraded to 4.5.0.2 - same as above, all seems to be working fine
• Upgraded to 4.5.0.3 - upon first library scan trigger, the "Removing item from database" start to appear and soon after metadata is wrong
• Restored backup and went back to 4.5.0.2 - everything seems to be working fine

 

So it looks like something introduced between 4.5.0.2 and 4.5.0.3 breaks my metadata for whatever reason. I'd appreciate any help where to go from here.

Edited April 9, 2020 by mprasil

[PenkethBoy 2063]

yes .3 was broken as i reported the issue

 

try .4  and .5 - not seeing anything odd with those

[mprasil 14]

yes .3 was broken as i reported the issue

 

try .4  and .5 - not seeing anything odd with those

This was not the case for me. All 4.5.0.3 and later releases are broken for me. Including .4 and .5

[Happy2Play 8281]

But it still comes back to a specific configurations on said system, as it does not affect all installs. 

 

So if one were to do a clean install without restoring of anything besides user and userdata of say version 4.3 they would not see this issue at all updating.  If you have really old installs, every new versions will/can be affected differently.

 

 

@@mprasil

 

If you created a new library what is the "CollapseSingleItemFolders" setting?

Edited April 9, 2020 by Happy2Play

[PenkethBoy 2063]

When i had problems with .3 - it was a very new db created during the 4.4 beta cycle less than a month old - so - it happens with new Db's as well as old

But it still comes back to a specific configurations on said system, as it does not affect all installs. 

 

So if one were to do a clean install without restoring of anything besides user and userdata of say version 4.3 they would not see this issue at all updating.  If you have really old installs, every new versions will/can be affected differently.

 

 

[mprasil 14]

@@mprasil

 

If you created a new library what is the "CollapseSingleItemFolders" setting?

It creates library with the "CollapseSingleItemFolders" set to true:

# fgrep CollapseSingleItemFolders root/default/Testlib/options.xml 
  <CollapseSingleItemFolders>true</CollapseSingleItemFolders>

 

[Happy2Play 8281]

 

It creates library with the "CollapseSingleItemFolders" set to true:

# fgrep CollapseSingleItemFolders root/default/Testlib/options.xml 
  <CollapseSingleItemFolders>true</CollapseSingleItemFolders>

So the absence of this setting in in your libraries will play a factor in current versions.

[mprasil 14]

So the absence of this setting in in your libraries will play a factor in current versions.

I have some other libraries that have it set to "false" (they were all created at about the same time AFAIK) that do not seem to have this problem. Is the absence of the setting something I can fix?

[Happy2Play 8281]

From my understand if <CollapseVideoFolders> is false all <CollapseSingleItemFolders> are independent of each other so each library can be true or false.  But if <CollapseVideoFolders> is true all <CollapseSingleItemFolders> are irrelevant and provides the collapsedview.

 

In the end all new installs are both set to true.  But from a old install standpoint the could means a destructive change.

 

At some point Devs are going to have to say a clean/new install is require and you can only restore user and userdata to get everyone on the same track.  Or attempt to explain why this happened on said update.

[AmericanCrisis 28]

could I suggest that you enclose  the "- 4k" and "- 1080p" inside square brackets [4k] / [1080p]. Anything inside [ ] is disregarded by the metadata search function thus providing faster and more accurate recognition of your movies should problems like this arise.

 

Can you clarify?

 

Do I configure like this:

 

• Title (year) - [4k]
• Title (year) - [1080p]

Or:

• Title (year) [4k]
• Title (year) [1080p]

Or:

• Title (year) [ - 4k]
• Title (year) [ - 1080p]