Jump to content
Netfool

ANSWERED Vanishing Collections

Recommended Posts

Netfool

Problem on an Nvidia Shield Pro running Emby Server 4.4.3.0. 

Sometime over the past 72 hours all of the collections on my Emby Server webUI and TV UI disappeared:

1226170776_ScreenShot2020-08-11at11_25_33AM.thumb.png.463736dd4c7d353b29a555056b3c4757.png

While they are missing from the UI, they are still in the "internal storage":

832109109_ScreenShot2020-08-11at11_44_24AM.png.f917919eab88710a6185885025dd867d.png

The highlighted collection.xml file is attached, and the paths it contains appear to be correct:

1977068990_ScreenShot2020-08-11at11_48_17AM.png.183ff658c1e0fdbe284a8d871332f7d8.png

How do I get the server to recognize the collections again?

 

collection.xml

Share this post


Link to post
Share on other sites
Happy2Play

What happens if you manually create a test collection?

Does anything appear in the Collections library on the home screen?

Share this post


Link to post
Share on other sites
Netfool
34 minutes ago, Happy2Play said:

What happens if you manually create a test collection?

Does anything appear in the Collections library on the home screen?

Yes, the new collection appears as expected and when returning to the Movies listing the individual movies were replaced by the collection. It also appears as expected in the Folders view.  The folder and collection.xml file are in the expected place in the internal storage directory listing along with all of the older collections which are now ignored.

Share this post


Link to post
Share on other sites
Happy2Play

Might need a server log, but if a new collection did not bring back the existing collections, that usually points to either a xml item path issue or a database issue with that path.

Share this post


Link to post
Share on other sites
Netfool
33 minutes ago, Happy2Play said:

Might need a server log, but if a new collection did not bring back the existing collections, that usually points to either a xml item path issue or a database issue with that path.

The last few server logs attached.  What should I grep for in this case?     ....and where is the database, and what format is it in.   sqlite?

embyserver-63732675305.txt embyserver.txt embyserver-63732700800.txt embyserver-63732748908.txt

Share this post


Link to post
Share on other sites
Happy2Play

Looks like this collection was created here with a Ids=53066

2020-08-11 13:41:10.504 Info HttpServer: HTTP POST http://otto:8096/emby/Collections?Name=Charlie%20Chan%20Movies&Ids=53066&UserId=a54c8eabcbc2428d9980d15322e4f121

 

Don't really know the android at all but to me looking at the embyserver.txt a collection is deleted but the server continues to keep looking a that id 47512

2020-08-11 13:41:44.600 Info HttpServer: HTTP GET http://otto:8096/emby/Items/47512/DeleteInfo. UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:80.0) Gecko/20100101 Firefox/80.0
2020-08-11 13:41:44.606 Info HttpServer: HTTP Response 200 to 172.20.20.203. Time: 5ms. http://otto:8096/emby/Items/47512/DeleteInfo
2020-08-11 13:41:55.044 Info HttpServer: HTTP DELETE http://otto:8096/emby/Items/47512. UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:80.0) Gecko/20100101 Firefox/80.0
2020-08-11 13:41:55.046 Info App: Removing item from database, Type: BoxSet, Name: Charlie Chan Movies, Path: /storage/emulated/0/Android/data/com.emby.embyserver/files/data/collections/Charlie Chan Movies [boxset], Id: 47512
2020-08-11 13:41:55.048 Info App: Deleting path /storage/emulated/0/Android/data/com.emby.embyserver/files/metadata/library/21/21f507e7a9155de1c777026f80278a81
2020-08-11 13:41:55.074 Info App: Deleting path /storage/emulated/0/Android/data/com.emby.embyserver/files/data/collections/Charlie Chan Movies [boxset]
2020-08-11 13:41:55.163 Info HttpServer: HTTP Response 204 to 172.20.20.203. Time: 120ms. http://otto:8096/emby/Items/47512

2020-08-11 13:42:01.068 Info HttpServer: HTTP GET http://otto:8096/emby/Users/a54c8eabcbc2428d9980d15322e4f121/Items/47512. UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:80.0) Gecko/20100101 Firefox/80.0
2020-08-11 13:42:01.092 Error HttpServer: Error processing request
	*** Error Report ***
	Version: 4.4.3.0
	Command line: /data/app/com.emby.embyserver-Y5G4QDr94uOBcEFGYcQVsQ==/base.apk
	Operating system: Unix 4.9.140.711
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: False
	Runtime: file:///mscorlib.dll
	System.Environment.Version: 4.0.50524.0
	Processor count: 4
	Program data path: /storage/emulated/0/Android/data/com.emby.embyserver/files
	Application directory: /data/user/0/com.emby.embyserver
	Mono: 6.12.0 (2020-02/165f4b03417)
	Android Version: 9-REL - Base:  - SDK: P
	Build ID: PPR1.180610.011 - Incremental: 4199437_1915.8582 Patch-Level: 2019-12-05
	Fingerprint: NVIDIA/mdarcy/mdarcy:9/PPR1.180610.011/4199437_1915.8582:user/release-keys
	Model: SHIELD Android TV - NVIDIA/NVIDIA
	Hardware: mdarcy/mdarcy/darcy/darcy
	SupportedAbis: arm64-v8a, armeabi-v7a, armeabi
	System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object
	  at Emby.Server.Implementations.Dto.DtoService.AttachPeople (MediaBrowser.Model.Dto.BaseItemDto dto, MediaBrowser.Controller.Entities.BaseItem item, MediaBrowser.Controller.Dto.DtoOptions options) [0x00014] in <c2c18c31369c438da8f2b08911328c4d>:0 
	  at Emby.Server.Implementations.Dto.DtoService.GetBaseItemDtoInternal (MediaBrowser.Controller.Entities.BaseItem item, MediaBrowser.Controller.Dto.DtoOptions options, MediaBrowser.Controller.Entities.User user) [0x0002e] in <c2c18c31369c438da8f2b08911328c4d>:0 
	  at Emby.Server.Implementations.Dto.DtoService.GetBaseItemDto (MediaBrowser.Controller.Entities.BaseItem item, MediaBrowser.Controller.Dto.DtoOptions options, MediaBrowser.Controller.Entities.User user) [0x00000] in <c2c18c31369c438da8f2b08911328c4d>:0 
	  at MediaBrowser.Api.UserLibrary.UserLibraryService.Get (MediaBrowser.Api.UserLibrary.GetItem request) [0x000f7] in <412630a1177c491e9d64fd85eb0a625f>:0 
	  at Emby.Server.Implementations.Services.ServiceController.GetTaskResult (System.Threading.Tasks.Task task) [0x00081] in <c2c18c31369c438da8f2b08911328c4d>:0 
	  at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync (Emby.Server.Implementations.HttpServer.HttpListenerHost appHost, MediaBrowser.Model.Services.IRequest httpReq, MediaBrowser.Model.Services.IResponse httpRes, Emby.Server.Implementations.Services.RestPath restPath, System.String responseContentType, System.Threading.CancellationToken cancellationToken) [0x001da] in <c2c18c31369c438da8f2b08911328c4d>:0 
	  at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler (MediaBrowser.Model.Services.IRequest httpReq, System.ReadOnlyMemory`1[T] urlString, System.ReadOnlyMemory`1[T] localPath, System.Threading.CancellationToken cancellationToken) [0x00b52] in <c2c18c31369c438da8f2b08911328c4d>:0 
	Source: mscorlib
	TargetSite: Void Throw()
	
2020-08-11 13:42:01.097 Info HttpServer: HTTP Response 500 to 172.20.20.203. Time: 29ms. http://otto:8096/emby/Users/a54c8eabcbc2428d9980d15322e4f121/Items/47512

Then it looks like it is removed again under a different id 78290.

2020-08-11 13:53:20.079 Info App: Removing item from database, Type: BoxSet, Name: Charlie Chan Movies, Path: /storage/emulated/0/Android/data/com.emby.embyserver/files/data/collections/Charlie Chan Movies [boxset], Id: 78290

I can only assume conflicting database entries.  

 

27 minutes ago, Netfool said:

and where is the database, and what format is it in.   sqlite?

The library.db is in the same folder the collections are in as shown above.  I would ensure you have a backup copy of the db before looking around.  Emby has to be offline to open current database.  And yes to sqlite.

Share this post


Link to post
Share on other sites
Luke

If you run a library scan does the new collection disappear again?

Share this post


Link to post
Share on other sites
Netfool
56 minutes ago, Happy2Play said:

looking at the embyserver.txt a collection is deleted but the server continues to keep looking a that id 47512

OK.  That deletion at 13:41 was me trying an experiment right after having created the test collection you suggested.

I re-created one of the vanished collections by adding one of the movies that was in the old collection.  I was hoping that might bring back the whole set, but it didn't.  It did find the poster image I had for the old one.  So it looks like when I deleted that it deleted both the new and old db entries.  It also deleted the directory with that collection.xml and poster image in it.

 

Share this post


Link to post
Share on other sites
Netfool
7 minutes ago, Luke said:

If you run a library scan does the new collection disappear again?

No, did a complete library scan from the dashboard scheduled tasks menu, and it's still there.

Share this post


Link to post
Share on other sites
Netfool
Posted (edited)

@Luke @Happy2Play I can't find a clear definition of what's included in "System Configuration" and "Library Definition" in the context of restoring from a backup. Which one includes library.db?   ...and I'm guessing that library.db-shm and library.db-whl don't need to be restored simultaneously with library.db.   I did see the instructions for manual restoration but that sounds like a recipe for unintended consequences.

Edited by Netfool

Share this post


Link to post
Share on other sites
Happy2Play
1 hour ago, Netfool said:

@Luke @Happy2Play I can't find a clear definition of what's included in "System Configuration" and "Library Definition" in the context of restoring from a backup. Which one includes library.db?   ...and I'm guessing that library.db-shm and library.db-whl don't need to be restored simultaneously with library.db.   I did see the instructions for manual restoration but that sounds like a recipe for unintended consequences.

Emby does not backup and restore databases or Emby stored images.  The plugin only does configuration.

As for the additional library.db files, they only exist when the database is open so no they are not needed.

Yes manual restoration can be problematic as almost everything in the library.db would need to rebuild since other content such as the \metadata folder is no longer present, but in the end it is all about what your intentions are as technically users only need the usersdata table in a restore..  Ideally you would/should maintain a system backup, but from a Emby standpoint if you backed up the programdata folder for Emby you have everything.

Program data path: /storage/emulated/0/Android/data/com.emby.embyserver/files

Have you looked around in the database yet for the missing collections?

Share this post


Link to post
Share on other sites
Netfool
Posted (edited)
1 hour ago, Happy2Play said:

Ideally you would/should maintain a system backup, but from a Emby standpoint if you backed up the programdata folder for Emby you have everything.

I agree, but this is an Nvidia Shield.  All of the back-up & restore apps in the Google play store seem to be made for phones.   About the only way I can figure to do a system backup is to script something on a linux box that makes an SMB connection and then does a file copy of "internal".   If I set it to run during the wee small hours of the morning and not have any scheduled task run around then, I might be able to avoid backing up a moving target, but it's not a given.  It would be better to be able to shut down the Emby Server and then copy it all, but there isn't any way to script a shutdown and restart as far as I can tell. 

 

1 hour ago, Happy2Play said:

Have you looked around in the database yet for the missing collections?

I did, but the table naming is so obtuse I'm not sure where to look, and without a better picture of the schema, it's puzzling.  Is the DB structure documented anywhere?

Edited by Netfool

Share this post


Link to post
Share on other sites
Happy2Play
17 minutes ago, Netfool said:

did, but the table naming is so obtuse I'm not sure where to look, and without a better picture of the schema, it's puzzling.  Is the DB structure documented anywhere?

No that I know of, don't know what you are using to look at the db but I go to the mediaitems table and filter the paths column.

example 

type 3 is Folder

type 5 is each Movie

type 10 is each trailer

type 9 is the virtual boxset/collection

db.thumb.jpg.7e166cbd6e1bf3df3b9bed742a8b623d.jpg

 

Share this post


Link to post
Share on other sites
Netfool

@Happy2Play
 

AHA!  That's an immense help!  Thank you!   I'm using the DB4S browser for SQLite on MacOS.

Share this post


Link to post
Share on other sites
Netfool

@Happy2Play

It looks to me like they're there.   ...and the paths seem correct to me.

1783169602_ScreenShot2020-08-12at5_21_29PM.png.9b9b2afc71441b632dcd27b73becaf88.png

 

Share this post


Link to post
Share on other sites
Happy2Play
1 minute ago, Netfool said:

@Happy2Play

It looks to me like they're there.   ...and the paths seem correct to me.

1783169602_ScreenShot2020-08-12at5_21_29PM.png.9b9b2afc71441b632dcd27b73becaf88.png

 

Right but you have to look at the bigger picture as the type will be there.  Pick a specific example that is missing and do what I did in my screenshot.  I am guess there is a rogue entry somewhere.

Share this post


Link to post
Share on other sites
Netfool
5 hours ago, Happy2Play said:

Pick a specific example that is missing and do what I did in my screenshot.

OK... I did miss that you were filtering on a fragment of the path, but....

I believe your example is not representative of the general situation:

  1. All of the movies in the collection are also in the same folder.  That seems redundant.
  2. The substring "12 rou" is in the collection name, the folder name and the movie name.

So if I filter for "Marx" in the path I get:

892414869_ScreenShot2020-08-12at11_44_31PM.png.8d51d65c43e36f9a6c0bd9b3b626cad3.png

The complete xml file for that collection is attached, but it contains:

550414118_ScreenShot2020-08-12at11_57_09PM.thumb.png.1cfa8363a723a024fa8d4327491df6d6.png

If I filter on "At the Circus" in the path I get:

1081982290_ScreenShot2020-08-12at11_49_05PM.thumb.png.98e16fc0a0a397ff9ad9bba84dd69870.png

That shows me the trailer for the Marx Brothers movie At the Circus (1939) is there (and type 5 not type 10), but the movie doesn't seem to to be in the database, however:

1431632749_ScreenShot2020-08-13at12_04_23AM.thumb.png.da50922e475d479e6a68cdd19f969c46.png

...and...

1324941941_ScreenShot2020-08-13at12_08_02AM.thumb.png.a36d6b82033bcbac9e77b709e5f8a643.png

So pretty clearly the database is borked.  Is there a way to rebuild the database from the content without reloading the whole server?

...and yes, I did do a full library scan from the scheduled tasks menu just before I wrote this post.

collection.xml

Share this post


Link to post
Share on other sites
Happy2Play
3 hours ago, Netfool said:

That shows me the trailer for the Marx Brothers movie At the Circus (1939) is there (and type 5 not type 10), but the movie doesn't seem to to be in the database,

It is showing the correct information as you look at the path for type 5 you see the movie.  The data column is just providing additional info for that item.  Type 10 is only there if your items contain Local Trailers.

3 hours ago, Netfool said:

Is there a way to rebuild the database from the content without reloading the whole server?

Ideally I would use the server configuration backup plugin so I had all my userdata.  Shut down Emby, delete or rename the library.db, start Emby and do a Full Library Scan.

Depending on how old your original library.db is you could possibly manually import the userdatas table from old to new database if you don't use plugin.

Share this post


Link to post
Share on other sites
Netfool
8 hours ago, Happy2Play said:

 I would use the server configuration backup plugin so I had all my userdata.  Shut down Emby, delete or rename the library.db, start Emby and do a Full Library Scan.

Great!  Thank you!   That sounds like a plan. 

I do have the config backup plugin running, although it's only been running for the last week or two.   I did mention that in the "Things I Wish I Knew..." post as nothing in the set-up instructions warns you you have to dive into the plugins page to configure it before anything will happen.

I'll report back here.

Share this post


Link to post
Share on other sites
Netfool
12 hours ago, Happy2Play said:

I would use the server configuration backup plugin so I had all my userdata.  Shut down Emby, delete or rename the library.db, start Emby and do a Full Library Scan.

 

That worked!  ...but not without unexpected side effects.  It created a new library named Recordings that pointed to the same folder and one I had named Live TV DVR for recordings from the HDHomeRun.  When I deleted the Recordings library, all the content disappeared from Live TV DVR library until I did a library scan of just that library.   It does take some time for the first full library scan, but that's to be expected, as the entire library.db needs to be recreated.

The good news is that all of the formerly inoperative Collections came back.   The bad news is that after looking at the Collections view in the Movies library once all of the various views vanished from the UI.  So now the Movies library listing looks like this:

1042994303_ScreenShot2020-08-13at4_30_24PM.thumb.png.df9a859f184b7fe2bf6aac47e3821469.png

...and the ●●● menu looks like this:

1172499107_ScreenShot2020-08-13at4_30_57PM.thumb.png.3f159972cd6326c39c4bb9229c93d9da.png

...when it used to look like this:

689929679_ScreenShot2020-08-13at1_14_34PM.thumb.png.d8ceb373791726a8c146a5c9b4b0da34.png

That looks like I must have inadvertently turned off all the alternate views, but I can't figure out where that setting is.  

Any suggestions?

 

Share this post


Link to post
Share on other sites
Happy2Play

Wild guess, did your libraries come back as their proper content type?  I am guessing they are now Mixed Content.  Dashboard-Library

If that did happen, pretty sure I have a work around but would require a new database again.

  • Like 1

Share this post


Link to post
Share on other sites
Netfool
17 minutes ago, Happy2Play said:

I am guessing they are now Mixed Content. 

They did indeed!  I didn't notice that.  A new database isn't that big a deal.  The process is simple... a bit slow, but not painful.

What's the work-around?

Share this post


Link to post
Share on other sites
Happy2Play
17 minutes ago, Netfool said:

They did indeed!  I didn't notice that.  A new database isn't that big a deal.  The process is simple... a bit slow, but not painful.

What's the work-around?

Place a content type .collection file in each folder, the delete your library.db, start and scan.

 

/storage/emulated/0/Android/data/com.emby.embyserver/files/root/default/ "each library folder"

 

https://emby.media/community/index.php?/topic/80502-backup-plugin-not-restoring-library-properly/page/3/&tab=comments#comment-884434

 

  • Like 1

Share this post


Link to post
Share on other sites
Netfool
21 hours ago, Happy2Play said:

Place a content type .collection file in each folder, the delete your library.db, start and scan.

That appears to have worked!   One odd thing is that it created a new multi-content Library called "Recordings".  That has the same path as my live tv recordings Library called "Live TV DVR".    I'm guessing I can just delete that in the Libraries page in the dashboard.

But that raises a question.  Where is Emby Server expecting live tv recordings to be?   If I go to Live TV and click on the Recordings heading I get a "No Items Found" page:

158378282_ScreenShot2020-08-14at3_26_27PM.png.244717616c6a2e511fb75c47963bba12.png

  It looks like I need to set a path somewhere, but I can't find where that's set.

Share this post


Link to post
Share on other sites
cayars

Hi Netfool,

Try the DVR menu option on the left in web admin then you can put in a recording path in "Default recording path:"

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...