Large library imported overnight, no media displayed this morning

[bruor 43]

I am in the process of converting from Jellyfin and hitting a major snag after doing some testing on a limited subset of my library and finding everything works beautifully. 

Overnight I let Emby import all media (12614 movies and 2463 series containing 126397 episodes), this morning I'm unable to access any of the libraries, the initial sections for the library are displayed, but if I click on any of them, there's no response in the client at all.  I seem able to admin the server but just can't browse my media library to try to play anything.  Live TV is inaccessible as well. 

The server isn't giving me anything to go from in the logs to try to determine the cause of the issue to see if there's something I should try to do a little differently.

Edited January 20, 2023 by bruor
clean up language

[GrimReaper 4740]

17 minutes ago, bruor said:

Overnight I let Emby import all media (12614 movies and 2463 series containing 126397 episodes)

I'd say for that amount of media overnight ain't enough time to import - is Scan media library task still running? If not, what does it say in Scheduled Task for how long it ran?

[bruor 43]

It is not still running.  It looks like it may have run again since the initial import.  It says last run was 4 hours ago and took 42 mins.

[GrimReaper 4740]

Do you store NFOs/artwork next to your media?

[rbjtech 5284]

A log would show what is happening, but the scheduled task (in the scheduled task admin section) does not necessarily indicate the task has completed.

For a library that size - on a new install - depending on your internet speed and throttling by the providers - you are looking at multiple hours best case, multiple days worst case.

there are various things that can be done to speed things up -local nfo files, removing providers, postponing chapter thumbnails etc - are all options - which is where @GrimReaper is likely going with this ..

 

Edited January 20, 2023 by rbjtech

[rbjtech 5284]

To add, you scan also individually scan per library if you wish - just select it from the dropdown on the library itself in the web browser.

Edited January 20, 2023 by rbjtech

[bruor 43]

1 hour ago, GrimReaper said:

Do you store NFOs/artwork next to your media?

No, the media folder is a dynamically generated/update set of STRM files that are ripped from an M3U.  Out of date files and empty directories are purged each time the M3U is rescanned. I'd have to add handling to check for and skip deletion of the info files for which STRM files exist etc. 

I will likely implement a file creation throttle so that it doesn't generate the entire set in one run, instead only adding X new STRM files per run so I can slowly add content to the system.  

I've got gigabit here and will try a manual scan before I attempt to remove the STRM folders from the library and rescan to shrink it. 

Edited January 20, 2023 by bruor

[bruor 43]

1 hour ago, rbjtech said:

To add, you scan also individually scan per library if you wish - just select it from the dropdown on the library itself in the web browser.

I'm unable to do this,  it's almost like the UI doesn't load entirely,  clicking on the 3 dot icon that appears on the library does nothing.  I have to go through the server settings page to attempt a library rescan. 

[Luke 42078]

Please attach the emby server log from when you tried to do that. Thanks.

[bruor 43]

They are too large to attach here, uploading to google drive for you.  May need to give them a few minutes, should be 519MB total.
 

Edit: DM sent to Luke with the URL

 

Edited January 20, 2023 by bruor

[rbjtech 5284]

1. In each library settings - remove all the Fetchers

2. Generate preview thumbnails = never (temporary)

3. In schedule - remove all the schedules for Scan library and scan metadata.

4. Re-start emby

5. After the re-start - come back to us on if you have control back.

6. If ok - then try a manual library scan as above.

7. no metadata is going to be got at this stage - but all the items should at least be listed and be available to play.

8. now add back the Fetchers

9. finally add back the thumbnails (this will take a while ..)

 

Edited January 20, 2023 by rbjtech

[bruor 43]

For now I've been able to get it working again.  I removed the STRM folders from the libraries, bounced the server and ran a rescan.  I removed all the files from the STRM folders, added them back to the libraries empty, ran a scan, then added around 10% of the media back and have started a rescan.  Once done and stable, I'll add a little more and rescan etc until I get it caught up. 

I don't think it was stuck downloading metadata or thumbnails because there was only the EmbyServer.exe process consuming roughly 100% of one core, but no disk or network activity.  After the service restarted it was idle, I'd only see it consume CPU when I tried to access the webUI or an android app, at which point it would go 100% usage for 5-10 minutes and then drop idle again. 

I've also made a change now to the databse settings, I told it to allow a 2048MB cache size in ram since I have the free memory and the SSD in this system isn't the fastest. 

Edited January 20, 2023 by bruor

[rbjtech 5284]

4 minutes ago, bruor said:

For now I've been able to get it working again.  I removed the STRM folders from the libraries, bounced the server and ran a rescan.  I removed all the files from the STRM folders, added them back to the libraries empty, ran a scan, then added around 10% of the media back and have started a rescan.  Once done and stable, I'll add a little more and rescan etc until I get it caught up. 

I don't think it was stuck downloading metadata or thumbnails because there was only the EmbyServer.exe process consuming roughly 100% of one core, but no disk or network activity.  After the service restarted it was idle, I'd only see it consume CPU when I tried to access the webUI or an android app, at which point it would go 100% usage for 5-10 minutes and then drop idle again. 

I've also made a change now to the databse settings, I told it to allow a 2048MB cache size in ram since I have the free memory and the SSD in this system isn't the fastest. 

ok - that's the right approach - the logs were showing a huge amount of access errors to services - so a slow build of the database is a good way to proceed with this.

[bruor 43]

I've got my first batch of media imported and it's definitely causing a delay when the home screen is loaded.  It takes around 4 seconds to switch from this: 

 

to this: 

 

3672 movies,  136 series 4933 episodes

Removing the TV folder from the library results in a load time of approx 1.5 sec.  

Thinking I might change my approach and include only the TV subfolders for the shows that I want to be in EMBY,  is there a limit to the number of folders that can be added to a library?  How does it handle things if a folder gets deleted? 

OR, if I enable "show missing episodes" can I get the new missing episodes when a series releases new ones to show up in the "continue watching" list so I can be notified that I need to copy the strm files into the library folder? 


 

Edited January 20, 2023 by bruor

[Happy2Play 9780]

What is your library.db size?

What is your Database Cache Size setting? App Settings-Database

[bruor 43]

27 minutes ago, Happy2Play said:

What is your library.db size?

What is your Database Cache Size setting? App Settings-Database

I'm not sure if it would have shrunk because I've removed media and trying to find a new way to work with it. 

library.db = 962MB

Cache size = 2048MB

[Happy2Play 9780]

2 minutes ago, bruor said:

I'm not sure if it would have shrunk because I've removed media and trying to find a new way to work with it. 

library.db = 962MB

Cache size = 2048MB

Looks good, you would potentially have to vacuum the database for it to shrink.  I am not sure why you are getting these slow loads, server log may show more information.

[bruor 43]

I have done some more testing here.  If I disable "continue watching" the UI loads instantly. 

My assumption is that "continue watching" query for movies is a lot less intense than the same query against a TV Series. 

How is the "continue watching" list composed under the hood? 

[Happy2Play 9780]

2 minutes ago, bruor said:

I have done some more testing here.  If I disable "continue watching" the UI loads instantly. 

My assumption is that "continue watching" query for movies is a lot less intense than the same query against a TV Series. 

How is the "continue watching" list composed under the hood? 

Depends on your setting as it tracks all items with Resume points and Next Up items for all Shows the user is watching.

Have you tried CW and Next Up (Legacy) rows?

[bruor 43]

Yes, having it enabled at all is problematic.

I was looking more for an explanation of how it searches the DB for the items to compose the view.  Was curious if it has to churn through every episode and do comparisons etc. 

[Happy2Play 9780]

4 minutes ago, bruor said:

Yes, having it enabled at all is problematic.

I was looking more for an explanation of how it searches the DB for the items to compose the view.  Was curious if it has to churn through every episode and do comparisons etc. 

I guess you can go to your Emby system.xml and enable <LogAllQueryTimes> and restart Emby to see the queries.  Ensure you have debug logging enabled also.  Or the devs will have to better explain.

[rbjtech 5284]

12 hours ago, Happy2Play said:

Looks good, you would potentially have to vacuum the database for it to shrink.  I am not sure why you are getting these slow loads, server log may show more information.

Agreed - I would vacuum the DB (see the db settings in the menu) - that is a large db size for such a small library (at the moment).

As a comparison with movies of 1200 and episodes of 27K - my dB size is 350Mb.

 

[bruor 43]

Vacuum of the DB brought it down to 560MB. 

I've relented and wrote a script that will push any "favorite shows" from the STRM folder to an EmbySTRM folder nightly.  If the series disappears in the STRM folder it'll delete it in the EmbySTRM folder.   This is the best thing I could come up with so that new episodes are added to, or removed series are purged from, the system automatically. 

I'll see if the devs can weigh in here before I try to enable logging and dig into the way it might be working.  If the continue watching results for episodes has to be calculated on the fly currently, perhaps the current or next episode could be stored in a sort of header record for the series and it could be calculated during playback events, library scans, or trakt.tv syncs so that pulling the "continue watching" list is a bit more efficient. 

Edited January 21, 2023 by bruor

[neik 873]

On 1/20/2023 at 4:47 PM, rbjtech said:

there are various things that can be done to speed things up -local nfo files, removing providers, postponing chapter thumbnails etc - are all options - which is where @GrimReaper is likely going with this ..

 

On 1/20/2023 at 7:35 PM, rbjtech said:

1. In each library settings - remove all the Fetchers

2. Generate preview thumbnails = never (temporary)

3. In schedule - remove all the schedules for Scan library and scan metadata.

4. Re-start emby

5. After the re-start - come back to us on if you have control back.

6. If ok - then try a manual library scan as above.

7. no metadata is going to be got at this stage - but all the items should at least be listed and be available to play.

8. now add back the Fetchers

9. finally add back the thumbnails (this will take a while ..)

 

@cayarsthis could be a great input for all a knowledgebase page.

I will be moving my server to new hardware and had to look through the forum to find rbj's posts.

Just a suggestion...

@rbjtech, thank you for the input. 

[Happy2Play 9780]

Not sure about the KB but could be in Tutorials/Guides section.