Jump to content
wickedlemon

Slow - almost non-existent - scan

Recommended Posts

wickedlemon

Hi

 

New user and first post so hopefully I'm posting in the right place and it's not full of (too) stupid questions!

 

I recently installed Emby Server on Windows 10, just to see what it actually did and get a feel of how it actually worked. I've now decided to set up Emby on a Raspberry Pi 2. I don't need it for streaming/transcoding, purely for pulling metadata, so I'm not concerned about any of the Pi's power limitations.

 

I installed 4.0.0.2 on a fresh install of Raspbian and started to set it up, adding libraries, plugins as I had done under Windows. Then I noticed that the scanning wasn't going as fast. Nowhere near as fast.

 

Back to a fresh Raspbian/Emby install and this time no plugins and a minimal library - just 12 TV episodes to test. First run and I gave up before it had even completed. The episodes already had all the nfo and thumb images before the scan started but it was still taking ages. Log entries showed ffprobe entries for each file spaced anywhere between 5 and 20 minutes apart.

 

Removed the library, nfos and images and tried again and it's still not working. Dashboard still says its scanning, but there doesn't appear to have been a log entry related to scanning for over half an hour. in both instances there just seems to be huge time periods of Emby not doing anything, or at least not logging anything.

 

Any suggestions? There doesn't seem to be an errors in the log and I had no issues scanning under Windows.

 

Two logs attached. 1 when folder had nfo/thumbs already and 2 with no existing local metadata.

embyserver1.txt

embyserver2.txt

Share this post


Link to post
Share on other sites
wickedlemon

So today I've fired my Windows Emby install up. Deleted folder and removed all nfo files and scanned to refresh all data. It's created 456 nfo files over 2 minutes, compared to the Pi which was taking between 20 minutes and 2 hours between nfos overnight.

Share this post


Link to post
Share on other sites
Luke

Hi, have you completed a full scan on the Pi?

Share this post


Link to post
Share on other sites
wickedlemon

No. Trying again now. I've upgraded to 4.0.1.0 in case that helped but it hasn't.

 

Meanwhile. overnight, my Windows install finished scanning my entire TV library, while being set to download all images and actors and store nfo/images with files.

Share this post


Link to post
Share on other sites
Luke

Ok, once you complete a full scan it should be much faster after that. thanks.

Share this post


Link to post
Share on other sites
wickedlemon

Ok, once you complete a full scan it should be much faster after that. thanks.

 

Thanks for the reply, but in trawling through these forums trying to troubleshoot this issue I've seen this be your stock answer to many problems. I appreciate what you're doing with this software and really don't want to seem rude or ungrateful, but let's be honest: This answer doesn't help.

 

My initial scan will never finish. It was adding on average one tv episode per hour. All my files have images/nfos next to them so it should be way quicker. That Windows is 1000s of times quicker over the Pi shows there is a problem with the library scan.

 

If you need more info to help you troubleshoot then please ask. Do you have any idea why there appears to be such a huge time gap between scanning each item? I tried enabling debug logging and that didn't make any difference. There are huge gaps in the log so what is Emby doing during this time and why isn't it logging anything?

 

Thanks again for your time. 

Share this post


Link to post
Share on other sites
mastrmind11

have you tried a small library with a few items?  did it finish?  if so, just add a few files/folders to the library at a clip.  Comparing a full blown desktop CPU to a rpi isn't a useful comparison.

Share this post


Link to post
Share on other sites
wickedlemon

have you tried a small library with a few items?  did it finish?  if so, just add a few files/folders to the library at a clip.  Comparing a full blown desktop CPU to a rpi isn't a useful comparison.

 

I tried with one tv series of 12 episodes (see the logs I posted) and still gave up. 

 

If I shouldn't be comparing a full blown desktop to a Pi then can you explain why? I really do want to understand why the Pi scan is taking so long but there is nothing in the logs to show me why and there is no maxing out of cpu or ram during scanning. What is Emby doing that is so resource intensive in the background? I'm not using Emby for playback only for library management. As a comparison I notice no little difference in speed when Kodi does a library scan on Pi vs PC.

Share this post


Link to post
Share on other sites
mastrmind11

I tried with one tv series of 12 episodes (see the logs I posted) and still gave up. 

 

If I shouldn't be comparing a full blown desktop to a Pi then can you explain why? I really do want to understand why the Pi scan is taking so long but there is nothing in the logs to show me why and there is no maxing out of cpu or ram during scanning. What is Emby doing that is so resource intensive in the background? I'm not using Emby for playback only for library management. As a comparison I notice no little difference in speed when Kodi does a library scan on Pi vs PC.

hmm.  strange.  and you're writing back to a nas?

Share this post


Link to post
Share on other sites
wickedlemon

hmm.  strange.  and you're writing back to a nas?

 

Yes, although I'm not sure it should be writing anything now as everything has nfo/images already from the Windows scan.

Share this post


Link to post
Share on other sites
mastrmind11

Yes, although I'm not sure it should be writing anything now as everything has nfo/images already from the Windows scan.

Well v4 did bring a bunch of db/meta changes, but still.  And you're hard wired to the nas on both your windows and rpi setups?

Share this post


Link to post
Share on other sites
wickedlemon

Everything's hard wired. I wonder if trying an older version of Emby would be worth it? Purely to see if it was a lot quicker.

 

I've just deleted the library and added one path with about 4 series in and started a scan. CPU is hovering around 9%, ram 30%.

Share this post


Link to post
Share on other sites
mastrmind11

try tailing the log while its scanning to get a better idea of what it's doing (or not).

Share this post


Link to post
Share on other sites
Luke

My initial scan will never finish

 

That's what many say, until it ends up finishing. We need to improve the progress reporting of the scan so that you can get a clear idea of what it's doing. Thanks.

Share this post


Link to post
Share on other sites
wickedlemon

That's what many say, until it ends up finishing. We need to improve the progress reporting of the scan so that you can get a clear idea of what it's doing. Thanks.

 

I'm not going by the percentage, but the fact that it is taking 20mins to 2 hours per episode. I can easily monitor this by deleting all nfos and setting it to create nfos.

Share this post


Link to post
Share on other sites
wickedlemon

try tailing the log while its scanning to get a better idea of what it's doing (or not).

 

My first log is exactly that. You can clearly see it 'not' doing anything for long periods of time.

Share this post


Link to post
Share on other sites
wickedlemon

Going to try a fresh Raspbian image and maybe add one file to the library. Wish me luck  :)

  • Like 1

Share this post


Link to post
Share on other sites
Luke

Thanks, let us know how you get on.

Share this post


Link to post
Share on other sites
wickedlemon

Ok, you asked Luke, so here you go:

 

Fresh Raspbian. Installed 4.0.1.0 with no added plugins.

 

Added TV Shows folder on NAS drive with only one episode (fyi TV show has 1 series and 12 eps total).

460MB mkv file with nfo files for show, season and episode as well as poster, thumbnail etc all from a previous Emby scan.

 

Changed scan settings to make it as quick as possible:

Enable real time monitoring N
Metadata savers: NFO Y
Save artwork into media folders Y
Download images in advance N
Enable chapter image extraction N
 
Saved folder and scan started at 0742. Completed in 24m40s
 
Added a second file and scanned again. Scan started at 0811. Completed in 9m21s
 
Added a third file and scanned again. Scan started at 0823. Completed in 97m13s
 
Added a fourth file and scanned again. Scan started at 1005. Completed in 162m23s
 
Yep, that's 2 hours and forty-two minutes for 4 files!  :o

 

Probably unrelated, but one thing I did notice was the season.nfo file seemed to keep being rewritten with the same contents. Maybe once for each file scanned?

 

Anyway, the four logs are attached. Have fun! I'd love to know why it's taking so long.

embyserver-1 episode (1 new).txt

embyserver-2 episodes (1 new).txt

embyserver-3 episodes (1 new).txt

embyserver-4 episodes (1 new).txt

Share this post


Link to post
Share on other sites
Okeur75

Hi,

 

I took a look at the logs but I'm no expert...

However I can see your library is mount on /mnt/NAS_1. Even if it's hardwired as you said, could you check an "iotop" during the rescan ?

Maybe some background stuff is writing on your NAS in the same time and prevents Emby from doing the same...

Share this post


Link to post
Share on other sites
wickedlemon

Hi,

 

I took a look at the logs but I'm no expert...

However I can see your library is mount on /mnt/NAS_1. Even if it's hardwired as you said, could you check an "iotop" during the rescan ?

Maybe some background stuff is writing on your NAS in the same time and prevents Emby from doing the same...

 

I've ran Ioptop while scanning but TBH I'm not exactly sure what I'm looking for. There doesn't seem to be a huge amount of activity from Emby or any other process showing up.  Emby isn't (or shouldn't) be doing much in the way of writing to the NAS as all the nfos and images are already there. 

Share this post


Link to post
Share on other sites
Okeur75

I've ran Ioptop while scanning but TBH I'm not exactly sure what I'm looking for. There doesn't seem to be a huge amount of activity from Emby or any other process showing up.  Emby isn't (or shouldn't) be doing much in the way of writing to the NAS as all the nfos and images are already there. 

As soon as you see nothing reaching 100% IOWAIT it's ok.

 

I just wanted to make sure the issue was not system (and more specifically storage) related.

Share this post


Link to post
Share on other sites
wickedlemon

As soon as you see nothing reaching 100% IOWAIT it's ok.

 

I just wanted to make sure the issue was not system (and more specifically storage) related.

 

It may only be a Pi, but it's not stressing one bit.

Share this post


Link to post
Share on other sites
wickedlemon

Decided just to dump a whole folder in and we'll see how we get on.

 

Scan started: 25th Jan 1630 GMT

Share this post


Link to post
Share on other sites
wickedlemon

Decided just to dump a whole folder in and we'll see how we get on.

 

Scan started: 25th Jan 1630 GMT

 

Still running. 41 episodes added after 24 hours.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...