Library Scan 20 mins Approx Version 3.2.36.0

[joelang6126 9]

My library scan always takes about 20 mins on average to scan hoovering the 67%-69% area before completion.

It always seems to take this long no matter which version of Emby I have used in the past.

 

Is there a known reason for this on Linux based devices?

 

If I attached a log would it be evident why it always hangs for a bit?

[PenkethBoy 2068]

Depends - i would guess as you have not mentioned any other issues that it is normal without any info on your setup

 

What model QNAP?

Which version Emby server?

How many Movies and how many tv episodes do you have?

 

Most QNAP have low power CPU's so that could be your limiting factor

 

I notice a similar point where the scan slows down - But - that is when emby gets to the TV library and as it has a large number of episodes it takes longer to scan and the % does not increase quickly

Edited December 20, 2017 by PenkethBoy

[joelang6126 9]

TS-EC1080 Xeon E3-1246 3.50 Ghz

 

Emby Version 3.2.36.0

 

Meta is stored in NFOs beside the media.

 

Is there a statistics part of Emby where I can see the number of Films/Epissodes? I do have alot!

 

I know there are 1511 Films.

Edited December 20, 2017 by joelang6126

[PenkethBoy 2068]

the Statistic plugin will help with numbers or the report plugin

 

as you have a xeon then power should not be a problem

 

I assume the media is on the NAS?

How do you have your disks configured?

What disks are they?

Are you using any external disks?

Do you have a read cache?

what version of the qnap firmware are you on?

 

My 853A does a library scan of ~270 movies and ~3400 episodes in approx 2 minutes and it only a celeron cpu - this is a scan with no new or changed media

if i adds some new movies etc it can go up to 15mins or so

My media is across an 8x Raid5 

Edited December 20, 2017 by PenkethBoy

[joelang6126 9]

The media is on the NAS

9 x WD60EFRX 6Tb Reds Raid 6

No externals

FW 4.3.3.0378

 

No read cache current in operation but I do have the module a 256Gb mSATA installed for future use.

[PenkethBoy 2068]

Ok sounds good

 

Run a scan - let it finish

then run it again straight after - does the time reduce?

 

If you search your logs do you see any errors reported - during the time of the scan(s)?

[joelang6126 9]

Scan took 33 mins

 

Scanning immediately after gets to 69.3% and hangs again very slow increasing. Ends at 21mins 17 seconds.

 

The cpu usage never gets over 4% while scanning, isnt there a way on increasing this?

[PenkethBoy 2068]

The amount of cpu usage cannot be increased but the speed of your disks/array is more important

Any errors in the logs while the scan is running?

 

Are you seeing any errors in the interface?

 

Have you checked for any errors on the NAS itself in the system log?

have you run a scrub on the array recently?

IIRC the QNAP help desk tool allows you to test the speed of your disk/array

 

the 69% thing is not an error its doing your TV series as there are probably thousands of episodes (based on your movie count) thats why it appears to slow down

 

In your movie and TV library setup - turn off metadata download and chapter extraction - then run the scan again as a test - if its significantly quicker it might be a network issue/provider issue - small chance but might be it

 

if there are no errors in your emby log and your disks/array is healthy then i suspect its the amount of media files you have thats dictating the time

 

20 minute is not that long - why do you want it to be faster?

 

Might be worth upgrading to 50.0 after you have done the checks - so things are consistent - i doubt the upgrade will affect the scan times

Edited December 20, 2017 by PenkethBoy

[joelang6126 9]

I cant see anything obvious in the logs and nothing wrong on the NAs side of things.

 

Ill run a scrub to see if that helps, I suspect it is the amount of files I have causing the slow scans.

 

I seen reports of other uses quoting much smaller scan times so I was wondering if something

was wrong.

 

Thanks for your help.

[Luke 42077]

Yes I would suggest updating to the latest version, thanks.

[joelang6126 9]

Upgraded to Version 3.2.60.2 beta

 

No change in scanning times.

[Luke 42077]

Can you please attach an emby server log? thanks.

[joelang6126 9]

Please find attached.

Log.txt

[Luke 42077]

Hi there, to improve the performance of your library scan, consider the following:

 

Uninstall the following plugins:

• Emby.Kodi Sync Queue 2.0.0.0
• Messages 1.0.0.0
• Statistics

Then shutdown all kodi apps to prevent them from trying to sync, because it appears to me that one or more them is trying to sync and this is placing a high load on the server. This is causing the library scan to take longer, and therefore you end up blaming it on the server release as the source of the problem.

 

Have you tried the new EmbyCon addon? 

 

https://emby.media/community/index.php?/topic/46651-embycon-kodi-addon/

 

No syncing required, which means no additional load placed on your server. It's just browse and play, very simple. Please let us know if this helps. Thanks.

[Quiptix 2]

I have a 569 Pro and it always takes around 20 minutes. From the 3 plugins mentioned I only have Statistics installed.

 

Just to add my observations...

 

I noticed that Emby tries to download subtitles while scanning the library. As it is also a seperate task this doesn't seem logic for existing media.

For new media it makes sense to download subtitles.

 

Make this configurable I suppose. (While make a feature suggestion)

Edited December 29, 2017 by Quiptix

[joelang6126 9]

Hi there, to improve the performance of your library scan, consider the following:

 

Uninstall the following plugins:

• Emby.Kodi Sync Queue 2.0.0.0
• Messages 1.0.0.0
• Statistics

Then shutdown all kodi apps to prevent them from trying to sync, because it appears to me that one or more them is trying to sync and this is placing a high load on the server. This is causing the library scan to take longer, and therefore you end up blaming it on the server release as the source of the problem.

 

Have you tried the new EmbyCon addon? 

 

https://emby.media/community/index.php?/topic/46651-embycon-kodi-addon/

 

No syncing required, which means no additional load placed on your server. It's just browse and play, very simple. Please let us know if this helps. Thanks.

 

Luke I have removed all three plugins, Emby still takes around 20-23 mins to scan. I noticed post elsewhere about Linux based Emby installations taking the same amount of time.

 

I will be moving to a Windows based Emby installation in the next few weeks.

[PenkethBoy 2068]

well not all linux scans take 20+ minutes

 

on my qnap with its lowly celeron takes 79 seconds

the same library (copy) on my 2012r2 server with an i7 takes 28 seconds

 

not too shabby for the QNAP which does have faster disks

 

even if it did take 20 minutes - not sure i would be that bothered

 

each to their own

[joelang6126 9]

How big is your library?

 

I have 1582 films and 19480 episodes.

 

Also PenkethBoy, you made two posts at the start of 2017 stating that your scan time was 20 mins and then 5 mins....what's changed now thats its down to 79 seconds?

Edited December 30, 2017 by joelang6126

[Luke 42077]

well not all linux scans take 20+ minutes

 

on my qnap with its lowly celeron takes 79 seconds

the same library (copy) on my 2012r2 server with an i7 takes 28 seconds

 

not too shabby for the QNAP which does have faster disks

 

even if it did take 20 minutes - not sure i would be that bothered

 

each to their own

 

thanks @@PenkethBoy !

[PenkethBoy 2068]

How big is your library?

 

I have 1582 films and 19480 episodes.

 

Also PenkethBoy, you made two posts at the start of 2017 stating that your scan time was 20 mins and then 5 mins....what's changed now thats its down to 79 seconds?

Hi

 

I have not looked up those posts but i suspect those times were from my old 459 Pro II (Atom cpu) with the mono package (IIRC) as i have only recently in the last couple of months put Emby on my 853A when the new .netcore came out. Also Luke has made improvements in the speed of scans and how they work since the beginning of the year and he has made some more very recently but they are not available in 60.2 as we need a new version update for qnap.

 

so more powerful cpu and 8x raid 5 rather than 4x raid 5 and also the 853A has HGST 7200 rpm drives which are significantly faster than the WD Red ones i had in the 459.

 

As for size of media library - mine is approx 18% of yours - so a linear extrapolation would give you about 7m+ if the speed was the same?

 

I have download metadata and chapter extraction on - so these will slow down a scan as an when it finds something new/changed

 

The 79 seconds is consistent as a scan ran over night taking the same time this is with disks configured to go into standby as well - which they do.

 

How much memory do you have in your QNAP?

[joelang6126 9]

Hi

 

I have not looked up those posts but i suspect those times were from my old 459 Pro II (Atom cpu) with the mono package (IIRC) as i have only recently in the last couple of months put Emby on my 853A when the new .netcore came out. Also Luke has made improvements in the speed of scans and how they work since the beginning of the year and he has made some more very recently but they are not available in 60.2 as we need a new version update for qnap.

 

so more powerful cpu and 8x raid 5 rather than 4x raid 5 and also the 853A has HGST 7200 rpm drives which are significantly faster than the WD Red ones i had in the 459.

 

As for size of media library - mine is approx 18% of yours - so a linear extrapolation would give you about 7m+ if the speed was the same?

 

I have download metadata and chapter extraction on - so these will slow down a scan as an when it finds something new/changed

 

The 79 seconds is consistent as a scan ran over night taking the same time this is with disks configured to go into standby as well - which they do.

 

How much memory do you have in your QNAP?

32Gb installed.

 

I think Luke is getting the wrong impression here, i'm not blaming any release as its always been around this time. I would just like to see if it can be improved and maybe give the Emby guys some feedback.

 

Emmby will be moving to my new dual xeon build in a few weeks anyway.

[PenkethBoy 2068]

I'm curious why yours would be so slow - your Xeon and memory are very similar to my 2012r2 server with 4790s i7 in passmark scores so horsepower is very unlikely to be the cause.

 

Did you run that scrub?

 

do you have a static/thick/thin volume?

 

mines Static as that gives the best performance although its not a night and day improvement

[PenkethBoy 2068]

FWIW - these are my disk speeds using the QNAP diagnostic tool

 

[joelang6126 9]

Yeah I did run the scrub at set it for weekly. No improvement. I can't remember how it's provisioned I'll check when I get back home. Almost sure it's thick.

[joelang6126 9]

It must be the size of the library that's the issue.