100% CPU-Usage while scan lib

[vk_digi 1]

Hello,

 

i think there must be a ugly bug in the lib-scan. As soon as the status bar reaches about 68% the cpu-usages stays at 100% for a long time until stauts is >72% (about). Is it possible the the cross check of files in file-system and the files in metadata is done via a sequential search instead of a binary search? i think there is somewhere such a problem because i can show you values here. 

i have 4 servers running on the same rig. first server as libs for movies and series for client1. the next one has files for client2. there is a third server having the movies and series that same for client1+client2. 

When client 1 as seen a file it is deleted. same for client2. for the third server all files delete in client1 or client2 will also deleted. in the end client3 has only files that both client1 and client2 have not seen yet.

the fourth server has all files that were ever in the libs for the rest of the clients. all files for client1-3 are hard-linked from base-file-system server 4.

 

for all servers the scan is done without chapters, chapter-pics and not without pre-load pics.

 

server for client-1: 464 Series with 9062 episodes and 203 movies. Time for scan is around 2 minutes

server for client-2: 573 Series with 11120 episodes and  1408 movies. Time for scan is around 80 minutes

server for client-3: 338 Series with 5910 episodes and 196 movies. Time for scan is around 1,5 minutes

server nr. 4 for all: 994 Series with 35849 episodes and 1455 movies. Time for scan is 90-120 minutes.

 

just remember the files for client1-3 are only a subset of the files for server nr. 4.

 

i tried to check logs (with debug on) to see if there is a hint which files are taking such a long time to scan. i was not able to see it.

All files have primary-pics and so on. as far as i can see there is no reason to download new files (pictures etc) from anywhere.

 

The only problem i see is that for each server one cpu (rig as 4/8) is used 100% from scan-progress  68% until it reaches some values >72% 

 

Any hint what i can do to speed this up and lower the cpu usage? 

 

While the scan is running there is very big lag to show pages within emby and trying to use the search function needs many seconds until the list is shown and more time until the search is done.

 

sorry for bad english. i am no native english speaker.

 

bye

 

ps: please do NOT comment about the 4 servers running: emby-premiere shows all green and all within allowed limits!

pps: the reason for doing it this way is: client-1 my wife is using kodi when watching alone. i am client-2 and using kodi to watch "alone". kodi is asking for delete files when watched. we are doing this. when my wife AND i want to watch togehter we want to emby/kodi only to show us the movies/series/episodes that we BOTH have not seen already. when we have watch a file togehter kodi is deleting this file and scripts in the background also delete this files in file-base for my wife (client 1) and my own (client-2). That way we have a sync over 2 users watching often alone and from time to time together without sitting in front of TV and asking "have you seen this....or that...." and after 30 minutes or so find some file that both of us are at same position and watched state. 

[PenkethBoy 2068]

the 68% thing comes up a lot - it when emby does its "thinking" after determining whats changed since the last scan

 

as you only assigned a single cpu to each server - this is why it appears at 100% - if you give it more cores you will see the interface become more responsive - wont speed up the scan much as the "thinking" time from what i see is a single thread process

 

Are you following the wiki on how to structure your movies and series. As this can affect the scan times a lot.

[vk_digi 1]

Hello Again,

 

i am not aware of limiting count of CPU to emby. Where can i set this to use i.e. 2 virt. Cores? About the structure of the lib: It is exact as shown in wiki and done with "auto organize"

 

i.e: 

 

Serie-A

  01

       EPI_S01E01

       EPI_S01E02

  02

       EPI_S02E01

       EPI_S02E02

Serie-B

...... and so on

 

 

 

 

Thanks for your answer.

 

bye

[Luke 42077]

@@vk_digi, it won't affect the library scan. It's only for playback transcoding. If you can attach the emby server log we can offer some advice. Thanks.

[PenkethBoy 2068]

if i understand correctly - you have your servers in VM's - if so them in your VM management software you define how many cores the VM uses

[vk_digi 1]

Hello,

 

those 4 servers are running "native" on the linux-rig. they use different ports and different path for data and so on. 

 

i restarted the server for client-2. 573 Series with 11120 episodes and  1408 movies. started the "scan media lib" in task section. the scan needed 66 Minutes this time. after that i  started the server once again. i attached here the logfile for the part off scanning.

 

i can see some exceptions before i started the scan. i think those are the gone session for the 3 connected clients after the restart. 2 time kodi (fire tv stick and a fire tv) and the admin session it self via chrome.

 

for the scan itself i see only the start message and after 66 min the stop messages. this is the reason for not understanding what is going on here.

 

and as before: from 67,8% until the end one cpu at 100% nearly all the time.

 

greetz

volker

Log.txt

Edited March 10, 2018 by vk_digi

[Luke 42077]

Is this a new emby server installation?

[vk_digi 1]

Hello,

 

new? do not understand what "new" means in this context. This server was fresh installed around nov. 2017. Since then there are only normal updates done that came with ubuntu-repo. 

 

was this the question? if not, please be more verbal what i need to tell.

 

the server for client-1 and client-3 (see first posting) are setup same time. those need only 1-2 minutes for a scan. the server nr- 4 with all files needs 90-120 minutes as written. this one is setup around 05/2017 or so.

 

 

bye volker

[Luke 42077]

Ok thanks, that's fine. I would just be patient and allow it to finish. The % indicator is based on number of files, and certain kinds of media are faster than others so that's why it jumps out quick and then slows down.

[vk_digi 1]

Hello,

 

i think you got me wrong. The server is up to date. The scan runs every 12 hours. even if there is no new file after last scan, the scan takes 65-75 minutes. and in that time cpu is 100%. 

 

bye volker

[Luke 42077]

Well, the odd thing is that there appears to be very little activity logged during the scan which suggests that it's just taking that long to deep scan your file system and compare it to the database. What you could try to do is remove the kodi sync plugin and the statistics plugin and see if that helps.

[vk_digi 1]

hello,

 

i remove for a test both of them in the past. no difference at all. please can you take a deep think about what i have written about the problem with time raising expotentiell and not liniar?

 

what i mean: if you have a database with 1000 enties and a file-system with this 1000 files (all up to date) and you search for every single file in database against all the 100 files in filesystem (i.e. a table with the vaules for sure) you will do 1000 * 1000 compares. result is 1.000.000

if you have a database with 10.000 files and file-system with 10.000 files you will do a search 10.000 times against 10.000 files with result 100.000.000 compares.

This is not 10 times as much coming from 1000 enties to 10.000 entires this is factor 100. 

 

For sure you know what i mean. as far as i have seen in a quick check of the source i think you do the search and compare recursiv for each lib->dir->and all the childs against the values in the file-system.

 

this will result in n**2 time and not n*x time needed. correct me if i think it is not correct what i try to tell here.

 

but 1,5 hours to update the database (scan media libs) without the need of any chance is not that fast  i would think.

 

 

right now i set up a fresh beta server and feed this one with the files step for step and let the system update all. we will see maybe what the show-stopper is here.

 

it is not that easy to install a beta along 4 stable server on same rig. need to be careful

 

greetz

volker

[Luke 42077]

Ok, let me know how the beta compares. It's not because there are actually any changes in the beta, but it's running on better technology (.net runtime as opposed to mono runtime). Thanks.

[vk_digi 1]

Hello,

 

so tests are done. first i setup only the 1400 movies and did a scan with the beta. this takes nearly 3 hours. downloaded many pics and info. okay. 

than after first scan was complete i did again a scan. this one needs about 60 minutes. there were NO series in the lib only the movies. so no better chance with beta.

 

after that i rebuild my file-system this way.

 

before:

 

archiv

  movies

       movie-1.mkv

       movie-2.mkv

......and this way up to movie-1408

 

afterwards:

 

archiv

     movies

         movie-1

               movie-1.mkv

         movie-2

               movie-2.mkv

      ........

 

        movie-1408

               movie-1408.mkv

 

start new scan. this needs about 20 minutes. okay all files were moved to new location

 

 

and NOW::::: started again a scan. 

 

duration: 30 seconds

 

what the fu..... restarted server. start scan again: duration 30 seconds

 

maybe you can find the problem with this info. btw while the fast scan status 67,8 until >70 is also 100% cpu but only for some seconds.

 

and just right now i did a scan again: this time 12 seconds.

 

bye volker

[Luke 42077]

Thanks for the info.

[vk_digi 1]

Hello,

 

i add now over night (here is germany 23:35) the series for this testserver. i think this will run the whole night.

 

tomorrow i can tell about the total time compare movies+series.

 

One for me very important question: for server client1-3 i need no "watched status" because all watched files are deleted. BUT for the server nr.4 for all other clients i need this status. if i move the movies as shown above into each movie in it's own directory, than as far as i can see this the status about watched for each client is lost because of now file-id / position.

 

is there any chance to save the status? maybe with a script running against the database with a sqlite-client? the filenames for all movies will remain the same but they are moved to new dir. Any chance?

 

greetz volker

[vk_digi 1]

Hello,

 

after adding now all series and movies for this server the scan needs < 1Minute (45-55 seconds) instead of 60-75 Minutes. I think this not that bad.

My problem from above: What can i do when reorg the file-base as needed for movies WITHOUT losing the Watched-State for the movies per user-base.

 

I did a quick check with sqlite and it seems not simple to get along the diffent item-ids, user-ids and the state. Job is not easier when in the sqlite data base only a blob with json data is found.

 

so let me ask for a different way: Will you be willing to correct this ugly bug with time for scan needed movies is factor > 50 in near future? If so it would be the lazy way for me to let the files as they are.

 

bye volker

[ebr 16169]

My problem from above: What can i do when reorg the file-base as needed for movies WITHOUT losing the Watched-State for the movies per user-base.

 

As long as the movies are properly identified both before and after the move, you will not lose watched status.

[vk_digi 1]

Hello,

 

 

As long as the movies are properly identified both before and after the move, you will not lose watched status.

 

i did a quick test. Move some of the movies after set them to watched. rescanned the lib and check the status again. The status is lost. Bad News.

[ebr 16169]

Hello,

 

 

 

i did a quick test. Move some of the movies after set them to watched. rescanned the lib and check the status again. The status is lost. Bad News.

 

Did the items have tmdb and or imdb IDs in their metadata both before and after?  If so, which ones?  Are they the same?

[vk_digi 1]

Hello,

 

Did the items have tmdb and or imdb IDs in their metadata both before and after?  If so, which ones?  Are they the same?

 

Here are Screenshots before the move and rescan:

 

 

 

And here are screenshot AFTER the move an rescan. 

 

 

 

Funny thing: Moving the Files back again to the place they were while setting "watched" and doing rescan those files have the watched-flag again. But not at the moved file-location.

[ebr 16169]

And you are positive those IDs existed at the time the item was marked watched...?

 

Can you test with a new item you mark watched for the first time?

[vk_digi 1]

And you are positive those IDs existed at the time the item was marked watched...?

 

Can you test with a new item you mark watched for the first time?

 

Hello,

 

sorry yes i am sure. did you read that i wrote "move them back again they got watched again"?

[vk_digi 1]

Hello,

 

i tested this again. After a server restart and a new rescan it works. i do not understand this, but now i will move the movies in subdirs per movie and cross fingers that the watch status will work as in my tests.

 

bye

[Luke 42077]

Thanks, we'll take a look at this.