Another slowscan topic

[ossii 2]

I tried to search a bit on the forum, but I didn't find any answers as of yet.

I'm struggling with a very, very... slow library scan. The scheduled rescan started 06:30AM and its still running now at 1:22PM (@56% now).

 

I'm trying to switch over from Kodi as backend, which handled these rescan a lot faster, full refresh took about 3hours tops.

So must be something that can be tuned here.

 

The reason for the long scans even on kodi is the remote location of the content, setup is as follows:

 

Remote location has the storage with the content, good internet connection.

Local setup:

- Directories mounted up with sshfs (sshfs user@@remote.ip:/mnt/nas /mnt/remote_nas -p 1901 -o ro,allow_other,uid=0,gid=0)

- Mounted directories remounted with rar2fs (this I don't need to do with kodi, so this is the only difference) rar2fs -o allow_other --seek-length=1 /mnt/remote_nas/movies /mnt/rar2fs/movies (same for other things)

- emby-server (Version 3.1.5.0) indexes the content of the rar2fs filesystem

- "Download images in advance" is turned off

 

Content structure: media is in separate directories matching the name

 

Local system:

Debian Stretch

Intel® Core i5-4690K CPU @ 3.50GHz

32GB RAM

emby running on a relatively new WD Red sata disk

Edited February 4, 2017 by ossii

[mastrmind11 722]

Are you talking about the initial Emby library scan, or the sync to Kodi?  The latter can be handled by the Fast Sync plugin.  The former just takes a while on the first go.

[ossii 2]

Are you talking about the initial Emby library scan, or the sync to Kodi?  The latter can be handled by the Fast Sync plugin.  The former just takes a while on the first go.

The initial Emby library scan and the scheduled rescan for new content. The scan I mentioned that started 06:30 and still now running at 13:57 is a "simple" rescan for new content, currently at 57.3%

Edited February 4, 2017 by ossii

[Luke 42077]

Please provide the emby server log. you can learn how to do that here:

 

https://emby.media/community/index.php?/topic/739-how-to-report-a-problem/

 

Thanks !

[ossii 2]

Please provide the emby server log. you can learn how to do that here:

 

https://emby.media/community/index.php?/topic/739-how-to-report-a-problem/

 

Thanks !

Restarted the server now and started a scan right away, with debug logging turned on.

I'll post the log tomorrow when the rescan is done.

[Luke 42077]

Thanks for the update.

[ossii 2]

Thanks for the update.

Here are the logs

Edited February 5, 2017 by ossii

[Luke 42077]

since you've installed the tvmaze provider, please try removing that. thanks.

[ossii 2]

Ok, disabled the downloading of subtitles as well as I saw there was a lot problems with that.

Rescan time: Last ran an hour ago, taking 11 hours 36 minutes and 46 seconds.

 

Will report when this run is done

Edited February 5, 2017 by ossii

[Luke 42077]

There aren't any problems with downloading of subtitles but it can cause the scan to take longer.

[ossii 2]

Last ran 12 minutes ago, taking 10 hours 54 minutes and 58 seconds.

So wen't a bit faster with tvmaze plugin off and downloading of subs off as well, but we're still at 3-4x the time kodi used to do the same job.

 

There is quite a bit of these:

2017-02-06 00:08:07.4501 Error App: Error in The Open Movie Database
	*** Error Report ***
	Version: 3.1.5.0
	Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /mnt/various/emby/server -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
	Operating system: Unix 4.9.0.1
	64-Bit OS: True
	64-Bit Process: True
	Mono: 4.6.2 (Debian 4.6.2.7+dfsg-1)
	Processor count: 4
	Program data path: /mnt/various/emby/server
	Application directory: /usr/lib/emby-server/bin
	MediaBrowser.Model.Net.HttpException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (503) Service Temporarily Unavailable.

Which understandably will make things slower if it's going to get a timeout (and maybe x amount of retries) on 2000+ movies

[Luke 42077]

Yes it can. And you can learn about that on the Omdb website:

 

http://www.omdbapi.com/

[ossii 2]

Yes it can. And you can learn about that on the Omdb website:

 

http://www.omdbapi.com/

Thought the error mentioned on the website there was the answer to all errors in the log, but I do see a lot of 403 (Forbidden) errors as well now.

2017-02-07 14:37:37.8234 Error HttpClient: Error getting response from https://www.omdbapi.com/?i=tt0095327&plot=full&tomatoes=true&r=json
	*** Error Report ***
	Version: 3.2.0.0
	Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
	Operating system: Unix 4.9.0.1
	64-Bit OS: True
	64-Bit Process: True
	Mono: 4.6.2 (Debian 4.6.2.7+dfsg-1)
	Processor count: 8
	Program data path: /var/lib/emby-server
	Application directory: /usr/lib/emby-server/bin
	System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

I see if I grab that URL and paste in my regular browser it works fine, but slow, but works.

[Luke 42077]

@@ossii you just need to be patient and wait it out. When the website is updated to say all systems go, then we can re-evaluate. thanks !

[ossii 2]

@@Luke How does emby determine name of tv shows, episode number and such? Will it base it on the dirname if that gives the needed info?

If dirnames are Some.Show.S01E03.WEBRip.1080p.x264-FANCY it should be relatively quick to at least do the initial pass on creating the library, since you have all info there, showname, season, episode and resolution.

 

-Ossii

[Luke 42077]

From the file name.