Library scan times

[kjp4756 41]

I'm running emby server 3.0.5781.5 in a standard jail on freenas 9.3.1.  I have 1670 media files (tv and movies only) in 3 libraries; movies, wife's movies, tv.  My media dataset is on 2 3TB WD Red drives in a zfs mirror.  A library scan takes 11 minutes to complete no matter if things changed or not.

 

I'm seeing people post who have much larger libraries than I where their scans take under a minute.  I'm wondering if this is a problem with my system or it's just a freebsd issue in general.

 

Can some people who run the freebsd ports version or the freenas plugin post how large their total library is and how long a scan takes?

 

I have posted about this issue before and posted logs but never heard anything more after posting logs.  I'm basically wondering if this is a common freebsd thing.  If so, I can accept that.

 

Thanks!

 

EDIT: Attached is a log.  I was wrong, it's 18 minutes for a scan.  It looks like it might be the TV shows causing the delay.  It appears to be refreshing every TV episode for some reason.  I'm not sure if that's by design or not.

server-63585536633.txt

Edited December 12, 2015 by kjp4756

[pclausen 41]

Mine doesn't do that.  Here are my scan times:

 

 

I have 2,351 movies consuming 33TB of space and 9,253 TV episodes consuming 19 TB.

 

I'm actually quite amazed at how quickly Emby running as a plugin under FreeNAS can scan my library.

 

I have 9 movie media folders and a single TV media folder with 165 TV Series parent folders within it.

Edited December 12, 2015 by pclausen

[Luke 37090]

I think the speed at which it can access the storage is more important than the operating system of the server

[kjp4756 41]

Mine doesn't do that.  Here are my scan times:

 

 

I have 2,351 movies consuming 33TB of space and 9,253 TV episodes consuming 19 TB.

 

I'm actually quite amazed at how quickly Emby running as a plugin under FreeNAS can scan my library.

 

I have 9 movie media folders and a single TV media folder with 165 TV Series parent folders within it.

Heh well not exactly a fair comparison with your system against mine.  I've got an e3 v2 with 16gb ddr3.

 

Are yo using NFO files?  I'm not and I wonder if that could be part of the slowness.

 

I just set up the emby freenas plugin.  I'm currently doing the initial scan on that.  I'll see how long a scan takes after that.  There may be something weird with my ports install.

 

UPDATE: Library scans are still slow with the clean install of the emby plugin with default settings.  I guess that rules out any sort of configuration issue or database issue.  It must just be my system then.  I can saturate my gigabit connection writing to my server so I'd think it would be fast enough storage.

Edited December 13, 2015 by kjp4756

[pclausen 41]

I do use NFO files.  I pretty much have all the bells & whistles turned on for metadata:

 

 

 

 

[kjp4756 41]

I'm not sure what's going on with mine.  I don't use chapter image extraction.  I turned off fanart.tv.  It's still slow.  During a library scan I ran iostat and there is hardly any disk activity.  It's not a storage issue.  CPU usage is also under 5% during a scan.

 

Guess I'll have to just live with it.

 

This is while a scan is running and my wife is watching a show on the kodi box (capable of ~150MB/s average)

       tty            ada0             ada1             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0    61 20.94  72  1.47  20.43  74  1.48  22  0  0  0 77
   0   183  0.00   0  0.00   4.00   1  0.00  26  0  0  0 74
   0    61  0.00   0  0.00   4.00   1  0.00  25  0  0  0 75
   0    61  4.00   1  0.00   0.00   0  0.00  25  0  0  0 75
   0    61 35.00  12  0.41  34.77  13  0.44  26  0  0  0 74
   0    61 18.17  59  1.05  17.93  58  1.01  25  0  0  0 75
   0    61  0.00   0  0.00   0.00   0  0.00  25  0  0  0 74
   0    61  0.00   0  0.00   4.00   3  0.01  24  0  0  0 76
   0    61  0.00   0  0.00   4.00   1  0.00  25  0  1  0 75
   0    61 34.51  35  1.18  33.44  36  1.17  25  0  1  0 74
   0    61 12.74  38  0.47  12.30  40  0.48  24  0  0  0 75
   0    61  3.17  24  0.07   3.20  25  0.08  25  0  1  0 74
   0    61  4.00   1  0.00   4.00   1  0.00  25  0  0  0 75
   0    61  4.00   1  0.00   4.00   1  0.00  25  0  0  0 75
   0    61 22.43  74  1.62  22.68  73  1.62  25  0  0  0 75
   0    61  2.67   6  0.02   2.86   7  0.02  24  0  1  0 75
   0    61  0.00   0  0.00   0.00   0  0.00  25  0  1  0 74
   0    61  4.00   1  0.00   0.00   0  0.00  25  0  0  0 75
   0    61  4.00   1  0.00   0.00   0  0.00  25  0  1  0 74

Edited December 13, 2015 by kjp4756

[kjp4756 41]

I think I'm making some progress here.  There definitely is a problem some where with my movies.

 

I created a new instance of the freenas emby plugin.  Added only my movies.  Did the initial scan which took a while.  After that I started another scan.  The scan reaches about 60% quickly then it slows down.  I decided to fire up truss to see what was happening for system calls.  It appears ffprobe is being called then I get a TON of bad file descriptor errors.  It does this the whole time the scan appears to stall.

 

 

 

24761: access("/usr/pbi/emby-amd64/bin/ffprobe",4) = 0 (0x0)
24761: access("/usr/pbi/emby-amd64/bin/ffprobe",1) = 0 (0x0)
24761: stat("/usr/pbi/emby-amd64/bin/ffprobe",{ mode=-rwxr-xr-x ,inode=56554,size=143088,blksize=131072 }) = 0 (0x0)
24761: open("/usr/pbi/emby-amd64/bin/ffprobe",O_RDONLY,00) = 62 (0x3e)
24761: lseek(62,0x0,SEEK_END)    = 143088 (0x22ef0)
24761: lseek(62,0x0,SEEK_SET)    = 0 (0x0)
24761: read(62,"\^?E",2)    = 2 (0x2)
24761: close(62)     = 0 (0x0)
24761: access("/usr/pbi/emby-amd64/bin/ffprobe",1) = 0 (0x0)
24761: stat("/usr/pbi/emby-amd64/bin/ffprobe",{ mode=-rwxr-xr-x ,inode=56554,size=143088,blksize=131072 }) = 0 (0x0)
24761: pipe(0x7fffff7b45a0)    = 0 (0x0)
24761: sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGEMT|SIGFPE|SIGKILL|SIGBUS|SIGSEGV|SIGSYS|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0)
24761: fork()      = 26395 (0x671b)
24761: nanosleep({0.010000000 })   = 0 (0x0)
24761: sigprocmask(SIG_SETMASK,0x0,0x0)   = 0 (0x0)
24761: ioctl(42,FIONREAD,0xff537934)   = 0 (0x0)
24761: ioctl(43,FIONREAD,0xff537934)   = 0 (0x0)
24761: write(63,">",1)     = 1 (0x1)
24761: close(62)     = 0 (0x0)
24761: close(63)     = 0 (0x0)
24761: close(55)     = 0 (0x0)
24761: close(59)     = 0 (0x0)
24761: close(61)     = 0 (0x0)
24761: _umtx_op(0x93c0fc,0x13,0x0,0x0,0x7fffff135d90,0x0) = 0 (0x0)
24761: _umtx_op(0x93c0fc,0x14,0x0,0x0,0x0,0x93c0fc) = 0 (0x0)
24761: kevent(19,{0x3c,EVFILT_READ,EV_ADD|EV_ENABLE|EV_ONESHOT,0,0x0,0x0},1,0x0,0,0x0) = 0 (0x0)
26395: read(62,">",1)     = 1 (0x1)
26395: close(62)     = 0 (0x0)
26395: close(63)     = 0 (0x0)
26395: dup2(0x37,0x0,0x1,0x0,0x0,0x0)   = 0 (0x0)
26395: dup2(0x3b,0x1,0x1,0x0,0x0,0x0)   = 1 (0x1)
26395: dup2(0x3d,0x2,0x1,0x0,0x0,0x0)   = 2 (0x2)
26395: getrlimit(RLIMIT_NOFILE,{ cur=469782,max=469782 }) = 0 (0x0)
26395: close(469781)     ERR#9 'Bad file descriptor'
26395: close(469780)     ERR#9 'Bad file descriptor'
26395: close(469779)     ERR#9 'Bad file descriptor'
26395: close(469778)     ERR#9 'Bad file descriptor'
26395: close(469777)     ERR#9 'Bad file descriptor'
26395: close(469776)     ERR#9 'Bad file descriptor'
26395: close(469775)     ERR#9 'Bad file descriptor'
26395: close(469774)     ERR#9 'Bad file descriptor'
26395: close(469773)     ERR#9 'Bad file descriptor'
26395: close(469772)     ERR#9 'Bad file descriptor'
26395: close(469771)     ERR#9 'Bad file descriptor'
26395: close(469770)     ERR#9 'Bad file descriptor'
26395: close(469769)     ERR#9 'Bad file descriptor'
26395: close(469768)     ERR#9 'Bad file descriptor'
26395: close(469767)     ERR#9 'Bad file descriptor'

[kjp4756 41]

@"pclausen"

 

What does your top output look like when a scan is running?  Here is mine.  I enabled threads view with 'H'.  One thread maxed out.  That doesn't look right to me.