problems scanning massive library

[mayk 1]

Hello group,

 

I've been digging around but can't find anything of use so far.

 

The intended use of emby for me is our personal pictures and vids.  Now comes the catch, it's massive.  I'm trying to use a mixed content library to scan it , but also tried a pictures only and a home vids only.

It just doenst float.. so far..  it populates some folders, but only about 1 % orso.

 

My structure ( if you can call it that ) is :

 

pictures - year - name/subject

 

sample :

 

pictures

  - 2000

             - 2000 02 Holliday in winter

             - 2000 06 holliday in summer

  - 2001 - 2002 05 baby born

  - 2001 - 2002 06 hollday

 

So it's year with event inside.

 

It contains pictures in jpg format , and vids in  mov and avi.  The total size is around 500 GIG ( 350 GIG pictures ) , counting around 75.000 pictures. ..

 

I'm in the process of moving the library, so it's not even complete on the destination being indexed by emby, but it breaks..

 

the source is another server, and i experimented with cifs mount and NFS mount wich made no difference , and it's gigabit uplinked. ( NFS dedicated GB link between systems) . The emby system is identical to the serving system.. Emby is the only software running active on this server, quad core atom 1.6 with 4 GB ram and ssd .

 

Any tip, suggestions for this..  not in it for a freebee , willing to buy lifetime but only want to buy a product that works in my environment.. :-)

 

Also tried plex ( sorry to cursu ) it indexes about 90% of the folders , but also not all but lot's more then emby...  Also tried different users for emby to run.. no change.... :-(

 

thanx  !!!

Edited December 11, 2015 by mayk

[Luke 42077]

So you think content is missing due to library size? photos are not recognized at all in mixed libraries - they are really just mixed movies and tv. a photo library supports both photos and videos, which sounds closer to what you want.

[jonours 3]

I have an ever larger collection of pictures and home videos and have not ran into any issue.  I have 131,138 photos & 1,514 home videos which take up just over 500 GB.  My set up is similar to yours with the exception that I separate my picture and home video library’s but they are stored the same as yours, Year\sub folder.   One difference that I may have is that I add each year as its own folder to the library but I am not sure if that makes any difference. Each subfolder shows as an album in the viewers.  I do not remember how long the initial scan took but it did take a while.  You could try adding a year at a time to see if that completes.    

 

 

[mayk 1]

Thanx for the info.

 

The fact that a content photo can also contain video is cool, and suits my need indeed.. :-) thanx..

 

For the original problem , perhaps i don't onderstand the inner workings.  To my presumption ( dumb ass brain and comparing with for example plex) , when i hit the scan library button i see it running.. xx % completed. It finishes rather quickly compared to, again plex.  My assumption was hey, they have a different more efficient way to index stuff. Cool..  It would finish for the entire bulk after about 15 - 20 minutes, and i would suspect it would run for a few hours. 

 

Now to the question... When the scan completes, and i look at the processes running, i see a lot of mono-sgen optimize=all threads .. The optimize part is if i recall for thumb generation..  Could it be that ( to help me understand) the scan for example generates a list of what needs to be processed, and in the background emby-server will eat trough the list of things ?

 

Running it on a smaller set of data ( one year) seems to work, still wating on some folders to have visable content after about 45 minutes but hey, if this is the way it works , and it works , then let it run for a few days....

 

Hoping to understand better..

 

Thanx

 

greets and a great weekend,

 

Mayk

Edited December 12, 2015 by mayk

[Luke 42077]

i'm not exactly sure what you mean but thumbnails for client apps are produced on demand. the scan is probing media info.

[mayk 1]

Hi,

 

Well, when the media scan finishes, there are a lot of mono-sgen processes running with a --optimize=all parameter. Since the scan is complete, i see the folder(s) and subfolder(s) .. but not pictures beside a few..  The log isn't generating any rules at that moment ( also none when running media scan ) .

So my gues was that i needed to wait and let the mono-sgen processed finish.. but i'm starting to believe i'm wrong in that assumption.  They are still running , but i'm still not getting any content...

I added a second library, containing a few small vids and a couple of pics (total about 100 mb) .. scan finished, and 15 minutes later i'm still missing content..  I can't grasp what's going on, or what it's doing ...  also..sometimes all cores are maxed out on mono-sgen while i'm not doing anything ...

 

i'm puzzled...

[mayk 1]

Test it now with a local disk.. Transfered all the data to a newly attached disk on the emby server itself and it's scanning now. 

 

In comparison to earlier , when running a scan trough a nfs mount of a cifs mount, this time the scan takes much more time, what makes me believe it will be better. 

 

Also, i found a log (besides the /var/log/embyserver.log) that contains more info ( /va/rlib/emby-server/logs) . Interesting to follow with debug enabled.  It's hitting a few errors creating xml files...  Error App: Error parsing xml file /mnt/hdd/2013/December/2013 12 - Iphone patricia/2013-12-28 15.23.55.xml

I've checked and for the testrun i've set a recursive 777 over all the content.

 

Will post a follow up once the scan finishes..

 

thanx

[mayk 1]

back again...  i'm starting to thing emby is not for me...  with all the content on a local disk ... still the same problems.  If someone has a good suggestion please..  For some reason plex does work, over cifs , nfs and now checking with local disk ( not that it matters but to see speed differences) , but a like emby , it's easy , the mobile app looks much better ( is slower then plex ) ..

 

time for a beer....