Optimizing scan times

[woenk 62]

Redid my hypervisor/ fileserver a few months ago and my scan times went up from roughly 10 to 20 minutes (about 200k files).

Pretty sure it is not Emby (no updates for months ) , Ubuntu went up from 20.04 to 22.04 both on the hypervisor and emby VM. 

Does not seem to be the network connection, getting about 9 Gbit/s with iperf and about 150MBytes/s from the NFS shares (WD DC HDDS with 7200 rpm).

Those NFS shares are on BTFRS, fstab and exports were  pasted from the previous  installation ( Xubuntu ), server now is a "clean"  installation wiht just ssh/nfs/kvm/smb .

As stated, it is for me not a problem caused by Emby but rather something I do not see

Also redid my firewall (a Sophos XG VM), which did not help.

VM is runnning with 8 cores and 24 GB Ram, and DB Cache is at 8 GB......absotely no chance its Embys fault

So I am simply searching for those bloody obvious answers google could not give me  

I am thankful for any sugesstion ... or like we germans say " I can not see the the forest because of alll those trees"  .

[Neminem 1518]

Hmmm I have seen vm network drivers behaving bad.

You might look into that.

Also check you dns, ntp.

[woenk 62]

13 hours ago, jaycedk said:

Hmmm I have seen vm network drivers behaving bad.

You might look into that.

Also check you dns, ntp.

Would expect quite some few dropped blocks if it was the network drivers, normally using virtio but just for the lols I also tried E1000, no change there. But maybe I should play around there a bit more.

Because DNS can allways break I access the NFS shares via IP

NTP is done on the firewall and both hypervisor and VM are in sync.

Also tried passing thought the cores directly to the VM which did also not help.

The only thing I noticed that the fan spins up when doing a scan and becomes a bit audible, but the CPU usage is only around 20% and temperature stays at around 50-55 C, its around 40C in idle.

Also tried some NFS caching on the client side, but that did also nothing.

[woenk 62]

Seems like the new scheduled Task Vaccum database did the trick and scan times are down to 9 minutes once more. Did initiate that manually several times for several months and did not help.

Still, 9 minutes when nothing has change seems a bit long...any suggestions what I could further optimize ?

[Spaceboy 2573]

the scan has to check every file to see if anything has changed, 200k files 9minutes seems reasonable to me - nothing you can do to optimise that other than reducing your library size/ file count.

btw i dream of a nine minute library scan, more like 90mins for me

[woenk 62]

On 20/04/2023 at 14:28, Spaceboy said:

the scan has to check every file to see if anything has changed, 200k files 9minutes seems reasonable to me - nothing you can do to optimise that other than reducing your library size/ file count.

btw i dream of a nine minute library scan, more like 90mins for me

well... I got it down even more...with all that "use SSDs" I totally forgot that HDDs may need some defragmentation, even more if one uses BTRFS.Also got some more drive space to spare....who woulf have thought that roughly 15TB can be so fragmented that they use over 70GB more space

Fastest right now after one pass of defragmentation on 2 drives is just shy of going below 6 minutes. Think I will do a few more passes