reduce ffprobe cpu usage

[chart100 1]

I have massive music library, when first adding it ffprobe sit at around 30%-50% CPU for more than 48 hours 

What settings can I turn off to stop or reduce ffprobe from scanning all the files? (e.g image downloaders? etc)

[Luke 42083]

Hi, did you enable the video preview thumbnail feature? That will have the biggest impact on the library scan.

[FrostByte 5392]

If adding multiple libraries add them one at a time, especially the huge music one.  I don't think turning music metadata off temporarily will help much with ffprobe.  You could temporarily move some of the music in a temp folder and add parts of it back to the actual library folder a little at a time.  Then scheduling scans at night may help cpu load during your normal use times.  Turning thumbnail creation off initially will help with movie, etc libraries.

[chart100 1]

thanks for the tips @Luke and @FrostByte

I think some of the the issue was having my music held on a separate network share and having Realtime scanning enabled. I have moved it to a more powerful server with the disk attached directly.

I'll let it run and see if it resolves the problem.

[Luke 42083]

If you still feel there's an issue then please attach the emby server log from the time of the scan. Thanks.

[chart100 1]

is it possible to select taglib https://taglib.org/ as the active scanner instead of ffprobe as I've run a taglib scan for something else and it barely uses any cpu. or is that perhaps a feature request I can ask for?

[Luke 42083]

We do use something similar to that for photos.

[Luke 42083]

22 hours ago, Luke said:

If you still feel there's an issue then please attach the emby server log from the time of the scan. Thanks.

This is whati would suggest.

[softworkz 5073]

13 hours ago, chart100 said:

is it possible to select taglib https://taglib.org/ as the active scanner instead of ffprobe as I've run a taglib scan for something else and it barely uses any cpu. or is that perhaps a feature request I can ask for?

We can't do this for audio or video files because actual playback will often (unless direct-play) go through ffmpeg, and for that, we need to know how ffmpeg/ffprobe "sees" a media file. Also we need precise and accurate information about media files.

On 4/18/2022 at 10:33 AM, chart100 said:

can I turn off to stop or reduce ffprobe from scanning all the files?

There seems to be a lot of misconception 

How did you come to the idea that ffprobe would be some kind of evil that would just consume CPU cycles for doing unnecessary stuff?

TagLib can handle 12 kinds of audio files and all it can do for those is "reading tags".
ffprobe/ffmpeg supports decoding of about 200 audio codecs and tag reading is just a tiny part of its capabilities. 

Tags can be wrong, and tags are often wrong.
Tags can be present, but don't need to be present.
Tags provide additional information, but it's not the elementary information that is required to properly identify, handle and play an audio file.
ffprobe can do that. TagLib can't. What TagLib can do is probably < 1% of what ffprobe can do - just to make the relation clear between these two.

Another misconception

Image extraction is not done through ffprobe. It's done with ffmpeg. 

Performance

On my local machine, ffprobe (with the params that Emby is using and when run in the context of Emby) takes between 20 and 30ms on a single audio file and when processing a large amount of audio files, the total from file-to-file is no more than 35ms (with all metadata retrieval disabled).

This means: 

• 28 per second
• 1680 per minute
• 100k per hour
• almost 5 Million within 48h

@chart100

I have a few questions, because you mentioned high CPU load of ffprobe, but I'm wondering how this could have happened. If it was about slow I/O because of your library being located on a network location, this would surely slow down the library scanning speed massively. But it wouldn't increase the CPU load of ffprobe. And hence I'm wondering whether something else might be going wrong at your side...

• Are you on Windows or Linux?
• How did you integrate your library - where was it located? (which kind of network device)
• How did you watch/determine the CPU load of ffprobe?
• Was it a persistent process (PID)?
  (or frequently changing process ids?)

 

Edited April 20, 2022 by softworkz