Library scan when files are missing // FFprobe using a lot of bandwidth scanning reintroduced files

[TonioRoffo 4]

Hello,

 

I have a question about the library scan.

 

i'm a bit surprised that a library scan takes a long time if part of the library (for example, a missing disk) is gone.  It looks like a lot of data about the files was removed from the library - but not everything, as the seen/unseen data is still correct.

 

Does Emby immediatly remove all metadata for a file if it goes missing (at the next scan?) or does this take a while?

 

In my case the disk was missing for >7 days and the database had to rebuild.  I also see a lot of disk activity at that time - does ffprobe read the whole file or just parts of it?

 

Can I control somehow, how fast data for missing files is purged from the database?  (In plex this could be done manually, missing files got a cross and only were purged when manually told to do so)

 

Thanks!

 

PS.  It looks indeed that all my files are in the library but all my metadata is missing, causing hundreds of gigabytes of download.  I'd love to know how to avoid this in the future.

Edited May 17, 2020 by TonioRoffo

[Luke 42086]

Hi there, can we please go over an example? Thanks !

[TonioRoffo 4]

Hi there, can we please go over an example? Thanks !

 

One mount with approx 7TB of data got disconnected from a pool of disks around 11/5/2020.  I didn't notice until 17/5/2020.

 

After re-adding the  mount, I performed a rescan of two libraries.  This went fast until "90%" and then slowed down and caused massive bandwidth usage on my internet connection.  Judging by the logs, redownloading metadata.

 

The mount is cloud-based.

 

Snippet of the logs during this, from Emby 4.5.0.11 beta:

2020-05-17 18:22:45.846 Info MediaProbeManager: ProcessRun 'ffprobe' Execute: /opt/emby-server/bin/ffprobe -i file:"/media/XBMCDATA/XBMCDATA/Videos/HDMovies/A/?????.mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2020-05-17 18:22:49.290 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 0
2020-05-17 18:22:49.301 Info App: MovieDbProvider: Finding id for item: ?????
2020-05-17 18:22:49.302 Info HttpClient: GET https://api.themoviedb.org/3/search/movie?api_key=f6bd687ffa63cd282b6ff2c6877f2669&query=?????&language=en
2020-05-17 18:22:49.601 Info HttpClient: GET https://api.themoviedb.org/3/movie/404579?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=casts,releases,images,keywords,trailers&language=en&include_image_language=en,null
2020-05-17 18:22:49.900 Info HttpClient: GET https://api.themoviedb.org/3/movie/404579?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=casts,releases,images,keywords,trailers
2020-05-17 18:22:50.094 Info HttpClient: GET https://webservice.fanart.tv/v3/movies/404579?api_key=5c6b04c68e904cfed1e6cbc9a9e683d4 

This seems to trigger about 100 to 200mb of data download per entry.  In total this ended up being in excess of 100GB download(!) to redownload the metadata.  

 

This happens because of missing files - I'd like to know how I can avoid my metadata being removed from the database in case this happens again.

 

Thanks.

Edited May 18, 2020 by TonioRoffo

[crusher11 1102]

Where do you store your metadata and images?

[TonioRoffo 4]

Where do you store your metadata and images?

 

I didn't change any settings concerning this.  Store NFO is off.  I think it's all database?

[crusher11 1102]

Then that's why it needs to re-download. If you store it with your media, it's all there for Emby to read when the drive is reconnected.

[TonioRoffo 4]

Then that's why it needs to re-download. If you store it with your media, it's all there for Emby to read when the drive is reconnected.

 

OK, but how quickly does Emby get rid of the metadata?  Instantly?  At the next scan?  This is problematic.

 

Also, browsing the forum, I've seen that storing the metadata with the media, only starts for new media when enabled.

 

How can I retroactively export the metadata in the database to the media folders?

 

Thanks

[Luke 42086]

OK, but how quickly does Emby get rid of the metadata?  Instantly?  At the next scan?  This is problematic.

What do you mean by get rid of? Why would the server get rid of metadata?

 

 

 

How can I retroactively export the metadata in the database to the media folders?

Try refreshing the metadata on an individual title, or parent folder of titles.