Jump to content

Minimize media analysis (rclone/gdrive)


Chiefmas

Recommended Posts

Chiefmas

I'm experimenting with the addition of Google Drive, mounted via rclone. I don't have any problems there, as far as accessing. But when I add the new path, it seems like emby is going to run an ffprobe/ffmpeg analysis (I see both processes running, which seems odd but that's a separate thing). Anyway, people that are doing this, did you just let it do it? Otherwise, what settings are you using to minimize that. It's going to be pretty obnoxious if Emby is going to want to deep scan every file on the path like that (as it means I'll effectively pull everything back down as it gets put in gdrive).

tldr; what settings to use minimize Emby processing of new files that are on remote storage?

Link to comment
Share on other sites

Hey - 

 Emby doesn't pull the entire file down when it scans a library. I don't save any artwork or metadata to the libraries, though. Library scans on my mounted drives take under 30 seconds.

Use the latest version of rclone with vfs mode set to full and set a cache size and location that works for you. Otherwise, the default settings for rclone mounting a drive seem to work pretty well for streaming with emby.

 

John

Link to comment
Share on other sites

Chiefmas
36 minutes ago, jad3675 said:

 Emby doesn't pull the entire file down when it scans a library. I don't save any artwork or metadata to the libraries, though. Library scans on my mounted drives take under 30 seconds.

Well, that's what I'm trying to figure out. It's not the library scan, it seems to be processing the file in some way, which is causing rclone to pull everything down. As I said, I see an ffprobe and an ffmpeg process running. If either of those were touching the file, they'd try and open it which would probably pull the file down to the local cache. That's why I was asking about setting to make sure Emby isn't trying to extract thumbnails or some such. I thought I turned that all off, but maybe I missed something.

I'm using vfs full with a local cache now, I don't think it's anything in my rclone settings that's triggering it.

Thanks!

 

Link to comment
Share on other sites

HI, it's true yes that it runs ffprobe, but that doesn't require pulling down the entire media file.

If you don't use nfo files, you could remove the nfo plugin and that will eliminate the checks for those files. That may not give you much of an improvement, but it's something you can do.

Quote

That's why I was asking about setting to make sure Emby isn't trying to extract thumbnails or some such. 

Did you explore the metadata options for each library that you created?

Link to comment
Share on other sites

Hi, Yes if thumbnail extraction is being used that will cause a lot of activity on your GDrive account.

You likely don't want to do this.  If you do want the bif files created and used one approach could be to load the movies first on your local system in a test library then copy the movie & bif file up to Gdrive.  You would want to try this with 5 to 10 files to see how it works out for you.

Link to comment
Share on other sites

Chiefmas

I did review the library settings, but I'm double checking...I have the metadata in nfo unchecked, and I have thumbnail creation set to never. Ah, I think I just found it. Subtitle download is requiring a hash match. That must be what the ffmpeg process was doing. I'll uncheck that and give it a go again.

Thanks!

 

Link to comment
Share on other sites

If not using NFO then also remove the plugin.

If you don't need subtitles for every movie turn off those automatic features as well.  You can pull the subtitles down for movies/shows as needed.

Link to comment
Share on other sites

Chiefmas
2 hours ago, cayars said:

If not using NFO then also remove the plugin.

If you don't need subtitles for every movie turn off those automatic features as well.  You can pull the subtitles down for movies/shows as needed.

Will do. I had liked the subtitle feature, but it's just a convenience for me, so I'll switch to an on-demand based approach.

Thanks!

 

Link to comment
Share on other sites

  • 2 weeks later...
Chiefmas

So an update, I'm still trying to figure this out.

I've created a test library with 6 files in them. I moved a set of files into a new folder, so that no previous caching or anything has happened.

In Emby, settings for the library, thumbail generation is set to never. Open Subtitles and require hash match are both unchecked, but I left Save Metadata Into Media Folders checked, but I assumed that shouldn't be a problem unless it also causes additional data extracted from scanning the file to happen.

Real time monitoring is enabled, as are the default metadata downloaders. A question here, is metadata download done from just the file name/meta data pulled from ffprobe?

Ok, so after all that, my rclone cache folder has a complete copy of all 6 files. rclone forums are saying that it will only pull what is requested, and they sound like that it should be at the block level not the file level. They are saying it should only have partials, but I tested the files, they are fully allocated and complete, I can play the cached copies. So based on what they are saying, something Emby is doing is causing a full scan of the file.

ffprobe commands in the log are failing with timeouts, which does seem to suggest that ffprobe is waiting on the file to become available through the file provider, but it's not clear to me why the whole file is coming down unless something in the way ffprobe requests it is causing it. I ran the same command against my local file while watching activity monitors, and the ffprobe data came back instantly, and didn't seem like it had to scan the entire file, which I expected.

So, all that said, can anyone think of any reasons that the entire contents of a file would be requested, that could be causing the entire file to get pulled down? I need to stop the behavior or it will likely limit how useful Emby will be to me in the future.

Thanks!

 

Link to comment
Share on other sites

Quote

Real time monitoring is enabled, as are the default metadata downloaders. A question here, is metadata download done from just the file name/meta data pulled from ffprobe?

file and folder names, as well as any pre-existing metadata files you might have.

The probing doesn't need the complete file, but I imagine it will need a portion of both the beginning and end of it.

Link to comment
Share on other sites

Chiefmas

Thanks! Hopefully this is my last update. Apparently the feature of only pulling partial files to meet requests which I was under the impression was in my rclone version (1.52.3) only came out on Windows at least, very recently, in 1.53. I've upgraded my rclone, and that does appear to have gotten things working much more in line with what I am shooting for.

Thanks again!

 

Link to comment
Share on other sites

  • 4 weeks later...
blueicegr
On 9/29/2020 at 3:17 AM, jad3675 said:

Hey - 

 Emby doesn't pull the entire file down when it scans a library. I don't save any artwork or metadata to the libraries, though. Library scans on my mounted drives take under 30 seconds.

Use the latest version of rclone with vfs mode set to full and set a cache size and location that works for you. Otherwise, the default settings for rclone mounting a drive seem to work pretty well for streaming with emby.

 

John

Hello John,

can you please share with us the rclone config you use?

Thank you

 

Link to comment
Share on other sites

Chiefmas
3 hours ago, blueicegr said:

 

can you please share with us the rclone config you use?

 

 

You should take a read at Animosity022's thread at the rclone forums . There's an extra step or two that you should be taking for optimal performance that you'll want to do, to be able to configure rclone for the most reliable performance. Setting up a dedicated Google API/client ID for rclone, for example. There's not a single, global config, that I've seen that's universally, with apologies to the Mandalorian, "The Way", that I've seen, though many of them will be similar. His thread is most commonly referenced on the rclone forums for this topic (it's a pinned topic in the guides, at the forum level, so that should tell you something)

This thread

Also, he links to his GitHub where he's got all the settings he uses along with an explination of why they are that way, so it's good to help understand what's going on.

Link to comment
Share on other sites

jad3675
On 11/4/2020 at 7:59 AM, blueicegr said:

Hello John,

can you please share with us the rclone config you use?

Thank you

 

Pretty much what Chiefmas posted. Nothing esoteric, just a full vfs mount with a 500GB cache.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...