Monitor library not detecting changes to catalogue when using mergefs

[As777 22]

I have been playing around and learning Emby server on linux for a few weeks now and have finally settled on a way of using everything as I like.

I am running several Emby servers in Docker on my Raspberry PI 4 and have created folders using mergefs to join several folders. I then specify the mount points in my Docker compose file.

It appears, though, that the server is unable to detect changes to the catalogue when pointing to a mergefs folder. I have to manually scan to pick up new files.

[mastrmind11 717]

how is your folder mounted?

[As777 22]

2 hours ago, mastrmind11 said:

how is your folder mounted?

With these options

defaults,allow_other,cache.files=off,use_ino

[As777 22]

Testing this from the mergefs FAQ

"

It does. If you're trying to put Plex's config / metadata / database on mergerfs you can't set cache.files=off because Plex is using sqlite3 with mmap enabled. Shared mmap is not supported by Linux's FUSE implementation when page caching is disabled. To fix this place the data elsewhere (preferable) or enable cache.files (with dropcacheonclose=true). Sqlite3 does not need mmap but the developer needs to fall back to standard IO if mmap fails.

If the issue is that scanning doesn't seem to pick up media then be sure to set func.getattr=newest though generally a full scan will pick up all media anyway."

 

*UPDATE* doesn't seem to help. Neither Plex nor Emby find new files unless I manually trigger a scan.

Edited August 3, 2021 by askog666
update

[Q-Droid 654]

What are the underlying filesystems in use?

[As777 22]

7 hours ago, Q-Droid said:

What are the underlying filesystems in use?

The filesystem I am writing to is BTRFS and I write to the actual drive not the mergefs share. I am wondering if this was the right way to go since I am experiencing performance issues on my Pi since moving to this filesystem on some of my drives. I think I will go to ext4 instead and test.

I am going to remove mergefs for now at least and see if that helps both my problems.

[As777 22]

Auto monitor working again after pointing the library directly to the share instead of the mergefs share. Seems there is a problem picking up new content through mergefs.

[Q-Droid 654]

You wrote "share" more than once above. Do you have network shares mounted on the rPi? Emby uses inotify for RTM but this works on local POSIX filesystems, not network shares.

Can you post your Emby log? What isn't clear is if mergerfs supports inotify or if the merged path has too many directories which would show up in the log.

 

[mastrmind11 717]

^^ what I was getting at when I asked you how your drives were mounted.

[As777 22]

 

I want to use merge fs as a convenient overview and a single map in Emby but I want to decide where the actual data is written to. If I write to mergefs it decides where the content will go.

I think the problem is that I am sending content not to the actual mergefs folder but directly to the local storage folder and that writing to the local folder does not trigger any event that Emby reacts to when pointing to the mergefs folder, thus does not "detect" any changes.

I am in the process of restructuring my server right now and converting to ext4 as I have other issues I need to sort and will report back.

On 8/4/2021 at 1:36 PM, Q-Droid said:

You wrote "share" more than once above. Do you have network shares mounted on the rPi? Emby uses inotify for RTM but this works on local POSIX filesystems, not network shares.

Can you post your Emby log? What isn't clear is if mergerfs supports inotify or if the merged path has too many directories which would show up in the log.

 

Quote

 

 

Edited August 6, 2021 by askog666

[Luke 37116]

Let us know how you get on. Thanks.

[As777 22]

So after an overhaul of my system I am pleased to say the monitoring of the library appears to be working perfectly with mergefs.

It now works regardless of whether you write to the merged folder or one of the source folders, which is very pleasing.

I did however change my filesystem from btrfs to ext4 for my entire catalogue and used the following options when making my mergefs folder. I am unsure if btrfs is a problem or not but i went back to ext4 after having performance problems on my Pi 4 and it seems to have helped.

defaults,allow_other,cache.files=partial,use_ino

It appears "cache.files" needs to be set to at least partial for anything relying on mmap. Hopefully this will help others if they find the same situation.

Edited August 10, 2021 by askog666

[Luke 37116]

Thanks for the feedback !

[embylad892746 15]

On 8/10/2021 at 8:12 AM, As777 said:

So after an overhaul of my system I am pleased to say the monitoring of the library appears to be working perfectly with mergefs.

It now works regardless of whether you write to the merged folder or one of the source folders, which is very pleasing.

I did however change my filesystem from btrfs to ext4 for my entire catalogue and used the following options when making my mergefs folder. I am unsure if btrfs is a problem or not but i went back to ext4 after having performance problems on my Pi 4 and it seems to have helped.

defaults,allow_other,cache.files=partial,use_ino

It appears "cache.files" needs to be set to at least partial for anything relying on mmap. Hopefully this will help others if they find the same situation.

Thanks so much for posting this. This fixed the issue for me as well.