Jump to content

Real time monitoring & NFS or CIFS


andrew0404

Recommended Posts

andrew0404

Hey guys,

 

I have emby beta running on ubuntu.  Real time monitoring is working great, but I am wanting to move my emby server to another machine that has more cycles available for transcoding.  That will mean using NFS or CIFS to connect to my file server.  It doesn't seem that real time monitoring is working over NFS, which is understandable.  Is there any way of sending a notification to Emby to force it to scan a single new entry, or get real time monitoring working in this situation?

 

Really the ideal solution here would be to have emby on my file server and have transcoding agents on other machines to offload this work.  Has there been any thought about adding this functionality?

 

Thanks for a continually improving great product!

Link to comment
Share on other sites

Hi, yes we do have an API endpoint you could use. That would be the best way to notify about an individual change.

Link to comment
Share on other sites

  • 1 month later...
andrew0404

Ok, Ive had some time to do some testing with the notifications from Radarr/Sonarr and it isn't working like I would expect.  I swear it worked the first time I tried it.

 

I can verify that a notification and update command is being sent from Radarr/Sonarr to Emby, but nothing seems to be happening with it.  I have tried this with both the Beta server and the Latest Stable server.  I'm currently on stable 3.5.3.0

 

Here is the debug log for the notification of a single movie being added to the library.  This is the first movie being added to an empty library:

 

 

2019-01-07 18:55:59.086 Info HttpServer: HTTP POST http://emby-server:8096/mediabrowser/Notifications/Admin. UserAgent: Radarr/0.2.0.1217 (Linux 4.15)

2019-01-07 18:55:59.091 Info HttpServer: HTTP Response 204 to 10.0.0.3. Time: 5ms. http://emby-server:8096/mediabrowser/Notifications/Admin
2019-01-07 18:55:59.099 Info HttpServer: HTTP POST http://emby-server:8096/mediabrowser/Library/Movies/Updated?ImdbId=tt0060345. UserAgent: Radarr/0.2.0.1217 (Linux 4.15)
2019-01-07 18:55:59.102 Info HttpServer: HTTP Response 204 to 10.0.0.3. Time: 3ms. http://emby-server:8096/mediabrowser/Library/Movies/Updated?ImdbId=tt0060345
 
 
 
No other log entries are produced for this operation.  I have debug logging enabled.
 
Any advice would be appreciated.  I know that this could be incorrect usage of the API from the client end, but looking at the logs it seems like Emby is getting it correctly.
 
Thanks,
Edited by andrew0404
Link to comment
Share on other sites

mastrmind11

I have my NAS drives mounted as NFS shares on ubuntu 16.04 and the real time monitor works perfectly.  What issue are you having?  What's your fstab look like?

Link to comment
Share on other sites

andrew0404

I am running in docker containers with docker nfs named volume mounts for different nfs mount locations.  So far real time monitoring has not worked and in fact the docker volumes have issues where the nfs mounts get stale and I have to restart docker.  I am probably going to change them from NFS to CIFS and see if that works better.  Perhaps realtime monitoring will work with that, but in case it doesn't, it seems like the notifications from Radarr/Sonarr should be sufficient to force emby to update a single entry on download.  That would be all that I need to get it to work.

 

I am creating my nfs volumes with ansible like this:

- name: Create NFS docker volume
  docker_volume:
    name: "{{ item.name }}"
    driver: local
    driver_options:
      type: nfs
      device: ':{{ item.path }}'
      o: 'addr={{ storage_host }},vers=3,nolock,noacl,nocto,noatime,nodiratime,tcp,actimeo=1,rw'
    state: present
  with_items: '{{ volumes }}'
  when: remote_mount|bool

I added most of the NFS options in attempts to fix the stale file handle issue that I am having, and while it helped, it didn't fix the issue.  It is possible that one or more of those options are affecting real time monitoring.  I'm not sure.

 

I am using an attachable swarm overlay network so that I can have my emby container running on an isolated host but it still appears local on the docker network.

Edited by andrew0404
Link to comment
Share on other sites

mastrmind11

ah, yeah that's a bit more complex than I was anticipating.  I am also running emby in docker but have my nfs shares mounted via fstab, which works great.  However I do agree w/ you irt the radarr/sonarr api trigger (which, also work for me btw).  

Link to comment
Share on other sites

andrew0404

Which version of emby/sonarr/radarr are you using?

I am on Emby server 3.5.3.0, Radarr Ver. 0.2.0.1217, Sonarr Ver. 2.0.0.5252

I am using an ext4 file system.

 

I have an entry for Emby in the Settings->Connect tab with the correct API and everything set to yes (notify on everything), and Update set to yes.  I have an empty tag filter, and Testing the connection returns success.

 

I have made sure that the path is identical in all of my containers, so a file path from sonarr will point to the same file in the emby container.

 

I originally had my host do a NFS mount and I just shared those mounts locally with docker file mounts, which is much simpler and straight forward, but I started having stale file handles, and my docker containers sometimes wouldn't see new files unless I restarted the container.  So I switched to mounting the NFS volumes directly in the containers to try and get around that.  It works better, but like I said I still have stale file handles sometimes.

 

I just made a change to my containers to limit cpu/memory in each one because I was having random failures when it was trying to start all of my containers at once (docker service restart or on boot up).  I haven't noticed any stale file handles since then, but Its only been a day or so.

 

My hosts are running Ubutu 18.04, right now in vagrant VMs for testing.

Edited by andrew0404
Link to comment
Share on other sites

mastrmind11

I'm running the same versions, but am running a zfs file system.  What happens if you just spin up a vanilla emby docker and point it to vanilla nfs mount points, just for testing purposes?

Link to comment
Share on other sites

andrew0404

I haven't done it in a while, since I modified my ansible setup to use nfs volume mounts, but I can test it.  I have 2 test configurations, one that uses several virtual disks that simulate my physical setup for testing snapraid and other stuff, and a linked configuration that links to my live data via NFS.  Thats where I noticed all of the stale file handles.  My live system is on Ubuntu 14.04, so maybe that caused the stale file handle issue.  I'll give this a shot between my two 18.04 test systems and see if realtime monitoring works and if I get any stale file handles.  May take a few days for them to show up.

 

I'm still interested in what I might be doing wrong with the notifications, though.  It really seems like it should be working.

Edited by andrew0404
Link to comment
Share on other sites

Stokkes

Does Emby's API allow to scan for a single item or it has the scan the whole library? Coming from Plex, I'm trying to identify like you're doing for Sonarr/Radarr to scan a particular folder.

 

Scanning a whole library (especially TV shows) when 1 episode is added would cause a backlog (too many API requests for library updates overlapping each other, since each scan takes about 5 minutes).

 

Haven't seen it though in the API. Don't mean to hijack the thread, I have a similar setup and haven't figured out how to notify Emby when a new episode is added so that it gets added to Emby immediately.

 

Thanks

Link to comment
Share on other sites

andrew0404

So spinning up a new emby server with a simple volume mount to my mounted NFS location still does not work.  My fstab for the nfs mount:

 

server1://storage /storage nfs intr,rw 0 0

My mount point for my docker container is to /storage/Media, and it successfully mounts and I can see the files there.  I configured the new API key for radarr and the notification is received, but still no update.

Also I do have realtime monitoring enabled, and that didn't queue an update either.  Depending on what I need to do for the next few days I may leave this configuration running to test for stale file handles, but at the very least this doesn't get around the problem.  I may also try CIFS if I have a chance.

 

Is there any way to do some deeper debugging besides the Enable Debug Logging checkbox?

 

 

Stokkes, as far as I understand it, the API should be able to get Emby to update a single item without queuing an entire library scan.  The Realtime monitoring will do the same thing, but it only works on some filesystems.

Edited by andrew0404
Link to comment
Share on other sites

andrew0404

Thanks for the info Luke.  So looking into INotify, it seems that both NFS and CIFS are not supported.  I am curious how you got it to work mastrmind11.  Are you using inotify-tools and a custom script or can you verify that it is working out of the box?

 

Also Luke, is there something that I can do to help debug the API notification issue?  Does the log entries that I posted look like they should force an update?

Link to comment
Share on other sites

andrew0404

Did some further testing.  Notifications work for Sonarr when the TV show already exists in Emby, so new episodes are picked up correctly.

 

The problems seem to be adding the first episode of a new show, or adding a new movie.  Neither of these notification trigger an update.  Is that a different API call that Sonarr/Radarr are not using correctly?

Link to comment
Share on other sites

andrew0404

I figured as much.  I'll try to work on that end.  Thanks Luke.

 

Now that I'm looking there I already see issues opened for the problem.  No idea when it will be addressed though, sorry for all the fuss.  Should have looked there first.

Edited by andrew0404
Link to comment
Share on other sites

mastrmind11

Thanks for the info Luke.  So looking into INotify, it seems that both NFS and CIFS are not supported.  I am curious how you got it to work mastrmind11.  Are you using inotify-tools and a custom script or can you verify that it is working out of the box?

 

Also Luke, is there something that I can do to help debug the API notification issue?  Does the log entries that I posted look like they should force an update?

Worked out of the box.  Only thing I can think of is that ZFS uses a different implementation of NFS since ZFS handles NFS sharing internally?  

Link to comment
Share on other sites

andrew0404

That must be it.  I have seen your posts in other threads about NFS.  From what I can tell INotify will not work with NFS or CIFS.  Must be something with ZFS that makes it work.  I am not familiar with ZFS and I have a huge library that I would have to convert to it in place, so that is a problem.  But I might test it with my vagrant setup if I get a chance.

 

I did find a workaround until Sonarr updates to the new API.  You can create custom scripts for sonarr/radarr.  I created this one and you just put your host url/port and API key in the arguments.  Its working beautifully!  Both Sonarr/Radarr docker containers have Curl in them, so dropping the script into a location that each container can access and pointing the custom script to it is all you need to do.

#!/bin/bash
while [ $# -gt 0 ]; do
  case "$1" in
  --api=*)
    apiKey="${1#*=}"
    ;;
  --url=*)
    url="${1#*=}"
    ;;
  *)
      printf "***************************\n"
      printf "* Error: Invalid argument.*\n"
      printf "***************************\n"
      exit 1
  esac
  shift
done

if [ -z "$apiKey" ]
then
      printf "*******************************\n"
      printf "* Error: No API Key specified.*\n"
      printf "*******************************\n"
      exit 1
fi

if [ -z "$url" ]
then
      printf "********************************\n"
      printf "* Error: No Emby URL specified.*\n"
      printf "********************************\n"
      exit 1
fi

if [ -z "$sonarr_eventtype" ] && [ -z "$radarr_eventtype" ]
then
      printf "******************************************************************\n"
      printf "* Error: Must be called as a custom script from Sonarr or Radarr.*\n"
      printf "******************************************************************\n"
      exit 1
fi

#SONARR
if [ "$sonarr_eventtype" == "Download" ];  then
  UpdateType="Series"
  Path="$sonarr_episodefile_path"
fi

if [ "$sonarr_eventtype" == "Rename" ]; then
  UpdateType="Series"
  Path="$sonarr_series_path"
fi

# RADARR
if [ "$radarr_eventtype" == "Download" ]; then
  UpdateType="Movie"
  Path="$radarr_movie_path"
fi

if [ "$radarr_eventtype" == "Rename" ]; then
  UpdateType="Movie"
  Path="$radarr_movie_path"
fi

if [ -z "$UpdateType" ] || [ -z "$Path" ]
then
      printf "*************************************************\n"
      printf "* Error: Script unsupported for this event type.*\n"
      printf "*************************************************\n"
      exit 1
fi

curl -s -X POST "$url/emby/Library/Media/Updated?api_key=$apiKey" -H "accept: */*" -H "Content-Type: application/json" -d "{ \"Updates\": [ { \"Path\": \"$Path\", \"UpdateType\" \"$UpdateType\" } ]}"

Edited by andrew0404
  • Like 1
Link to comment
Share on other sites

Stokkes

@@andrew0404 You read my mind my friend. I was just about to write a very similar bash script.

 

However, for those who copy/paste, Sonarr/Radarr will fail the test since the script doesn't have a check for eventttype == "Test", so you won't be able to save it (at least I wasn't).

 

Here's a very quickly revised script:

#!/bin/bash
while [ $# -gt 0 ]; do
  case "$1" in
  --api=*)
    apiKey="${1#*=}"
    ;;
  --url=*)
    url="${1#*=}"
    ;;
  *)
      printf "***************************\n"
      printf "* Error: Invalid argument.*\n"
      printf "***************************\n"
      exit 1
  esac
  shift
done

if [ -z "$apiKey" ]
then
      printf "*******************************\n"
      printf "* Error: No API Key specified.*\n"
      printf "*******************************\n"
      exit 1
fi

if [ -z "$url" ]
then
      printf "********************************\n"
      printf "* Error: No Emby URL specified.*\n"
      printf "********************************\n"
      exit 1
fi

if [ -z "$sonarr_eventtype" ] && [ -z "$radarr_eventtype" ]
then
      printf "******************************************************************\n"
      printf "* Error: Must be called as a custom script from Sonarr or Radarr.*\n"
      printf "******************************************************************\n"
      exit 1
fi

#SONARR
if [ "$sonarr_eventtype" == "Download" ];  then
  UpdateType="Series"
  Path="$sonarr_episodefile_path"
fi

if [ "$sonarr_eventtype" == "Rename" ]; then
  UpdateType="Series"
  Path="$sonarr_series_path"
fi

if [ "$sonarr_eventtype" == "Test" ]; then
  exit 0
fi

# RADARR
if [ "$radarr_eventtype" == "Download" ]; then
  UpdateType="Movie"
  Path="$radarr_movie_path"
fi

if [ "$radarr_eventtype" == "Rename" ]; then
  UpdateType="Movie"
  Path="$radarr_movie_path"
fi

if [ "$radarr_eventtype" == "Test" ]; then
  exit 0
fi

if [ -z "$UpdateType" ] || [ -z "$Path" ]
then
      printf "*************************************************\n"
      printf "* Error: Script unsupported for this event type.*\n"
      printf "*************************************************\n"
      exit 1
fi

curl -s -X POST "$url/emby/Library/Media/Updated?api_key=$apiKey" -H "accept: */*" -H "Content-Type: application/json" -d "{ \"Updates\": [ { \"Path\": \"$Path\", \"UpdateType\" \"$UpdateType\" } ]}"
  • Like 1
Link to comment
Share on other sites

We're going to see if we can help out with both Sonarr and Radar and help get them updated to the new API. Thanks guys.

Link to comment
Share on other sites

  • 1 month later...
harrv

 

@@andrew0404 You read my mind my friend. I was just about to write a very similar bash script.

 

However, for those who copy/paste, Sonarr/Radarr will fail the test since the script doesn't have a check for eventttype == "Test", so you won't be able to save it (at least I wasn't).

 

Here's a very quickly revised script:

// script was here

 

Thank you @@andrew0404 and @@Stokkes! I would like to deploy this, but I think I'm going to need path substitution. Sonarr and Radarr run on a Mac mini with the media accessible from a mounted volume, and Emby runs on Windows 10 and uses a SMB UNC path to access the media. For example, here are the paths to the series "Example Series":

 

Sonarr: /Volumes/media/tv/Example Series

Emby: \\D2015\mediafull\tv\Example Series

 

I have three TV libraries though, and Sonarr is configured to use all three for different series, so at the same level as "tv" in the example above I actually have three directories named tv, tv-pilots, and tv-drew, and each of those contains series.

 

I could really use some help with how to implement that in this script, if either of you are willing.

 

By the way, regarding Luke's comment that they're going to help Sonarr and Radarr properly implement Emby connections, I see right now that there is a pull request in Sonarr, yet unmerged, to update to the newer Emby API, but it doesn't appear to handle path substitution either. Judging from a comment made on the pull request, that seems to be a reason it hasn't been merged yet.

Edited by harrv
Link to comment
Share on other sites

They should still merge it because even without that last step, it's still in a better place than the previous method.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...