Server - Option to ignore media in scans

[dcol 165]

On 4/28/2019 at 10:54 AM, Luke said:

If you don't want it scanned, then why add it to the library in the first place? Or is this for temporary usage?

because this is a media player. Some of us have static libraries with no need to update it

[jmansell90 4]

18 hours ago, Hxemby001 said:

I understand now.. probably should have read a little better.. OBVIOUS why it wasn't posted.  LOL

What's funny is I did try that and panicked when my entire media library disappeared.  I was hoping it would just stop *updating* the folder, not remove all that was inside it.
 

18 hours ago, Happy2Play said:

This Excludes media/folder from Emby altogether.  They want to see the media, just no long have it scanned.

But at the same time Emby is only looking for changes when a media folder is scanned.  So if nothing changes nothing is updated.  That would be why my library scan takes 2-3 minutes, but overall setup applies as all my media is on the same system running Emby.

It may be looking for changes only, but in a library with over a million media files nested under no less than 100k child directories (In the case of TV shows, tons of nested directories) that is a LOT of scanning for changes.  Sure, this is only extreme cases like myself... but every media hoarder is going to run into this problem eventually.  I'm curious what filesystem you use and the size of your library though, for comparison.

I have 4 CIFS shared NTFS filesystems on an NFS partition alongside an XFS formatted iSCSI SAN for my newer media.  The scanning of my XFS filesystem shared into the Emby container goes by pretty quickly, but that is a smaller library too.  The scanning of the CIFS folders takes over an hour.  Part of this could be latency with CIFS and grabbing the filetree but again I don't know the method Emby uses to scan.  I'm definitely pushing it hard, though.

[rbjtech 4264]

As a comparison - I use SATA direct attached storage (via UNC shares in Emby) and can do a full scan of ~1,500 films and ~21,000 tv episodes in approx 2 mins 30 seconds - on an aging i5 Gen1 CPU.

So it may well be the CIFS/SMB/Network that is a big overhead here.  

Maybe some others can comment on the scanning duration (with approx library size) using a NAS via CIFS/SMB for example ?

Edited March 31, 2021 by rbjtech

[Veggav 8]

My google skills are terrible for not finding this thread. 

So, let me be one more to request this feature.

[dcol 165]

Adding this feature is a total 'No Brainer'. Not sure why it hasn't been done already.

[bigjohn 655]

7 minutes ago, dcol said:

Adding this feature is a total 'No Brainer'. Not sure why it hasn't been done already.

It likely hasn't been done because it is not nearly as simple as it sounds.

[rbjtech 4264]

I must be missing something here.

a) Turn off the scheduled scan (all areas) - just remove the scheduled task.

b) If you NEED to scan a library - then click the ... and select 'Scan LIBRARY' ..  It will just scan that library, nothing else will be scanned..

?

Edited April 27, 2021 by rbjtech

[Veggav 8]

17 minutes ago, rbjtech said:

I must be missing something here.

a) Turn off the scheduled scan (all areas) - just remove the scheduled task.

b) If you NEED to scan a library - then click the ... and select 'Scan LIBRARY' ..  It will just scan that library, nothing else will be scanned..

?

 

This breaks the entire automation. 

Let me put it this way:

 

Do you have wife? kids?

[rbjtech 4264]

Both thanks !

Once scanned - Why do you feel the need to scan your library again in the first place ?  

As I add new media, it gets added - I haven't scanned my libraries using either a scheduled scan (because I turned it off - by default, I believe it's set to run on startup and again once a week?) or an individual scan - I simply have no need.

Why Automate something that doesn't need to run ?

 

[dcol 165]

I may be naive, but I did a fair amount of programming in the past and think that before a library is scanned it can be checked for a flag set whether it should skip that library. Emby does library scans one at a time and not multiple libraries at once. The other reason for not implementing this is if the programmer is no longer available.

[dcol 165]

5 minutes ago, rbjtech said:

Both thanks !

Once scanned - Why do you feel the need to scan your library again in the first place ?  

As I add new media, it gets added - I haven't scanned my libraries using either a scheduled scan (because I turned it off - by default, I believe it's set to run on startup and again once a week?) or an individual scan - I simply have no need.

Why Automate something that doesn't need to run ?

 

Scanning certain libraries is important to renew metadata. Also the new media add doesn't work in certain configurations, ie some network shares etc. When you have a static library that does not need scanning with 500K+ items, It would be nice to not have Emby work for hours on it.

Edited April 27, 2021 by dcol

[Luke 37064]

Some UI will have to be re-thought, as well as some internal mechanisms will have to be reworked. It's nothing overly complicated, but just a matter of getting to it. This falls more into the age old question of "why isn't everything I want done already".

[Veggav 8]

3 minutes ago, rbjtech said:

Both thanks !

Once scanned - Why do you feel the need to scan your library again in the first place ?  

As I add new media, it gets added - I haven't scanned my libraries using either a scheduled scan (because I turned it off - by default, I believe it's set to run on startup and again once a week?) or an individual scan - I simply have no need.

Why Automate something that doesn't need to run ?

 

 

So you understand that asking your kid to scan a library isn't the best solution, right?

The thing is real time monitoring isn't perfect. I have it failing me in 2 different scenarios:

Downloaded content on my unraid NAS box and my google drive mounted share with rclone.

 

And when you have a huge library and maybe this library isn't local scaning takes forever. 

I do think this can't be too difficult to add.

Create a similar condition like .ignore but .donotscan 

If .donotscan

{

Break;

}

There's a tons of ways to check for a file in C++. 

[rbjtech 4264]

1 minute ago, dcol said:

Scanning certain libraries is important to renew metadata. Also the new media add doesn't work in certain configurations, ie NFS shares etc. When you have a static library that does not need scanning with 500K+ items, It would be nice to not have Emby work for hours on it.

ok - on NFS, I wasn't aware it didn't recognise new items - thanks.

If emby does recognise the new media - then same per library option is there for just metadata too ...

[Veggav 8]

3 minutes ago, Luke said:

Some UI will have to be re-thought, as well as some internal mechanisms will have to be reworked. It's nothing overly complicated, but just a matter of getting to it. This falls more into the age old question of "why isn't everything I want done already".

 

Just break the operation in the same way as you do with ignore. 

Copy and paste the .ignore function, give it another name and break it.

 

Or, maybe, if we could set the priority of scaned libraries just leave the one you don't want scanned to be the last one and in the cron job just put a stop timer of 10 minutes.

[teddybear75 11]

19 hours ago, Luke said:

This falls more into the age old question of "why isn't everything I want done already".

I thought that was the definition of "Feature requests"

Kidding aside, i too hope this gets implemented at some point. It would be a nice bonus to the already great experience that Emby is

[Veggav 8]

Ok, figured out how to do what I want:

First, real time monitoring with rclone mount is not possible so a scan is needed indeed.

Second this is what I'm doing:

1. Disable any schedule library scan

2. you will need something running scripts on linux, I have an unraid box running user scripts. 

3. install curl

4. Example:

curl -d '{"Updates":[{"Path":"/movies/movies/TV Shows Recent/","UpdateType":"Created "}]}' -H "Content-Type: application/json" -X POST http://serverip:8096/emby/Library/Media/Updated?api_key=YOURAPIKEY

5. In bold are the things you need to change for your particular setup. 

[teddybear75 11]

 

On 03/05/2021 at 01:38, Veggav said:

Ok, figured out how to do what I want:

First, real time monitoring with rclone mount is not possible so a scan is needed indeed.

Second this is what I'm doing:

1. Disable any schedule library scan

2. you will need something running scripts on linux, I have an unraid box running user scripts. 

3. install curl

4. Example:

curl -d '{"Updates":[{"Path":"/movies/movies/TV Shows Recent/","UpdateType":"Created "}]}' -H "Content-Type: application/json" -X POST http://serverip:8096/emby/Library/Media/Updated?api_key=YOURAPIKEY

5. In bold are the things you need to change for your particular setup. 

Hello.

I tried this without luck, i might have the path wrong, can you show me an other example of a path? I have my shares on network paths. So i was thinking something like this for path: /Libraryname/\\ipadress\foldername/  or /Libraryname/ipadress/foldername/ but this did not seem to work.

I get no error message from curl, but nothing happens, it seems to run okay but nothing happens, or at least the library specified does not update.

Generated a new api key under API keys.

Edited May 5, 2021 by teddybear75

[Veggav 8]

4 hours ago, teddybear75 said:

 

Hello.

I tried this without luck, i might have the path wrong, can you show me an other example of a path? I have my shares on network paths. So i was thinking something like this for path: /Libraryname/\\ipadress\foldername/  or /Libraryname/ipadress/foldername/ but this did not seem to work.

I get no error message from curl, but nothing happens, it seems to run okay but nothing happens, or at least the library specified does not update.

Generated a new api key under API keys.

 

You might need to wait a little for it to start or you might need send the command twice, happened to me once. 

The path is full path that you add when adding a new library. 

[jmansell90 4]

3 hours ago, Veggav said:

 

You might need to wait a little for it to start or you might need send the command twice, happened to me once. 

The path is full path that you add when adding a new library. 

I wasn't able to get this working either.  You did however give me an idea and lucky enough, there is a way to do this via API.  In the UI if you go into the media library and go to Folders, you can click on the "options" of the folder and scan.  Turns out, this scans that particular folder for new content while ignoring the others.  So I grabbed the API call and simplified it a touch (It  may be able to be further simplified) but I got this:

 

curl 'http://EMBY-IP/emby/Items/FOLDER-ID/Refresh?Recursive=true&ImageRefreshMode=Default&MetadataRefreshMode=Default&ReplaceAllImages=false&ReplaceAllMetadata=false&api_key=API-KEY'   -X 'POST'   -H 'Connection: keep-alive'   -H 'Content-Length: 0'   -H 'Accept: */*'   --compressed   --insecure

While this is not as intuitive to customize as your example, it's not an extreme task to get the info needed.  Of course, replace EMBY-IP with your instance IP address and API-KEY with your API key.  The hard part is the FOLDER-ID.  What you'll need to do is browse to Emby, and go into the library in question.  Select the Folders tab and click on the folder you want to scan.  Look at the URL, you should have something like this:

 

http://IP-ADDRESS/web/index.html#!/list/list.html?parentId=49264&serverId=SERVERID

The parentID in the URL is your FOLDER-ID

In my tests I put an empty file named for The Mask (1994) into "New Movies" folder and The Wolfman (2010) into my "Static Movies" folder.  When I scanned using the "New Movies" FolderID I got The Mask in my library moments later but The Wolfman did not appear.  I repeated this for about 10min and The Wolfman never appeared.  Then I changed to the Static Movies FolderID and ran the curl command once and instantly The Wolfman appeared in my library.

Could somebody a little better with curl and APIs maybe strip the command down a bit to be less of a mouthful?

[Veggav 8]

32 minutes ago, jmansell90 said:

I wasn't able to get this working either.  You did however give me an idea and lucky enough, there is a way to do this via API.  In the UI if you go into the media library and go to Folders, you can click on the "options" of the folder and scan.  Turns out, this scans that particular folder for new content while ignoring the others.  So I grabbed the API call and simplified it a touch (It  may be able to be further simplified) but I got this:
 

 

It's very similar, maybe on my setup this works as expected because I'm using an Emby docker on unraid server. 

This API call I posted is used by Sonarr so it should work. 

 

Anyway, as we can see this is very very trivial to do. 

[dcol 165]

I thought this was a thread about stopping scans not how to automate them. Anyone with very large libraries needs an option to ignore scans per library.

[rbjtech 4264]

...you turn off all the scan's using the scheduler and turn on the ones you need using the above API calls - it achieves the same goal.

Edited May 5, 2021 by rbjtech

[jmansell90 4]

19 minutes ago, dcol said:

I thought this was a thread about stopping scans not how to automate them. Anyone with very large libraries needs an option to ignore scans per library.

This is a workaround until such a feature can be implemented.

Disable ALL scanning in emby.  Emby will no longer tell itself to scan anything.
Use API calls to scan from outside emby, and only the folders you want to scan
Use cron or some other scheduling tool to automatically run the scan.

We still want the ability to ignore folders from scans so this weird backwards solution wont be necessary.  BUT if you need this function while you wait, this achieves that if you have the resources to do so.
 

44 minutes ago, Veggav said:

 

It's very similar, maybe on my setup this works as expected because I'm using an Emby docker on unraid server. 

This API call I posted is used by Sonarr so it should work. 

 

Anyway, as we can see this is very very trivial to do. 

I am using a similar setup, so I'm not sure why it doesn't work for me.  But at least we have multiple options.

[dcol 165]

1 hour ago, rbjtech said:

...you turn off all the scan's using the scheduler and turn on the ones you need using the above API calls - it achieves the same goal.

And how would I go about that in a Windows environment?