Jump to content
Oxide

Scan media library - Option for offline files

Recommended Posts

Oxide

Would it be possible to add an option to "Scan media library" that aborts the scan if the file server it's scanning is offline, currently it marks the media as offline and this can create chaos in the Kodi plugin as it has to resync all the "changes"

  • Like 8

Share this post


Link to post
Share on other sites
MSattler

I think this would actually be ideal.  

 

Just went through this today when 1 of 2 media servers went down.   It ripped all content out of the library, and then re-added it when it came back, and redownloaded all coverart =(

  • Like 1

Share this post


Link to post
Share on other sites
Luke

Yes I think our offline handling could use some improvement.

  • Like 1

Share this post


Link to post
Share on other sites
MrWebsmith

I agree as well...I'd love something to scan but not mark or remove it intentionally...even if my content is temporarily offline at the time.

  • Like 1

Share this post


Link to post
Share on other sites
AdrianW

Bump.

 

Please implement something to prevent the server from spending hours (and hours) scanning content that is offline. I have one log file of over 3 million lines because the server woke up in the middle of the night and performed a scan when neither of my NAS boxes were on (I do not have Scan Media Library scheduled at all).

 

The server should really check that a share is available before proceeding to scan down a particular path. If the share can't be accessed - then simply skip to the next.

 

Here's a request I made for this last year: Scan Library - when specific path is unavailable

  • Like 2

Share this post


Link to post
Share on other sites
Oxide

Just bumping this request, my unraid server hung almost 2 days and Emby is still scanning the info back  :huh: This also causes havoc with the Kodi plugin  :(

 

@@Luke any chance of adding an option to skip library scans when the paths are not available? Pretty please  ;)

  • Like 2

Share this post


Link to post
Share on other sites
Oxide

Another bump to this feature request.


 


We had a power failure and the Emby server started up once power was restored, but the server with the media didn't. I estimate it will now take 2 days to reimport all the movies and series again as with the media server being offline Emby has removed the media from all the libraries.


RiAARo9iL.gif


 


@@Luke and members of the dev team please look at implementing an option in the library section that we can select that if the source directory is offline then skip any media scans. This is really needed for the sanity of your users.


Songs-About-Going-Crazy-630x420.jpg


  • Like 1

Share this post


Link to post
Share on other sites
ebr

I agree we can improve how this is handled now but you can minimize the pain by storing your metadata with your media.  That will make total re-scans much faster.

  • Like 1

Share this post


Link to post
Share on other sites
funwithmedia

I agree we can improve how this is handled now but you can minimize the pain by storing your metadata with your media.  That will make total re-scans much faster.

That is good to know!

Share this post


Link to post
Share on other sites
AdrianW

...dev team please look at implementing an option in the library section that we can select that if the source directory is offline then skip any media scans.

I don't think we need a feature to select that a source directory is offline - Emby should just automatically detect that it is offline.

  • Like 2

Share this post


Link to post
Share on other sites
ebr

I don't think we need a feature to select that a source directory is offline - Emby should just automatically detect that it is offline.

 

And what is the difference between "offline" and "permanently removed"...?

Share this post


Link to post
Share on other sites
AdrianW

And what is the difference between "offline" and "permanently removed"...?

 

If you've permanently removed a source directory, you'd go to your library setup and remove the folders from the library.

Share this post


Link to post
Share on other sites
ebr

If you've permanently removed a source directory, you'd go to your library setup and remove the folders from the library.

 

I guess what we'd have to do is test each defined source folder for each library before the scan starts.  IIRC, that's the kind of thing we ended up doing for this in MB 2.x.

  • Like 1

Share this post


Link to post
Share on other sites
AdrianW

I guess what we'd have to do is test each defined source folder for each library before the scan starts.  IIRC, that's the kind of thing we ended up doing for this in MB 2.x.

 

Yes. That's basically what I suggested in the first post of this thread (from July 2015).

Share this post


Link to post
Share on other sites
dcook

If an Admin removes a Library folder location then Emby should remove the items from the library.

 

If the library folder location has not been removed but is offline or otherwise not reachable, Emby should not assume its been removed and remove all the items, it should skip and try again on the next scheduled scan.

 

@@ebr Storing metadata with media is not the correct answer for this issue, as that would mean that your storage location would need to be accessed all the time, which is slower then a local SSH.   Its simply much faster to keep metadata on a local SSH disk.

Edited by dcook
  • Like 1

Share this post


Link to post
Share on other sites
mastrmind11

Yes. That's basically what I suggested in the first post of this thread (from July 2015).

Or just simply have a list of IPs that get pinged before the scan runs, like I suggested in this FR https://emby.media/community/index.php?/topic/42630-server-ping-nas-offset/

 

ping times out a defined number of times or over a defined duration, scan gets skipped until next scheduled scan (lather rinse repeat).

Share this post


Link to post
Share on other sites
ebr

@@ebr Storing metadata with media is not the correct answer for this issue, as that would mean that your storage location would need to be accessed all the time, which is slower then a local SSH.   Its simply much faster to keep metadata on a local SSH disk.

 

I did not suggest it was a solution but, rather, it would make it less painful on a rescan.

 

Also, your assumption about metadata location affecting performance is not strictly true.  We don't have to access the metadata files for normal operation (except for images but then even only sometimes).

Share this post


Link to post
Share on other sites
dcook

@@ebr, the performance difference between storing metadata on a local SSH or on a NAS or network share has been proven many times, Do you want me to go through and find all the posts?

There is a valid reason why everyone is putting cache and metadata on local fast SSH.

 

I really think you guys should stop pushing the "Storing metadata with media" as its not the ideal solution

 

 

 

 

I did not suggest it was a solution but, rather, it would make it less painful on a rescan.

 

Also, your assumption about metadata location affecting performance is not strictly true.  We don't have to access the metadata files for normal operation (except for images but then even only sometimes).

Share this post


Link to post
Share on other sites
ebr

Cache and metadata location are two very different things.  Yes, the cache should be on an SSD if you have that option.  And, if you wish to, you can also store your metadata elsewhere.  It is just that there is a definite trade-off to that and one of those trade-offs is that it can potentially take much longer to rebuild your library in any instance where it may need to be done (and there are several beyond just this one).

Share this post


Link to post
Share on other sites
dcook

Cache is only good for stuff that you have viewed, you can't generate a cache for everything, so by having your metadata on fast SSH makes a big difference on speed, otherwise Emby has to contact the NAS or file share and wait until the drives spin up, which makes things slower.

 

If there was a way to build the entire cache first then I would agree that metadata could be moved to slower storage, but since that is not possible the best option is to keep cache and metadata on fast ssd storage.

 

Point is, that you should not need to rebuild your library if for some reason your NAS or Media share is "unavailable"  

Why would Emby just assume that if its unavailable its ok to delete everything?

 

It should skip and retry later.

Only if an Admin actually goes in a removes a folder from a library or an entire library should the media be deleted.

 

 

 

Cache and metadata location are two very different things.  Yes, the cache should be on an SSD if you have that option.  And, if you wish to, you can also store your metadata elsewhere.  It is just that there is a definite trade-off to that and one of those trade-offs is that it can potentially take much longer to rebuild your library in any instance where it may need to be done (and there are several beyond just this one).

Edited by dcook

Share this post


Link to post
Share on other sites
drashna

Bump.

 

Because this still isn't implemented (as far as I'm aware). 

 

I agree we can improve how this is handled now but you can minimize the pain by storing your metadata with your media.  That will make total re-scans much faster.

 

ROFLMAO

 

/cries

 

 

I have roughly 200 movies (and growing) and 700 tv shows on my server.    Worse is that there is *something* funky going on with it (for a while now) that causes it to hang, which causes the shares to be inaccessible until I reboot the system.    This happens inconsistently, and there doesn't appear to be a specific trigger.  (and no, reinstalling isn't an option). 

 

The worse part, is that this causes the shares to appear offline, and then Emby starts scanning the library and destroys the library. 

 

And then I have to rescan the library, which takes close to a week to complete.  A WEEK. 

 

Because of this .... I've gotten to the point where I have a script to copy the db files (SQLite, I believe) to another location weekly, and then will just copy over the backed up databases to the original location and hope for the best. 

  • Like 1

Share this post


Link to post
Share on other sites
Happy2Play

Bump.

 

Because this still isn't implemented (as far as I'm aware). 

 

 

ROFLMAO

 

/cries

 

 

I have roughly 200 movies (and growing) and 700 tv shows on my server.    Worse is that there is *something* funky going on with it (for a while now) that causes it to hang, which causes the shares to be inaccessible until I reboot the system.    This happens inconsistently, and there doesn't appear to be a specific trigger.  (and no, reinstalling isn't an option). 

 

The worse part, is that this causes the shares to appear offline, and then Emby starts scanning the library and destroys the library. 

 

And then I have to rescan the library, which takes close to a week to complete.  A WEEK. 

 

Because of this .... I've gotten to the point where I have a script to copy the db files (SQLite, I believe) to another location weekly, and then will just copy over the backed up databases to the original location and hope for the best. 

 

Hmm, I know everyone's setup is different but a week​ is extreme. 

Share this post


Link to post
Share on other sites
drashna

Hmm, I know everyone's setup is different but a week​ is extreme. 

 

Yeah, well, maybe not a week.  But still 3-5 days, depending.  

 

But this is approximately 60TB of media, now.  

Share this post


Link to post
Share on other sites
ebr

Yeah, well, maybe not a week.  But still 3-5 days, depending.  

 

But this is approximately 60TB of media, now.  

 

And that is storing the metadata with the media?

Share this post


Link to post
Share on other sites
drashna

And that is storing the metadata with the media?

 

Yes.  All the metadata is stored locally.  Images, NFO files, etc.  

 

The "Save artwork and metadata into media folders" and "Download images in advance" options are enabled.

And I'm not sure if this is the problem ... or not, but the the chapter image extraction options are enabled. Including the "during library scan" option. 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...