Jump to content


Photo

Emby Server - Preserve Watched Status of External-ID-less Files (such as Extras) When Moved

server watch watched move locations extras specials movie

  • Please log in to reply
18 replies to this topic

#1 funwithmedia OFFLINE  

funwithmedia

    Advanced Member

  • Members
  • 378 posts
  • Local time: 06:38 AM

Posted 06 September 2017 - 08:15 PM

Per another thread ( https://emby.media/c...ry/#entry484965 ) video files that do not have an external ID (eg, TheMovieDB, IMDB) associated with them in Emby will lose their Watched status if you ever move the files to another location. This is because currently the only way to determine the uniqueness of the file (if it does not have an external ID) is to use the full path to the file.

 

Now, most items will have an external ID, but the big exception can be Movie Extras and TV Specials. Some TV Specials will have an external ID. And some Movie Extras do exist in the IMDB and TheMovieDB, but currently there is no metadata lookup available for Movie Extras ( https://emby.media/c...kup-for-extras/ ). So if a Movie and it's associated Extras is ever moved, the Watched status for those extras will be lost.

 

This feature request is for there to be some better method of identifying files such that those files which do not have external ID's can still be correctly identified (and thus Watched status preserved) even if they are moved. I am most interested in this for Movie Extras.

 

I'm not sure what the best route for achieving this would be, but wanted to put the idea out here for consideration.

 

Thanks!

:)


  • jnheinz likes this

#2 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124564 posts
  • Local time: 06:38 AM

Posted 06 September 2017 - 10:55 PM

nfo is probably the best route, but nfo is only designed to hold watch data for one user. we'd probably want to add something proprietary into the file to hold data for all users.



#3 jnheinz OFFLINE  

jnheinz

    Advanced Member

  • Members
  • 143 posts
  • Local time: 04:38 AM
  • LocationBurnsville, MN

Posted 07 September 2017 - 10:21 AM

+1 to this.  I had to manually migrate my watched status for stuff like this.  This applies for sporting events too -- no external ID.  Even if were just the file name or an arbitrary ID generated for these, it would be better than the full file path. 



#4 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 43767 posts
  • Local time: 05:38 AM

Posted 07 September 2017 - 10:24 AM

+1 to this.  I had to manually migrate my watched status for stuff like this.  This applies for sporting events too -- no external ID.  Even if were just the file name or an arbitrary ID generated for these, it would be better than the full file path. 

 

What arbitrary ID could be tied back to the item when it is moved to another location?

 

The issue with trying to save this data with some other (not truly unique) ID would be that it would then make it possible for the watched data to be overwritten by something else that somehow matches the same ID.  IMO that would be a worse outcome than the issue here (which should be quite rare).

 

As far as saving in nfo, well, we've been down that path before and had to disable it due to all the problems we had with other programs wiping that data out and then people reported that Emby had "lost" their watched status.

 

So, I'm just not sure there is a perfect answer here but we are definitely open to ideas.



#5 jnheinz OFFLINE  

jnheinz

    Advanced Member

  • Members
  • 143 posts
  • Local time: 04:38 AM
  • LocationBurnsville, MN

Posted 07 September 2017 - 10:39 AM

The ID would have to have a prefix that would be unique to this endeavor.  Not just a random hash.  Definitely makes Emby feel a little less portable when watched status for everything can't be transferred.  Migration woes.



#6 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 43767 posts
  • Local time: 05:38 AM

Posted 07 September 2017 - 10:46 AM

The ID would have to have a prefix that would be unique to this endeavor.  Not just a random hash. 

 

That's what we have now.  It isn't a random hash it is something we are positive will relate to that exact file.

 

There is probably a way to skin this cat though and we don't have to discuss implementation specifics here.  Luke and I would have to figure it out if there is enough support for this feature.



#7 funwithmedia OFFLINE  

funwithmedia

    Advanced Member

  • Members
  • 378 posts
  • Local time: 06:38 AM

Posted 07 September 2017 - 11:11 AM

Glad to see there's interest in this. And I totally agree with the various points regarding the challenges of implementing this.

 

A thought I'll throw into the mix: If we created a hash value of each file (using something like http://corz.org/wind...tware/checksum/ ) we could be certain of the identity of the file (if we use a hashing algorithm that is free of collisions). The obvious downside of this is that this process is computationally expensive (especially the larger the file gets). So perhaps when a file is added (that has no external ID) it would start with the current location-based id, but then there could be a periodic server process/task that goes through and creates hashes of those files which then becomes the new, permanent ID (and we'd probably want to provide the user with a way to force this process manually). Then, if it finds that a hash-ID already exists in the DB it associates that already present DB data with this file. I would recommend the DB be setup such that if a user chooses to have multiple instances of the same file (which I've run into a few times with certain DVDs), which would thus all have the same file hash, that they all get linked to the same DB data (eg, If I watch the Extra in one place, it makes sense for it to show as Watched when a copy of the same exact Extra file is in another location).

 

Obviously, all of the above could be much further refined. And if it is helpful I'm certainly happy to brainstorm further with folks (I like finding creative, workable solutions to things). :)



#8 jnheinz OFFLINE  

jnheinz

    Advanced Member

  • Members
  • 143 posts
  • Local time: 04:38 AM
  • LocationBurnsville, MN

Posted 07 September 2017 - 11:17 AM

That was actually my first thought, an MD5 checksum or something similar.  Maybe could be performed using a scheduled task to mitigate the barrage of disk activity to generate the checksums at undesirable times?  As @funwithmedia suggested, I am just brainstorming too.



#9 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 43767 posts
  • Local time: 05:38 AM

Posted 07 September 2017 - 11:32 AM

Using a file checksum (as well as slowing things down) also will then lose the data when you replace the file with one of a different quality.  I think that is probably more likely than the situation we are trying to solve here (moving everything to new locations).



#10 funwithmedia OFFLINE  

funwithmedia

    Advanced Member

  • Members
  • 378 posts
  • Local time: 06:38 AM

Posted 07 September 2017 - 11:39 AM

Using a file checksum (as well as slowing things down) also will then lose the data when you replace the file with one of a different quality.  I think that is probably more likely than the situation we are trying to solve here (moving everything to new locations).

Fair point. I hadn't considered that.

 

Perhaps we're coming at this all wrong: Maybe instead we should be thinking about having a method/process whereby we can tell Emby, "Hey, I'm moving these files to this location -- please keep track of them"? I'd have to think some about how that might work, as I think a typical user behavior is to just move their files (independent of Emby) and then only go into Emby Server if something hasn't worked right with the move (ie, lost Watched status). Perhaps if there could be a user setting to have Emby retain data for old/missing items for a user-defined amount of time. And then we could use that data in conjunction with some sort of dialog for matching up moved items manually to old/missing items in the DB. That sort of manual process would go most quickly the less parts of the path one has changed (eg, if you've only changed the root, but not the later parts of the path, in theory you could match up larger batches at a time).



#11 bigeyez OFFLINE  

bigeyez

    Advanced Member

  • Members
  • 159 posts
  • Local time: 05:38 AM

Posted 30 December 2017 - 11:12 PM

​It would be very useful if Emby detects moved files (files that are moved within the library folders). It seems like I loose the watched tag of a movie that I watched  if I move the file to some another place :( So now I rtefrain from reorganizing my library because I am afraid that I will loose my stats.


Edited by bigeyez, 30 December 2017 - 11:12 PM.


#12 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124564 posts
  • Local time: 06:38 AM

Posted 30 December 2017 - 11:16 PM

We store watch data based on things like IMDb and movie db IDs. As long as the metadata remains the same thing should be able to move movies, TV and music.

#13 bigeyez OFFLINE  

bigeyez

    Advanced Member

  • Members
  • 159 posts
  • Local time: 05:38 AM

Posted 31 December 2017 - 12:34 AM

​Well, I never watch movies, shows, tv etc. I use Emby to watch some tutorials I download from Youtube and such. Does that mean that moves cant be detected?


Edited by bigeyez, 31 December 2017 - 12:34 AM.


#14 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124564 posts
  • Local time: 06:38 AM

Posted 31 December 2017 - 02:01 AM

yea those will be a little more difficult. storing watch data in nfo is always possible, although we've been down that road before and had problems.



#15 bigeyez OFFLINE  

bigeyez

    Advanced Member

  • Members
  • 159 posts
  • Local time: 05:38 AM

Posted 31 December 2017 - 01:46 PM

​Can the files get hashed to be uniq so it does not matter where they are?

 

yea those will be a little more difficult. storing watch data in nfo is always possible, although we've been down that road before and had problems.


Edited by bigeyez, 31 December 2017 - 01:47 PM.


#16 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124564 posts
  • Local time: 06:38 AM

Posted 31 December 2017 - 01:48 PM

It's possible yes.
  • bigeyez likes this

#17 funwithmedia OFFLINE  

funwithmedia

    Advanced Member

  • Members
  • 378 posts
  • Local time: 06:38 AM

Posted 06 January 2018 - 01:15 PM

This thread is possibly(?) a duplicate of the following related thread:

https://emby.media/c...ras-when-moved/



#18 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124564 posts
  • Local time: 06:38 AM

Posted 06 January 2018 - 06:04 PM

@bigeyez i have merged your topic into here.



#19 bigeyez OFFLINE  

bigeyez

    Advanced Member

  • Members
  • 159 posts
  • Local time: 05:38 AM

Posted 13 January 2018 - 04:32 AM

Why not provide moving mechanisms in the Emby (akin to delete) gui for now, until a better way is figued  out? That way Emby will already know the files are moved? So we would need some rudimentatry move and rename kind of thing.







Also tagged with one or more of these keywords: server, watch, watched, move, locations, extras, specials, movie

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users