Jump to content


Photo

We need a simple REAL 100% Migration & Why save metadata with media is bad!

migration metadata nfo database

  • Please log in to reply
30 replies to this topic

#1 kaizo OFFLINE  

kaizo

    Member

  • Members
  • 10 posts
  • Local time: 09:58 AM

Posted 20 October 2018 - 05:45 AM

n7ATcub.jpg
 
Let's talk about a real full migration and the separation of metadata and mediafiles.
 
I know i know, you system architects of emby have had discussions and thoughts about this topic enough i guess...But i want to ask you to maybe rethink one or two things about the structure of emby.
 
But hold on, before you think: great, another non-programming average Joe who thinks he knows things better than we do... just let me explain.
 
The fact that i am here right now, writing this post about emby is because: i love it  :wub: 
As a long time kodi-user i instantly fell in love with emby the moment i installed it and started using it.
 
I mean, kodi gives me some more options and fine-control on how I want to structure my library where emby is limiting me, and emby has some little flaws here and there. But the overall feeling about emby is just awesome!
 
So, i ditched Kodi for emby and never looked back...until now.
I'm in a similar position as @Kirk137 in his post https://emby.media/c...ation-strategy/
or like @flort in this post https://emby.media/c...-changed-paths/
Due to some hardware and software-changes, i'm in the process of having to migrate from a emby macOS installation to a Ubuntu-based Linux installation.
 
Even though my case is about macOS/Linux-migration, my problem is related with the general migration-process and metadata-management, and therefore belongs in the "general"-forum.
 
So, as besides of the root-path from my mediafiles, nothing has changed and i thought: this won't be a big deal...just copying the database, correcting the rootpath for the mediafiles and boom it's done. Should only take one hour at best. But this was thought too short.
 
I followed the instructions from https://github.com/M...iki/wiki/Backup
and was a little shocked when i read this in the documentation:
 



This will not backup library contents and metadata. To keep a permanent copy of metadata, we suggest enabling saving of local metadata to media folders.

 
How can it be that a software like emby, so powerful and well written/supported, has no proper 100% backup/migration-process?
 
I followed the instructions for a manual backup anyway and now i have a "partially" migrated/backed up new database. After the re-indexing scan of 1-2 hours, the files are there and users and everything else - but the manually edited, handcrafted movie-entries are ALL gone. And this is a huge bummer as i putted in A LOT OF TIME identifying the movies/videos/shows by hand while remaining their original filenames!(sometimes this is possible sometimes not)
 
So, the whole migration-process took 4-5 hours by now and still isn't complete/finished and satisfying. And right now i don't know where to go or what to do to get this done.
 
I for myself and i don't think i'm alone in this - need a proper way to backup and migrate a whole installation. And with whole installation i mean whole. This should not only include:

  • Server configuration
  • Users
  • User data (playstates, favorites, etc).
  • Installed plugins
  • Plugin settings
  • Playlists

but also:

  • library/index
  • metadata
  • views
  • full cache(people,pictures,movieinfos,artistinfos)
  • Manually edited, handcrafted movie-entries.

When nothing besides the rootpath of my mediafiles and the paths to the config files have changed, i don't really understand nor can't i find good reasons why i should have to put my hardware, and my nerves to so much stress(io-workloads) re-indexing, re-downloading, re-editing and re-configuring the whole library when the data are already available somewhere(old machine).
 
I mean, i can understand that from a programmers perspective a migration/backup-process is no simple task. But from a users-perspective it should not be extremely more than:
2kje55.jpg
 
Ok, ok, i know. It's not just a single button-press and everything is magically done in 2 minutes. And i am fine with copying config files by hand, and running well documented pre-written SQL-queries etc etc. But in the end a migration/backup-process should be 100%, and not just 50-60%-ish.
 
And now comes the part about emby, that i completely dislike and reject: The structure of handling data and media.
 
The idea about saving metadata with the mediafiles might seem appealing and easy first of all...but when you think about it its not that great at all. Here's why.
 
I don't want any tool than my file-manager and my brain to get in the way of my structure and naming-conventions. Therefore i only mount my media-folders read-only for emby, kodi and other tools. I do this because i don't want any bug, user or coincidence to threatening me with the possibility of messing, deleting, renaming or restructuring my media-files/folders.
 
I need to preserve the structure and names of my files and folders to some degree for archiving-and a lot of other reasons. So i don't want emby or other tools to write nfo-files all over the place or rename anything.
 
Yes i know about the possibility of recognizing files by hash even when the filenames have changed, but this is no option as i need to preserve the original filenames. When you think about it, why would you want mix up two categories of datatypes(data and media)? For me personally it feels way more natural when i have the two separated.
I have one place for data and one for media.
 
The other really big problem i have is when this idea of "saving local metadata to media folders" is the only possible way of a given migration-process. Without another given option it feels like getting told: You ether swallow our nfo-mess in your file-structure or you have to redo the whole indexing and editing-process with every crash or system-migration! Ouch!
 
Most of other media-libraries like itunes, kodi, subsonic etc. separate the two datatypes while maintaining full migration-ability.
 
Sometimes emby is also forcing me to rename files/folders just to get indexed correctly. For example when episodes of a show are named E01 instead of EP01...or when a show/anime hasn't episode-numbers splitted by seasons etc.
 
While you are reading this, you might be thinking: What do you want from me?!
Let me tell you:

  • First of all, i need help. Medic please! I need to get this migration-process done without the need of having to re-edit all the rare movies and videos i edited by hand(took me  weeks to months!) I need a way(detailed instructed how-to) on how to migrate 100% of my library from one machine to another.
  • Give us the ability to just change the root-path of already indexed files after a migration. This should avoid the need of reindexing. Maybe implement a routine that searches for filenames in case of files are restructured(but saves the metadata until the user deletes the metadata).
  • I please you to rethink/rewrite/expand you backup/migration-process. So it becomes easier, complete and hassle/io-stress-free.
  • I please you to overthink your thoughts on the handling of data (metadata,user,movie etc) and mediafiles. A clear separation is needed imho. A separation would also make a full true migration-process easier i guess ¯\_(ツ)_/¯
  • Please give us the ability to index custom naming-conventions without the need of renaming files and or folders. Kodi is great starting point as it has given me this possibility.

I'm pretty sure that i missed a lot of thought's i had but this is already long enough.
I hope i described everything clearly, if not don't hesitate to ask me questions.
I also hope that other users voice their opinion about this issues.
 
Thank you for emby and for your attention.
Cheers kaizo


Edited by kaizo, 20 October 2018 - 07:48 AM.


#2 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 2936 posts
  • Local time: 04:58 AM

Posted 20 October 2018 - 07:59 AM

First keep in mind that the software can't be 100% for everyone as we all have different needs.
Without getting into a debate (not my point) but if you did save meta-data to the media folders you wouldn't be here with this post as your new install would pick up this information during a re-scan.
 
That's sort of the point of this feature to allow you to rebuild the database and keep your custom meta-data.  It also allows you to use 3rd party tools that can change poster art and such outside of Emby which can make life easy.
 
With that said even in your present situation you can still do the migration if you copy all the proper directories to the new host.  You can MANUALLY run some queries against your DB to change paths and point things to their new location.  You just need to know some basic SQL.  The database is not very big or have many tables so this is pretty easy.
 
Except for any possible SQL changes the instructions located at https://github.com/M...iki/wiki/Backup cover what needs to be done for a migration.  Those directions are pretty spot on if the file locations stay the same.  If they change locations then the SQL update commands you write will help to change "old location" to "new location".
 
You should not loose any data if done correctly like this so the concerns you have with keeping said old data/meta is not a concern as that will migrate.


#3 kaizo OFFLINE  

kaizo

    Member

  • Members
  • 10 posts
  • Local time: 09:58 AM

Posted 20 October 2018 - 11:02 AM

First keep in mind that the software can't be 100% for everyone as we all have different needs.

 

Exactly, you absolutely have a valid point with this. Because of the different needs, that's exactly why there can't and shouldn't be a "one-size-fits-them-all"-way.

Give people the choice if they want to save metadata in database or with the mediafiles (and still have the possibility of a proper full-migration), and everyone can be happy. But having no choice at all, is a disadvantage in my perspective.

 

No matter if i we both are on the same side of how to store metadata, i think software and or systems in general should adapt to the needs of the users, and not the other way around.

I mean, of course, you always have to change your own behavior to some extent to work with some new system, and i think i already did this.

 

Before emby i had folders with mixed content for animes, comedy, shows, movies, documentaries etc...even with different season markers for episodes (s01ep02, s01e2,ep1-ep199 etc)

and it worked absolutely well with kodi. I had to identify every movie and show by hand, but my file-structure stayed untouched! Yeay!

 

Then came emby, and mixed content folders was a mess...so i had to change the folder structure to divide between movies and shows...fine.

Some shows were not picked up for correct indexing because the show wasn't ordered in seasons, or only had "e01" marking instead of "s01e01". So i began to change the filenames and startet to sort some animes in seasons even though they aren't intended to so...i did this, fine.

When there is a movie that has the year at first position of the title in the filename, emby doesn't pick that up correctly...so i have to change the filename, or edit the entry by hand...i did this, fine.

Not enough that emby forces me to switch my file-structures and non-destructively renaming files, now it want me to spit metadata into my files and folders? For the sake of making a "smooth" migration/backup?

 

NO! This is where i stop to bend myself and my habits only because emby wants me to.

And i think when the creators of emby want to earn money, my respect and grow a great userbase they should make us all at least a little happy - not just the people that reflect their own opinions.

 

nIbSc7i.png

 

Without getting into a debate (not my point) but if you did save meta-data to the media folders you wouldn't be here with this post as your new install would pick up this information during a re-scan.

 

No, not really. I would still be here posting my request because of the cluster-shoot spitting metadata into the mediafiles-structure. Just to be clear about the "migration"-process. The described "migration"-process from the wiki is more like a rebuild-process than a migration.Because it involves a complete re-indexing instead of a quick re-scan as you stated. I say that a good/true migration-process doesn't need a complete re-indexing of the whole library. A wizard that helps me with the change of the root-paths followed by a quick re-scan(path correct) should be all it takes imho. Also could the pick up of this information could also easily came from the database instead of the file-structure, so that's not a good argument.

 

 

That's sort of the point of this feature to allow you to rebuild the database and keep your custom meta-data.  It also allows you to use 3rd party tools that can change poster art and such outside of Emby which can make life easy.

 

I would rather call this a flawed system with missing options than a "feature". As stated above, a true migration should work without rebuilding the database. My custom meta-data could also be saved into the database instead of the filesystem of my mediafiles, so ¯\_(ツ)_/¯. 3rd party tools can also read and write to/from the database instead of the filesytem. Why should the life for 3rd party tools be easy, but complicated for me as a user when i try to migrate?

 

 

Except for any possible SQL changes the instructions located at https://github.com/M...iki/wiki/Backup cover what needs to be done for a migration.  Those directions are pretty spot on if the file locations stay the same.  If they change locations then the SQL update commands you write will help to change "old location" to "new location".

 

Sadly that's not the case...my custom made entries are gone on the new machine after i followed the instructions :( I still have the old machine and database working, but i don't know where to start or what to do exactly as the wiki only tells me: save metadata with mediafiles for migration, or leave.

 

 

With that said even in your present situation you can still do the migration if you copy all the proper directories to the new host.  You can MANUALLY run some queries against your DB to change paths and point things to their new location.  You just need to know some basic SQL.  The database is not very big or have many tables so this is pretty easy.

 

Awesome :o If such a complete database migration process is possible, why it isn't documented in the wiki or anywhere else? As i know nothing about SQL or the structure of databases, could you guide me thru this? I mean i know, that this probably isn't a big deal for you as devs/tech-people...but for a normal user it totally is. There should be a wizard for a full system/database migration for non tech-users in my opinion. At least a full guide with all the necessary queries, would be nice.



#4 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 20 October 2018 - 01:40 PM

With new installations on 3.6+ you should be able to move the server around as much as you want.


  • kaizo likes this

#5 kaizo OFFLINE  

kaizo

    Member

  • Members
  • 10 posts
  • Local time: 09:58 AM

Posted 20 October 2018 - 05:33 PM

With new installations on 3.6+ you should be able to move the server around as much as you want.

Sounds awesome :) does this new freedoms implicate saving metadata with mediafiles or is this feature also possible with separated metadata?

 

But wait, does such big changes in the db-structure mean that everyone who wants to upgrade to 3.6+ needs to re-index/rebuild the library?

Please let me know, because if this is the case i won't migrate until the release of 3.6.

Btw was an eta for the 3.6 mentioned anywhere before?

 

Thanks in advance



#6 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 20 October 2018 - 05:36 PM

We don't have ETA's, sorry, and yes, there will be an upgrade procedure that happens on first run.


  • kaizo likes this

#7 tdiaz OFFLINE  

tdiaz

    Member

  • Members
  • 14 posts
  • Local time: 12:58 AM

Posted 26 October 2018 - 08:06 PM

Whats interesting is .. I effectively migrated from one server to another by installing the newer FreeBSD release, started it, entered a username/connected the server to my Emby account.

 

Quit it,

Moved /var/db/emby-server to a new location (actually, created a symbolic link and placed the actual directory on a server) .. restarted it and viola.

 

Upgraded from 3.3.x to 3.5.x .. and moved all the stuff. Done. Easy peasy. :)



#8 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1996 posts
  • Local time: 01:58 AM

Posted 26 October 2018 - 08:19 PM

Enhance the migration tools all you like.  Metadata with media is superior.  If my metadata is with my media not only do I solve almost everything these enhancements can bring, but I can switch from Emby to any other tool I like.  While that last isn't likely at the moment, I like having the option.



#9 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 26 October 2018 - 08:37 PM

Yes the newer your server installation is, the easier you will be able to pick up and move it as improvements have been made along the way. Really old installations may have some issues. Thanks.

#10 Lyfesaver OFFLINE  

Lyfesaver

    Advanced Member

  • Members
  • 242 posts
  • Local time: 03:58 AM

Posted 27 October 2018 - 06:35 AM

[DELETED BY POSTER] 


Edited by Lyfesaver, 27 October 2018 - 07:57 AM.


#11 kaizo OFFLINE  

kaizo

    Member

  • Members
  • 10 posts
  • Local time: 09:58 AM

Posted 15 January 2019 - 05:43 AM

 

With that said even in your present situation you can still do the migration if you copy all the proper directories to the new host.  You can MANUALLY run some queries against your DB to change paths and point things to their new location.  You just need to know some basic SQL.  The database is not very big or have many tables so this is pretty easy.
 
Except for any possible SQL changes the instructions located at https://github.com/M...iki/wiki/Backup cover what needs to be done for a migration.  Those directions are pretty spot on if the file locations stay the same.  If they change locations then the SQL update commands you write will help to change "old location" to "new location".
 
You should not loose any data if done correctly like this so the concerns you have with keeping said old data/meta is not a concern as that will migrate.

 

 

I took the time and did what cayars said. I changed the paths thru sqlitebrowser(yikes) by hand via some googled queries.

It was a mess, it wasn't any fun and at the end it didn't work out. After the "migration" of this "changed-by-hand" db none of the media-entries in the library can be found anymore.

So, im stil stuck on the status quo old machine, and noting has changed.

 

No user should have to jump through hoops simply for a migration process!

 

But not enough...as i once accidentally mounted the medialibrary writeable instead of read-only, emby began to overwrite my original NFO-files and puke new NFO files across my whole medialibrary.

Emby did this because after an update "save metadata into mediadirectories" was enabled in the backend by itself.

Some updates ago i only had the option "Metadatastorage: NFO" disabled(no nfos were written). After an emby-update(dont remember which) the option "Save Artwork and Metadata into Media Folders" was added and enabled by default without asking me. Emby began to start the nfo-mess and when i recognized i disabled everything i could find about nfo-metadata in every media folder. Im majorly pissed right now!

 

How should anyone trust your software when it messes things up like this? This is unacceptable!

 

I was going to get a lifetime premiere membership but after this frustrating and unproductive experience here and the lack of interest from the team/support for the metadata/migration problematic, i won't support this project with my money. I as a user, feel left alone with the problems your software produces and feel ignored about some really important missing options/functions.

 

ATM i feel better off using my money to support bug bounties for forks of this software instead of giving it to people who don't seem to care about my needs as a user.

Maybe i will come back at some point in time and look if anything has changed in this project.

 

Hopefully the ignorance will be gone by that time.

Good luck to anyone


Edited by kaizo, 15 January 2019 - 08:19 AM.


#12 Spaceboy OFFLINE  

Spaceboy

    Advanced Member

  • Members
  • 3870 posts
  • Local time: 09:58 AM

Posted 15 January 2019 - 08:39 AM

The problem is you are in the vast minority so unless you can bring others around to your way of thinking, which you haven’t yet, your request will be quite near the bottom of an extremely long list. If you decide emby is the tool for you based on how it actually works then I would say fitting to “the system” for a one time setup is worth it

#13 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 02:08 PM

 

 

No user should have to jump through hoops simply for a migration process!

 

I agree, and going forward, if you install 4.0 fresh, you will be able to pick up the installation and move it around as much as you like.



#14 Perplexed OFFLINE  

Perplexed

    Advanced Member

  • Members
  • 73 posts
  • Local time: 01:58 AM

Posted 15 January 2019 - 02:15 PM

I was going to get a lifetime premiere membership but ...

 

I feel for your ordeal, but honestly, I wish I had a dollar for every time I read this sentence (I would have bought the product if only... fill in the blanks) in any "subscription optional" based forum I have every visited... I'd be swimming in a Scrooge like money bin for sure...


  • d00zah likes this

#15 BAlGaInTl OFFLINE  

BAlGaInTl

    Advanced Member

  • Members
  • 697 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 03:48 PM

I agree, and going forward, if you install 4.0 fresh, you will be able to pick up the installation and move it around as much as you like.


So in essence, this solution already exists. Just not a direct path to it from old versions.

@kaizo, so they aren't ignoring... the improvement you are seeking is already there.

Everything that you are describing that you don't like that Emby does, to most, is the essence of what a media server does. It organizes and updates everything automatically. Seems like what you want is just a media viewer instead of a complete management system. Nothing wrong with that, it just isn't what Emby is. Setting everything to read only by Emby eliminates many of its finest features. One of my biggest use cases is as a PVR. That wouldn't be possible if Emby was read only.

#16 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 03:50 PM

Right, ideally, this would have been possible with older versions, but we have found issues with them relating to this. Thanks.



#17 kaizo OFFLINE  

kaizo

    Member

  • Members
  • 10 posts
  • Local time: 09:58 AM

Posted 15 January 2019 - 05:13 PM

I agree, and going forward, if you install 4.0 fresh, you will be able to pick up the installation and move it around as much as you like.

Just to be totally clear here.

You can confirm that emby from version 4.0 ongoing full fill the following conditions:

  1. Does not write or overwrite nfo files into my media source folders.
  2. Does not activate the "feature" "save metadata with media files" by itself/automatically (accidentally or intentionally)
  3. Gives the ability for a full proper migration (full sqlite db, watch status, users, settings, metadata - the whole emby installation without media files) from one system (ie Macos) to another system(ie Linux) whitout the need of write or overwrite nfo files into my media source folders
  4. Let me run a setup with mediafiles and metadata clearly separated (Don't forces a metadata handling model on me for the argument of maintaining a healthy migration-ability)

If this is the case then i have to admit that i would be positively surprised! But there would be one single question left. I have red today anywhere in a comment that emby is going closed source? If this is true, that would sadly be dealbreaker to me. Is this really true, and if so - why?

 

I feel for your ordeal, but honestly, I wish I had a dollar for every time I read this sentence (I would have bought the product if only... fill in the blanks) in any "subscription optional" based forum I have every visited... I'd be swimming in a Scrooge like money bin for sure...

Thanks for your sympathy. And haha, i really had to laugh when i have red your post. Your post was really funny, even though i am totally for real. Just bought a subsonic license lately, and i put my money where my mouth is. But i agree, a lot of people say this. But not everyone seems to be for real.

 

@kaizo, so they aren't ignoring... the improvement you are seeking is already there.

Everything that you are describing that you don't like that Emby does, to most, is the essence of what a media server does. It organizes and updates everything automatically. Seems like what you want is just a media viewer instead of a complete management system. Nothing wrong with that, it just isn't what Emby is. Setting everything to read only by Emby eliminates many of its finest features. One of my biggest use cases is as a PVR. That wouldn't be possible if Emby was read only.

Yeah i get what you mean with your idea of emby "finest features", but to be honest - i dont use PVR, live-tv or something like that and i couldn't care less. Emby/Kodi is the only reason i dont have to watch TV anymore. But that is another topic and i don't want to get into right now. And no, a simple media-viewer would not meet all my requirements. Emby is good, Kodi seems to meet a lot more functions BUT lacks the feature of a web-interface and server/client-model, so...


Edited by kaizo, 15 January 2019 - 05:28 PM.


#18 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 05:41 PM

 

 

  1. Does not write or overwrite nfo files into my media source folders.
  2. Does not activate the "feature" "save metadata with media files" by itself/automatically (accidentally or intentionally)

 

Only if you enable nfo metadata saving, which is off by default.



#19 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 05:42 PM

 

 

  1. Let me run a setup with mediafiles and metadata clearly separated (Don't forces a metadata handling model on me for the argument of maintaining a healthy migration-ability)

 

Well we've always had this. This is the default and has always been. Local metadata saving has always been off by default.



#20 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 136138 posts
  • Local time: 04:58 AM

Posted 15 January 2019 - 05:45 PM

 

 

  1. Gives the ability for a full proper migration (full sqlite db, watch status, users, settings, metadata - the whole emby installation without media files) from one system (ie Macos) to another system(ie Linux) whitout the need of write or overwrite nfo files into my media source folders

 

You cannot move the installation across operating systems because there are native binaries built specifically for each OS & cpu architecture. What you can do is move the contents of the server's program data folder, since that's really what you're after anyway.







Also tagged with one or more of these keywords: migration, metadata, nfo, database

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users