Rollback Update / Recovery Mode

[Lighthammer 80]

Per request from Luke, I am breaking up the suggestions from this thread:

https://emby.media/community/index.php?/topic/80816-cumulative-suggestions-from-lighthammer-end-of-2019/

 

I'll add additional information for each topic as needed.

 

Rollback Update / Recovery Mode

Over the years, we've had some serious server breaking updates that have crippled major parts of the server. While we *DO* have the backup feature for those of us with Premium Access, I think Emby is in dire need for a easy-to-access recovery mode that allows rollbacks to a point before updates.

 

Just about all the tools are already in place to do this, save the interface.

 

I believe moving forward, Emby should create a restore point before updates are applied regardless of your membership status.

The only "real work" needed to get this off the group is to create a simplified rollback interface and to drop it in the dashboard.

 

Additionally, Backup really needs a home in the Administration Panel without having to go look for it under Plugins. I'd probably agree this should/could remain a premium feature, but it probably should be moved away from the Plugins Panel with additional coding saying something to the effect of "Get Premium to access this feature"

 

[Happy2Play 8282]

But wouldn't your OS already do this for free assuming you have a backup plan in place?  

 

For this recovery mode to work a complete backup of your Emby-Server programdata folder would need to be maintained as some updated have database changes that are destructive with no rollback option do to changes.

 

Everyone's setup will be different depending on metadata/image saved locations, so recovery sizes will vary.  Then the amount of time this would take to recover/rollback potentially several hundred thousand files and folders.  My primary server has a 75Gb metadata folder alone.

[Dibbes 431]

But wouldn't your OS already do this for free assuming you have a backup plan in place?  

 

For this recovery mode to work a complete backup of your Emby-Server programdata folder would need to be maintained as some updated have database changes that are destructive with no rollback option do to changes.

 

Everyone's setup will be different depending on metadata/image saved locations, so recovery sizes will vary.  Then the amount of time this would take to recover/rollback potentially several hundred thousand files and folders.  My primary server has a 75Gb metadata folder alone.

 

I just spent two weeks recovering from a failed SSD where the backup plugin proved to be more of a hindrance than a time-saver, so yes, I'd love a proper recovery tool...

 

Also, a roll-back to a previous server version shouldn't necessarily have to involve metadata. Just server and plugin versions.

[Happy2Play 8282]

I just spent two weeks recovering from a failed SSD where the backup plugin proved to be more of a hindrance than a time-saver, so yes, I'd love a proper recovery tool...

 

Also, a roll-back to a previous server version shouldn't necessarily have to involve metadata. Just server and plugin versions.

Well if there are database updates that cause rescan/rebuilds then a full restore is needed as it affects item ids in the database which affect all /metadata/library items.

 

So databases along with metadata are needed, so a a full folder restore.  So in some cases can be say 75+Gb backup sizes to include several minutes for backups and restores of several hundred thousand files and folders.  In my case almost a million files.

[Dibbes 431]

Well if there are database updates that cause rescan/rebuilds then a full restore is needed as it affects item ids in the database which affect all /metadata/library items.

 

So databases along with metadata are needed, so a a full folder restore.  So in some cases can be say 75+Gb backup sizes to include several minutes for backups and restores of several hundred thousand files and folders.  In my case almost a million files.

 

In case of a database update, yes. but then this should be build into the install/upgrade process.

[Lighthammer 80]

But wouldn't your OS already do this for free assuming you have a backup plan in place?  

 

For this recovery mode to work a complete backup of your Emby-Server programdata folder would need to be maintained as some updated have database changes that are destructive with no rollback option do to changes.

 

Everyone's setup will be different depending on metadata/image saved locations, so recovery sizes will vary.  Then the amount of time this would take to recover/rollback potentially several hundred thousand files and folders.  My primary server has a 75Gb metadata folder alone.

 

So you want me to rely on doing a full restore on my OS in the event that the development of a single application has a problem?

 

Emby is already actively creating a system folder backup. This already seems like the next logical step.

 

Additionally, this has been a standard in web apps for a very long period. uBulletin is probably considered an aging CMS these days and have had it since 2012.

[Happy2Play 8282]

Well I can restore a folder not a entire OS if needed.  You have to have the entire programdata folder to do a true recovery, as this is the true way to recover from any issues you have in the update.  So to me it is zero difference then a OS backup or a true backup plan.

Edited December 29, 2019 by Happy2Play

[Happy2Play 8282]

In case of a database update, yes. but then this should be build into the install/upgrade process.

From a install process you have nothing so it does not apply.   Now a upgrade is another story as currently only the server is updated and your existing databases may change with that update which in turn affect all the central metadata.

[Dibbes 431]

From a install process you have nothing so it does not apply.   Now a upgrade is another story as currently only the server is updated and your existing databases may change with that update which in turn affect all the central metadata.

 

This is mincing words, mate... I see that I need to describe all interpretations of a word here otherwise the interpretation with the most ambiguity will be taken. Anyway, an upgrade is also installed, so yes, install process. Since the person who creates the package knows (or should know) if the server upgrade is going to do a database change, it's not far fetched to build in an option in the installer (or upgrader) to copy the current data to a to-be-specified location so that it can be restored with the same installer (or upgrader) or even from a separate tool (plugin or otherwise).

 

Honestly, I do not see what your reluctance is here. You reacted similarly to any issues I had with the backup plugin and I haven't even listed these all as feedback does not seem to be welcome on this specific topic.

[Lighthammer 80]

 if the server upgrade is going to do a database change, it's not far fetched to build in an option in the installer

 

I really want to highlight this very specific point.

 

Before applying an update, we really need to know what we're getting into and make some educated decisions weather we want to let the update happen or wait to see if there are any hiccups.

 

It's very unpopular to say, but its true to say that Emby has had some MASSIVE changes that broke how things work for some users/admins in the past. I think the record isn't as bad as Windows 10 at this point, but I've had some major hiccups.

 

I run my personal emby from a server that now has 500 TB. A hiccup for me can mean *A LOT* of lost data or *A LOT* of fixes needed.

 

Considering the number of times Emby has caused me massive issues that I had to spend a week or two in my free time fixing problems over the years has been pretty signifigant. Heck, Emby is ENTIRELY the reason that my server has a hot swappable server drive (which isn't as bad as having a hot swappable Boot Drive for when Microsoft pushes some crazy update that breaks everything thanks to their lack of QA).

 

But in a roundabout way, a signifigant problem that exists is that Emby doesn't have a large dedicated QA team and the team they do have, I'd wager, only does Smoke Tests. Its not like they spend weeks before a major release doing regression to make sure all the old bugs didn't crop up again. Feel free to correct me, but I don't think the Emby Team even uses production software or bug tracking software outside of various spreadsheets / notepad docs (if I'm wrong, I'll be surprised, but very happy).

[Lighthammer 80]

Well I can restore a folder not a entire OS if needed.  You have to have the entire programdata folder to do a true recovery, as this is the true way to recover from any issues you have in the update.  So to me it is zero difference then a OS backup or a true backup plan.

So, I want to take a moment and explain MY backup plan and illustrate how it is excessive for the average person grabbing Emby to make a small home media server:

 

- I have two M.Satas. The Primary is mirrored to the Secondary on boot. These act as my primary drives in this usecase is primarily for when Windows decided to push a bad update.

- Have two Satas dedicated to most of my server technologies (Emby being one of them). They are also mirrored on boot for when Emby explodes.

- I load Emby (and other servers) STRICTLY to Optane Memory and run EXCLUSIVELY from those (in case Emby explodes).

- I have Acronis Server Running in the background that keeps a backup copy of each Server Folder independently and maintains a platinum backup every 30 days and then maintains a rollback of each server every 3 days saved to a raid that's only purpose is backups.

- Acronis Server also does a backup of my primary drive every day.

- Other then that, I run Raid 5 with some specific folders running as a nested Raid 7.

 

So YEAAAAAAAAA ----

 

EXCESSIVE

 

But I want to stress that this is so incredibly excessive because I've had Emby (and Windows) blow up in some super significant ways over the years. Sadly, both Luke and Ebr have taken the brunt of it when an update messed something up really bad and I was really pissed over it.

 

My overall point is very few people in the community are going to have the resources to setup such a crazy server rack as I have and PROBABLY don't even know how to do most of whats setup.

 

As a community, we struggle with people even having a raid setup in many cases and I'm not even talking about JBODs, I'm talking about legit just assigning folders from various drives.

 

For a lot of people, Emby might be the first server they ever try to setup or manage. The community needs to do a lot better for these folks.

 

IIRC, I am now entering my 8th year running an Emby server. Besides User Groups being a paramount need for Emby, I think a rollback/recovery environment is something the server REALLY needs unless Emby is ready commit to having a MASSIVE QA team of at least 10 people (possibly 20; that seems like it would only be needed, at most temporarily) to ensure each release is absolutely flawless --- but even then, the ability to recover your installation needs to be there.

 

That being said, it does spring to mind that it might be easier to look at having the installer have a recovery button and have it physically do the backup / rollback if needed. I wouldn't even mind seeing the installer make a TAR or RAR file during the update with a simple UID.

Edited December 29, 2019 by Lighthammer