What backup software?

[CharleyVarrick 277]

Hopefully @@MSattler can give you a few pointers.

@@CBers

hopefully, right now I don't know which of ARQ5 or Amazon is causing it.

Funny, I abandoned Crashplan 5 years ago after 3 days and getting 0.001% done, Had signed up for a yearly plan but they refunded me without a fuss.

Edited May 2, 2017 by jlr19

[CharleyVarrick 277]

Seeing how right now the ARQ5/Amazon combo doesn't cut it, I went on the Backblaze website and they have a speedtest to provide an ETA.

My ISP plan being 120/20, I was expecting similar results: Download is similar (and irrelevent) but upload barely reaches 5 or 6 mbps, with 80ms latency.  That would give me about 65g/day. Having maybe 27Tb to do, I'm looking at least 18 months.

I then went to speedtest.net for a 2nd opinion and got 130 down an 22 up (my isp does gives a 10% overhead). Lends me to believe that the backup servers I'm uploading to are too far away and/or throttling.

 

EDIT: I stopped (if I can call it that) ARQ5/Amazon, then setup CrashPlan and started a trial backup (just 1x 6tb drive with about 4.5Tb of stuff on it) to their cloud: I'm getting 3mbps with a 73 day ETA. About as bad as years ago. Back then I had a 30/10 plan. I'll let it run overnight and look at the ETA tomorrow morning.

 

What good is unlimited backup and decent internet plan when data trickles in to their data centers like this  ?

Edited May 2, 2017 by jlr19

[MSattler 387]

I am a bit puzzeled here about the backup strategy, I do the OS and the pertinent installations on the C-Drive of my NUC that is an HTPC, so that in a very short time replacing the M2 500GB drive and reinstalling the OS drive, that I can understand.

 

But my media servers have capacity of 48TB to preserve that I run UnRaid with two parity drives. Should up to two drives fail at any given time just place in 2 new drives and run the restore and I am back on line. Don't think it is worth backing up tens of TBs on-line. Wether it is worth to have tens of TBs is a different story.

 

 

If it costs me $60 a year via Amazon Cloud Drive, why not?  Especially if you are already using it as a off-site backup for important things like family albums, documents, etc.

[MSattler 387]

Seeing how right now the ARQ5/Amazon combo doesn't cut it, I went on the Backblaze website and they have a speedtest to provide an ETA.

My ISP plan being 120/20, I was expecting similar results: Download is similar (and irrelevent) but upload barely reaches 5 or 6 mbps, with 80ms latency.  That would give me about 65g/day. Having maybe 27Tb to do, I'm looking at least a year.

I then went to speedtest.net for a 2nd opinion and got 130 down an 22 up (my isp does gives a 10% overhead). Lends me to believe that the backup servers I'm uploading to are too far away and/or throttling.

 

EDIT: I stopped (if I can call it that) ARQ5/Amazon, then setup CrashPlan and started a trial backup (just 1x 6tb drive with about 4.5Tb of stuff on it) to their cloud: I'm getting 3mbps with a 73 day ETA. About as bad as years ago. Back then I had a 30/10 plan. I'll let it run overnight and look at the ETA tomorrow morning.

 

What good is unlimited backup and decent internet plan when data trickles in to their data centers like this  ?

 

Interesting, sometimes Amazon is a tad slower, but I have no issues backing up.  I wonder perhaps if it has something to do with where you are located and which backup pod you are being put on?

 

I have noticed on the VM it runs slower.  The other downside is every time you start a backup, it scans through everything it has already backed up, so you don't want to interrupt the scans very often else it spends a ton of time scanning.   

[CBers 6771]

The other downside is every time you start a backup, it scans through everything it has already backed up, so you don't want to interrupt the scans very often else it spends a ton of time scanning.

So it's tree-walking every time.

 

Is there no way of disabling that? There should be an option somewhere.

 

I assume these are incremental backups each time?

[CharleyVarrick 277]

@@CBers

ARQ has limited options, none ressembling "stop scanning the damn files and get on with uploading"

Does Amazon have any options?

 

@@MSattler

It indeed looks like it's location related, I'm in Canada, not too far from Montreal.

To answer your other questions (from the other thread)

Nothing else was using network

ARQ5 version I have is 5.7.13

 

I have resumed ARQ/Amazon backup, right now I have them and Crashplan running simultaneously on same hard drive.

I'll let them run side-by-side a bit and see if one seems to pickup its game.

 

Windows Task Mgr still reporting a mildly fluctuating 2.4% network usage

Before starting, I was worried online backup would bring my internet connection to its knees (make it unusable for other tasks), I was ready to choke it down (scheduling), but as it stands, it's barely making a dent.

[CharleyVarrick 277]

@@MSattler

Would you know of any way I could pinpoint if the bottleneck is ARQ or Amazon

 

About ARQ5, I wish it would provide a speed reading and possibly an ETA, those are pretty basic infos left out.

Unless Amazon can provide something in those lines (?)

[CharleyVarrick 277]

After discussing my issues with coworkers already using Amazon, one of them recalled he had to try a few backup clients before getting decent performance.

As he's a Linux penguin, I didn't asked what client he ended up with.

Anyone has experience (good or bad) with the following:

Needs to be unlimited, cloud compatible, and preferably freeware.

 

 

EDIT: Found this link, with many more alternatives

https://www.lifewire.com/free-backup-software-tools-2617964

Edited May 6, 2017 by jlr19

[CharleyVarrick 277]

I have resumed ARQ/Amazon backup, right now I have them and Crashplan running simultaneously on same hard drive.

I'll let them run side-by-side a bit and see if one seems to pickup its game.

ARQ5 completed a 500gb test backup in three and a half day. That's about 143gb/day

Crashplan meanwhile has managed to backup 243gb in five days. A dismal 48gb a day...

 

While Crashplan is definitely user-friendly and well established, it (still) seems unfit for larger backup, unless one doesn't mind completing an initial backup in months if not years, if ever.

 

As for ARQ5, the interface is crude, and while it does performs faster than Crashplan (a small feat), there's room for improvement.

I just don't see myself paying 50$ now for what looks (to me) like a very unpolished product.

 

I am still studying the list of clients (above post). I'll start new test backups, favouring open source freeware solution. Reports to follow.

Edited May 6, 2017 by jlr19

[CharleyVarrick 277]

I started a Duplicati to Amazon 500gb test backup. Duplicati has a modern web interface, It doesn't look like its doing much but I do monitor 50mb data chunks piling up every minute on Amazon's side. Task Mgr reports merely 1% peaks network usage; that can't be good.

 

 

Duplicati is donation-ware, optional. Read somewhere that its support level is based on donation size.

Edited May 6, 2017 by jlr19

[CBers 6771]

@@jlr19 Have you looked at pure cloud syncing software, such as Google Drive, Microsoft's One Drive or Dropbox?

 

As mentioned elsewhere, The is a sister app to DriveBender called CloudXtender.

 

You can have multiple cloud accounts attached to CloudXtender. Have a read of the link.

 

Unless of course you want complete FULL backups in the cloud.

[CharleyVarrick 277]

Unless of course you want complete FULL backups in the cloud.

Yes, full cloud backup, and having presently around 27tb, I would need an unlimited plan.

[CBers 6771]

Yes, full cloud backup, and having presently around 27tb, I would need an unlimited plan.

Really? Not just the latest?

[CharleyVarrick 277]

@@CBers

Latest what?

 

While CloudXtender does appear to have some of the features I am looking for, I'm not sure it can be considered a backup to cloud client per say.

[CBers 6771]

@@CBers

Latest what?

Latest copy of your data.

 

By FULL backups, I mean multiple point-in-time copies of your data, as opposed to LATEST only, which is the latest copy of a file that has changed.

 

I only keep LATEST copies of my data using ROBOCOPY.

[CharleyVarrick 277]

Oh, sorry, I haven't had my 2nd cup of jo yet.

That's a very good question you're asking me, which I have given little thought to yet.

 

I would venture it would be a mix of full and latest:

Media files could be just latest, as they're (usually) stable, at least the main file. nfo seems to be updated all the time (judging by my own experience with Emby)

I'm not sure I need 15 nfo versions in backup, not that they take up much place.

 

Not sure how this feature is called, but if I delete (voluntarily or not) the original file, I don't wan't the backup copy to be automatically deleted as well.

Otherwise, what's the point in backup, right?

 

Other files types could/should be full (versioning)

Edited May 6, 2017 by jlr19

[CBers 6771]

I thought that's what you'd say

 

Copying complete datasets between local servers is easy, but to the cloud is dependent on the internet and everything in between.

 

How many FULL backups would you want to keep in the cloud and for how long?

 

What frequency do you want and would you start a backup only when the previous one completed?

 

Perhaps you could look at a monthly FULL backup and daily or weekly INCREMENTAL backups for speed.

 

So many questions; so many answers

[CharleyVarrick 277]

Perhaps you could look at a monthly FULL backup and daily or weekly INCREMENTAL backups for speed.

When you say full monthly backup, I'm unsure what it exactly means.

Say I have a 4gb "static" movie file. I wouldn't want nor expect a duplicate copy of it every month.

I am guessing a FULL monthly backup does only what's new and what's changed, and under no circumstance what's unchanged from previous job.

Edited May 6, 2017 by jlr19

[CBers 6771]

A FULL backup of everything, whether it's changed or not.

 

So in your description, I'd say that was a sync backup, where a file is inly backed up if it changes, or is new.

 

But, if you delete it on purpose or by accident, then when the next sync process runs, the file in the secondary storage is also deleted.

 

You seem to want a backup of new and changed files, but never delete it automatically. That's gonna be difficult to implement, unless you concoct something specific to do that.

[CharleyVarrick 277]

You seem to want a backup of new and changed files, but never delete it automatically. That's gonna be difficult to implement, unless you concoct something specific to do that.

That sums it up pretty nicely.

 

Full initial backup of 27Tb and counting will require months of 24/7 uploading, by the looks of it.

Once this is done, new/modified data might "only" be 500-750gb/month.

Edited May 6, 2017 by jlr19

[CBers 6771]

That sums it up pretty nicely.

 

Full initial backup of 27Tb and counting will require months of 24/7 uploading, by the looks of it.

That begs the question, do you really need it all backed up

 

I don't bother backing up my movies or TV shows, just photos (to Google Photos, plus I copy them to a 2nd PC) and music (to Google music).

 

Important personal and business files are also copied to the same 2nd PC as my Photos.

[CharleyVarrick 277]

I have been building this collection of mine for over 17 years. A boatload of resources, money, efforts and time went into it.

So the answer for me is yes. I already have the hardware to more than double the present size of my collection, providing my present local backup goes cloud.

I figure it's win-win, as I increase my storage space significantly (future proof), and in the making, increase my backup security (catastrophe proof).

 

As it stands, should my house go down in flame next week, or get burglarized, I would loose 97% of those last 17 years (original and local copy).

That would hurt beyond understanding! The remaining 3% which would be unaffected is the Google stuff (photos/music).

 

Considering all of the above, Amazon unlimited cloud drive for 60$ a year sounds like a pretty sweet deal, to me anyway.

[CBers 6771]

You never housekeep?

[CharleyVarrick 277]

What you mean by housekeep?

[CharleyVarrick 277]

CloudBerry test backup underway, its the fastest I've seen yet by far, but there's a big issue: NO encryption, at least not to Amazon Cloud.

 

Too bad, it was very user friendly, nice clean design, and the fastest so far.

Unless I can pre-encrypt my media before uploading it, I will regretfully pull the plug.

 

EDIT:

CloudBerry Backup comes in many versions.

I was using the freeware's, that one doesn't support encryption, but every other version does.

But they have small data limit, which you can upgrade ($$$) I may check it out but with my present need, it probably wont be cost effective.

 

And I'm not sure I agree with the general idea of more capacity=more expensive, as after all, they're not providing the storage, Amazon does.

I'm providing the necessary bandwidth, the hardware and electrical power, why do they care about task size? 

Edited May 6, 2017 by jlr19