GRM "cloud" backup group

Depends on how much space we end up contributing, but considering the kind of upload bandwidth most of us have I don't think you'd have to worry about hitting your cap

Mezzanine wrote: Could you take a small step backwards and describe what the benefits and purpose of a shared backup is for the less technically savvy among us?

It looks like a great way for someone like China to get you to put all your important stuff on their server farm so they can peruse at their own pace.

NOHOME wrote:

Mezzanine wrote: Could you take a small step backwards and describe what the benefits and purpose of a shared backup is for the less technically savvy among us?

It looks like a great way for someone like China to get you to put all your important stuff on their server farm so they can peruse at their own pace.

Haha no. First, the data is going to be physically stored only on the disks of the people participating. Second, it's going to be encrypted in such a way that China would have a hard time perusing it even if we stored it on their server farm and gave them permission to have a look around.

Also, if China wanted to steal some cloud data, they would likely target the motherload(s) on large corporate servers, dropbox, google, one-drive, etc. Not a few measly terabytes of worthless data geeks are backing up with one another.

I can probably spare 1TB or more, but can adjust to meet whatever is decided on. Currently have a drive on my Kodi PI2, but would probably get another setup like that to dedicate to this & webserver so I'll buy whatever size is necessary.

What settings we use depends on how many people we have to start with, so if anyone else wants to join, the time to commit is now. I was really hoping for at least 10. So far we have me, ProDarwin and Rusted_Busted_Spit.

Here are the maximum space contribution numbers so far:

Me: 500GB (I'll need to make some hardware changes to contribute more than 250GB...no more empty drive bays!)

Rusted_Busted_Spit: 500GB

ProDarwin: 1TB

I can be in at the same level as ProDarwin.

Does drive age/speed matter? I came into possession of a free 500gb drive, no SATA.

Speed or age are no problem, in fact the system can't tell a RAMdisk from a cold-war-era hard drive

Hey Moxnix & PHeller, what are your upload bandwidths like? I'm still looking for someone to run the helper server.

Max space contribution update:

Me: 500GB

Rusted_Busted_Spit: 500GB

ProDarwin: 1TB

moxnix: 1TB

PHeller: 500GB

Five would be a good number of users to start with.

Edit: I assume we're talking "drivemaker's gigabytes" here, since most of us are talking about the size of hard drives.

So... what does this little system do and why should I join?

RossD wrote: So... what does this little system do and why should I join?

Creates an off-site backup of your files. You'll be the off-site backup for someone else.

E36 M3 hits the fan and your house burns down (and all the hard drives in it), drives all crash, etc. - all you have to do is log back into this cloud with a computer and you can recover all your important stuff.

Distributed & highly secure cloud storage, useful for offsite backups among other things, check out my post about 1/4 way down page 1.

So it's not all mirrored data, we'd each have individual storage on someone's system?

RossD wrote: So it's not all mirrored data, we'd each have individual storage on someone's system?

We could do mirroring with a small number of users (like 3 or less), but what we'll be doing is like a RAID 5/6 setup where the data is spread across all the devices (or, servers in this case) in such a way that a certain number of devices can fail without causing any data loss.

Hey Gameboy!

I'm interested in this. I got myself a commercial grade NAS and 4x2TB drives to run on it a while ago, but I haven't set it up because I'm afraid it might lock me into some proprietary disk format and it'll be too big to take with me when I inevitably hop an ocean again.

Thing is I have pretty bad internet. I have unlimited bandwidth but it's only 6Mb combined (probably 1Mb up or less) and it randomly goes offline for half an hour at least once or twice a day. Granted it's better than any home internet plan I've had since ~2010 but it still sucks. Would that be a problem?

Also what kind of space per user would there be? My most critical stuff - probably 60 GB of photos and 40 GB of self-made music projects - isnt too big but it would be nice to be able to upload some of the other random bits I've got with no backups.

I'm happy to toss a 500GB or 1TB drive in a thin client if I can contribute. The NAS rig sucks too much power for me to leave on 24/7.

Edit: I HIGHLY recommend RAID6 over RAID5. In fact I recommend RAID1 over either of those. If a disk on RAID5 goes out while another disk is being rebuilt, you lose the whole thing. RAID5 made sense back when a high capacity SCSI drive was 18GB and cost thousands, but not so much in the cheap terabyte era. I'd go so far to say RAID5 would be a deal breaker for me.

Double edit: also how does adding stuff to this thing work? Can I copy files to my local drive and let them percolate up to the cloud on their own time? Or would I have to leave a my laptop occupied for days while it 'uploads'?

It's not truly RAID5/6 just a similar concept. In fact the default setting is that it can tolerate 70% of the disks going down!

Your intermittent connection would hardly be noticeable.

The amount of space depends on our erasure coding (like "RAID level") setting and the number of users. For example, with the stock 30% required/70% failure allowed setting, it would be 30% of the total disk space divided by the number of users - with an equal share amount and one share per user, that means 30% of the space you contribute is available to you.

Edit: It works a lot like an SFTP/SCP connection (you can even connect with an SFTP/SCP client), meaning that at least your server will have to be running while the files upload, and if you want any automatic synchronization you'll have to script that yourself.

Jay wrote: Double edit: also how does adding stuff to this thing work? Can I copy files to my local drive and let them percolate up to the cloud on their own time? Or would I have to leave a my laptop occupied for days while it 'uploads'?

It should be just like Dropbox / One Drive / etc. Certain folders, or partitions or entire drives on your system will be marked as synced folders. You use them like you would any normal folder. Changed files are synchronized with the 'cloud' constantly.

Edit: Wait, what? The above is how BT Sync works. I assumed Tahoe-LAFS would work the same way.

Ah, okay. Got the wrong idea. I saw RAID5 and got tunnel vision. :P

Sorry, a few more technical questions :

What I mean with the upload/download method is, do I upload to my local machine and have it trickle up to the cloud, or do I upload to the cloud and have it trickle back down? Sounds like it's the first thing, which would be fine with me.

What happens if the machine I'm running this on goes down permanently and I have to replace it, can I just swap the drive over to something new?

Also does this require an OS to run or is it its own OS? (Linux based?) Do I need a strictly x86/PC-compatible machine or could I use something like a PPC G4 or an old AppleTV?

ProDarwin wrote: Edit: Wait, what? The above is how BT Sync works. I assumed Tahoe-LAFS would work the same way.

Nope it doesn't, yet. Support for this feature is still in development. Right now your best option might be a cronjob running a "tahoe mirror" command on the directory you want to keep in sync.

Jay wrote: What I mean with the upload/download method is, do I upload to my local machine and have it trickle up to the cloud, or do I upload to the cloud and have it trickle back down? Sounds like it's the first thing, which would be fine with me.

It's more like the second one, assuming you want to upload the data and then access it remotely. Again just like using an FTP share. You'll probably want to access your files through your server.

Jay wrote: What happens if the machine I'm running this on goes down permanently and I have to replace it, can I just swap the drive over to something new?

The only really important thing you need to have to maintain access is your aliases file. These are the keys to the kingdom. If your storage drive is lost it will re-download data from all the other nodes that are still up (similar to a RAID 5/6 rebuild)

Jay wrote: Also does this require an OS to run or is it its own OS? (Linux based?) Do I need a strictly x86/PC-compatible machine or could I use something like a PPC G4 or an old AppleTV?

It requires an OS, it'll run on the big 3 desktop OSes among other things. It's not architecture-dependent.

GameboyRMH wrote:

ProDarwin wrote: Edit: Wait, what? The above is how BT Sync works. I assumed Tahoe-LAFS would work the same way.

Nope it doesn't, yet. Support for this feature is still in development. Right now your best option might be a cronjob running a "tahoe mirror" command on the directory you want to keep in sync.

That's pretty annoying.

I also wasn't having much luck finding anything about it on the Pi :( I'm sure it can be done, but I don't want to have to do a bunch of manual setup because I suck at that sort of thing. None of my other machines are always on, or even consistently on.

Was hoping we could just have an image file hosted. Download, burn to SD card, plug in HD, change a few config settings, and you're up and running. I guess if we are all running different hardware it complicates things.

Installation on the pi should be the same as on any Linux desktop PC without a repo, just unzip and run.

Keep in mind that automatic synchronization also means automatic damage propagation, which is bad for a backup, but if BT Sync-like operation is what you want, the cron job will do it. I could write it for you if you want, should be a one-liner.

So, Jay, do we have you in for 1TB max? If we each contribute 500GB we're looking at about 140-240GB of storage per user depending on how many people we end up with and what erasure coding level we choose (worst case in terms of users, we may end up allowing 50% failure, but that's good news in terms of usable disk space).

Giving this another bump if anyone else wants to join in. The next 2 weeks are going to be ultra-busy for me, with the last events in autocross and offroad rally over the next 2 weekends, but I want to get things going.

Rusted_Busted_Spit, I need you to install tahoe-lafs and configure an introducer server, set up port forwarding (using the port from the introducer.port file that will be generated), and then PM the introducer FURL (from the private/introducer.furl file that will be generated) to me, ProDarwin, moxnix and PHeller. We want to keep this string private, it's the closest thing we have to a username & password to join the group.

I will get working on it. I will try to get it done this week. I have a couple of server installs for work but I will try and get it done.