Over-engineer your own file-backup solution

Until yesterday all my digital life was backed up on a physical drive using Time Machine. Occasionally I would feel guilty about that TimeMachine reminder that I haven’t backed-up in a while and would rummage through my desk for the external hd that I would occasionally use to backup my main laptop.

Lame.

I could have just decided to pay a service like backblaze but nah, I want to be in control of this data so I decided I was going to roll my own solution.

There are 3 main folders I care about, everything else is in a remote git repo. I want an encrypted/private s3 bucket, one IAM user should be able to put and get files from the bucket, and every 20 days I would like all the files moved to long-term storage in Glacier.

I had just started using Terraform in a limited fashion so I wanted to use this as an opportunity to tinker with it.

First up I created an empty git repo in Keybase bjoernw where I keep the terraform files along with a backup script.

Now some terraform. Create a file called providers.tf with the following:

provider "aws" {
  profile = "personal"
  region = "${var.region}"
  version = "~> 1.29"
}

terraform {
  backend "local" {
    path = "terraform-backend/terraform.tfstate"
  }
}

Here we setup our provider, tell terraform which aws profile (i.e. personal) to use (as defined in ~/.aws/credentials), and tell terraform where to save its state. I could have used a variety of backends but I wanted to keep it simple and just commit the state into git whenever I make changes.

Next lets create a file called s3.tf where we will setup everything related to the s3 bucket.

resource "aws_s3_bucket" "backup-bucket" {
  bucket = "${var.backup-bucket-name}"
  acl = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = "${aws_kms_key.backupKey.arn}"
        sse_algorithm = "aws:kms"
      }
    }
  }

  lifecycle_rule {
    id = "main"
    enabled = true

    transition {
      days = 30
      storage_class = "GLACIER"
    }

  }
  tags {
    Name = "Backups"
  }
}

resource "aws_kms_key" "backupKey" {
  description = "This key is used to encrypt backup bucket objects"
  deletion_window_in_days = 30
  tags {
    Name="BackupBucketKey"
  }
}

resource "aws_kms_alias" "key_alias" {
  name = "alias/${var.key_name}"
  target_key_id = "${aws_kms_key.backupKey.id}"
}

We tell s3 that we want a private bucket, encrypted using the key backupKey, and a lifecycle rule that ships our s3 files to Glacier on a monthly basis. At the bottom we create our backup key and assign an alias to it.

Now lets add some permissions.

resource "aws_s3_bucket_policy" "backup-bucket-policy" {
  bucket = "${aws_s3_bucket.backup-bucket.id}"
  policy = "${data.aws_iam_policy_document.default.json}"
}

data "aws_iam_policy_document" "default" {
  statement = [{
    actions = ["s3:*"]

    resources = ["${aws_s3_bucket.backup-bucket.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = ["${var.main-user}"]
    }
  }]

}

Here we created a S3 bucket policy for this bucket which gives a “main-user”, all permissions on that bucket.

Last we need to create our variables.tf file, which holds variables used throughout this terraform module.

variable "region" {
  default = "us-west-2"
}

variable "key_name" {
  description = "Name of the key"
  default = "backup-s3-bucket-key"
}

variable "backup-bucket-name" {
  default = "your-personal-backup-bucket"
}

variable "main-user" {
  default = "arn:aws:iam::5623203034562:user/someUser"
}

Run terraform to create the infrastructure.
terraform plan
terraform apply

Now we need something that efficiently syncs local files with an s3 bucket. We could go the traditional route and use rsync but I’ve always wanted to try a tool called rclone. It does everything I need.

Features:

brew install rclone && rclone config should get you up an running. Now alls that’s left is to create a shell script that backs up what you want backed up. Here is my version:

backup.sh

#!/usr/bin/env bash
rclone sync ~/Pictures personal:your-personal-backup-bucket/pictures
rclone sync ~/Documents personal:your-personal-backup-bucket/documents
rclone sync ~/books personal:your-personal-backup-bucket/books

Then run ./backup.sh and voila, now it’s painfully obvious how slow your upload speed is, so take your laptop to work and run it from there ;) The neat thing about rclone is that it picks up where it left off so you can interrupt it without losing progress.

Happy over-engineering!

-Bjoern

 
32
Kudos
 
32
Kudos

Now read this

Using Hazelcast and RxJava to build agnostic clients

I am currently refactoring a suite of services that are all Hazelcast cache instances and part of a cache cluster. One of my first steps was to look at client-service communication endpoints and see whether I can’t decouple clients and... Continue →