A Backup Strategy through the Power of Ansible and Restic

Installing Updates on Windows Server systems using Ansible

29. April 2024

Automate Your Proxmox VM Deployment with Ansible

13. July 2024

Table of Contents

Ansible As a Free Software company we also have to deal with Windows servers. Especially in the operation of these Windows server systems we try to use Free Software components on them. In this article we are going to explain a combination of Restic and Ansible to Backup a Windows server system.

Restic is a Free Software which provides fast, secure, and efficient data backup solutions. It supports various backends for storing backups, including local file systems, cloud storage services, and SFTP servers. With features like encryption, deduplication, and incremental backups, Restic ensures that data is not only safely backed up but also optimally stored, reducing storage requirements and backup time.

Ansible is an automation tool used for application deployment, configuration management, and task automation. It uses YAML for its playbook scripts, making it easy to define tasks and ensure idempotent operations, meaning repeated tasks will produce the same results, ensuring consistency across environments. Setting up Ansible is not scope of this article

Backup Strategy Snapshot or File-based?

Normally we are using snapshot backups to reduce our system-wide recovery efforts. These snapshots capture the complete state of a system at any given moment, ensuring we can rapidly restore entire systems when necessary. However, we recognize their limitations for individual file or database restoration, which can become complex without tailored solutions, despite the adaptability offered by serval cloud platforms and for example TrueNAS.

To address the need for more granular recovery capabilities, we integrate Restic into our backup strategy. Restic allows us to perform precise, file-level backups, enabling selective data recovery that complements our broader snapshot backup approach. With Restic, we enhance our data protection framework by adding a layer of flexibility and security, leveraging features like encryption and deduplication for efficient and secure data management.

That would mean we need to drive both, snapshot and file based backups. Lets focus on the Restic part.

Restic and Windows – VSS for Comprehensive Backups

Restic uses the Volume Shadow Copy Service (VSS) in the Windows environment, enabling it to back up open and locked files seamlessly. This integration ensures that Restic can capture a consistent snapshot of the filesystem, vital for databases and services requiring uninterrupted operation. With automated management of VSS snapshots, Restic combines efficiency with thorough data protection.

Restic allows to use features like encryption and deduplication well known on GNU/Linux on Windows as well as amd it is able to handle the complexities of a Windows filesystem through VSS.

A Brief Note on Borg Backup
While Restic is tailored for cross-platform use with advanced Windows support, Borg Backup offers specialized efficiency for GNU/Linux and UNIX-like systems. Known for its deduplication, compression, and encryption, Borg provides a reliable backup solution where Windows support isn’t a primary concern, making it a strong option for non-Windows environments.

Introduction to Restic

Next, we will explain the steps needed to configure Restic for using an SFTP backend, simplifying the process. More complex backends are thinkable for example you can also use S3 Storage like Minio. You can find a list on Restic-Docs

Here’s a guide to setting up Restic on Windows, along with steps to create a backup repository, perform backups, and restore data.

Setup Restic on Windows

Deploy Restic with Chocolatey

We use an ansible playbook also to install Restic. There are multiple ways to install Restic by using a package manager like Scoop or Chocolatey or by copying the binary from Github. We use an Ansible playbook to install Chocolatey and Restic at the same time.

Playbook

---
# Deploy Restic on Windows
- name: Install Chocolatey and Restic
  hosts: restic-showcase
  gather_facts: false
  become: true

  tasks:
  - name: Install Chocolatey - a Windows Package Manager
    win_shell: |
      Set-ExecutionPolicy Bypass -Scope Process -Force
      [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
      iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

  - name: Install Restic via Chocolatey
    win_shell: |
      choco install -y restic

The playblook can be executed as follows:

➜ ansible-playbook -i hosts.ini restic-install.yaml
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [Install Chocolatey and Restic] ****************************************************************************************************************************************************

TASK [Install Chocolatey - a Windows Package Manager] ***********************************************************************************************************************************
changed: [restic-host]

TASK [Install Restic via Chocolatey] ****************************************************************************************************************************************************
changed: [restic-host]

PLAY RECAP ******************************************************************************************************************************************************************************
restic-host                : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Creating a Backup Repository

To provide a clear and detailed guide, the setup process is explained step-by-step through manual operations.

We’re focusing on a Windows Server 2022 environment that requires backing up to an SFTP server, which we’ve named datacart. Before we can establish the backup repository, it’s essential to generate an SSH key for authentication against our datacart server.

Please, open PowerShell or Command Prompt on the Windows Server to follow the subsequent steps.

Create a SSH key on the Windows Server:

PS > ssh-keygen -t ed25519

Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\Administrator/.ssh/id_ed25519):
Created directory 'C:\Users\Administrator/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\Administrator/.ssh/id_ed25519.
Your public key has been saved in C:\Users\Administrator/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:LVSjzmZIHG5sHXKW3oF9Inyl2bLljntFSG6krOTMfGY administrator@restic
The key's randomart image is:
+--[ED25519 256]--+
|      o.++o..    |
|     + *=+==+    |
|      B.+=+Oo.   |
|     + =o.+=+ .  |
|      .*S.o...   |
|       o*.Eo  .  |
|         +. ..   |
|           ..    |
|          ..     |
+----[SHA256]-----+

Transfer the public to the backup target:

We already have a predefined user on datacart named rbackup.

PS > cat .\.ssh\id_ed25519.pub |ssh rbackup@datacart "cat - >> .ssh/authorized_keys"
The authenticity of host 'datacart (172.16.77.1)' can't be established.
ECDSA key fingerprint is SHA256:xFUeMRuhqAnT75s9UwX1VTwqvKusWcrc9eCZEOohMy0.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'datacart,172.16.77.1' (ECDSA) to the list of known hosts.
rbackup@datacart's password:

Ensure the SFTP connection is working

It is working when you are not prompted for a password anymore.
```
PS > sftp rbackup@datacart
Connected to datacart.
sftp> quit
```
Generate a random string

Restic is well known for its encryption feature to keep to data secure even if it is hosted in an area not well known by you. Hence, we create a random string which is being used as the encryption password for you backuped data.
```
PS > [System.Web.Security.Membership]::GeneratePassword(32, 2)
yG8mm7yd:p(7Ef}?2zF#4u0N/PXflz[G
```
Create the RESTIC_PASSWORD environment variable to carry the password. We use it only temporarily to create the repo. Later it is known only by the Ansible server.
```
PS > $env:RESTIC_PASSWORD = "yG8mm7yd:p(7Ef}?2zF#4u0N/PXflz[G"
```

Initialize the Repository:

Let’s create the Restic repository.

PS > restic -r sftp:rbackup@datacart:windows init
created restic repository e6c986d035 at sftp:rbackup@datacart:windows

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is irrecoverably lost.

The command will the repo named windows on the host datacart via the SFTP user rbackup. It reads the password from the enviroment variable.

Performing a Test Backup

Performing a test backup to your SFTP repository. Ensure $RESTIC_PASSWORD is set.

   PS > restic -r sftp:rbackup@datacart:windows backup C:\ --use-fs-snapshot
   repository e6c986d0 opened (version 2, compression level auto)
   no parent snapshot found, will read all files

   creating VSS snapshot for [c:\]
   successfully created snapshot for [c:\]

   Files:       205650 new,     0 changed,     0 unmodified
   Dirs:        59450 new,     0 changed,     0 unmodified
   Added to the repository: 13.784 GiB (6.634 GiB stored)

   processed 205650 files, 21.464 GiB in 16:51
   snapshot a30e4628 saved

Due to the use of --use-fs-snapshot Restic will create a native Windows Volume Shadow Copy Service (VSS) snapshot.

Performing a Test Restore

To restore files from your backup:

List Available Snapshots:

Find the ID of the snapshot you want to restore:


PS > restic -r sftp:rbackup@datacart:windows snapshots
repository e6c986d0 opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths
------------------------------------------------------------
a30e4628  2024-02-22 14:55:39  restic                  C:\
------------------------------------------------------------

It did find a single snapshot which was created before.

Restore Files:

Use the snapshot ID to restore files:

PS > restic -r sftp:rbackup@datacart:windows --target c:\restore restore a30e4628:/C/Windows/System32/
repository e6c986d0 opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
restoring  to c:\restore
Summary: Restored 16883 files/dirs (2.920 GiB) in 1:30

Due to the specification of a --target c:\restore Restic proove by itself if the target directory does exist, if not it will be created. Several methods do exist to restore a single path. One is to give the path right behind the snapshot ID.

You can also proof the content of the snapshot by using the ls option of Restic:

PS > restic -r sftp:rbackup@datacart:windows ls a30e4628
repository e6c986d0 opened (version 2, compression level auto)
[0:00] 100.00%  2 / 2 index files loaded
snapshot a30e4628 of [C:\] filtered by [] at 2024-02-22 14:55:39.8986101 +0100 CET):
/C
/C/$Recycle.Bin
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-1000
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-1000/desktop.ini
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-500
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-500/$IP018UJ
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-500/$RP018UJ
/C/$Recycle.Bin/S-1-5-21-3884125958-3418005504-2446353653-500/desktop.ini
/C/$WinREAgent
/C/$WinREAgent/Scratch
/C/Documents and Settings
[...]

Bringing Ansible Into The Game

Leveraging Ansible in tandem with Restic on Windows provides an efficient and streamlined approach for managing backups across various systems. This integration allows users to benefit from centralized scheduling using tools such as Ansible AWX or Semaphore, facilitating the orchestration of backup operations from a single point of control. Additionally, the centralized management of Restic’s encryption keys enhances security, as the encryption keys are not stored on the target systems themselves. This architecture not only simplifies the backup process but also significantly strengthens data protection measures.

Introducing a Backup Script

We stripped down the backup script to show the procedure.

      $resticRepo = "sftp:rbackup@datacart:windows/"
      $resticPass = "yG8mm7yd:p(7Ef}?2zF#4u0N/PXflz[G"
      $backupPaths = @("C:\")

      function BackupPath {
          param (
              [string]$pathToBackup
          )
          & restic -r $resticRepo backup $pathToBackup --use-fs-snapshot *>&1
      }

      $env:RESTIC_PASSWORD = $resticPass

      foreach ($path in $backupPaths) {
          Write "---"
          BackupPath -pathToBackup $path
      }

As evident, the encryption password is embedded directly in the script, which is not our desired approach. While transferring the script to the server would allow it to execute, this method necessitates managing the password individually on each host, which is impractical.

Creating an Ansible Playbook

Ansible offers the flexibility to run the backup script with a distinct set of parameters on each target host. We will store the repository and backup paths in Ansible variables. Additionally, we plan to utilize an Ansible Vault to securely store the Restic encryption password.

You can also replace the Ansible Vault with a key server (i. e. HashiCorp Vault) or any other technic to hold your secrets.

The following is executed in your automation environment.

Creating the Ansible Vault:

➜ ansible-vault encrypt_string 'yG8mm7yd:p(7Ef}?2zF#4u0N/PXflz[G' --name 'restic_pass' > restic_pass.yaml
New Vault password: 
Confirm New Vault password:

You have to specificy a Vault password.

Creating the Final Playbook

---
# Restic Showcase: Backup Windows Server 2022
- name: Restic - Backup Windows Server 2022
  hosts: restic-showcase
  gather_facts: false
  become: true
  vars_files:
    - restic_pass.yaml
  vars:
    restic_repo: "sftp:rbackup@datacart:windows/"
    backup_paths: ['C:',]

  tasks:
  - name: Execute Restic Backup Script
    ansible.windows.win_shell: |
      $resticRepo = "{{ restic_repo }}"
      $resticPass = "{{ restic_pass }}"
      $backupPaths = "{{ backup_paths | join('\", \"') }}"
      $backupPaths = @($backupPaths -split ',')

      function BackupPath {
          param (
              [string]$pathToBackup
          )
          & restic -r $resticRepo backup $pathToBackup --use-fs-snapshot *>&1
      }

      $env:RESTIC_PASSWORD = $resticPass

      foreach ($path in $backupPaths) {
          Write "---"
          Write "Backing up $path..."
          BackupPath -pathToBackup $path
          Write "$path backup completed."
      }
    register: restic_result

  - debug:
      var: restic_result.stdout_lines

Executing the Playbook

You will be asked about the Vault password. Which can be automated as well which is not scope of this article.

➜ ansible-playbook -i hosts.ini win-backup.yaml --ask-vault-pass
Vault password: 

PLAY [Restic - Backup Windows Server 2022] ***************************************************************************************************************************

TASK [Execute Restic Backup Script] **********************************************************************************************************************************
changed: [restic-host]

TASK [debug] *********************************************************************************************************************************************************
ok: [restic-host] => {
    "restic_result.stdout_lines": [
        "---",
        "Backing up C:\\...",
        "using parent snapshot f4264bc7",
        "creating VSS snapshot for [c:\\]",
        "successfully created snapshot for [c:\\]",
        "",
        "Files:           1 new,    31 changed, 221126 unmodified",
        "Dirs:            0 new,    62 changed, 61484 unmodified",
        "Added to the repository: 16.484 MiB (1.772 MiB stored)",
        "",
        "processed 221158 files, 24.368 GiB in 0:43",
        "snapshot 217cbdf8 saved",
        "C:\\ backup completed."
    ]
}

PLAY RECAP ***********************************************************************************************************************************************************
restic-host                : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Conclusion

In conclusion, Ansible and Restic complement each other exceptionally well, offering a robust and flexible solution for managing backups across diverse environments. Throughout this discussion, we’ve highlighted the essential steps to integrate these powerful tools, demonstrating their compatibility and ease of use together. While this article focused on direct interaction between Ansible and Restic, it’s worth noting that for those looking to further automate and scale their backup solutions, platforms such as Ansible AWX and Semaphore present valuable options. Although exploring these automation platforms falls beyond the scope of this article, they underscore the potential for extending the capabilities of Ansible and Restic, reinforcing the seamless synergy between them for comprehensive backup management.