Rancher on AWS, Backup to S3 with IRSA for Authentication

2023-07-04T00:00:00+02:00

This is the second Article of the Series Integrate Rancher with AWS services, I'm going to demonstrate how to perform backups from Rancher to S3 by using IAM Roles for Service Accounts (IRSA) instead of EC2 Instance IAM Roles or AWS access keys.

Update: The recording of my talk Rancher integration with AWS services: possibilities, challenges, outlook (abstract and slide-deck) at openSUSE Conference 23 is online and covers parts of this article as well.

media.ccc.de (includes options to download video and audio)
youtube.com

Terminology
Rancher Backup
- Overview
- IRSA Configuration
Conclusion

Terminology

Kubernetes objects and annotations

I assume you are aware what Kubernetes objects and annotations are, if not, that's your chance to brush up your knowledge, I will use these terms to explain the configuration of Rancher Backup.

Helm chart, repository, release

Again, I assume you are aware of Helm in general and also what a chart, repository and release is. I'm going to to use these terms later.

IAM Role and IRSA

To learn more about AWS IAM Roles and IRSA, I recommend to checkout the first Article of this series: What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

Rancher Backup

Overview

Rancher provides the backup-restore-operator, it can be used to perform scheduled and encrypted backups of all Rancher resources. Amazon S3 is a supported target, which is a high available and resilient location for backups. IRSA can be used for authentication by adding the serviceAccount annotation during the installation.

The official documentation about Backing up Rancher provides further information about the functionality and general setup.

IRSA Configuration

The backup operator already supports IRSA but it's not covered in the official Rancher Documentation yet. It's sufficient to add three additional lines as part of the installation to activate it:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-backup-role

The complete values file for Helm could look like this for example:

s3:
  bucketName: my-rancher-backup-bucket
  credentialSecretName: ''
  credentialSecretNamespace: ''
  enabled: true
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-backup-role

Besides the serviceAccount annotation, the initial IRSA setup for the cluster and the creation of the IAM Role, with a link to the service account, used by the Rancher backup operator, is required.

The default service account name is rancher-backup in the namespace cattle-resources-system.

Further information can be found in What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

Conclusion

At first it was a little tricky to figure out if and how the Rancher backup operator supports IRSA, as I realized that the necessary code change was already merged a while ago, I was surprised, it's not mentioned in the documentation yet.

The actual configuration was then quite straight forward and similar as in examples I found in the Amazon EKS documentation.

By default, long-term access key credentials are used by Rancher, which I would recommend to avoid, use short-term tokes as provided via IRSA instead, I outlined the why already in the first article of this series ;)

In the next article of this series, I will show you how to push log files from Rancher to CloudWatch and - again - to authenticate with IRSA instead of EC2 Instance IAM Roles or access keys.

Article series Integrate Rancher with AWS services:

What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?
Rancher on AWS, Backup to S3 with IRSA for Authentication
Rancher on AWS, Logging to CloudWatch with IRSA for Authentication
Rancher on AWS, SAML Authentication with AWS IAM Identity Center as SAML IdP (coming soon)
Rancher on AWS, GitOps with Fleet and AWS CodeCommit (coming soon)

SUSE Manager / Uyuni - Database Backup with "smdba" fails on symlink

2021-08-31T00:00:00+02:00

I performed a Upgrade of our SUSE Manager Instance at work to Version 4.2 and in parallel adjusted the disk layout a bit.

The exact details about all the changes are not that interesting, more the fact which issue I faced after changing /var/spacewalk/db-backup to be a symlink into a sub-folder of a new mountpoint.

Backups are created and handled by the smdba tool, which performs an Owner and Permission check as soon a Backup was started. The path is defined by the argument "--backup-dir", default as mentioned in the official docs is /var/spacewalk/db-backup, but as of today it doesn't follow symlinks.

So what happens: It will check the owner and permissions of the symlink instead the target folder. Due to the limitations that chmod can't be performed on symlinks, it stays 777, smdba will always fail when comparing with /var/lib/pgsql/data.

To solve this issue and make smdba symlink aware, some small adjustments on the code of /usr/lib/python3.6/site-packages/smdba/postgresqlgate.py are required.

I created a GitHub Issue (Archive: [1], [2]) to report my findings and discuss a solution.

Based on the feedback and suggestions from Victor and Michael, I created a Pull Request to get the fix hopefully included in a future release :)

diff --git a/src/smdba/postgresqlgate.py b/src/smdba/postgresqlgate.py
index 5bcb86e..e28b8f2 100644
--- a/src/smdba/postgresqlgate.py
+++ b/src/smdba/postgresqlgate.py
@@ -767,6 +767,11 @@ def do_backup_hot(self, *opts: str, **args: str) -> None:  # pylint: disable=W06
         if 'enable' in args.keys():
             # Check destination only in case user is enabling the backup
             if args.get('enable') == 'on':
+                # Save original value in temporary variable to re-assign after Permission Check
+                args_backup_dir_orig = args['backup-dir']
+                # If backup-dir is Symlink, use target instead
+                while os.path.islink(args['backup-dir']):
+                    args['backup-dir'] = os.readlink(args_backup_dir_orig)
                 # Same owner?
                 if os.lstat(args['backup-dir']).st_uid != os.lstat(self.config['pcnf_pg_data']).st_uid \
                        or os.lstat(args['backup-dir']).st_gid != os.lstat(self.config['pcnf_pg_data']).st_gid:
@@ -777,6 +782,8 @@ def do_backup_hot(self, *opts: str, **args: str) -> None:  # pylint: disable=W06
                 if oct(os.lstat(args['backup-dir']).st_mode & 0o777) != oct(os.lstat(self.config['pcnf_pg_data']).st_mode & 0o777):
                     raise GateException("The \"%s\" directory must have the same permissions as \"%s\" directory."
                                         % (args['backup-dir'], self.config['pcnf_pg_data']))
+                # Avoid issues at a later point due to different paths by setting original value
+                args['backup-dir'] = args_backup_dir_orig
             self._perform_enable_backups(**args)

         if 'source' in args.keys():

I know that I could also use a different path in the "--backup-dir" argument, which already points to the new location. But in my opinion it's helpful to stick with paths that are mentioned in the official documentation when there are multiple administrators.

Doesn't matter how good your internal KB is, in case of an issues or when new people takeover, project / vendor guides mostly have a higher precedence.

Also supporting Symlinks avoid that configured backup cronjobs silently failing in case someone has the idea to move the folder away and configure a symlink, like me ;)

The Wombelix Post - Backup

Rancher on AWS, Backup to S3 with IRSA for Authentication

SUSE Manager / Uyuni - Database Backup with "smdba" fails on symlink