The Wombelix Post - Kubernetes

AWS Neuron Kernel Modules on OpenShift with KMM

2025-07-24T00:00:00+02:00

As part of my work at AWS, I've been diving deep into running AI/ML workloads with the AWS Neuron SDK on Inferentia and Trainium instances on Red Hat OpenShift Services on AWS (ROSA). What started as exploring concepts quickly turned into building actual tooling to solve real problems.

The challenge: how do you get AWS Neuron drivers working reliably across different OpenShift versions? The solution turned out to be more interesting than I initially thought.

I developed and published kmod-with-kmm-for-ai-chips-on-aws, a project that automates building Kernel Module Management (KMM) compatible container images for AWS Neuron drivers.

Running specialized hardware like AWS Inferentia and Trainium chips requires kernel modules. Kernel modules are tied to specific kernel versions, and new driver releases need to be built against different kernel versions. This creates ongoing maintenance work that grows with each supported platform version.

Red Hat's Kernel Module Management (KMM) operator solves this by automating kernel module lifecycle management. But you still need to build the container images that contain your modules for each kernel version.

That's where the Driver Toolkit comes in - it provides the exact kernel headers and build environment for each OpenShift release.

I fully automated this using GitHub Actions with nightly scans for new OpenShift releases, automated builds triggered by driver or mapping changes, and minimal container images containing only busybox and the compiled kernel module.

GitHub Actions proved surprisingly powerful for this automation. The dual tagging strategy works well - images get both kernel-specific and OpenShift-specific tags for different use cases. The build script also works locally for development, automatically detecting the environment and adapting accordingly.

This removes the manual work of tracking releases and building modules for each kernel version. The process now happens automatically. The approach is reusable for other specialized hardware drivers on OpenShift.

If you're running specialized hardware on OpenShift, KMM and the Driver Toolkit are powerful tools once you understand how to use them effectively.

SUSE Certified Administrator (SCA) in Longhorn 1.5 - Cloud native Storage

2024-10-07T00:00:00+02:00

I passed the new SUSE Certified Administrator (SCA) in Longhorn 1.5 Exam today! Finally back at 100% SUSE Certified after almost a month since it was released ;) Now I'm waiting for the SUSE Training Team to create and publish a new Tech Exam :)

I used the official SUSE Course Longhorn Deployment and Operations - LHN201v1.5 to prepare and found it sufficient. A couple of questions were a bit tricky from a wording perspective and not well reflected in the Course content. But the majority was of questions was well structured and balanced.

Never heard of Longhorn before? It's SUSEs Cloud native distributed block storage for Kubernetes, to learn more visit: https://longhorn.io

PS: Yeah, I know, Longhorn was also the codename for an operating system that spectacularly failed many many years ago ;)

Summary: This certification validates fundamental knowledge of Longhorn, including installation, configuration, management, and troubleshooting of Longhorn clusters. It tests the candidate's ability to utilize Longhorn's features such as automated backups, disaster recovery, and volume snapshots to ensure high availability and data protection in Kubernetes environments.

Source & Copyright: https://www.suse.com

Skills

Longhorn Features
Longhorn Architecture
Installation Requirements
Installation with Helm
Installation as a Rancher Application
Longhorn Web UI Features
Enabling and Disabling Nodes
Adding a new Node in Cluster
Node Maintenance
Removing a Node from Cluster
Adding Individual Disk Partitions
Removing Disk Storage
Adding LVM Storage
Creating a Volume
Using Trim to Reclaim Storage Space
Resizing a Volume
Tuning Storage Performance by Data Locality
Defining Replica Counts and Location
Replica Balancing for even Distribution of Resources
Installation of Prometheus and Grafana for Monitoring Longhorn’s Vital Metrics
Integration of Longhorn Metrics with the Rancher Monitoring System
Configuration of Longhorn Alert Rules
Recovering from Failures
Manually Creating a Snapshot of a Volume
Restoring a Volume from a Snapshot Manually
Deleting a Volume Snapshot
Working with Recurring Volume Snapshots
Manually Creating a Backup of a Volume
Restoring a Volume from a Backup
Working with Recurring Volume Backups
Working with Disaster Recovery Volumes
Upgrading Longhorn Manager
Upgrading Longhorn Engine

Source & Copyright: https://www.suse.com

Certificate

Downloads
- Certificate (ID 133) (PDF, 41K)
Links
- Verify Certificate
- SUSE Badges

How-to ace the CKAD, CKA and CKS exam in just one month

2023-09-30T00:00:00+02:00

Between 31st July and 21st August 2023, I passed the CKA, CKAD and CKS Exam. It was quite a ride and required some discipline. Everything boils down to a lot of hands-on practice.

I know that something like this isn't common, most people need much more time to prepare, which is totally fine. I do a lot of different trainings and certification exams on a regular basis. I'm also a quick learner, have a lot of knowledge across a vast variaty of topics. During the years, I developed mechanisms to effectively prepare and take exams. But I'm not a role model, take all the time you need, don't rush it!

Key learning resources

The CKA, CKAD and CKS courses from Mumshad Mannambeth are pretty good. Quality is exactly in that order, CKA is superb, CKAD is ok and CKS the lowest. They are available as subscription on KodeKloud and from time to time on-sale for ~$15 on Udemy.

killercoda.com gives you free access to lab environments to practice various scenarios. Make use of it! When the Exam date comes closer, you should get bored by this excercises and just know what to-do while reading the instructions.

killer.sh with every Exam you purchase, you will get two session vouchers for free. This should be your last preparation step before the actual Exam. As soon you feel ready, use the killer.sh exam simulator to validate your knowledge. Think of it like the real thing, block time, focus on the tasks, keep an eye on the timer. Afterwards take a close look on the questions that you had wrong. The environment is available for 24h, try to solve failed questions again. If necessary practice again with KodeKloud and/or killercoda. Take the second killer.sh session a day or two before your Exam as final preparation. When you ace killer.sh, you ace the actual Exam too :)

"common" knowledge

You can call it common knowledge, basic knowledge or whatever name you think fits for all those skills you should already have or brush up real quickly, because you will need them for the kubernetes trifecta.

Linux

You should feel at home in a shell like bash, navigating around through different folder, becoming root, starting/stopping services, copying or editing files, all this should be your daily business.

If I ask you to start and enable service xyz, you should right away have something like systemctl enable --now xyz in mind.

Copy file abc.yaml to /etc/ on server node01, for sure you know that you can achieve this by running something like scp ./abc.yaml node01:/etc/.

YAML

Widely used markup language, Kubernetes Manifests are written in YAML but also Ansible Playbooks and much more. You should familiar with the overall syntax. How to spot syntax errors and perform the correct indentation.

vim

Vim is everywhere and in most shells set as default editor. I highly recommend you brush up your skills. Navigating and basic editing should be something you are familiar with. YAML friendly settings in your .vimrc are life safers. The good news: In your Exam environment, that's pre-configured already :)

Some relevant settings from my vim rc file:

set fileencoding=UTF-8
set syntax=on
set ts=2
set sw=2
set sts=2
set et nu

kubectl

During the Exams, you use kubectl, or the alias k, all the time. It is key that you life and breathe it, for example:

Which sub-commands are available?
How to find and show resources?
How can you do a dry-run and redirect the output as YAML into a file?
When should you use it to create/edit a resource and when write a Manifest?

Focus during your study on these things and practice, practice, practice!

Get used to type quickly and use tab-completion. Copy & Paste is fine but I recommend to do it only for names or values. This way you avoid typos but don't slow yourself down.

Before the Exam

Read all instructions about the PSI testing environment carefully! They have comprehensive FAQ that will cover everything. It is important that you do exactly what they expect. Otherwise your risk that your Exam will be terminated by the proctor.

Perform a complete system test and fix any issues. Trust me, you don't want that stress on the Exam day.

Remove everything from your Desk. If you have an external monitor and camera, I suggest you use them! The display resolution should be as large as possible, I used a 4k display. Don't try it with the standard Laptop Display, it will not provide enough space, you have to scroll around all the time.

Exam day

You did all the practice and preparation, go relaxed into the Exam. Get a good night sleep, stay hydrated but keep in mind that you can't pause the Exam. So don't drink too much and use the Restroom before you start. You going to focus for quite some time, your brain needs energy. I always eat a Banana right before sitting the Exam.

Reading and understanding is key. Ensure you fully understand what you have to do. For example, you might have a question about creating a backup of etcd. Then there will be additional details like the node where you have to do it. The path of the etcd database that you have to backup. The path where you have to store it. All those details matter, don't waste points by overseeing parts of the instructions.

Prioritize questions with a high point count and start with them. Read the instructions very carefully, if you feel comfortable, do it right away, if not flag it. Pick the next question and repeat. Don't waste time on questions you can't solve. Don't waste time with 5 questions that give you 1 point when you can solve 2 questions with 10 points each in the same time. If there is time left at the end, go for the low score questions. But solving the majority of high score question will make you pass.

Conclusion

I found it extremely valuable to take all three Exams. I learned so much during the preparation. It helped me a lot in my Job as well. I would do it again and if you decide to tackle them, I wish you all the best!

CKS: Certified Kubernetes Security Specialist

2023-08-21T00:00:00+02:00

I did it, after already passing CKA and CKAD, I completed the trifecta today by passing the CKS: Certified Kubernetes Security Specialist as well. Taking and passing all three within one Month was challenging, at some point a little stressful, but very rewarding and satisfying!

As mentioned in the CKA and CKAD, post already, no real problems with PSI, everything went quite smooth again, nothing to complain about.

The Exam was, as expected, on a whole other level compared to CKA and CKAD. The requirement to take the CKS is to hold an active CKA, which you going to need. All the additional knowledge you have to gain about various security tools and best-practices is massive. But also extremely valuable, preparing for the Exam and diving deep into Kubernetes security made me realize how many attack vectors are out there I wasn't even thinking of so far. For me it was an eye opener and I'm glad I decided to tackle all three exams, I learned more in a month as I did on my own within the last year.

I took some Red Hat Exams in 2021 and loved the fact that they are also performance-based, I'm three specialist certifications short to become a Red Hat Certified Architect, so I guess some Red Hat OpenShift Certifications are next on my list :)

Summary: Earners of this designation are accomplished Kubernetes practitioners (as evidenced by holding the CKA credential). Earners demonstrated the requisite abilities to secure container-based applications and Kubernetes platforms during build, deployment and runtime, and are qualified to perform these tasks in a professional setting.

Source & Copyright: https://www.credly.com

Skills

API objects
AWS Cloud Security Services
Cloud
Custom Resource Definitions
Helm
Ingress
Kubernetes
Logging and Troubleshooting
Open Source Software
Orchestration
Scheduling
Security Policies
Services
Site Reliability Engineer
System Administrator
Volumes

Source & Copyright: https://www.credly.com

Certificate

Downloads
- Certificate - The Linux Foundation (ID LF-fvagino0m0) (PDF, 499K)
- Certificate - Credly (PDF, 117K)
- Badge (PNG, 110K)
Links
- Credly Badge (Archive: [1], [2])
- Certificate verification (The Linux Foundation)

CKAD: Certified Kubernetes Application Developer

2023-08-07T00:00:00+02:00

And the next one, after passing the CKA a week ago, I cleared CKAD: Certified Kubernetes Application Developer today, which makes two out of three, next stop CKS!

I wrote a bit about the experience with PSI already last week in the CKA Post, it was very similar this week, swift check-in process, quite relaxed proctor, no real technical issues with the exam environment, my setup was exactly the same.

From an Exam perspective it was really cool to dive deep into different deployment scenarios, how to configure and troubleshoot them, gave me even more confidence with Kubernetes and using kubectl. In hindsight, it would've probably better to take the CKAD before the CKA Exam. Now that I know what's covered and how the Exams are, that feels like a more logical order, CKAD is overall a little easier as CKA.

It was again performance-based, as all Kubernetes Exam from The Linux Foundation, and I can't get enough of it. This type of exam is just so much more fun, engaging and challenging as question-based ones. From that perspective I'm a little sad that I only have one other Exam ahead.

I'm now going to prepare for the Certified Kubernetes Security Specialist (CKS), after all I heard, that's a complete different beast compared to CKA and CKAD. Let's see if I can make it in two weeks and how different it really will be.

Summary: Earners of this designation demonstrated the skills, knowledge and competencies to perform the responsibilities of a Kubernetes Application Developer. Earners are able to define application resources and use core primitives to build, monitor, and troubleshoot scalable applications and tools in Kubernetes. The skills and knowledge demonstrated by earners include Core Concepts, Configuration, Multi-Container Pods, Observability, Pod Design, Services & Networking, State Persistence.

Source & Copyright: https://www.credly.com

Skills

cloud native applications
ConfigMaps
Containers
Docker
Java
Kubernetes
Node.js
Open Source Software
Orchestration
Python
rkt
Secrets
Software Developer
Volumes

Source & Copyright: https://www.credly.com

Certificate

Downloads
- Certificate - The Linux Foundation (ID LF-pl8i8vvtrb) (PDF, 714K)
- Certificate - Credly (PDF, 142K)
- Badge (PNG, 122K)
Links
- Credly Badge (Archive: [1], [2])
- Certificate verification (The Linux Foundation)

CKA: Certified Kubernetes Administrator

2023-07-31T00:00:00+02:00

My goal is to get all three Kubernetes Certifications from The Linux Foundation (CKA, CKAD, CKS), today I started this journey by passing the first Exam (CKA) and becoming a CKA: Certified Kubernetes Administrator.

It's the first performance-based exam since almost two years and I have to admit, I missed it! For whatever reason, it became standard in the industry to go with question-based exams instead. I understand that those are easier to deliver and might even attract more people, but in my opinion, demonstrating real Hands-On experience during an Exam is way more valuable and requires a lot of practice.

Don't get me wrong, every Exam has it's values, there is not a single one I took were I not learned something new during preparation. But at least for me, I gained the deepest level of understanding and the most real world experience from practical exams.

I did a lot with Kubernetes before and of course knew already much about it, but preparing for the CKA Exam gave me a massive boost in my kubectl skills and pushed my knowledge about all the different components and how to interact with objects to a new level. Personally, I never felt that comfortable with Kubernetes before and can recommend the CKA to everyone who works with Kubernetes.

A lot of people complain about the Exam provider PSI and the exam environment, especially the PSI secure browser, it's poor performance and a bad experience with the proctors. For me things went actually very well, I used the MacBook Pro M1 with an external 27" Monitor, external Webcam (Logitech C925e), wired Keyboard and Mouse. As per policy, the MacBook was closed during the Exam, before I launched the PSI secure browser I connected all external devices and closed the lid. Monitor and Webcam were recognized as expected, scan of my ID was no problem with the Logitech Webcam. I have another Monitor on my desk, which I unplugged and turned around. I removed anything else (e.g. smartphone charger, pens, paper). The proctor was fine with the external monitor that way, I showed my whole desk, in more detail the region below keyboard, mouse and macbook. I have some stickers on my macbook and had to zoom in to show that there are no notes or something. Then the rest of my room, the walls, ceiling, below my desk and so on. All this went fine, no complains from the proctor, I just followed the instructions and was ready to take the exam after around 15 minutes, which is a good time and comparable to PearsonVUE and Questionmark. Also the actual exam environment, remote desktop delivered through the PSI secure browser, was working without much issues. I had a few times a brief "lag" and the sticky key problem - you press a key once and it gets repeated multiple times, hitting k becomes kkkkkkkkkk for example - but most of the time I could use the environment just the same way as I would run the commands on my local machine.

So my recommendations in regards to PSI: Ensure the computer you use has a supported operating system and enough cpu/ram resources. Keep your desk clean as good as possible but don't worry too much about if there are still some cables or a second monitor, just ensure that the proctor can clearly see that it's impossible for you to use. Follow the instructions of the Proctor, arguing will not help even if they might ask "stupid questions", just answer, explain what something is or what it's used for and show it in the camera, everything will be fine. Check your internet connectivity, in my case it's a 400mbit up/down fibre connection which did great, just ensure you have some bandwidth and that not the rest of the household is streaming Netflix in 4k in parallel ;). Last but not least, focus on your monitor and don't look around too much, that's something the PSI system will otherwise recognized and trigger the proctor, worst case they close your Exam.

Summary: Earners of this designation demonstrated the skills, knowledge and competencies to perform the responsibilities of a Kubernetes Administrator. Earners demonstrated proficiency in Application Lifecycle Management, Installation, Configuration & Validation, Core Concepts, Networking, Scheduling, Security, Cluster Maintenance, Logging / Monitoring, Storage, and Troubleshooting

Source & Copyright: https://www.credly.com

Skills

API objects
Cloud
Custom Resource Definitions
Helm
Ingress
Kubernetes
Logging and Troubleshooting
Open Source Software
Orchestration
Scheduling
Security Policies
Services
Site Reliability Engineer
System Administrator
Volumes

Source & Copyright: https://www.credly.com

Certificate

Downloads
- Certificate - The Linux Foundation (ID LF-0v1fpn4ify) (PDF, 704K)
- Certificate - Credly (PDF, 128K)
- Badge (PNG, 110K)
Links
- Credly Badge (Archive: [1], [2])
- Certificate verification (The Linux Foundation)

Rancher on AWS, Logging to CloudWatch with IRSA for Authentication

2023-07-05T00:00:00+02:00

This is the third Article of the Series Integrate Rancher with AWS services, I'm focusing on Logging to CloudWatch from Rancher by using IAM Roles for Service Accounts (IRSA) to authenticate to avoid long-term credentials.

Update: The recording of my talk Rancher integration with AWS services: possibilities, challenges, outlook (abstract and slide-deck) at openSUSE Conference 23 is online and covers parts of this article as well.

media.ccc.de (includes options to download video and audio)
youtube.com

Terminology
Rancher Logging
Conclusion

Terminology

I assume you have a basic level of understanding about Kubernetes objects and annotations as well as Helm charts, repositories, releases. If you want to brush up your knowledge, links to resources about those topics are part of the second article of this series: Rancher on AWS, Backup to S3 with IRSA for Authentication

To learn more about AWS IAM Roles and IRSA, I recommend to checkout the first Article of this series: What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

Rancher Logging

Overview

Rancher provides the rancher-logging Helm chart, which is based on the kube-logging operator, it's using Fluent Bit to collect and Fluentd to forward the logs. One of the supported targets is Amazon CloudWatch.

With enhanced cloud provider logging, logs from Amazon EKS will be collected and pushed to CloudWatch as well.

IRSA is technically supported but the necessary serviceAccount annotation need to be added after the installation in a separate step.

The official documentation about Rancher Integration with Logging Services provides further information about the functionality and general installation.

IAM Policy

You need a IAM Policy to create later the IAM Role linked to a Kubernetes service account. An example how such a policy could look like to push logs to CloudWatch, based on the out_cloudwatch_logs example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

IRSA Configuration

The kube-logging operator already supports IRSA but it can't be configured as part of the Helm installation with the rancher-logging Helm chart yet. The annotation need to be added afterwards to the Logging Resources, this is different compared to Rancher Backup, where this is possible directly as part of the installation.

I created a GitHub Pull Request to include the serviceAccount annotation in the Helm chart.

In the meantime, you have to edit the Logging Resources rancher-logging-root and (if Rancher is running on Amazon EKS with enabled enhanced cloud logging) rancher-logging-eks in namespace cattle-logging-system manually.

The configuration is not covered in the official Rancher Documentation yet. Six lines need to be added per Logging Resource, each with it's own role-arn:

spec:
  fluentd:
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-logging-role

Besides the serviceAccount annotation, the initial IRSA setup for the cluster and the creation of the IAM Role, with a link to the service account, used by the kube-logging operator, is required.

The names of the service accounts rancher-logging will create are rancher-logging-root-fluentd and (in case of EKS and enhanced logging) rancher-logging-root-fluentd, also in namespace cattle-logging-system. You have to create a IAM Role and link it to those accounts by using eksctl as shown in the first article What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?.

ClusterOutput

To push logs to the actual target, Fluentd is using so called Outputs. There are two types, Output (namespace) and ClusterOutput (cluster-wide), further reading.

Following an example how a basic ClusterOutput could look like, I named it testcloudwatchoutput and it will push all available logs to the group rancher-demo-cluster-log-group and stream rancher-demo-cluster-log-stream in Amazon CloudWatch of AWS Region us-east-1.

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  creationTimestamp: '2023-05-17T14:03:16Z'
  generation: 4
  managedFields:
    - apiVersion: logging.banzaicloud.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:active: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-05-17T14:03:16Z'
    - apiVersion: logging.banzaicloud.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:cloudwatch:
            .: {}
            f:auto_create_stream: {}
            f:log_group_name: {}
            f:log_stream_name: {}
            f:region: {}
      manager: rancher
      operation: Update
      time: '2023-05-18T10:14:12Z'
  name: testcloudwatchout
  namespace: cattle-logging-system
  resourceVersion: '16031275'
  uid: a4bd1852-eca8-487b-bdc8-47d9966e6da2
spec:
  cloudwatch:
    auto_create_stream: true
    log_group_name: rancher-demo-cluster-log-group
    log_stream_name: rancher-demo-cluster-log-stream
    region: us-east-1
status:
  active: true

Conclusion

Logging is a complex field and very dependent on the individual requirements and use-case. I recommend to invest some time to learn about kube-logging, to write down the goals you want to achieve with your log setup and work backwards from there to perform the actual configuration.

This is way beyond the scope of this Article, my main intention was to show how to bring IRSA into the mix.

Compared to Rancher Backup, it's a little more effort to get IRSA working, mainly because of the missing support in the Helm chart, but it's worth it and not too complicated.

I hope my contribution to the rancher-logging Helm chart will help to improve the user experience. As soon a pull request that allows the serviceAccount annotation, was merged, I plan to also submit a PR to get the Rancher Documentation updated accordingly. Configuring such a feature, based on security best practices, should be as easy as possible to achieve broad adoption :)

In the next article of this series, I take a break from IRSA and will talk about login to Rancher via SAML Authentication by using AWS IAM Identity Center as SAML identity provider.

Article series Integrate Rancher with AWS services:

What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?
Rancher on AWS, Backup to S3 with IRSA for Authentication
Rancher on AWS, Logging to CloudWatch with IRSA for Authentication
Rancher on AWS, SAML Authentication with AWS IAM Identity Center as SAML IdP (coming soon)
Rancher on AWS, GitOps with Fleet and AWS CodeCommit (coming soon)

Rancher on AWS, Backup to S3 with IRSA for Authentication

2023-07-04T00:00:00+02:00

This is the second Article of the Series Integrate Rancher with AWS services, I'm going to demonstrate how to perform backups from Rancher to S3 by using IAM Roles for Service Accounts (IRSA) instead of EC2 Instance IAM Roles or AWS access keys.

media.ccc.de (includes options to download video and audio)
youtube.com

Terminology
Rancher Backup
- Overview
- IRSA Configuration
Conclusion

Terminology

Kubernetes objects and annotations

I assume you are aware what Kubernetes objects and annotations are, if not, that's your chance to brush up your knowledge, I will use these terms to explain the configuration of Rancher Backup.

Helm chart, repository, release

Again, I assume you are aware of Helm in general and also what a chart, repository and release is. I'm going to to use these terms later.

IAM Role and IRSA

To learn more about AWS IAM Roles and IRSA, I recommend to checkout the first Article of this series: What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

Rancher Backup

Overview

Rancher provides the backup-restore-operator, it can be used to perform scheduled and encrypted backups of all Rancher resources. Amazon S3 is a supported target, which is a high available and resilient location for backups. IRSA can be used for authentication by adding the serviceAccount annotation during the installation.

The official documentation about Backing up Rancher provides further information about the functionality and general setup.

IRSA Configuration

The backup operator already supports IRSA but it's not covered in the official Rancher Documentation yet. It's sufficient to add three additional lines as part of the installation to activate it:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-backup-role

The complete values file for Helm could look like this for example:

s3:
  bucketName: my-rancher-backup-bucket
  credentialSecretName: ''
  credentialSecretNamespace: ''
  enabled: true
  endpoint: s3.us-east-1.amazonaws.com
  region: us-east-1
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-backup-role

Besides the serviceAccount annotation, the initial IRSA setup for the cluster and the creation of the IAM Role, with a link to the service account, used by the Rancher backup operator, is required.

The default service account name is rancher-backup in the namespace cattle-resources-system.

Further information can be found in What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

Conclusion

At first it was a little tricky to figure out if and how the Rancher backup operator supports IRSA, as I realized that the necessary code change was already merged a while ago, I was surprised, it's not mentioned in the documentation yet.

The actual configuration was then quite straight forward and similar as in examples I found in the Amazon EKS documentation.

By default, long-term access key credentials are used by Rancher, which I would recommend to avoid, use short-term tokes as provided via IRSA instead, I outlined the why already in the first article of this series ;)

In the next article of this series, I will show you how to push log files from Rancher to CloudWatch and - again - to authenticate with IRSA instead of EC2 Instance IAM Roles or access keys.

Article series Integrate Rancher with AWS services:

What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?
Rancher on AWS, Backup to S3 with IRSA for Authentication
Rancher on AWS, Logging to CloudWatch with IRSA for Authentication
Rancher on AWS, SAML Authentication with AWS IAM Identity Center as SAML IdP (coming soon)
Rancher on AWS, GitOps with Fleet and AWS CodeCommit (coming soon)

What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?

2023-07-03T00:00:00+02:00

This is the first Article of the Series Integrate Rancher with AWS services, I did quite a lot with Rancher on Amazon Web Services recently and want to share some of my experiences about the necessary configuration to interact with AWS services for Backup, Logging and Authentication.

I will cover IAM roles for service accounts (Archive: [1], [2]), what it is and how it works under the hood on Amazon Elastic Kubernetes Service (EKS), in this post.

Let's start with a brief overview of the identity and access management challenges that can be solved by using IRSA and some important terminologies.

media.ccc.de (includes options to download video and audio)
youtube.com

Terminology
IAM Roles for Service Accounts (IRSA)
- IRSA under the hood
- Pod Identity Webhook
Conclusion

Terminology

AWS IAM Role and Policy

To grant an application access to AWS services, you need an AWS IAM Role and Policy. A IAM Role contains a IAM Permission policy and a IAM Trust relationship. The policy defines what can be done with the specified service, for example, uploading files to a Amazon Simple Storage Service (S3) Bucket. The relationship defines who can assume the IAM Role to perform the actions based on the attached permission policy.

IAM Permission policy >> IAM Role << IAM Trust relationship

Amazon Elastic Compute Cloud (EC2) Instance IAM Role

When a IAM Role is attached to an EC2 Instance, every application running on this specific instance has the same level of permissions. If you have one single application running, it's good practice to grant permissions that way. But imagine you have multiple applications running, for example in different containers like in a Kubernetes Environment, then all of them inherit the same set of permissions.

For Example: The IAM Role grants write access to a S3 Bucket, because container #1 is supposed to upload files, by using an EC2 Instance IAM Role, container #2 and container #3 can also write to the same S3 Bucket. Those container run a complete different application, maybe even outside your own control. Worst case, this could cause data corruption, data loss or even a data leak.

container 1 --|     IAM Role
              |        |
container 2 --|-- EC2 Instance -- S3 Bucket
              |
container 3 --|

AWS IAM access key credentials

Long-term credentials should be avoided whenever possible from a security best practice point of view. AWS access keys are linked to a IAM user and have no expire date. They could be leaked or shared between multiple applications and therefore might have to broad permissions assigned.

I think it's obvious why those type of credentials should not be used, in my opinion not even during development. Start early in the process to replace long-term with short-term credentials!

IAM Roles for Service Accounts (IRSA)

Now that I explained why EC2 Instance IAM Roles and IAM access keys are not a good idea in a Kubernetes environment, what's the alternative? IRSA to the rescue ;) With this feature, IAM Roles are assigned to Kubernetes Service Accounts, which are then linked to specific pods. That way you grant granular permissions on a per service basis inside your Kubernetes cluster.

It's an AWS feature and available out-of-the-box on Amazon EKS. the open source solution that makes the dynamic configuration and assignment of temporary credentials possible is Amazon EKS Pod Identity Webhook.

AWS CLI and AWS SDK both fully support IRSA, every application inside a Pod that leverages the CLI or SDK to interact with AWS services, can use IRSA instead of, for example, access keys with none or just minor code changes.

IRSA under the hood

Personally I'm super excited about IRSA and how it works, I think it should be used for every Kubernetes workload on AWS, there is really no reason at all to stick with EC2 Instance IAM Roles or even access key credentials.

That's why I want to dive deeper into how IRSA works and what magic is happening under the hood.

A reference between the EKS Cluster and IAM is established via OIDC. This is a one-time setup per cluster and can be done via eksctl. Example:
```
eksctl utils associate-iam-oidc-provider --cluster <CUSTER_NAME> --approve
```
A reference between a Kubernetes service account and a IAM Role has to be created. This can be done via eksctl in two ways, either with an account and the role managed by EKS, or by assigning a role to an existing Kubernetes service account. In case of Rancher, it's option two because Rancher creates and manages Kubernetes service accounts on it's own. Example:
```
eksctl create iamserviceaccount --name <SA_NAME> --namespace <NS_NAME> --cluster <CLUSTER_NAME> \
  --role-name <ROLE_NAME> --attach-policy-arn <IAM_POLICY_ARN> --approve --role-only
```
The Kubernetes resource is configured with a appropriate service account annotation, for example as part of a installation via Helm or by adjusting a Manifest.
As soon a Pod with a service account annotation comes up, the Pod Identity Webhook will be triggered and reconfigure (mutate) the Pod to use IRSA
The Pod assumes the specified IAM Role and connects to the AWS Security Token Service
AWS STS verifies the request by contacting AWS IAM
If the request could be verified and is valid, AWS STS assigns temporary credentials

AWS CLI and applications leveraging the AWS SDK, can now interact with AWS services based on the permissions of the IAM Role, without the need of EC2 Instance IAM Roles or long-term access key credentials.

As soon the temporary credentials are expired, the process automatically starts over from step #5 to get a new set of temporary credentials, there is no manual interaction required as soon step #1 till #3 are completed.

Further information and examples can be found in the AWS Documentation about IRSA on Amazon EKS.

Pod Identity Webhook

I explained that the Pod Identity Webhook performs a reconfiguration / mutation in step #4 of the IRSA Architecture Diagram. What happens in this step, is that the following Environment variables and Volumes are added to the Pod:

env:
- name: AWS_DEFAULT_REGION
  value: us-west-2
- name: AWS_REGION
  value: us-west-2
- name: AWS_ROLE_ARN
  value: "arn:aws:iam::111122223333:role/s3-reader"
- name: AWS_WEB_IDENTITY_TOKEN_FILE
  value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
- name: AWS_STS_REGIONAL_ENDPOINTS
  value: "regional"

volumeMounts:
- mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
  name: aws-token

volumes:
- name: aws-token
  projected:
    sources:
    - serviceAccountToken:
        audience: "sts.amazonaws.com"
        expirationSeconds: 86400
        path: token

The region and role arn values are example data and set according to your IAM Role configuration performed in step #2. The Environment variables are used by AWS CLI or SDK to understand that the authentication need to be performed via a token, which is available in the mount aws-token. The content of this mount is updated based on the response from AWS STS in step #7.

Conclusion

IRSA (Archive: [1], [2]) is in my opinion a very elegant way to increase security and address some of the most critical IAM challenges. The initial setup on Amazon EKS is normally done within a few minutes, afterwards it's a solution that just works.

Given the fact that the Pod Identity Webhook component is open source, it's also possible to use IRSA on other Kubernetes clusters, which are deployed directly on EC2 and using a CNCF compliant distribution.

With this deep dive into IRSA I wanted to share the benefits and help you to better understand the upcoming articles of this series about Backup and Logging with Rancher on AWS, where it's about how to get it working with IRSA and to avoid long-term credentials.

Article series Integrate Rancher with AWS services:

What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?
Rancher on AWS, Backup to S3 with IRSA for Authentication
Rancher on AWS, Logging to CloudWatch with IRSA for Authentication
Rancher on AWS, SAML Authentication with AWS IAM Identity Center as SAML IdP (coming soon)
Rancher on AWS, GitOps with Fleet and AWS CodeCommit (coming soon)

SUSE Certified Deployment Specialist in Rancher Kubernetes Engine 2

2023-06-21T00:00:00+02:00

Second day at SUSECON 23 in Munich, second Exam passed: SCDS in Rancher Kubernetes Engine 2. After passing SCA in NeuVector 5 yesterday, I can remove another one from my list. Let's see if I can also clear SCDS in Rancher Manager 2.7 for Rancher Prime tomorrow.

Summary: Thorough understanding in deploying, backing up, restoring, and upgrading Kubernetes clusters with RKE2 in a professional setting.

Source & Copyright: https://www.suse.com

Skills

Requirements for RKE2 Deployment
RKE2 Architecture and Design Concepts
Installation of RKE2 Clusters
RKE2 Installation Methods
RKE2 Manual Installation
RKE2 Installation with the Installer Script
RKE2 Configuration Options
Multi-node and Highly Available RKE2 Clusters
Installation of RKE2 Clusters with Auto Deployed Services
Kubernetes Addons
Helm Integration in RKE2
Configuring Deployment of Addons During the RKE2 Cluster Install
Installation of RKE2 Clusters with Advanced Networking
Cluster Networking Options in RKE2
Configuring Cluster Networking in RKE2
Configuring RKE2 to Provide Multiple Network Interfaces to Pods
Installation of CIS Hardened RKE2 Clusters
Installation of Air-gapped RKE2
Basic etcd Concepts
RKE2 Backup and Recovery Concepts
Back Up an RKE2 Cluster
Restore an RKE2 Cluster
Upgrading RKE2 Clusters
Upgrading RKE2 Clusters in an Air-gapped Environment

Source & Copyright: https://www.suse.com

Certificate

Downloads
- Certificate (ID 105) (PDF, 40K)
Links
- Verify Certificate
- SUSE Badges

SUSE Certified Deployment Specialist in SUSE Rancher and Kubernetes Distributions

2022-11-07T00:00:00+01:00

Never stop learning! Today I passed the Exam SUSE Certified Deployment Specialist in SUSE Rancher and Kubernetes Distributions, it's my second SUSE Rancher and Kubernetes related Certification, preparing for the Exam helped me to get an in-depth understanding of deploying and configuring various Rancher Kubernetes Distributions (RKE, RKE2, K3s), great extension to the SCA SUSE Rancher 2.6 I received a few days ago.

Summary: Knowledge how to deploy Kubernetes clusters with RKE, RKE2 and K3S, both from the command line and from SUSE Rancher as well as how to deploy Rancher on Kubernetes, also how to backup and restore RKE and RKE2 clusters from the command line and back up RKE with SUSE Rancher.

Source & Copyright: https://www.suse.com

Skills

Deploy RKE
- Plan a New Cluster
- Get Started with Deployment
- Configure RKE
- Deploy Kubernetes with RKE
- Work with config.yaml
- Add Nodes
- Troubleshoot
Deploy RKE2 and K3s
- Requirements
- Nodes
- Understand RKE2 Architecture
- Configure RKE2
- Deploy theInitial RKE2 Cluster
- Add Nodes
- Troubleshoot
- RKE2 and K3s
Deploy SUSE Rancher
- Deploy SUSE Rancher with HELM
- Deploy with Terraform
- Other Deployment Options
Deploy RKE with SUSE Rancher
- Deploy RKE on Existing Servers
- Deploy RKE on New Nodes on an Infrastructure Provider
- Deploy RKE with a Hosted Kubernetes Provider
Backup and Restore RKE and RKE2 Clusters
- etcd and Failure Recovery
- Back up and Restore RKE Cluster
- Backup and Restore RKE2 Cluster

Source & Copyright: https://www.suse.com

Certificate

Downloads
- Certificate (ID 134) (PDF, 121K)
Links
- Verify Certificate

SUSE Support Accreditation - SUSE Rancher

2022-11-07T00:00:00+01:00

There was only on Certification left on my SUSE Learning Path, SPSA SUSE Rancher, which I received today, after meeting the Pre-Requirements by holding the SUSE Rancher SCA and SCDS Certification.

It was, similar as SPSA SLES great strengthen my knowledge about troubleshooting, tools and best practices to interact with the SUSE Support Team with a focus on SUSE Rancher.

Skills

Maintenance
Basic Health Checks
Troubleshooting
Gathering System Information for support
Interacting with SUSE
Support FAQs

Source & Copyright: https://www.suse.com

Certificate

Downloads
- Certificate (ID 100) (PDF, 121K)
Links
- Verify Certificate