The Wombelix Post

AWS IAM Role and Policy

To grant an application access to AWS services, you need an AWS IAM Role and Policy. A IAM Role contains a IAM Permission policy and a IAM Trust relationship. The policy defines what can be done with the specified service, for example, uploading files to a Amazon Simple Storage Service (S3) Bucket. The relationship defines who can assume the IAM Role to perform the actions based on the attached permission policy.

IAM Permission policy >> IAM Role << IAM Trust relationship

Amazon Elastic Compute Cloud (EC2) Instance IAM Role

When a IAM Role is attached to an EC2 Instance, every application running on this specific instance has the same level of permissions. If you have one single application running, it's good practice to grant permissions that way. But imagine you have multiple applications running, for example in different containers like in a Kubernetes Environment, then all of them inherit the same set of permissions.

For Example: The IAM Role grants write access to a S3 Bucket, because container #1 is supposed to upload files, by using an EC2 Instance IAM Role, container #2 and container #3 can also write to the same S3 Bucket. Those container run a complete different application, maybe even outside your own control. Worst case, this could cause data corruption, data loss or even a data leak.

container 1 --|     IAM Role
              |        |
container 2 --|-- EC2 Instance -- S3 Bucket
              |
container 3 --|

AWS IAM access key credentials

Long-term credentials should be avoided whenever possible from a security best practice point of view. AWS access keys are linked to a IAM user and have no expire date. They could be leaked or shared between multiple applications and therefore might have to broad permissions assigned.

I think it's obvious why those type of credentials should not be used, in my opinion not even during development. Start early in the process to replace long-term with short-term credentials!

IRSA under the hood

Personally I'm super excited about IRSA and how it works, I think it should be used for every Kubernetes workload on AWS, there is really no reason at all to stick with EC2 Instance IAM Roles or even access key credentials.

That's why I want to dive deeper into how IRSA works and what magic is happening under the hood.

1. A reference between the EKS Cluster and IAM is established via OIDC. This is a one-time setup per cluster and can be done via eksctl. Example:

   eksctl utils associate-iam-oidc-provider --cluster <CUSTER_NAME> --approve
   
   

2. A reference between a Kubernetes service account and a IAM Role has to be created. This can be done via eksctl in two ways, either with an account and the role managed by EKS, or by assigning a role to an existing Kubernetes service account. In case of Rancher, it's option two because Rancher creates and manages Kubernetes service accounts on it's own. Example:

   eksctl create iamserviceaccount --name <SA_NAME> --namespace <NS_NAME> --cluster <CLUSTER_NAME> \
     --role-name <ROLE_NAME> --attach-policy-arn <IAM_POLICY_ARN> --approve --role-only
   
   

3. The Kubernetes resource is configured with a appropriate service account annotation, for example as part of a installation via Helm or by adjusting a Manifest.

4. As soon a Pod with a service account annotation comes up, the Pod Identity Webhook will be triggered and reconfigure (mutate) the Pod to use IRSA

5. The Pod assumes the specified IAM Role and connects to the AWS Security Token Service

6. AWS STS verifies the request by contacting AWS IAM

7. If the request could be verified and is valid, AWS STS assigns temporary credentials

AWS CLI and applications leveraging the AWS SDK, can now interact with AWS services based on the permissions of the IAM Role, without the need of EC2 Instance IAM Roles or long-term access key credentials.

As soon the temporary credentials are expired, the process automatically starts over from step #5 to get a new set of temporary credentials, there is no manual interaction required as soon step #1 till #3 are completed.

Further information and examples can be found in the AWS Documentation about IRSA on Amazon EKS.

Pod Identity Webhook

I explained that the Pod Identity Webhook performs a reconfiguration / mutation in step #4 of the IRSA Architecture Diagram. What happens in this step, is that the following Environment variables and Volumes are added to the Pod:

env:
- name: AWS_DEFAULT_REGION
  value: us-west-2
- name: AWS_REGION
  value: us-west-2
- name: AWS_ROLE_ARN
  value: "arn:aws:iam::111122223333:role/s3-reader"
- name: AWS_WEB_IDENTITY_TOKEN_FILE
  value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
- name: AWS_STS_REGIONAL_ENDPOINTS
  value: "regional"

volumeMounts:
- mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/"
  name: aws-token

volumes:
- name: aws-token
  projected:
    sources:
    - serviceAccountToken:
        audience: "sts.amazonaws.com"
        expirationSeconds: 86400
        path: token

The region and role arn values are example data and set according to your IAM Role configuration performed in step #2. The Environment variables are used by AWS CLI or SDK to understand that the authentication need to be performed via a token, which is available in the mount aws-token. The content of this mount is updated based on the response from AWS STS in step #7.