The Wombelix Post

Overview

Rancher provides the rancher-logging Helm chart, which is based on the kube-logging operator, it's using Fluent Bit to collect and Fluentd to forward the logs. One of the supported targets is Amazon CloudWatch.

With enhanced cloud provider logging, logs from Amazon EKS will be collected and pushed to CloudWatch as well.

IRSA is technically supported but the necessary serviceAccount annotation need to be added after the installation in a separate step.

The official documentation about Rancher Integration with Logging Services provides further information about the functionality and general installation.

IAM Policy

You need a IAM Policy to create later the IAM Role linked to a Kubernetes service account. An example how such a policy could look like to push logs to CloudWatch, based on the out_cloudwatch_logs example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

IRSA Configuration

The kube-logging operator already supports IRSA but it can't be configured as part of the Helm installation with the rancher-logging Helm chart yet. The annotation need to be added afterwards to the Logging Resources, this is different compared to Rancher Backup, where this is possible directly as part of the installation.

I created a GitHub Pull Request to include the serviceAccount annotation in the Helm chart.

In the meantime, you have to edit the Logging Resources rancher-logging-root and (if Rancher is running on Amazon EKS with enabled enhanced cloud logging) rancher-logging-eks in namespace cattle-logging-system manually.

The configuration is not covered in the official Rancher Documentation yet. Six lines need to be added per Logging Resource, each with it's own role-arn:

spec:
  fluentd:
    serviceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/my-rancher-logging-role

Besides the serviceAccount annotation, the initial IRSA setup for the cluster and the creation of the IAM Role, with a link to the service account, used by the kube-logging operator, is required.

The names of the service accounts rancher-logging will create are rancher-logging-root-fluentd and (in case of EKS and enhanced logging) rancher-logging-root-fluentd, also in namespace cattle-logging-system. You have to create a IAM Role and link it to those accounts by using eksctl as shown in the first article What is IAM Roles for Service Accounts (IRSA) and Amazon EKS Pod Identity Webhook?.

ClusterOutput

To push logs to the actual target, Fluentd is using so called Outputs. There are two types, Output (namespace) and ClusterOutput (cluster-wide), further reading.

Following an example how a basic ClusterOutput could look like, I named it testcloudwatchoutput and it will push all available logs to the group rancher-demo-cluster-log-group and stream rancher-demo-cluster-log-stream in Amazon CloudWatch of AWS Region us-east-1.

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  creationTimestamp: '2023-05-17T14:03:16Z'
  generation: 4
  managedFields:
    - apiVersion: logging.banzaicloud.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:active: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-05-17T14:03:16Z'
    - apiVersion: logging.banzaicloud.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:cloudwatch:
            .: {}
            f:auto_create_stream: {}
            f:log_group_name: {}
            f:log_stream_name: {}
            f:region: {}
      manager: rancher
      operation: Update
      time: '2023-05-18T10:14:12Z'
  name: testcloudwatchout
  namespace: cattle-logging-system
  resourceVersion: '16031275'
  uid: a4bd1852-eca8-487b-bdc8-47d9966e6da2
spec:
  cloudwatch:
    auto_create_stream: true
    log_group_name: rancher-demo-cluster-log-group
    log_stream_name: rancher-demo-cluster-log-stream
    region: us-east-1
status:
  active: true