Sky Cone

You can use the Amazon EKS add-on to install Container Insights with enhanced observability for Amazon EKS. The add-on installs the CloudWatch agent to send infrastructure metrics from the cluster, installs Fluent Bit to send container logs, and also enables CloudWatch Application Signals to send application performance telemetry.

The Amazon EKS add-on is not supported for clusters running Kubernetes instead of Amazon EKS.

For more information about the Amazon CloudWatch Observability EKS add-on, see Install the CloudWatch agent by using the Amazon CloudWatch Observability EKS add-on.

To install the Amazon CloudWatch Observability EKS add-on
1. Create an IAM role, e.g., AmazonEKS_CloudWatch_Observability_Role.

2. Set up the necessary permissions by attaching the CloudWatchAgentServerPolicy and AWSXrayWriteOnlyAccess IAM policies to the role dedicated for this add-on.

Note
The aws iam attach-role-policy command in AWS CLI version 2 only allows you to attach one IAM policy to a role at a time. You cannot specify multiple policy ARNs in a single command.

Prepare a script to attach multiple IAM policies to the IAM role.
script.sh:

roleName="AmazonEKS_CloudWatch_Observability_Role"
declare -a policyArns=("arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy" "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess")

for policyArn in "${policyArns[@]}"
do
  aws iam attach-role-policy --role-name "$roleName" --policy-arn "$policyArn"
done

Grant permissions to the current user so that you can execute this script file.
% chmod 744 script.sh

% ./script.sh



2. Enter the following command to install the add-on:
% eks_cluster_name=<EKS cluster name>
% aws eks create-addon --cluster-name $eks_cluster_name --addon-name amazon-cloudwatch-observability --service-account-role-arn  "$(aws iam get-role --role-name AmazonEKS_CloudWatch_Observability_Role --query 'Role.Arn' --output text)"
% aws eks create-addon --cluster-name <EKS cluster name> --addon-name amazon-cloudwatch-observability --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role
An example output is as follows.

{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "<EKS cluster name>",
        "status": "CREATING",
        "addonVersion": "v1.2.1-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:us-west-2:111122223333:addon/<EKS cluster name>/amazon-cloudwatch-observability/e0c670e8-xxxx-xxxx-xxxx-cb53bd9b82c9",
        "createdAt": "2024-01-07T10:46:09.391000+08:00",
        "modifiedAt": "2024-01-07T10:46:09.427000+08:00",
        "serviceAccountRoleArn": "arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role",
        "tags": {}
    }
}

If encounter below error, add the "--resolve-conflicts Overwrite" flag.

Issue type	Description	Affected resources
ConfigurationConflict	Conflicts found when trying to apply. Will not continue due to resolve conflicts mode. Conflicts: ClusterRole.rbac.authorization.k8s.io cloudwatch-agent-role - .rules	-

% aws eks create-addon --cluster-name <EKS cluster name> --addon-name amazon-cloudwatch-observability --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role --resolve-conflicts Overwrite

{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "<EKS cluster name>",
        "status": "CREATING",
        "addonVersion": "v1.2.1-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:us-west-2:111122223333:addon/<EKS cluster name>/amazon-cloudwatch-observability/e0c670e8-xxxx-xxxx-xxxx-cb53bd9b82c9",
        "createdAt": "2024-01-07T10:46:09.391000+08:00",
        "modifiedAt": "2024-01-07T17:26:10.505000+08:00",
        "serviceAccountRoleArn": "arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role",
        "tags": {}
    }
}

Encountered below error, update K8s service account and the trust policy of the IAM role.
% k get po -n amazon-cloudwatch

NAME                                                              READY   STATUS             RESTARTS      AGE
...
cloudwatch-agent-6wlbz                                            1/1     Running            1 (9s ago)    20s
cloudwatch-agent-hxsm6                                            1/1     Running            1 (9s ago)    20s
cloudwatch-agent-lh7sk                                            0/1     Pending            0             35m
cloudwatch-agent-m9mtp                                            1/1     Running            1 (10s ago)   20s
cloudwatch-agent-pchvn                                            0/1     CrashLoopBackOff   3 (15s ago)   97s
cloudwatch-agent-tbsqh                                            1/1     Running            1 (9s ago)    19s

% k logs -n amazon-cloudwatch cloudwatch-agent-xxxx

...
2024/01/07 10:05:03 Describe EC2 Tag Fail. Will retry the request: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::124011853020:assumed-role/AmazonEKS_CloudWatch_Observability_Role/1704621903483664002 is not authorized to perform: ec2:DescribeTags because no identity-based policy allows the ec2:DescribeTags action
	status code: 403, request id: 492e3f12-48b8-43e7-b633-32c62354c0f2
...

Determine the service account name to patch.
% k describe ds -n amazon-cloudwatch fluent-bit | grep "Service Account"

  Service Account:  cloudwatch-agent

% k describe ds -n amazon-cloudwatch cloudwatch-agent | grep "Service Account"

  Service Account:  cloudwatch-agent

% k get sa -n amazon-cloudwatch cloudwatch-agent -o jsonpath='{.metadata.annotations}'

{"eks.amazonaws.com/role-arn":"arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{},\"name\":\"cloudwatch-agent\",\"namespace\":\"amazon-cloudwatch\"}}\n"}%

If the ARN of the IAM role for cloudwatch-agent and fluent-bit to assume is not listed from the return message, follow below step. If not, skip the following step.
Prepare the file to patch the service account for patch-cw-agent-ds.yaml

metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/AmazonEKS_CloudWatch_Observability_Role

Update the trust policy.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/BFDB***D49F"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringEquals": {
					"oidc.eks.us-west-2.amazonaws.com/id/BFDB***D49F:aud": "sts.amazonaws.com",
					"oidc.eks.us-west-2.amazonaws.com/id/BFDB***D49F:sub": "system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"
				}
			}
		}
	]
}

Add the "oidc.eks.us-west-2.amazonaws.com/id/BFDB***D49F:sub" line under "Condition".

Note
Understanding the Components:

• oidc.eks.us-west-2.amazonaws.com/id/*****: This is the issuer URL of the OIDC provider for your EKS cluster. The ***** represents the unique ID of your EKS cluster. Each EKS cluster has its own OIDC provider with a unique URL.
• :sub: This stands for "subject", which is a standard claim in OpenID Connect. In the context of EKS, the sub field is used to represent the unique identifier of a Kubernetes service account in the format system:serviceaccount:<namespace>:<service-account-name>.

-

...
        Kube_Filter    kubernetes.labels.app=blog



[FILTER]
        Name                kubernetes
        ...
        Kube_Filter         labels=app=blog
...

-

...
    [FILTER]
        K8S-Logging.Exclude On
...

-
% k delete po -n amazon-cloudwatch -l k8s-app=fluent-bit

pod "fluent-bit-6xk4j" deleted
pod "fluent-bit-8xr6l" deleted
pod "fluent-bit-r2nrl" deleted
pod "fluent-bit-vkjxg" deleted

-

References

Install the Amazon CloudWatch Observability EKS add-on

https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml