EKS — Accessing AWS services from Fargate pods

Image copied from ItNext blog here

There is always need to access other AWS services from inside the pods launched by EKS on fargate.

One way to achieve this, is by mounting the AWS access/secret creds as environment vars in the pod.

apiVersion: v1
kind: Pod
metadata:
  name: bb
spec:
  containers:
  - image: busybox
    name: bb
    env:    
    - name: 'AWS_ACCESS_KEY_ID'
      value: '<AWS_ACCESS_KEY_ID>'
    - name: 'AWS_SECRET_ACCESS_KEY'
      value: '<AWS_SECRET_ACCESS_KEY>'
    - name: 'AWS_DEFAULT_REGION'
      value: '<AWS_DEFAULT_REGION>'

but these are secrets and they are better to be created as secrets and mounted as env in pod. You can create secret with some name <MY_AWS_SECRET> with below command:

kubectl create secret generic <MY_AWS_SECRET> --from-literal "AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>" --from-literal "AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>" --from-literal "AWS_DEFAULT_REGION=<AWS_DEFAULT_REGION>"

Use the pod spec envFrom key to mount all its keys as env in pod.

apiVersion: v1
kind: Pod
metadata:
  name: bb
spec:
  containers:
  - image: busybox
    name: bb
    envFrom:
    - secretRef:
        name: <SECRET_NAME>

If you do not want to manage secrets in Kubernetes for whatever reasons, you can use AWS Secrets manager to manage/rotate secrets as described at https://aws.amazon.com/blogs/containers/aws-secrets-controller-poc/ but the article suggests that this is not intended for production use. Also AWS secret manager has it’s own pricing for storing and accessing secrets. You could also encrypt secrets as explained here

All this aside, it’s always good practice to use iam roles attached to instance to access AWS services instead of access keys. This is described at https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

We will essentially be creating a serviceaccount in Kubernetes and associate this to an AWS iam role.

NOTE: One might think that we could just attach an iam role to fargate pod and access services using this, similar to an EC2 instance role. But as doc suggests, the Pod execution role which we attach to each fargate pod is only used by kubelet to register as Node with EKS cluster. The containers running in the Fargate pod cannot assume the IAM permissions associated with the pod execution role. Doc at https://docs.aws.amazon.com/eks/latest/userguide/pod-execution-role.html

Now let’s go do this:

1. Create an IAM OIDC provider for your cluster — Chances are you already have oidc provider in your cluster if you created it with eksctl. if not, you can easily enable it using above doc link. You will get an oidc connect url in EKS cluster details page in console if it’s enabled. Note down the oidc connect url, which will be used in later steps.
2. Create an IAM role and attach an IAM policy to it with the permissions that your service accounts need — Create an IAM role with web identity provider url(select the oidc url from above), attach necessary permissions and save the role ARN you got. Steps in the above doc should be straight forward.

NOTE: For service account name and namespace in the IAM role, put the name of the service account you intend to create. It’s ok even if the service account isn’t yet created.

3. Associate an IAM role with a service account — Create a service account in kubernetes with annotation which helps associate serviceaccount with the IAM role.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: <ROLE_ARN>
  name: <MY_SERVICE_ACCOUNT_NAME>
...

4. Enable pod to use the service account we created.

apiVersion: v1
kind: Pod
metadata:
  name: bb
spec:
  containers:
  - image: busybox
    name: bb
  serviceAccountName: <MY_SERVICE_ACCOUNT_NAME>

We need to delete and re-create pods to use this service account. To confirm if everything went alright with this process, you should be able to see the below env vars in pod.

AWS_ROLE_ARN=<ROLE_ARN>
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

If you have aws cli in your pod, you can just do:

aws sts get-caller-identity