By using the Kubernetes Secrets Store CSI Driver you can provide pods with secrets from the AWS Secret Manager. This allows you to use the features the Secrets Manager has to offer within your EKS cluster.

The Secrets Store CSI driver mounts secrets from external stores into your pods as volumes. Secret Store providers are available for AWS, Azure, Google and HashiCorp Vault. These providers allow secrets store integration with your Kubernetes cluster. This means your application doesn’t have to implement custom code to interact with these secret stores.

The Secrets Store CSI driver allows you to sync secrets with the Kubernetes Secrets by enabling the Secret Sync so they can be defined as environment variables in pods. Also supported is key rotation but as of writing this its still in Alpha.

To integrate the AWS Secret Manager with Kubernetes you use the ‘AWS Secrets and Configuration Provider’ (ASCP), a plugin for the Secrets Store CSI driver. The provider retrieves the secrets from the Secret Manager and parameters from the Parameters store and passes them to the Secrets Store CSI driver.

Installation guide

In the guide below you will find how to set up the CSI driver with the AWS Secrets Manager. The Secret sync is enabled, so you can define secrets from the Secrets Manager as environment variables in your pods.

1. Secrets Store CSI Driver

To get a more detailed installation guide for the Secrets Store CSI Driver see installation

Or run the following commands (syncSecret parameter is set to true):

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true --namespace kube-system

Optional helm parameters

The following features are not enabled by default and can be enabled by setting helm parameters.

For a list of all values that can be customized when running helm install see helm configuration

2. AWS Secrets and Configuration Provider (ASCP)

To install the ASCP use to following command:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

3. IAM role

Create an IAM role with the following policy to allow access to the Secret Manager. This IAM role will be attached to a Kubernetes ServiceAccount.

Minimal policy

Minimal permissions needed to get secrets from the Secret Manager.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:*:secretsmanager:*:*:secret:MySecret-??????"
        }
    ]
}

Policy for secrets encrypted with KMS

When secrets are encrypted with KMS the user requesting the secrets must be able to get the KMS key used to encrypt the secret. The permissions needed for decrypting the secrets are kms:GenerateDataKey & kms:Decrypt

To make sure the KMS key is only accessible by the Secrets Manager use the kms:ViaService condition key with the value secretsmanager.AWS_REGION.amazonaws.com.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "kms:GenerateDataKey",
              "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
              "StringEquals": {
                "kms:CallerAccount": [
                  "AWS_ACCOUNT_ID"
                ],
                "kms:ViaService": [
                  "secretsmanager.AWS_REGION.amazonaws.com"
                ]
              }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:*:secretsmanager:*:*:secret:MySecret-??????"
        }
    ]
}

4. Kubernetes service account

The ServiceAccount gives your pods access to the Secret Manager with the previously created role.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-manager-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>

5. Create SecretProviderClass

You configure one or more secrets you need, through the Kubernetes custom resource SecretProviderClass. The pod mounts the secrets as a volume from this custom resource.

The secretObjects is used to sync the secrets with Kubernetes Secrets. (only works when Secret Sync is enabled)

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: secrets-provider
spec:
  provider: aws
  secretObjects:
    - secretName: database_password
      type: Opaque
      data:
        - objectName: "MySecretPassword"
          key: password
  parameters:
    objects: |
      - objectName: arn:*:secretsmanager:*:*:secret:MySecret-??????
        objectAlias: "MySecretPassword"      

JSON formatted secret

If your secret from the AWS Secrets Manager is a JSON-formatted secret use jmesPath. This allows you to retrieve a specific key-value pair from the JSON.

Example JSON secret:

{
    "username": "username",
    "password": "password"
}

To retrieve the username and password from the JSON use jmesPath as follows:

  parameters:
    objects: |
      - objectName: arn:*:secretsmanager:*:*:secret:MySecret-??????
        jmesPath:
            - path: "username"
              objectAlias: "MySecretUsername"
            - path: "password"
              objectAlias: "MySecretPassword"      

6. Configure the volume for the pod

Below you will find a Kubernetes manifest which show how to add the SecretProviderClass as a volume. Making the secrets available as a file which can be found in /mnt/secrets-store. Or the secrets can be defined as environment variables in a pod.

kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  serviceAccountName: secret-manager-service-account # The ServiceAccount with permissions to access the Secret Manager secret 
  containers:
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
    env:
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: database_password # Name of the secret in Kubernetes Secrets, which has been set in the secretProviderClass 
          key: password
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "secrets-provider" # Name of the secretProviderClass

Sources

• GitHub: secrets-store-csi-driver
• Secrets Store CSI Driver helm configuration: helm configuration
• GitHub: secrets-store-csi-driver-provider-aws
• AWS Doc: Use Secrets Manager secrets in Amazon Elastic Kubernetes Service
• Doc: Kubernetes Secrets Store CSI Driver