Tibor Hercz
Writing about Cloud, architecture, AWS, GCP and software engineering.

What happens when you leak AWS credentials and how AWS minimizes the damage

April 5, 2023

I heard multiple times that AWS scans public GitHub repositories for AWS credentials and informs its users of the leaked credentials.

So I am curious to see this for myself, so I decided to intentionally leak AWS credentials to a Public GitHub repository. And show you the steps I took and how I got informed about the leaked credentials.

Setting up the credentials

I created a IAM user in my AWS account named test-user and generated an access key and secret for this user and attached a very limited policy to this user.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::aws-leaked-credentials"
        }
    ]
}

I then pushed the AWS access key and secret to GitHub see the commit here: Commit: 33b4d387e94e16b1c8b9277056299bdb02de3a4b.

Very quickly after pushing the credentials to GitHub various things happened which I will show below.

Timestamps and events after leaking the credentials to GitHub

Here you will find all the events that happened after pushing to GitHub.

12:33:12 - Pushed the credentials to GitHub

12:34:19 - The AWSCompromisedKeyQuarantineV2 policy is attached to the IAM user test-user by AWS

12:34:32 - Various List and Describe calls are made using the leaked credentials

12:35:08 - Received an email from AWS with the subject ‘ACTION REQUIRED: Your AWS Access Key is Exposed for AWS Account 12345678’

As you can see 1 minute and 7 seconds after leaking the credentials AWS added the AWSCompromisedKeyQuarantineV2 policy. Because AWS IAM is eventually consistent the malicious actor was able to perform various API calls even after the AWSCompromisedKeyQuarantineV2 policy has been added. Luckily this was short-lived because IAM has little delay for changes to take effect. 2 minutes after leaking the credentials I received an email from AWS informing me of the event and providing instructions on how to secure the account.

Below you will find detailed information about every event.

Policy attached by AWS

The AWSCompromisedKeyQuarantineV2 is attached to the IAM user test-user. This policy denies the most important actions. But if the leaked credentials have a lot of permissions the malicious actor could still do damage to systems running in the AWS account.

See the policy below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "cloudtrail:LookupEvents",
        "ec2:RequestSpotInstances",
        "ec2:RunInstances",
        "ec2:StartInstances",
        "iam:AddUserToGroup",
        "iam:AttachGroupPolicy",
        "iam:AttachRolePolicy",
        "iam:AttachUserPolicy",
        "iam:ChangePassword",
        "iam:CreateAccessKey",
        "iam:CreateInstanceProfile",
        "iam:CreateLoginProfile",
        "iam:CreatePolicyVersion",
        "iam:CreateRole",
        "iam:CreateUser",
        "iam:DetachUserPolicy",
        "iam:PassRole",
        "iam:PutGroupPolicy",
        "iam:PutRolePolicy",
        "iam:PutUserPermissionsBoundary",
        "iam:PutUserPolicy",
        "iam:SetDefaultPolicyVersion",
        "iam:UpdateAccessKey",
        "iam:UpdateAccountPasswordPolicy",
        "iam:UpdateAssumeRolePolicy",
        "iam:UpdateLoginProfile",
        "iam:UpdateUser",
        "lambda:AddLayerVersionPermission",
        "lambda:AddPermission",
        "lambda:CreateFunction",
        "lambda:GetPolicy",
        "lambda:ListTags",
        "lambda:PutProvisionedConcurrencyConfig",
        "lambda:TagResource",
        "lambda:UntagResource",
        "lambda:UpdateFunctionCode",
        "lightsail:Create*",
        "lightsail:Delete*",
        "lightsail:DownloadDefaultKeyPair",
        "lightsail:GetInstanceAccessDetails",
        "lightsail:Start*",
        "lightsail:Update*",
        "organizations:CreateAccount",
        "organizations:CreateOrganization",
        "organizations:InviteAccountToOrganization",
        "s3:DeleteBucket",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:PutLifecycleConfiguration",
        "s3:PutBucketAcl",
        "s3:PutBucketOwnershipControls",
        "s3:DeleteBucketPolicy",
        "s3:ObjectOwnerOverrideToBucketOwner",
        "s3:PutAccountPublicAccessBlock",
        "s3:PutBucketPolicy",
        "s3:ListAllMyBuckets",
        "ec2:PurchaseReservedInstancesOffering",
        "ec2:AcceptReservedInstancesExchangeQuote",
        "ec2:CreateReservedInstancesListing",
        "savingsplans:CreateSavingsPlan"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

CloudTrail event

In the us-east-1 CloudTrail logs you can see that a AttachUserPolicy API call is made to attach the AWSCompromisedKeyQuarantineV2 policy. In the event record it clearly states that this has been invoked by AWS "invokedBy": "AWS Internal".

CloudTrail event:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDAxxxxxxxxx",
        "arn": "arn:aws:iam::xxxxxxxxx:user/test-user",
        "accountId": "xxxxxxxxx",
        "accessKeyId": "ASIAxxxxxxxxx",
        "userName": "test-user",
        "sessionContext": {
            "sessionIssuer": {},
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2023-03-21T11:34:19Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "AWS Internal"
    },
    "eventTime": "2023-03-21T11:34:19Z",
    "eventSource": "iam.amazonaws.com",
    "eventName": "AttachUserPolicy",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "AWS Internal",
    "userAgent": "AWS Internal",
    "requestParameters": {
        "userName": "test-user",
        "policyArn": "arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantineV2"
    },
    "responseElements": null,
    "requestID": "7b163d6c-xxxxxxxxx",
    "eventID": "c3eda312-xxxxxxxxx",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "xxxxxxxxx",
    "eventCategory": "Management"
}

Malicious actor?

I found it surprising how quickly the malicious actor made some automated API calls with the leaked credentials. Meaning that they scan GitHub public repositories constantly for leaked credentials.

The user agent used was aws-sdk-go-v2/1.17.1 so they probably have an automated tool that detects leaked credentials and start scanning the AWS account for resources. In the image below you see the various API calls they made, luckily without any luck because they didn’t have the permissions for these API calls.

Below you will find the CloudTrail log of the ListCertificates API call:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDAxxxxxxxxx",
        "arn": "arn:aws:iam::xxxxxxxxx:user/test-user",
        "accountId": "xxxxxxxxx",
        "accessKeyId": "AKIAYUBS5O3BYC7WBJWO",
        "userName": "test-user"
    },
    "eventTime": "2023-03-21T11:34:36Z",
    "eventSource": "acm.amazonaws.com",
    "eventName": "ListCertificates",
    "awsRegion": "eu-west-1",
    "sourceIPAddress": "3.xxx.xx.xxx",
    "userAgent": "aws-sdk-go-v2/1.17.1 os/linux lang/go/1.18.10 md/GOOS/linux md/GOARCH/amd64 api/acm/1.15.2",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:iam::xxxxxxxxx:user/test-user is not authorized to perform: acm:ListCertificates because no identity-based policy allows the acm:ListCertificates action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "b83e5733-xxxxxxxxx",
    "eventID": "37d59144-xxxxxxxxx",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "xxxxxxxxx",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "acm.eu-west-1.amazonaws.com"
    }
}

Email from AWS

Within two minutes of leaking the credentials I received an email from AWS informing me that the credentials have been leaked. See the first part of the email below:

Hello,

We have become aware that the AWS Access Key AKIAYUBS5O3BYC7WBJWO , belonging to IAM User test-user , along with the corresponding Secret Key is publicly available online at https://github.com/tiborhercz/aws-leaked-credentials/blob/33b4d387e94e16b1c8b9277056299bdb02de3a4b/credentials.txt.

Your security is important to us and this exposure of your account’s IAM credentials poses a security risk to your AWS account, could lead to excessive charges from unauthorized activity, and violates the AWS Customer Agreement or other agreement with us governing your use of our Services.

To protect your account from excessive charges and unauthorized activity, we have applied the “AWSCompromisedKeyQuarantineV2” AWS Managed Policy (“Quarantine Policy”) to the IAM User listed above. The Quarantine Policy applied to the User protects your account by denying access to high risk actions like iam:CreateAccessKey and ec2:RunInstances. You can view all the actions denied by the policy by going here: https://console.aws.amazon.com/iam/home#policies/arn:aws:iam::aws:policy/AWSCompromisedKeyQuarantineV2$jsonEditor?section=permissions.

In the email they also give guidance on how to secure the account. They do this in four steps:

  1. Rotate and delete the exposed AWS Access Key AKIAYUBS5O3BYC7WBJWO
  2. Check your CloudTrail log for unsanctioned activity
  3. Review your AWS account for any unauthorized AWS usage
  4. You must respond to the existing Support Case or create a new one to confirm completion of the steps above in order to restore access to your account, prevent suspension, and apply for a billing adjustment, if applicable.

How does AWS achieve this

How does AWS respond this fast when credentials are leaked is something I am not 100% sure about, because I do not have confirmation from AWS itself.

How they could achieve this is by using GitHub ‘Secrets Scanning’ service and using the ‘Secret scanning alerts for partners’. GitHub will then report leaked secrets to these partners, when receiving this report AWS has some automated system that adds the AWSCompromisedKeyQuarantineV2 policy, sends the email and opens a support ticket.

Conclusion

When leaking AWS credentials to GitHub, AWS has safeguards in place to protect you. AWS and the malicious actors act impressively fast on the leaked credentials both taking action immediately. AWS informs the root user and adds a Deny policy to the IAM user. The malicious actor starts to scan the account for resources they can exploit.

It is very nice that AWS protects its users in this way, but we could better prevent the leaking of the credentials. There are various ways of preventing leaking the credentials.

Read more about the actions you can take to minimize the damage if the AWS credentials are leaked here Steps to take after leaking AWS credentials