As CircleCI continues to investigate the security incident affecting its continuous integration and continuous delivery (CI/CD) platform, enterprise defenders should also be hunting for signs of malicious activities on third-party applications integrated with CircleCI.
In its Jan. 4 disclosure, CircleCI urged users to rotate all secrets stored within the platform and to check internal logs for any signs of “unauthorized access” starting from Dec. 21, 2022. Since enterprises integrate software-as-a-service (SaaS) applications and other cloud providers, defenders should also hunt for signs of malicious behavior on those environments as well.
Step 1: Change Secrets
The first step is to change all passwords, secrets, access tokens, environment variables, and public-private keypairs because the attackers may have stolen them. When organizations integrate CircleCI with other SaaS and cloud providers, they provide CircleCI with those authentication tokens and secrets. The breach with CircleCI means the platform itself is compromised, as are all the SaaS platforms and cloud providers integrated with CircleCI because those credentials are now exposed.
CircleCI is offering a script CircleCI-Env-Inspector to export a JSON-formatted list of the names of CI secrets that need to be changed. The list would not contain the values of the secrets, CircleCI said.
To run this script, clone the repository and execute the run.sh file.
In subsequent updates, CircleCI said it has invalidated Project API tokens used by projects and that it has rotated all GitHub OAuth tokens on behalf of customers. Amazon Web Services is alerting customers via email with lists of potentially impacted tokens (subject line: [Action Required] CircleCI Security Alert to Rotate Access Keys.) that customers should change.
For organizations using TruffleHog, the log scanning feature outputs any passwords or API keys that may have been accidentally logged. Run TruffleHog with the following flags:
trufflehog circleci –token=<token>
Step 2: Check CircleCI for Suspicious Activity
CircleCI has made self-serve audit logs available to all customers, including free customers, through the platform’s user interface. Customers can query up to 30 days of data and have 30 days to download the resulting logs. CircleCI’s documentation outlines how to use the logs.
The logs provide information about actions taken, by which actor, on which target, and at what time, according to a threat hunting guide from Mitiga. Look for log entries indicating actions taken by a CircleCI user during the time between Dec. 21, 2022, and when the secrets were changed and updated. Actions attackers may be interested in are those for gaining access (user.logged_in) and maintaining persistence (project.ssh_key.create, project.api_token.create, user.create).
Step 3: Hunt for Malicious Actors in Third-Party Apps
The impact of the breach extends beyond CircleCI as it includes third-party applications that are integrated with the development platform, such as GitHub, Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Enterprise defenders need to hunt for signs of malicious activity across each of the integrated SaaS applications and cloud providers.
For GitHub: CircleCI authenticates to GitHub via PAT, an SSH key, or locally generated private and public keys. Defenders should check GitHub security log for suspicious GitHub activity – such as git.clone (copying the repository), git.fetch and git.pull (different ways of grabbing the code from the repository) – originating from CircleCI users, according to Mitiga’s threat hunting guide. The GitHub Audit logs provide information about the actions performed, who performed the action, and when it was performed. Check the GitHub Audit logs containing actor_location and look for abnormal connections and operations originating from new IP addresses.
For AWS: Look at API management events actions in AWS CloudTrail’s management activity logs. Search for events the CircleCI user shouldn’t be performing, such as suspicious reconnaissance actions (for example, ListBuckets GetCallerIdentitiy), AccessDenied events, and activity originating from unknown IP addresses and programmatic UserAgents (such as boto3 and CURL).
For GCP: Review Cloud Audit logs – Admin Activity audit logs, Data Access audit logs, and Policy Denied audit logs – via the Google Cloud console (Logs Explorer), the Google Cloud CLI, or the Logging API. Check which resources the service account used with CircleCI has permissions.
The API call:
searchAllIamPolicies
From the command line:
gcloud asset search-all-iam-policies
Search for abnormalities, such as an error severity record, weird timestamps, or unusual IP subnets, Mitiga recommends in its guide.
For Azure: Review sign-in errors and patterns in Azure Active Directory Sign-in logs and check for abnormalities, such as the date of the sign-in and the source IP address. The Azure Monitor activity log is a platform log in Azure providing information about subscription-level events such as when a resource is modified or a virtual machine is started. One thing to look for in this log is whether there are actions listed that are different from the ones the service account typically performs.
“Hunting for malicious actions done by compromised CI/CD tools in your organization is not trivial, because their scope goes beyond that CI/CD tool and affects other SaaS platforms integrated with it,” Mitiga’s team wrote in the guide.
