CircleCI incident adds to SecOps toil – TechTarget
January 5, 2023
posted on
Editor’s note: This story was updated to reflect additional information issued by CircleCI Thursday around 7 p.m. ET on its official blog post.
SecOps pros in CircleCI environments potentially faced hours of work to rotate all secrets data stored in their repositories in response to a security advisory from the SaaS CI/CD vendor this week.
CircleCI issued its first statement on its official blog and via email to users Wednesday about a security incident it was investigating between Dec. 21 and Jan. 4. The statement recommended that users “immediately rotate any and all secrets stored in CircleCI.”
The recommendation was made out of “an abundance of caution,” according to the statement, but no further details about the scope of the incident or how CircleCI has addressed it internally were given Wednesday. The vendor followed this with a set of instructions Thursday for how to perform a secrets rotation.
Around 7 p.m. ET Thursday, CircleCI provided an update on its official blog post that included a link to an open source tool for discovering secrets in CircleCI environments in response to customer requests and also made audit log access free for all customers up to 30 days. The blog update did not specify how the incident occurred or its scope, other than to say it was not related to a reliability update on Dec 21.
The post also said CircleCI had rotated all the access keys in its production environment and completed an audit of all system access, and re-emphasized that customers should “rotate any and all secrets stored in CircleCI” for all projects. These include OAuth tokens; project and user API tokens; project environment and context variables; project SSH keys; and runner tokens, according to a detailed list included in the update.
CircleCI also addressed criticism for issuing its Jan. 4 advisory well after U.S. East Coast business hours. “We understand that many of our North American customers experienced late nights and on-call rotations once our guidance to rotate secrets was released at 6:30 p.m. PT / 9:30 p.m. ET on Wednesday, January 4,” the update stated. “We erred on the side of getting information out as fast as possible to minimize any potential exposure time.”
When the initial recommendation was issued Jan. 4,many SecOps teams began the work of sifting through software development project repositories to find and update secrets, or privileged credentials used to authenticate and authorize access to systems.
One CircleCI user in the U.K. posted on social media that he’d been up doing this work at 4 a.m. Another responded to CircleCI’s Twitter post,calling for the company to make this information easier to locate in its UI.
“It’s certainly disruptive to end users and admins alike,” said Peter Wright, a systems engineer for a CircleCI customer in Los Angeles. “It also will trigger lots of work investigating if any potential break-ins happened during the window they stated … so it has a domino effect on support, admins, developers and security people at affected companies.”
In CircleCI’s platform, secrets can take the form of personal or project credentials, including API tokens used to broker access to specific software development projects. Project API tokens across the board have been “invalidated” and will need to be replaced, according to the CircleCI advisory.
CI is not only a critical component of many environments for productivity purposes, but it also lives in a sensitive part of your infrastructure. Peter WrightSystems engineer, CircleCI user
The potential effect of the incident on API access to source code is of particular concern, Wright said.
“CI is not only a critical component of many environments for productivity purposes, but it also lives in a sensitive part of your infrastructure,” he said. “It often has access both to your source code, as well as having the ability to deploy software. So it’s not unreasonable to worry that any stolen secrets could be used to gain access to customer environments in a privileged way.”
CircleCI pledges more detail on scope of incident
This is not the first CircleCI security incident to be reported in recent years. In 2019, it disclosed an incident involving a third-party analytics vendor in 2019. That disclosure contained specific details about the scope of the systems affected and how CircleCI had responded internally, information that was missing 24 hours after this week’s advisory.
“What makes this so scary is … not a lot of details from CircleCI yet on the scope of what happened, what steps they have done to remediate it and what signatures we can look for, [such as] IP addresses or things like that that an attacker may be using stolen credentials with,” Wright said.
CircleCI’s public statement included a commitment to add further updates as details became available. A spokesperson for the vendor said Thursday it was planning an update before the close of business on the West Coast, where it is headquartered.
This information will be crucial for SecOps pros to prioritize their response to this incident, said Melinda Marks, an analyst at Enterprise Strategy Group, a division of TechTarget.
“Just like if there is a breach of a company, if you use their service, you would go change your password,” Marks said. “In the case of the CircleCI breach, you don’t know how much access the attacker has to the repos, so you’d want to rotate all your secrets to minimize risk of the attacker … gaining access to your repo and code.”
Automatically rotating secrets, especially in cloud-hosted services, is a best practice for security hygiene, but like the timely application of software patches, isn’t always followed within enterprise organizations. Storing secrets in code repositories is also best avoided, but while 83% of 350 respondents to a recent ESG survey of enterprise organizations scan their Git repositories for risky secrets in code, 31% have had security incidents resulting from exposed secrets.
“People do typically scan, but they don’t know how to prioritize taking action, [and] a high percentage are still getting secrets stolen,” Marks said.
CircleCI’s advisory was issued just as a series of powerful storms hit Northern California this week, knocking out power and otherwise compromising some SecOps teams’ ability to access their systems. It also comes on the heels of other high-profile IT vendor security incidents, including a breach at ChatOps that vendor Slack reported on Dec 31.
“It’s been a busy few months with Rackspace, LastPass, CircleCI and Slack,” Wright said.
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
