Security Failures At TikTok's Virginia Data Centers, From … – Forbes

Other security issues at the sites have included unattended boxes of hard drives, illicit crypto mining, and a sanctioned supplier.

For years, TikTok has told lawmakers that the private data of its U.S. users is secured — and safe from potential influence or exfiltration — in a cluster of data centers located in Northern Virginia.

But interviews with seven current and former employees and more than 60 documents, photos and videos from the data centers reveal that the centers have faced security vulnerabilities ranging from unmarked flash drives plugged into servers to unescorted visitors to boxes of hard drives left unattended in hallways. Sources suggest that these challenges are the result of TikTok trying to grow its data storage capacity very quickly, and sometimes cutting corners along the way.

Documents, photos, and interviews also suggest that TikTok’s data center operations are still tightly enmeshed with ByteDance’s business in China. Among other suppliers, the data centers use servers produced by Inspur, a company that the Pentagon said in 2020 was controlled by the Chinese military and that the Commerce Department added to a sanctions list last month. Documents also show that as recently as last week, server work orders were sent to data center technicians by Beijing ByteDance Technology Co., Ltd., a ByteDance subsidiary partially owned by the Chinese government, which TikTok has repeatedly insisted has no control over its operations.

These revelations come at a critical moment for TikTok, which is facing a federal criminal investigation for surveilling journalists (including this reporter) and a threat from the Biden Administration that ByteDance must sell TikTok or face a full ban of the app in the U.S. A bipartisan coalition of lawmakers, along with the White House, have raised concerns that the Chinese government could use ByteDance’s control over TikTok to exfiltrate valuable data about American citizens or influence domestic or international civic discourse. (Disclosure: In a previous life, I held policy positions at Facebook and Spotify.)

“Each new story raises more concerns and provides additional examples of TikTok appearing to misrepresent its data security practices,” said Senator Mark Warner, who has led a Senate effort to ban TikTok in recent months.

The data centers use servers produced by Inspur, a company that the Pentagon said in 2020 was controlled by the Chinese military.

TikTok’s rebuttal to these concerns — and the threat of a potential ban — is a proposal known as Project Texas, under which TikTok would remove private U.S. user data from the Virginia servers and isolate it in a set of Texas-based data centers owned by Oracle. However, TikTok CEO Shou Zi Chew testified before a House Committee last month that U.S. user data is still “sitting in our servers in Virginia” today.

In response to a detailed list of questions from Forbes, TikTok spokesperson Maureen Shanahan acknowledged using Inspur servers, but said TikTok had “not procured from that vendor for quite some time.” Inspur has also worked with other major U.S. companies, including Microsoft, IBM and Intel. Inspur did not reply to a request for comment, nor did the Department of Commerce. The Department of Defense had not provided comment by press time.

Shanahan said the work orders from Beijing ByteDance Technology Co. were “an artifact of a ticketing system,” which “does not provide any access to user data,” and that TikTok began routing new U.S. user traffic to Oracle servers in June 2022. She also offered the following statement:

“In the past several years, we’ve increased our investments in people, processes, and technology to help safeguard our community, including establishing a team dedicated to data center operations, maintenance, and compliance.”

^{Unmarked Flash Drives, Unsecured Servers And Unescorted Visitors}

Like many tech giants, TikTok rents space in large data centers in Northern Virginia. TikTok’s data halls within these centers are managed in part by ByteDance and in part by contract workers from several data center management firms. (Two firms that work in the ByteDance centers did not reply to a request for comment.)

In January 2023, TikTok’s U.S. Data Security division — the new entity that, under Project Texas, will secure and provision access to U.S. user data — released a blog post stating: “Our Virginia data center includes physical and logical safety controls such as gated entry points, firewalls, and intrusion detection technologies.” But seven current and former employees who spoke to Forbes — anonymously for fear of retribution from ByteDance or their contracting company — said that security at the sites is variable and lax.

According to the employees, physical security systems vary by building, but most rely on a badge system. Company policy states that guests, including a regular rotation of delivery couriers, hardware vendors, electricians, and other professionals, must be escorted by an employee at all times. But in practice, according to four employees, that doesn’t always happen. “We do not have time to watch them all,” one said.

Four sources said they have seen unmarked, “unticketed” flash drives plugged into servers.

The employees described multiple recordkeeping systems used by the company to track server and other hardware repair done in the centers, including five separate internal ticketing tools. All the employees said they also received work requests through ByteDance’s workplace software, Lark.

Three sources, though, told Forbes that they were aware of modifications made to servers that were not reflected in any ticketing system, and four said they have seen unmarked, “unticketed” flash drives plugged into servers. TikTok said these claims were inconsistent with the company’s internal security monitoring.

Four sources also said that the company’s degaussers — machines used to wipe and destroy old hard drives — were often broken or jammed, requiring staff to take drives to other data centers to be disposed of. A person who had been placed in this position said, “Anyone with malicious intent could’ve just taken them, and we wouldn’t have known.” (TikTok acknowledged having this issue in the past, but said it has since been addressed.)

Photos provided by one source, who said they were taken in 2020, show hard drives left unattended in open boxes in the hallways of a Virginia data center. Details in the photos, including doors, floor tiles, ceiling tiles, wall paint, and server racks, match photos and videos provided by a second, unrelated source.

Inadequate investment in data center security is an industry-wide problem, according to Sanjukta Das Smith, Chair of Management Science and Systems at the University of Buffalo School of Management and an expert on data center resourcing. In an interview, she said, “Many times, security does tend to take a backseat, because in most client-facing interactions, the focus is more on the client’s experience: how fast are things loading on their devices, and what are the aesthetics of that?”

Despite these systemic challenges, the practices described by TikTok employees at times diverged from industry norms described by Smith. For example, in the data centers she has visited, Smith said, “even if you have prior authorization, once you go through check-in, then it’s not like someone is free to roam around. It’s an escorted experience.” But this is apparently not the case at TikTok’s centers, where one source said, “We never really knew when these people were going to show up, they just showed up with a ticket and asked to be cut access, and someone would cut them access and they’d go in and do whatever they do and then leave.”

Smith said she has never seen USB drives used in data centers, but that they would raise additional concerns, given how easy it is to install malicious software on them.

Bruce Schneier, a fellow at Harvard’s Berkman Klein Center for Internet & Technology and lecturer at the Harvard Kennedy School, cautioned against reading too far into any one unexplained occurrence. Analogizing to claims about voter fraud, he said, “it’s easy to find stuff that looks suspicious, but it’s a little like saying we found boxes of votes we don’t understand.”

Like Smith, Schneier noted that security problems are industry-wide. He mentioned Twitter, which he said had struggled with security because of its efforts to grow so quickly. As for TikTok, he said, “I’m sure there’s negligence. Like any big tech company, they care about profits, and security is expensive.”

^{Building Safety, Fire Hazards And Crypto Mining}

Forbes reporting also uncovered other issues with the Virginia data centers. Sources raised concerns about building safety — three described being occasionally asked to work in buildings that were still under construction, and said that in some buildings, door alarms go off so frequently that they are meaningless. (TikTok said door alarms are investigated on an as-needed basis.)

Photos and videos from inside the data centers also showed wooden pallets and cardboard boxes left by delivery couriers in the server rooms — a fire hazard when coupled with the occasional overheating server. Audio recordings of internal TikTok meetings note that heat in these data centers has been a problem before: In a September 2021 meeting, a Trust & Safety director can be heard describing an instance in which the Virginia servers overheated and U.S. user data was routed to servers in Singapore until the issue could be fixed.

“ByteDance just didn’t give a shit.”