The following is the Foreword and Part 1 of a three-part series revealing key highlights from Walmart Global Tech’s Media Day, compiled from a series of on-site tours, fireside chats, panels, roundtables and one-on-one interviews.
Bookended by security personnel in front and back, we were warned in no uncertain terms before entering Walmart’s East Data Center facility: Any attempt to bring an electronic device into the building would result in our immediate expulsion.
The caution was understandable. We were, after all, the first-ever visitors who were not a Walmart employee or vendor partner to step foot in this building, where the retail giant’s precious data is collected and processed. This was serious business for the $572.8 billion Fortune 1 company, who invited a small gaggle of reporters to Walmart headquarters in Bentonville, Arkansas, for Walmart Global Tech’s (WGT) inaugural media day.
Before the day was over, we also were treated to a tour of the on-site security operations center (SOC) and a forensics lab — both located in the David Glass Technology Center, the main headquarters for WGT, Walmart’s internal technology and business services division.
The event also served as an opportunity to talk to WGT’s top cyber and IT executives about their efforts to innovate as a security and tech leader in a manner that can scale with the retailer’s ongoing world dominance. With roughly 10,500 stores, plus various eCommerce websites under 46 banners in 24 countries, Walmart is not only king in the retail space — it also operates in the health and financial sectors, and has its own manufacturing, distribution and logistics operations to account for.
So why the sudden willingness to offer the world an inside glimpse of its security operations? According to senior vice president and global CISO Jerry Geisler, it’s all about Walmart’s desire to prove to customers and clients that the company is working hard to earn their trust.
As an omnichannel retailer, Walmart is aware of how the lines between digital and physical commerce have blurred.
“With my teams, when we’re thinking about digital trust, of course were thinking end-to-end. We’re thinking everywhere we would potentially interact with that customer, when and where they would share data with us, and then how we use that data,” Geisler said. “How is it that we use or protect the data that our customers choose to entrust us with? And more so: How we’re potentially using emerging tech… because we want to enable the business, and part of that enablement is ensuring that we don’t have disruptive events that erode trust.”
Part 1: Walmart special on SOCs: Touring the security & data centers
The first sight that caught our eye as we left the reception lobby and marched through the David Glass Technology Center were three trampolines sitting side by side in a cavernous interior space. Wow — is this the most fun place to work or what?
OK, so actually this was not an office perk, but rather the work of merchandisers who are constantly experimenting with products that ultimately wind up in the stores. And it’s not unusual to see all sorts of unique items being tested on site. There’s always something going on at the Walmart campus, and that’s certainly true on the technology side of things.
The data center: Despite being connected via an intermediary passageway, the East Data Center is technically inside a completely separate building from the DWTC, requiring a stringent security check-in process. Only about 1,000 people per year set foot inside (and typically an NDA is involved), but never anyone from the media — until this day.
First stop: the C-Floor server room, home to highly sensitive and invaluable threat data leveraged by the incident response team and OneLab forensics teams (other corporate data is stored on the A- and B-Floors).
This area featured multiple protections against potential physical harm, including a floor-to-ceiling caged partition, leak detection cables and sticky anti-contamination doormats to reduce electrostatic discharge and loose dust particles (“You’ll be working in your socks before you know it,” remarked tour leader Kevin McCoin, distinguished architect, systems engineering, referring to the shoe-stealing sticky mats). The floor was elevated three feet above the concrete base to allow for better air circulation flow, helping to conduct heat from the hot servers outside and cool air back in.
Moving further inside, we were escorted into a long corridor featuring the critical infrastructure plant facilities designed to keep the servers operating through a series of pumps, motors and industrial controls. There’s a mission statement on wall in this area: to “provide data center capability and operate data centers in a manner that sustains a highly available and uninterrupted business operation.”
Different sections of this facility are fully segregated so that if a problem occurs in one section, the others are not affected. The goal is nothing short of 100 percent uptime, and multiple redundancies are the key. If one particular mechanism or system experiences downtime, Walmart relies on built-in automation and AI/machine learning to switch to one of the redundancies, thus maintaining operational stability and continuity.
We next were ushered over to the B-Floor’s private cloud server room for a fleeting glimpse inside, although we were only allowed to peek from just outside. Just like on the C-Floor, the room works on a grid system — like a big game of Battleship — to help facility workers know exactly where to go to if a problem arises.
Walmart has made news lately for pursuing a hybrid cloud computing strategy that is increasingly relying on its own private cloud, while mixing in public cloud offerings and edge nodes for ultimately flexibility in what cloud-based resources it uses.
The SOC: Including this one, Walmart actually operates three SOCs around the world (with the other two in Reston, Virginia, and Bangalore, India). Inside the main hub, the walls were adorned with large screens displaying timely cyber news media articles, the latest vulnerabilities sorted by company and product, summaries of zero-day exploits, and other vital info. Just outside, a hanging whiteboard noted the number of days since the last significant security incident. However, it was completely wiped clean — perhaps in anticipation of the media’s arrival?
Walmart’s SOC processes roughly 6 trillion points of telemetry and monitors approximately 3 million IP addresses in its network, plus roughly 167,000 public and private GitHub repositories. This requires a tremendous amount of automated monitoring correlation, detection and response mechanisms to aid the various teams that collaborate on security operations.
Inside a conference room, the heads of these teams laid out for the media a fictionalized scenario in which an imaginary, financially motivated cyber actor sent fake browser updates to Walmart associates in an attempt to get them to open a malicious file infected with a RAT, infostealer or Cobalt Strike file — an intermediary step for a ransomware attack.
It starts with threat intelligence team who creates reports based on the latest available commercial feeds as well as its own collection of TTPs, IOCs and other data points gathered via frameworks like MITRE ATT&CK.
The threat hunting team uses this intelligence to proactively sniff out threat actors that might be using these identified techniques, and they coordinate with the anomalous endpoint behavior team to spot rare or unusual activity that is indicative of a possible threat. As Vice President of Security Operations Jason O’Dell explained, the corporate philosophy is: “Rare isn’t always bad, but bad should always be rare.”
And although threat actors are getting better at hiding their malicious traffic, “something’s gonna stand out,” said Vernon Habersetzer, senior enterprise expert, security incident management.
If an alert is generated through either hunting or through traditional detection and response, the SOC analyst team is on hand to analyze a potential threat and then either take action, elevate to incident response or declare a false positive. From these analysis results, Walmart can then create custom detections for future protection.
There’s also an engineering team that maintains and stewards the 6 trillion datapoints; a data assurance team that spearheads data loss prevention (they weren’t part of the simulation); and an incident response team that handles containment and threat eradication, much of which is automated if Walmart’s systems detect flagged behavior.
Finally, there’s a red team, which is tasked with attacking the organization throughout the year to see “how all the controls put in place… stack up,” noted Harold Ogden, red team senior director. Recently, Walmart made the strategic move to move the red team — which had been operating as a more segregated entity — in-house in order to create a more collaborative purple teaming environment. (For more on this, see SC Media’s video interview with Jason O’Dell.)
The forensics lab. The tools and machines found in WGT’s hardware forensics lab environment evoke, in essence, a living museum environment that showcases the chronology of device repair and data recovery. This is where malfunctioning or damaged devices and drives end up when the company needs to salvage data from them for reasons ranging from legal discovery to security investigations to personal file restoration. Altogether, the forensics teams fulfills more than 3,000 various requests per year.
Inside the room were soldering stations, microscopes, ultrasonic wire bonders, X-ray machines capable of peering into the 10 different layers of smartphones, and a clean room when a contaminant-free working environment is needed. Several monitors displayed extreme close-ups of chips and circuits from various electronics currently under examination (or perhaps placed there simply as a visual demonstration for us).
As hardware changes, improves and gets smaller, the equipment used to examine it must evolve as well. Over the years, Walmart has found that it’s ideal to handle such tasks internally, rather than rely on a third party, which presents complications related to chain of custody, data privacy and expense, explained Wayne Murphy, distinguished architect, systems engineering, and hardware recovery expert.
Indeed, that just as Walmart realized there were advantages to setting up its own distribution and manufacturing operations, Walmart has resolved to become a self-reliant corporation when it comes to data collection, cybersecurity and forensics. Considering its vast resources and infrastructure, Walmart probably could form its own managed IT and cyber services company if it so desired. And at this point, would it even be surprising if they did?
Stay tuned for Part 2 of SC Media’s Walmart Global Tech Media Day coverage, featuring the retailer’s latest cyber innovations and initiatives.
