As part of “shift left” to incorporate security discussions earlier in the software development lifecycle, organizations are beginning to look at threat modeling to identify security flaws in software design. With developers increasingly incorporating machine learning in their applications, threat modeling is necessary for identifying the risks to the organization.
“People are still grappling with the whole idea that when you use that very new technology [machine learning], it brings along a bunch of risk, as well,” says Gary McGraw, co-founder of Berryville Institute of Machine Learning. “I’ve been in the unenviable position of saying, ‘Well, there’s this risk, and there’s that risk, and the sky is falling,’ and everybody goes, ‘Well, what am I supposed to do about that?”
There have been a lot of conversations about machine learning risk for quite some time now, but the difficulty lies in figuring out how to address them, McGraw says. Threat modeling – identifying the types of threats that can cause harm to the organization – helps organizations think through security risks in machine learning systems such as data poisoning, input manipulation, and data extraction. If developers could understand the security flaws in their designs by threat modeling, it would reduce the time spent on security testing during development and before production. The NIST Guidelines on Minimum Standards for Developer Verification of Software recommends threat modeling to look for design-level security issues.
IriusRisk’s threat modeling tool addresses this challenge by automating both threat modeling and architecture risk analysis. Developers and security teams can import the code into the tool to generate diagrams and threat models. Threat modeling templates make threat modeling accessible even to those not familiar with diagramming tools or risk analysis.
And the newly launched AI & ML Security Library allows organizations using IriusRisk to threat model the machine learning system they are planning in order to understand what the security risks are, as well as how to mitigate those risks.
“We’re finally getting around to building machinery that people can use to address the risk and control the risk,” says McGraw, who is also a member of IriusRisk’s advisory board. “When you put machine learning into your [system] design, and you’re using IriusRisk, now you know what risks are involved and what to do about that.”
What ML Threat Modeling Looks Like
IriusRisk with the AI & ML Security Library help organizations ask necessary questions. For example:
- Asking where the data being used to train the machine learning model came from. It’s important to also ask whether anyone had the opportunity to embed incorrect or malicious data to make the machine do the wrong thing.
- Consider how the machine keeps learning once it is in production. Machine learning systems that are online and keep on learning from users are more dangerous than the ones that are not online. “It depends on who is using it. Is it your people? Is it bad people? Is it everybody on Twitter, or X?” McGraw says, noting there have been examples of past projects that had to be taken offline after it learned objectionable information.
- Ask if confidential information can be extracted from the machine. If you put confidential information into your machine learning algorithm, it is not protected by cryptographic means and can be extracted. “If you put the data in the machine, it’s in the machine,” McGraw says. “You need to think about making sure that people using your machine learning system cannot extract that confidential data.”
The AI & ML Security Library is based on the BIML ML Security Risk Framework, a taxonomy of machine learning threats as well as an architectural risk assessment of typical machine learning components developed by McGraw. The framework is designed to be used by developers, engineers, and designers creating applications and services that use machine learning in the early design and development phases of the project. With IriusRisk’s library, everybody who is using machine learning can now take advantage of BIML’s framework.
The AI & ML Security Library is available to both IriusRisk customers and those using the community edition of the platform.
Time to be Threat Modeling
The AI & ML Security Library was developed in response to interest from organizations in how to analyze and secure AI and ML systems, according to Stephen de Vries, CEO of IriusRisk. “We have seen a surge in interest from our customers in the finance and technology sectors for guidance on how to analyze, and secure design ML systems,” de Vries said in a statement. “Since these are often new projects that are still in the design phase, performing threat modeling here adds a lot of value, because those teams will very quickly understand where the security goalposts are – and what they need to do in order to get there.”
The library doesn’t help organizations who don’t have visibility into their machine learning usage. Just as organizations can have Shadow IT – where different business stakeholders set up their own servers and web applications without IT oversight – they can also have shadow machine learning, McGraw says. Different departments are trying out new applications and tools – but there is a gap between what individual employees are using and what risks IT and security teams know about.
“Everybody’s like, ‘I don’t think I have any machine learning in my organization,’” says McGraw. “But as soon as they find out that they do… they find it everywhere.”
Many organizations do not incorporate threat modeling during software design, and those that do rely on manual processes where a person analyzes the threats one at a time.
“If you have a mature threat modeling program and you’re using a tool like IriusRisk, you can also now handle machine learning. So the people that are already doing the best are going to do even better,” McGraw says. “What about the people who aren’t doing threat modeling? Maybe they should start. It’s not new. It’s time to do it.”
