Immuta – provider of the interoperable data security platform – has announced integrations and updates that will improve security for Databricks. Databricks is a software company founded by the creators of Apache Spark, which provides solutions for data warehouses and data lakes.
In the past few years, Databricks has come out with an innovative new architectural approach that combines warehouses with data lakes to form the new “lakehouse” model. It has also made significant investments in artificial intelligence (AI), which calls for more advanced security measures.
That is where Immuta comes in. Founded in 2015, Immuta is one of North America’s top data security platform providers, and it specializes in AI workflow protection. Its cloud-native data governance tools enable better security, easier discovery of sensitive data, and stronger access controls.
Thanks to this new update announced in June 2023, Databricks’ customers will be able to enjoy Immuta’s platform capabilities.
About the Immuta and Databricks Integration
Immuta and Databricks strengthened their partnership in May of this year when Databricks Ventures (the lakehouse company’s VC arm) made a sizable investment in security. While the amount was undisclosed, Immuta said that the funding would be used for product development.
So, it makes sense that the new updates would feature Databricks front and center, bringing customers a plethora of new features and capabilities.
The two companies are long-time partners, working together since the spring of 2018 when Immuta rolled out new tools for Apache SparkSQL. Since then, the partnership has grown and evolved, culminating in this latest upgrade. Key features include access control for AI workload protection and localized discovery of sensitive data.
The Need for AI Workload Protection
In 2023, it is clear that AI workload protection is the need of the hour. Much like the cloud once ushered in a massive digital revolution, we are witnessing the same for artificial intelligence. Over 90% of developers use AI tools at work, and a different survey reveals that 77% of companies use multiple third-party tools for AI workloads.
This makes systems like Databricks particularly vulnerable to security attacks and breaches. The company is making a strong AI play and intends to help customers cut costs and speed up innovation by unifying their data, analytics, and AI in one cloud platform.
But without adequate AI workload governance, this could mean a massive threat vector.
As a result, the company has recently ramped up its cybersecurity efforts. In addition to Immuta’s vital upgrades to its data security platform, Databricks has also joined hands with security companies BigID, Theom, and Hunters this year.
(Download Whitepaper: Protecting your Data End-to-End)
Immuta’s Enhanced Data Security Platform
Now, let us turn to the key changes you can expect from Immuta’s recent update.
1. Native integration with Databricks Unity Catalog
The Databricks Unity Catalog is among the company’s flagship offerings. It lets you govern structured and unstructured data as well as machine learning models through a three-level namespace that organizes your data assets.
Data at different levels can put a lot of pressure on engineers to wrangle access policies, especially as the models’ complexity grows. Immuta’s native integration will help in three ways – secure data, detect access issues, and discover sensitive information.
2. Security policy enforcement
Not only does the new and enhanced Immuta enable smarter security policies, but it also makes it easier to enforce them. You can grant or revoke access rights at the Unity Table level through native policy enforcement.
3. Centralized metadata management
Immuta will centralize metadata management for information stored in Databricks lakehouses. On the surface, this lets you automate and orchestrate policies. But digging deeper, it has several other benefits, like helping you discover sensitive data more easily. You can add tags like personally identifiable information (PII) to enforce strict policies on all the sensitive data you discover.
4. User activity monitoring
Immuta will let you monitor and record all the user activity happening on Databricks. This includes audit logs of user queries, who changed a policy and when, and user access summaries. Here, Immuta’s Unified Audit Model (UAM) comes into play. Kit ensures a consistent log structure and metadata across Unity Catalog and your main Immuta instance to simplify security.
5. Access control policies
Immuta introduces automation to the Databricks environment. When you build a single Immuta policy with a high-level intent, the data security platform will automatically create multiple policies to execute in Databricks. This significantly reduces your manual effort for defining access control policies.
The company states that using Immuta’s new integrations can reduce the number of roles and policies to manage by 93X.
Strengthening AI Workload Protection
While AI workload protection is not the only thing you can achieve with the Immuta-Databricks integration, it is definitely among the top use cases. As artificial intelligence scales, it is likely that you will want to migrate AI data to the cloud. Now that the Databricks Unity Catalog is secured, you can filter roles, mask columns, discover sensitive data, control AI data access, and much more. Migration processes will now happen with more confidence.
In addition to AI workload protection, Immuta ensures that these data processes comply with regulations such as HIPAA and GDPR.
The data security platform also adds another layer of trust to security analysis. Typically, when analyzing AI data workloads for vulnerabilities, you are inundated with potential risks. Many of these are false positives that only take up time with little outcomes. Immuta’s Detect feature assigns scores to AI data based on the degree of sensitivity.
You can prioritize the risks and set up real-time alerts to be notified when truly severe events threaten AI workload governance.
What Are the Business Benefits?
The newly announced solution would provide customers with the following benefits:
- Improved performance without compromising security: The Databricks Lakehouse architecture is compatible with any cloud of your choice. Now, you can combine this freedom with enhanced security from Immuta. This means you can migrate AI data assets to any cloud environment and ensure superior performance without compromising security.
- Streamlined collaboration and data sharing on Databricks: With Immuta, users can participate in secure collaboration on Databricks. This is because the integration maintains detailed audit trails with full transparency. The centralized metadata store can be accessed by relevant users. You can delegate the stewardship of policies and even define purpose-based access.
- Gaining a competitive edge with top-notch data security: Immuta brings cutting-edge security and AI workload protection to the Databricks environment. Unity Catalog users will be able to make the most of their data repositories, build new AI models, and solve complex business problems without worrying about security or compliance.
What Customers Have to Say
Immuta is a veteran when it comes to managing data governance and AI workload protection. For example, the company helped clinical research from Cognoa accelerate HIPAA-compliant AI and machine learning through its data security platform.
“With Immuta, you can basically emulate a user’s query and their permissions to see exactly what they see. You can see it in a governed way and ensure that no one gets access to something they’re not supposed to,” said Charlie Qin, Cognoa’s Data Platform Owner.
The new integration makes similar benefits available to Databricks Unity Catalog users. Here is what two of their shared customers have to say:
“Databricks helps us to manage that data and Immuta plays an important role in administering security and access control. As we look to innovate with new products and implement a multi-cloud strategy, we must treat the data properly – it must be governed,” said Jack Berkowitz, Chief Data Officer at ADP.
“Swedbank needed to build an enterprise-scale advanced analytics platform that would also enforce trust in our security, management, and access to data internally while protecting our customers’ assets and data. Immuta and Databricks have been instrumental in helping us build that vision and we are excited to see their partnership go to the next level,” said Vineeth Menon, Head of Data Lake Engineering at Swedbank.
Conclusion
In the AI era, cloud data infrastructure typically has three layers – the data lake, the data warehouse, and the data exchange. Immuta and Databricks come together to add a fourth pillar: data security in the cloud. AI requires information to be handled at an incredible scale and speed, which needs three things:
- Separating the policy from the platform
- Native and not retrofitted cloud data controls
- Leveraging attributes instead of roles to tag data
Immuta and Databricks realized that, over the years, their shared customers have started running complex workloads like machine learning and AI on the cloud. The new integration – with its Discover, Detect, and Secure components – enables better granularity and manageability for your data stores.
Existing customers can now avail of this integration.
