AI Unlimited, Teradata’s on-demand AI/ML and advanced analytic compute engine is now available in preview through Microsoft Fabric. This integration brings together Teradata's powerful AI compute capabilities with Microsoft Fabric's comprehensive data management and lifecycle applications, making it easier than ever for developers (data scientists, data engineers, and ML engineers) to rapidly explore, prototype, iterate, and operationalize at scale.  

Microsoft Fabric and AI Unlimited  

Microsoft Fabric is a unified “all-in-one" platform for data engineering, data science and data analysis needs. It brings together and builds upon Microsoft's existing suite of products offering a platform with unified governance and security across products that span from data ingestion, transformation, data warehouses and lake houses to report building and more.  

AI Unlimited is Teradata’s standalone product that natively integrates into Fabric giving you access ephemeral compute and Teradata’s SQL-native Analytics engine. AI Unlimited's engine is built for handling large-scale datasets and complex analytics with over 150 in database functions, enabling efficient analysis across billion-row datasets. 

Key Benefits of AI Unlimited via Microsoft Fabric: 

• Effortless data exploration and experimentation: Developers can easily explore, experiment and confidently work with data in OneLake or any cloud object store using delta lake or Iceberg open table formats.  

• In-Database analytics with a SaaS like experience: For those familiar with Teradata's efficient in-database analytics, AI Unlimited allows you to leverage these capabilities in a serverless-like environment without impacting production. This approach gives developers the freedom to explore and experiment without concerns about production workloads. 

• Seamless Integration with Microsoft Fabric’s Data Management: When paired with Fabric’s robust data management and data lifecycle capabilities, the Teradata engine provides an agile and frictionless experience for data exploration, experimentation and operationalization. 

• Cost-Effective, Scalable Solution: AI Unlimited is designed to be budget-friendly, with options to scale down as needed. It’s integrated into the Fabric ecosystem, giving you direct control over your resources and costs. Paid upgrades will be available in early 2025, and you can join the waitlist once you complete the preview!  

Accessing AI Unlimited on Microsoft Fabric 

To get started with AI Unlimited, visit the "Workloads" tab in Microsoft Fabric, where you’ll find third-party applications listed as “Workloads.” Search for and select the AI Unlimited card and request your admin to add it to your tenant for seamless integration. 

Accessing Data for use with AI Unlimited on Fabric

Within Fabric, setting up data is straightforward. You have many options for accessing data. For example, you can create a "New Lakehouse" and use "Shortcuts" to link directly to files in any Object Store, allowing you to query data without migrating it. This setup enables you to load and view your files immediately, create a Delta Parquet format table from the shortcut, and leverage Teradata's extensive suite of analytical functions for data exploration and preparation through AI Unlimited notebooks. 

Efficient Data Prep via Modular Functions on AI Unlimited 

Once data access is configured, data preparation becomes efficient with AI Unlimited and Teradata’s ClearScape Analytic functions. Through AI Unlimited notebooks you can use Teradata Vantage’s fit-and-transform functions which efficiently handle data cleaning, feature engineering, scaling features and processing in a single operation.  
 
A key aspect of this efficiency are Teradata’s fit tables, which serve as a modular blueprint that store transformation parameters. This modularity allows for consistent data preparation and feature engineering across datasets. Fit tables separate the logic of each transformation into reusable components (tables) to ensure all transformations across training, testing and production are consistent.  

For example, after the “fit tables” are defined, the TD_ColumnTransformer function will use those tables and apply multiple transformations in a single operation, improving performance by approximately 30%. 

Building Fit Tables for Data Transformation  

Fit tables are created via various functions and serve as modular blueprints that guide transformations. For example, you may employ the following fit functions to conduct data preparation and feature engineering to prepare a dataset for ML processing tasks like k-means clustering.  

• Outlier Removal: The TD_OutlierFilterFit function creates a fit table that defines parameters to identify and remove outliers. 

• Handling Missing Values: TD_SimpleImputeFit function creates a fit table that assigns default values to null entries, ensuring every record is included in the analysis.  

• Feature Engineering: The TD_NonLinearCombineFit function creates a fit table with the target columns and formula that uses non-linear combination of existing features, such as total sales per customer by multiplying total quantity by total price. 

• Scaling Data To ensure uniformity, Teradata’s TD_ScaleFit and TD_ScaleTransform functions normalize features (e.g., TotalPrice, TotalQuantity) onto a 0-1 scale, preparing data. 

These “Fit tables” are then used as input into a single TD_ColumnTransformer function, which will process all transformations in a single operation, giving approximately 30% performance improvement over running each transformation separately.  

Review the docs for additional feature engineering and data preparation fit functions. 

Streamlined Workflow for Test and Training Data 

Once data prep is complete, you can apply the same transformations to test data to evaluate your models. With fit tables set up, simply run the TD_ColumnTransformer function to prepare, engineer, and scale your test data for analysis. This approach mirrors operational workflows, allowing data to be prepared and analyzed almost immediately upon arrival. 

By leveraging AI Unlimited on Microsoft Fabric, organizations can streamline the entire data journey—from data preparation to model deployment—ensuring high performance, efficiency, and scalability in data-driven initiatives.