As organizations increasingly rely on AI to drive innovation and efficiency, protecting sensitive data has become both a strategic necessity and a regulatory mandate. Traditional security measures, often reactive and manual, no longer suffice. Instead, we now stand at the cusp of a new era where data governance is automatic, intelligent, and built to match the speed of AI. 

Let’s explore how AI-driven sensitive data protection is transforming data security. Then, discover how Striim’s AI agents are leading the way in this revolution. 

The New Age of Data Governance 

Despite the widespread deployment of multiple API security products, recent surveys reveal a staggering statistic: 92% of organizations experienced an API-related security incident in the last year, with 57% encountering multiple incidents. This alarming reality underscores the limitations of traditional security measures and highlights the urgent need for more intelligent, automated solutions.

Historically, safeguarding hackable data required a labor-intensive process—manual audits, constant monitoring, and a reactive approach to threats. However, the reality of today’s fast-moving data environment demands a radical shift. With the advent of AI-driven security, sensitive data can be detected, classified, and protected in real time. This proactive stance eliminates the need for constant manual oversight. Protecting sensitive data helps organizations work towards compliance and reduce the risk of human error.

Imagine a world where sensitive data moves through systems effortlessly, but never without oversight. Striim’s AI-powered approach ensures this by detecting and classifying data before it even reaches storage. Continuous scanning identifies sensitive data the moment it’s created—not after it’s stored—while proactive security mechanisms like real-time masking, encryption, or redaction safeguard the information from exposure. Striim enables businesses to instantly manage and protect sensitive data, making it possible to adhere to regulations like GDPR, CCPA, and HIPAA. The result? Data flows freely and securely, empowering businesses to focus on what matters most.

Enter Striim’s AI Agents Sentinel and Sherlock: Pioneering AI-Powered Data Governance

Striim’s AI agents, Sentinel and Sherlock, are pioneering tools that bring real-time, AI-powered governance to your data pipelines, increasing security without compromising performance.

Sherlock AI offers: 

• Source Operation: Identifies sensitive data before it enters data pipelines—even in third-party-managed databases and SaaS environments.
• Early Detection: Finds sensitive data before it moves, eliminating risk at the earliest stage.
• Comprehensive Visibility: Works seamlessly across SaaS, cloud, and third-party environments to ensure full visibility.
• Lightweight Scanning: Operates with zero performance impact, ensuring databases aren’t overloaded.
• Automated Classification: Classifies financial, health, and identity-related PII automatically, providing real-time security insights.
• Data Quality Monitoring: Detects data quality issues in real time, alerting teams when sensitive data appears in unintended locations.

Sentinel AI provides: 

• In-Motion Protection: Provides real-time detection and protection of sensitive data as it moves across systems.
• Accurate Detection: Spots PII anywhere in a record—even if it’s misplaced or mislabeled—beyond the scope of rules-based controls.
• Exposure Prevention: Prevents data exposure when transferring information from internal systems to external platforms for analytics or exchange.
• Compliance Support: Supports 25+ sensitive data types across the USA, Canada, UK, and India to support various compliance requirements.
• Automated Actions: Executes policy-based actions such as encryption and masking (partial, full, regex-based) automatically.
• Plug-and-Play UX: Easily integrated into your pipeline with a plug-and-play setup that requires only a few clicks.
• Regulatory Governance: Supports businesses on their journey to meet GDPR, CCPA, HIPAA, and other regulatory requirements.

Together, Sherlock AI and Sentinel AI work to prevent sensitive data exposure before it happens, ensuring your operations remain secure and that your team is in full control of its data.

How AI-Powered Data Governance Works

Our process begins with Sherlock AI, which proactively identifies sensitive data at its source—before it moves. By scanning both structured and unstructured data across SQL, NoSQL, SaaS, and cloud databases, it detects and automatically classifies financial, health, and identity-related information that may present compliance challenges. 

As data moves, Sentinel AI validates it in real time using advanced pattern recognition and NLP, catching any mislabeled or misplaced data that traditional rules-based systems might overlook. Sentinel AI then applies automated protection measures—encrypting, masking, or blocking data based on business policies—to secure its movement between internal and external systems and prevent unintended processing of regulated information. 

Sentinel delivers live reporting via real-time dashboards that continuously monitor sensitive data exposure, security actions, and compliance. It uses predefined identifiers to detect, log, and protect sensitive information, while AI-driven metadata tags each event for effective tracking and auditing. With support for schema evolution, Sentinel easily adapts to new data sources, ensuring ongoing AI-powered data governance.

This continuous monitoring helps organizations stay audit-ready and compliant. Real-time dashboards provide complete visibility into data protection efforts, and Sentinel generates audit logs that align regulations like GDPR, CCPA, HIPAA, and the EU AI Act. Additionally, it integrates with enterprise security tools such as SIEM, DLP, Datadog, and Snowflake Security to ensure a unified security framework.

The Impact of AI-Powered Automation on Data Governance 

By automating these processes, organizations no longer need to scramble after a potential data breach. Instead, security becomes a built-in feature of data management. Sensitive information is automatically shielded by AI agents as it moves through the enterprise ecosystem, whether in production environments, during testing, or throughout analytics workflows.

Automated authentication and connection processes also reduces strain on IT teams. This allows security professionals to shift their focus from routine monitoring to strategic initiatives, such as threat intelligence and proactive risk management. With Sentinel AI operating silently in the background, businesses can innovate without fear of compromising their sensitive data. 

By ensuring that sensitive data is protected, organizations can also enhance customer trust. In addition, streamlined security processes translate into improved operational efficiency. Data flows remain uninterrupted, and the risk of security incidents is drastically minimized.

Moving Forward in the AI Era

The AI era requires businesses to rethink traditional approaches to data security. With the speed at which data moves and the sophistication of modern cyber threats, it’s clear that reactive measures are no longer sufficient. Automated, intelligent solutions are not just an option—they are a necessity. 

Get a demo today and discover how Striim can help you better protect your data. 

 

Imagine searching for products on an online store by simply typing “best eco-friendly toys for toddlers under $50” and getting instant, accurate results—while the inventory is synchronized seamlessly across multiple databases. This blog dives into how we built a real-time AI-powered hybrid search system to make that vision a reality. Leveraging Striim’s advanced data streaming and real-time embedding generation capabilities, we tackled challenges like ensuring low-latency data synchronization, efficiently creating vector embeddings, and automating inventory updates.

We’ll walk you through the design decisions that balanced consistency, efficiency, and scalability and discuss opportunities to expand this solution to broader Retrieval-Augmented Generation (RAG) use cases. Whether you’re building cutting-edge AI search systems or optimizing hybrid cloud architectures, this post offers practical insights to elevate your projects.

What is RAG?

Retrieval-Augmented Generation (RAG) enhances the capabilities of large language models by incorporating external data retrieval into the generation process. It allows the model to fetch relevant documents or data dynamically, ensuring responses are more accurate and context-aware. RAG bridges the gap between static model knowledge and real-time information, which is crucial for applications that require up-to-date insights. This hybrid approach significantly improves response relevance, especially in domains like e-commerce and customer service.

Why Vector Embeddings and Similarity Search?

Vector embeddings translate natural language text into numerical representations that capture semantic meaning. This allows for efficient similarity searches, enabling the discovery of products even if queries differ from stored descriptions. Embedding-based search supports fuzziness, matching results that aren’t exact but are contextually relevant. This is essential for natural language search, as it interprets user intent beyond simple keyword matching. The combination of embeddings and similarity search improves the user experience by providing more accurate and diverse search results.

Why Real-time RAG Instead of Batch-Based Data Sync?

Real-time RAG ensures that inventory changes are reflected instantly in the search engine, eliminating stale or outdated results. Unlike batch-based sync, which can introduce latency, real-time pipelines offer continuous updates, improving accuracy for fast-moving inventory. This minimizes the risk of selling unavailable products and enhances customer satisfaction. Real-time synchronization also supports dynamic environments where product data changes frequently, aligning search capabilities with the latest inventory state.

How We Designed the Embedding Generator for Performance

In designing the Vector Embedding Generator, we addressed the challenges associated with token estimation, handling oversized input data, and managing edge cases such as null or empty input strings. These design considerations ensure that the embedding generation process remains robust, efficient, and compatible with various AI models.

Token Estimation and Handling Large Data

Google Vertex AI

Vertex AI simplifies handling large data inputs by silently truncating input data that exceeds the token limit and returning an embedding based on the truncated input. While this approach ensures that embeddings are always generated regardless of input size, it raises concerns about data loss affecting embedding quality. We have an ongoing effort to analyze how this truncation impacts embeddings and whether improvements can be made to mitigate potential quality issues.

OpenAI

OpenAI enforces strict token limits, returning an error if input data exceeds the threshold (e.g., 2048 or 3092 tokens). To handle this, we integrated a tokenizer library into the Embedding Generator’s backend to estimate token counts before sending data to the API. The process involves:

1. Token Count Estimation: Input strings are tokenized to determine the estimated token count.
2. Iterative Truncation: If the token count exceeds the model’s limit, we truncate the input to 75% of its current size and recalculate the token count. This loop continues until the token count falls within the model’s threshold.
3. Submission to Model: The truncated input is then sent to OpenAI for embedding generation.

For instance, if an OpenAI model has a token limit of 3092 and the estimated token count for incoming data is 4000, the system will truncate the input to approximately 3000 tokens (75%) and re-estimate. This iterative process ensures compliance with the token limit without manual intervention.

Handling Null or Empty Input

When generating embeddings, edge cases like null or empty input can result in API errors or undefined behavior. To prevent such scenarios, we adopted a solution inspired by discussions in the OpenAI developer forum: the use of a default vector.

Characteristics of the Default Vector:

• Dimensionality: Matches the size of embeddings generated by the specific model (e.g., 1536 dimensions for OpenAI’s text-embedding-ada-002).
• Structure: The vector contains a value of 1.0 at the first index, with all other indices set to 0.0.
  • Example:[1.0, 0.0, 0.0, … , 0.0]

By returning this default vector, we ensure the system gracefully handles cases where input data is invalid or missing, enabling downstream processes to continue operating without interruptions.

Summary of Implementation:

1. Preprocessing
   • Estimate token counts and handle truncation for models with strict token limits (OpenAI).
   • Allow silent truncation for models like Google Vertex AI but analyze its impact.
2. Error Handling
   • For null or empty data, return a default vector matching the model’s embedding dimensions.
3. Scalability
   • These mechanisms are integrated seamlessly into the Embedding Generator, ensuring compatibility across multiple data streams and models without manual intervention.

This design enables developers to generate embeddings confidently, knowing that token limits and edge cases are managed effectively.

Tutorial: Using the Embeddings Generator for Search

An e-commerce company aims to build an AI-powered hybrid search that enables users to describe their needs in natural language.

Their inventory management system is in Oracle database and the store front search database is maintained in the Azure PostgetSQL.

Current problem statements are:

1. Data Synchronization: Inventory data from Oracle must be replicated in real-time to the storefront’s search engine to ensure data consistency and avoid stale information.
2. Vector Embedding Generation: Product descriptions need vector embeddings to facilitate similarity searches. The storefront must support storing and querying these embeddings.
3. Real-Time Updates: Ongoing changes in the inventory (such as product details or stock updates) need to be reflected immediately in the search engine.
4. Embedding Updates: Updates to products in the inventory should trigger real-time embedding regeneration to prevent outdated or inaccurate similarity search results.

Solution

Striim has all the necessary features for the use case and the problem statements described.

1. Readers to capture initial snapshot and real-time changes from Oracle database.
2. Embedding generator to generate vector embeddings for text content.
3. Writers to deliver the data along with the embeddings to Postgres database.

Striim Features :

1. Readers: Capture the initial snapshot and ongoing real-time changes from the Oracle database.
2. Embedding Generator: Creates vector embeddings for product descriptions.
3. Writers: Deliver updated data and embeddings to the PostgreSQL database.

PostgreSQL with pgvector:

1. Supports storing vector embeddings as a specific data type.
2. Enables similarity search functionality directly within the database.

Note: The search engine for this solution is implemented in Python for demonstration purposes. OpenAI is used for embedding generation and for summarisation of the results.

Design Choices and Rationale

1. Striim for Data Integration: Chosen for its seamless real-time change capture (CDC) from Oracle to PostgreSQL.
   1. Alternative could be to have an independent application/script periodically but would be inefficient and expensive to have this developed for the initial load and change data capture, also would be difficult to maintain over a period of time for various data sources which might also need to be synced to the store front database (Postgres)
2. OpenAI Embeddings: Ensures high-quality embeddings compatible across pipeline stages.
3. Striim Embedding generator: Enables in-flight embedding generation while the data is being synced instead of having to move the data separately and then generate and update the embedding.
   1. Alternative is to have separate listeners/triggers/scripts in the search front database (Postgres) to generate and update the embeddings every time the data changes. This could be very expensive and not be very accurate as the data changes and the embedding changes can go out of sync.
4. pgvector: Facilitates native vector searches, reducing system complexity.
   1. Alternative design choices were to choose the standalone vector databases but they do not provide the flexibility pgvector provides as we can store the actual data and the embeddings in a relational database setup compared to vector databases where we need to maintain metadata for each vector database and cross lookup for actual data from a different database/source while summarising the similarity search results. Eg., the embeddings would be in one place which needs to be queried using similarity search query whereas price or rating-based filters need to be applied elsewhere.
5. Python Search Engine: Provides flexibility and integration simplicity.
   1. This is a convenient choice to make use of the python libraries, more details are included in the upcoming sections

Future Work

1. Expand embedding model options beyond OpenAI and make it generic
   1. This is already done for the Striim Embedding generator as it supports VertexAI as well. We could consider supporting self-hosted models.
2. Expand the support for non-textual input.
3. Expand the implementation to generically cover any use case and have the application interface integrated with Striim UI.
   1. Current implementation as a proof of concept is tightly coupled with the e-commerce use case and it’s data set.

Step-by-step instructions

Set up Striim Developer 5.0

1. Sign up for Striim developer edition for free at https://signup-developer.striim.com/.
2. Select Oracle CDC as the source and Database Writer as the target in the sign-up form.

Prepare the dataset

The dataset can be downloaded from this link:  https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/main/cloud-sql/postgres/pgvector/data/retail_toy_dataset.csv

(selected set of 800 toys from the Kaggle dataset – https://www.kaggle.com/datasets/promptcloud/walmart-product-details-2020)

Have the above data imported into Oracle. Please use the following table definition to import the data :

A peek into the data :

Create Embedding generator

Go to StriimAI -> Vector Embeddings Generator and create a new Embeddings Generator

1. Provide a unique name for the embedding generator.
2. Choose OpenAI as model provider
3. Enter the API key copied over from your OpenAI platform account – https://platform.openai.com/settings/organization/api-keys
4. Enter “text-embedding-ada-002” as the model
   1. The same model should be used in the Python code for converting the user query into embedding before similarity search as well.
5. Later in this blog, the above embedding generator will be used in the pipeline for generating vectors using a Striim builtin function – generateEmbeddings()

Setup the automated data pipeline from Oracle to Postgres

Go to Apps -> Create An App -> Choose Oracle as Source and Azure Postgres as Target

Follow the guided wizard flow to configure the pipeline with source, target connection details and the table selection. In the review screen, make sure to choose the “Save & Exit option”

Note: Please follow the prerequisites for Oracle CDC from Striim doc – https://striim.com/docs/en/configuring-oracle-to-use-oracle-reader.html

The table used in Postgres :

Please note that the embedding column is created additionally in Postgres.

Customize the Striim pipeline

1. Go back to the pipeline using Apps -> View All Apps
2. Open the IL app RealTimeAIDemo_IL
3. Open the reader configuration -> Disable “Create Schema” under “Schema and Data Handling” as we are creating the schema on Postgres already.

Click the output stream of the reader and add a Continuous Query component (CQ) to it to generate embeddings for the description column using the embedding generator we created above.

The CQ essentially puts the embeddings into the userdata section of the source  event, which can be used to write to the embedding column in Postgres table.

Here the output of the generateEmbedding() function will be placed as part of the ‘embedding’ section (as part of the user data)  in the source event.

Click to the target and change the input stream to the output of CQ (OutputWithEmbedding).  In the Tables property, also add a mapping for embedding column under Data Selection :

AIDEMO.PRODUCTS,aidemo.products ColumnMap(embedding=@USERDATA(embedding)

This will insert or update the ‘embedding’ column of the Postgres table with the generated vectors from the generateEmbedding() call. This column holds the vector value of the ‘description’ column value.

Perform the same steps for the CDC app in the pipeline as well.

1. Go back to the pipeline using Apps -> View All Apps
2. Open the CDC app RealTimeAIDemo_CDC and perform the same steps as done for the RealTimeAIDemo_IL application (i.e calling generateEmbedding function and columnMap())

Create Gen AI application using python

Next step is to build a python application, which does the following:

1. Accepts user query in natural text for searching a product
2. Converts the Query to embeddings using Open AI
3. Performs similarity search using PG vector extension functionality in Azure Database for PostgreSQL
4. Returns a response generated using LLM service with the top matching product details

1. Please click here to download this python application.
2. asyncpg is used to connect to Postgres
3. OpenAIEmbeddings (langchain.embeddings) is used to generate embeddings for the user query
   1. Please note that you need to use the same model in the Striim embedding generator
4. langchain.chains.summarize is used for summarisation (model used : gpt-3.5-turbo-instruct)
5. Query used to perform the similarity search :

1. 1. Similarity threshold of 0.7, max matches of 5 are used as experimentation but only one closest result is picked for summarisation

6. gradio is used to present in a simple UI

Start the pipeline to perform the initial snapshot load

Once the initial load is complete, the pipeline will automatically transition to CDC phase. Verify the data in Postgres table and confirm that embeddings are stored as well.

Run the similarity search python application and verify that it fetches the results using the similarity search of pgvector.

Capture the real-time changes from the Oracle database and generate on the fly vector conversion

Perform a simple change in your source Oracle database to make a better product description for the product with id : ’20e597d8836d9e5fa29f2bd877fc3e0a’

Striim pipeline would instantly capture the change and deliver it to the Postgres table along with the updated vector embedding.

Run the same query in the search interface to notice that a fresh result shows up :

Now the same query would result in a different product id since the query works against the recently updated data in the Oracle database.

There we go! Real-time integration and consistent data everywhere!

Conclusion

Experience the power of real-time RAG with Striim. Get a demo or start your free trial today to see how we enable context-aware, natural language-based search with real-time data consistency—delivering smarter, faster, and more responsive customer experiences!

References and credits

Inspired by the Google notebook which showcased the use case and also had reference to the curated dataset: Building AI-powered data-driven applications using pgvector, LangChain and LLMs