Tag: Artificial Intelligence

As organizations increasingly rely on AI to drive innovation and efficiency, protecting sensitive data has become both a strategic necessity and a regulatory mandate. Traditional security measures, often reactive and manual, no longer suffice. Instead, we now stand at the cusp of a new era where data governance is automatic, intelligent, and built to match the speed of AI. 

Let’s explore how AI-driven sensitive data protection is transforming data security. Then, discover how Striim’s AI agents are leading the way in this revolution. 

The New Age of Data Governance 

Despite the widespread deployment of multiple API security products, recent surveys reveal a staggering statistic: 92% of organizations experienced an API-related security incident in the last year, with 57% encountering multiple incidents. This alarming reality underscores the limitations of traditional security measures and highlights the urgent need for more intelligent, automated solutions.

Historically, safeguarding hackable data required a labor-intensive process—manual audits, constant monitoring, and a reactive approach to threats. However, the reality of today’s fast-moving data environment demands a radical shift. With the advent of AI-driven security, sensitive data can be detected, classified, and protected in real time. This proactive stance eliminates the need for constant manual oversight. Protecting sensitive data helps organizations work towards compliance and reduce the risk of human error.

Imagine a world where sensitive data moves through systems effortlessly, but never without oversight. Striim’s AI-powered approach ensures this by detecting and classifying data before it even reaches storage. Continuous scanning identifies sensitive data the moment it’s created—not after it’s stored—while proactive security mechanisms like real-time masking, encryption, or redaction safeguard the information from exposure. Striim enables businesses to instantly manage and protect sensitive data, making it possible to adhere to regulations like GDPR, CCPA, and HIPAA. The result? Data flows freely and securely, empowering businesses to focus on what matters most.

Enter Striim’s AI Agents Sentinel and Sherlock: Pioneering AI-Powered Data Governance

Striim’s AI agents, Sentinel and Sherlock, are pioneering tools that bring real-time, AI-powered governance to your data pipelines, increasing security without compromising performance.

Sherlock AI offers: 

• Source Operation: Identifies sensitive data before it enters data pipelines—even in third-party-managed databases and SaaS environments.
• Early Detection: Finds sensitive data before it moves, eliminating risk at the earliest stage.
• Comprehensive Visibility: Works seamlessly across SaaS, cloud, and third-party environments to ensure full visibility.
• Lightweight Scanning: Operates with zero performance impact, ensuring databases aren’t overloaded.
• Automated Classification: Classifies financial, health, and identity-related PII automatically, providing real-time security insights.
• Data Quality Monitoring: Detects data quality issues in real time, alerting teams when sensitive data appears in unintended locations.

Sentinel AI provides: 

• In-Motion Protection: Provides real-time detection and protection of sensitive data as it moves across systems.
• Accurate Detection: Spots PII anywhere in a record—even if it’s misplaced or mislabeled—beyond the scope of rules-based controls.
• Exposure Prevention: Prevents data exposure when transferring information from internal systems to external platforms for analytics or exchange.
• Compliance Support: Supports 25+ sensitive data types across the USA, Canada, UK, and India to support various compliance requirements.
• Automated Actions: Executes policy-based actions such as encryption and masking (partial, full, regex-based) automatically.
• Plug-and-Play UX: Easily integrated into your pipeline with a plug-and-play setup that requires only a few clicks.
• Regulatory Governance: Supports businesses on their journey to meet GDPR, CCPA, HIPAA, and other regulatory requirements.

Together, Sherlock AI and Sentinel AI work to prevent sensitive data exposure before it happens, ensuring your operations remain secure and that your team is in full control of its data.

How AI-Powered Data Governance Works

Our process begins with Sherlock AI, which proactively identifies sensitive data at its source—before it moves. By scanning both structured and unstructured data across SQL, NoSQL, SaaS, and cloud databases, it detects and automatically classifies financial, health, and identity-related information that may present compliance challenges. 

As data moves, Sentinel AI validates it in real time using advanced pattern recognition and NLP, catching any mislabeled or misplaced data that traditional rules-based systems might overlook. Sentinel AI then applies automated protection measures—encrypting, masking, or blocking data based on business policies—to secure its movement between internal and external systems and prevent unintended processing of regulated information. 

Sentinel delivers live reporting via real-time dashboards that continuously monitor sensitive data exposure, security actions, and compliance. It uses predefined identifiers to detect, log, and protect sensitive information, while AI-driven metadata tags each event for effective tracking and auditing. With support for schema evolution, Sentinel easily adapts to new data sources, ensuring ongoing AI-powered data governance.

This continuous monitoring helps organizations stay audit-ready and compliant. Real-time dashboards provide complete visibility into data protection efforts, and Sentinel generates audit logs that align regulations like GDPR, CCPA, HIPAA, and the EU AI Act. Additionally, it integrates with enterprise security tools such as SIEM, DLP, Datadog, and Snowflake Security to ensure a unified security framework.

The Impact of AI-Powered Automation on Data Governance 

By automating these processes, organizations no longer need to scramble after a potential data breach. Instead, security becomes a built-in feature of data management. Sensitive information is automatically shielded by AI agents as it moves through the enterprise ecosystem, whether in production environments, during testing, or throughout analytics workflows.

Automated authentication and connection processes also reduces strain on IT teams. This allows security professionals to shift their focus from routine monitoring to strategic initiatives, such as threat intelligence and proactive risk management. With Sentinel AI operating silently in the background, businesses can innovate without fear of compromising their sensitive data. 

By ensuring that sensitive data is protected, organizations can also enhance customer trust. In addition, streamlined security processes translate into improved operational efficiency. Data flows remain uninterrupted, and the risk of security incidents is drastically minimized.

Moving Forward in the AI Era

The AI era requires businesses to rethink traditional approaches to data security. With the speed at which data moves and the sophistication of modern cyber threats, it’s clear that reactive measures are no longer sufficient. Automated, intelligent solutions are not just an option—they are a necessity. 

Get a demo today and discover how Striim can help you better protect your data. 

 

Imagine searching for products on an online store by simply typing “best eco-friendly toys for toddlers under $50” and getting instant, accurate results—while the inventory is synchronized seamlessly across multiple databases. This blog dives into how we built a real-time AI-powered hybrid search system to make that vision a reality. Leveraging Striim’s advanced data streaming and real-time embedding generation capabilities, we tackled challenges like ensuring low-latency data synchronization, efficiently creating vector embeddings, and automating inventory updates.

We’ll walk you through the design decisions that balanced consistency, efficiency, and scalability and discuss opportunities to expand this solution to broader Retrieval-Augmented Generation (RAG) use cases. Whether you’re building cutting-edge AI search systems or optimizing hybrid cloud architectures, this post offers practical insights to elevate your projects.

What is RAG?

Retrieval-Augmented Generation (RAG) enhances the capabilities of large language models by incorporating external data retrieval into the generation process. It allows the model to fetch relevant documents or data dynamically, ensuring responses are more accurate and context-aware. RAG bridges the gap between static model knowledge and real-time information, which is crucial for applications that require up-to-date insights. This hybrid approach significantly improves response relevance, especially in domains like e-commerce and customer service.

Why Vector Embeddings and Similarity Search?

Vector embeddings translate natural language text into numerical representations that capture semantic meaning. This allows for efficient similarity searches, enabling the discovery of products even if queries differ from stored descriptions. Embedding-based search supports fuzziness, matching results that aren’t exact but are contextually relevant. This is essential for natural language search, as it interprets user intent beyond simple keyword matching. The combination of embeddings and similarity search improves the user experience by providing more accurate and diverse search results.

Why Real-time RAG Instead of Batch-Based Data Sync?

Real-time RAG ensures that inventory changes are reflected instantly in the search engine, eliminating stale or outdated results. Unlike batch-based sync, which can introduce latency, real-time pipelines offer continuous updates, improving accuracy for fast-moving inventory. This minimizes the risk of selling unavailable products and enhances customer satisfaction. Real-time synchronization also supports dynamic environments where product data changes frequently, aligning search capabilities with the latest inventory state.

How We Designed the Embedding Generator for Performance

In designing the Vector Embedding Generator, we addressed the challenges associated with token estimation, handling oversized input data, and managing edge cases such as null or empty input strings. These design considerations ensure that the embedding generation process remains robust, efficient, and compatible with various AI models.

Token Estimation and Handling Large Data

Google Vertex AI

Vertex AI simplifies handling large data inputs by silently truncating input data that exceeds the token limit and returning an embedding based on the truncated input. While this approach ensures that embeddings are always generated regardless of input size, it raises concerns about data loss affecting embedding quality. We have an ongoing effort to analyze how this truncation impacts embeddings and whether improvements can be made to mitigate potential quality issues.

OpenAI

OpenAI enforces strict token limits, returning an error if input data exceeds the threshold (e.g., 2048 or 3092 tokens). To handle this, we integrated a tokenizer library into the Embedding Generator’s backend to estimate token counts before sending data to the API. The process involves:

1. Token Count Estimation: Input strings are tokenized to determine the estimated token count.
2. Iterative Truncation: If the token count exceeds the model’s limit, we truncate the input to 75% of its current size and recalculate the token count. This loop continues until the token count falls within the model’s threshold.
3. Submission to Model: The truncated input is then sent to OpenAI for embedding generation.

For instance, if an OpenAI model has a token limit of 3092 and the estimated token count for incoming data is 4000, the system will truncate the input to approximately 3000 tokens (75%) and re-estimate. This iterative process ensures compliance with the token limit without manual intervention.

Handling Null or Empty Input

When generating embeddings, edge cases like null or empty input can result in API errors or undefined behavior. To prevent such scenarios, we adopted a solution inspired by discussions in the OpenAI developer forum: the use of a default vector.

Characteristics of the Default Vector:

• Dimensionality: Matches the size of embeddings generated by the specific model (e.g., 1536 dimensions for OpenAI’s text-embedding-ada-002).
• Structure: The vector contains a value of 1.0 at the first index, with all other indices set to 0.0.
  • Example:[1.0, 0.0, 0.0, … , 0.0]

By returning this default vector, we ensure the system gracefully handles cases where input data is invalid or missing, enabling downstream processes to continue operating without interruptions.

Summary of Implementation:

1. Preprocessing
   • Estimate token counts and handle truncation for models with strict token limits (OpenAI).
   • Allow silent truncation for models like Google Vertex AI but analyze its impact.
2. Error Handling
   • For null or empty data, return a default vector matching the model’s embedding dimensions.
3. Scalability
   • These mechanisms are integrated seamlessly into the Embedding Generator, ensuring compatibility across multiple data streams and models without manual intervention.

This design enables developers to generate embeddings confidently, knowing that token limits and edge cases are managed effectively.

Tutorial: Using the Embeddings Generator for Search

An e-commerce company aims to build an AI-powered hybrid search that enables users to describe their needs in natural language.

Their inventory management system is in Oracle database and the store front search database is maintained in the Azure PostgetSQL.

Current problem statements are:

1. Data Synchronization: Inventory data from Oracle must be replicated in real-time to the storefront’s search engine to ensure data consistency and avoid stale information.
2. Vector Embedding Generation: Product descriptions need vector embeddings to facilitate similarity searches. The storefront must support storing and querying these embeddings.
3. Real-Time Updates: Ongoing changes in the inventory (such as product details or stock updates) need to be reflected immediately in the search engine.
4. Embedding Updates: Updates to products in the inventory should trigger real-time embedding regeneration to prevent outdated or inaccurate similarity search results.

Solution

Striim has all the necessary features for the use case and the problem statements described.

1. Readers to capture initial snapshot and real-time changes from Oracle database.
2. Embedding generator to generate vector embeddings for text content.
3. Writers to deliver the data along with the embeddings to Postgres database.

Striim Features :

1. Readers: Capture the initial snapshot and ongoing real-time changes from the Oracle database.
2. Embedding Generator: Creates vector embeddings for product descriptions.
3. Writers: Deliver updated data and embeddings to the PostgreSQL database.

PostgreSQL with pgvector:

1. Supports storing vector embeddings as a specific data type.
2. Enables similarity search functionality directly within the database.

Note: The search engine for this solution is implemented in Python for demonstration purposes. OpenAI is used for embedding generation and for summarisation of the results.

Design Choices and Rationale

1. Striim for Data Integration: Chosen for its seamless real-time change capture (CDC) from Oracle to PostgreSQL.
   1. Alternative could be to have an independent application/script periodically but would be inefficient and expensive to have this developed for the initial load and change data capture, also would be difficult to maintain over a period of time for various data sources which might also need to be synced to the store front database (Postgres)
2. OpenAI Embeddings: Ensures high-quality embeddings compatible across pipeline stages.
3. Striim Embedding generator: Enables in-flight embedding generation while the data is being synced instead of having to move the data separately and then generate and update the embedding.
   1. Alternative is to have separate listeners/triggers/scripts in the search front database (Postgres) to generate and update the embeddings every time the data changes. This could be very expensive and not be very accurate as the data changes and the embedding changes can go out of sync.
4. pgvector: Facilitates native vector searches, reducing system complexity.
   1. Alternative design choices were to choose the standalone vector databases but they do not provide the flexibility pgvector provides as we can store the actual data and the embeddings in a relational database setup compared to vector databases where we need to maintain metadata for each vector database and cross lookup for actual data from a different database/source while summarising the similarity search results. Eg., the embeddings would be in one place which needs to be queried using similarity search query whereas price or rating-based filters need to be applied elsewhere.
5. Python Search Engine: Provides flexibility and integration simplicity.
   1. This is a convenient choice to make use of the python libraries, more details are included in the upcoming sections

Future Work

1. Expand embedding model options beyond OpenAI and make it generic
   1. This is already done for the Striim Embedding generator as it supports VertexAI as well. We could consider supporting self-hosted models.
2. Expand the support for non-textual input.
3. Expand the implementation to generically cover any use case and have the application interface integrated with Striim UI.
   1. Current implementation as a proof of concept is tightly coupled with the e-commerce use case and it’s data set.

Step-by-step instructions

Set up Striim Developer 5.0

1. Sign up for Striim developer edition for free at https://signup-developer.striim.com/.
2. Select Oracle CDC as the source and Database Writer as the target in the sign-up form.

Prepare the dataset

The dataset can be downloaded from this link:  https://raw.githubusercontent.com/GoogleCloudPlatform/python-docs-samples/main/cloud-sql/postgres/pgvector/data/retail_toy_dataset.csv

(selected set of 800 toys from the Kaggle dataset – https://www.kaggle.com/datasets/promptcloud/walmart-product-details-2020)

Have the above data imported into Oracle. Please use the following table definition to import the data :

A peek into the data :

Create Embedding generator

Go to StriimAI -> Vector Embeddings Generator and create a new Embeddings Generator

1. Provide a unique name for the embedding generator.
2. Choose OpenAI as model provider
3. Enter the API key copied over from your OpenAI platform account – https://platform.openai.com/settings/organization/api-keys
4. Enter “text-embedding-ada-002” as the model
   1. The same model should be used in the Python code for converting the user query into embedding before similarity search as well.
5. Later in this blog, the above embedding generator will be used in the pipeline for generating vectors using a Striim builtin function – generateEmbeddings()

Setup the automated data pipeline from Oracle to Postgres

Go to Apps -> Create An App -> Choose Oracle as Source and Azure Postgres as Target

Follow the guided wizard flow to configure the pipeline with source, target connection details and the table selection. In the review screen, make sure to choose the “Save & Exit option”

Note: Please follow the prerequisites for Oracle CDC from Striim doc – https://striim.com/docs/en/configuring-oracle-to-use-oracle-reader.html

The table used in Postgres :

Please note that the embedding column is created additionally in Postgres.

Customize the Striim pipeline

1. Go back to the pipeline using Apps -> View All Apps
2. Open the IL app RealTimeAIDemo_IL
3. Open the reader configuration -> Disable “Create Schema” under “Schema and Data Handling” as we are creating the schema on Postgres already.

Click the output stream of the reader and add a Continuous Query component (CQ) to it to generate embeddings for the description column using the embedding generator we created above.

The CQ essentially puts the embeddings into the userdata section of the source  event, which can be used to write to the embedding column in Postgres table.

Here the output of the generateEmbedding() function will be placed as part of the ‘embedding’ section (as part of the user data)  in the source event.

Click to the target and change the input stream to the output of CQ (OutputWithEmbedding).  In the Tables property, also add a mapping for embedding column under Data Selection :

AIDEMO.PRODUCTS,aidemo.products ColumnMap(embedding=@USERDATA(embedding)

This will insert or update the ‘embedding’ column of the Postgres table with the generated vectors from the generateEmbedding() call. This column holds the vector value of the ‘description’ column value.

Perform the same steps for the CDC app in the pipeline as well.

1. Go back to the pipeline using Apps -> View All Apps
2. Open the CDC app RealTimeAIDemo_CDC and perform the same steps as done for the RealTimeAIDemo_IL application (i.e calling generateEmbedding function and columnMap())

Create Gen AI application using python

Next step is to build a python application, which does the following:

1. Accepts user query in natural text for searching a product
2. Converts the Query to embeddings using Open AI
3. Performs similarity search using PG vector extension functionality in Azure Database for PostgreSQL
4. Returns a response generated using LLM service with the top matching product details

1. Please click here to download this python application.
2. asyncpg is used to connect to Postgres
3. OpenAIEmbeddings (langchain.embeddings) is used to generate embeddings for the user query
   1. Please note that you need to use the same model in the Striim embedding generator
4. langchain.chains.summarize is used for summarisation (model used : gpt-3.5-turbo-instruct)
5. Query used to perform the similarity search :

1. 1. Similarity threshold of 0.7, max matches of 5 are used as experimentation but only one closest result is picked for summarisation

6. gradio is used to present in a simple UI

Start the pipeline to perform the initial snapshot load

Once the initial load is complete, the pipeline will automatically transition to CDC phase. Verify the data in Postgres table and confirm that embeddings are stored as well.

Run the similarity search python application and verify that it fetches the results using the similarity search of pgvector.

Capture the real-time changes from the Oracle database and generate on the fly vector conversion

Perform a simple change in your source Oracle database to make a better product description for the product with id : ’20e597d8836d9e5fa29f2bd877fc3e0a’

Striim pipeline would instantly capture the change and deliver it to the Postgres table along with the updated vector embedding.

Run the same query in the search interface to notice that a fresh result shows up :

Now the same query would result in a different product id since the query works against the recently updated data in the Oracle database.

There we go! Real-time integration and consistent data everywhere!

Conclusion

Experience the power of real-time RAG with Striim. Get a demo or start your free trial today to see how we enable context-aware, natural language-based search with real-time data consistency—delivering smarter, faster, and more responsive customer experiences!

References and credits

Inspired by the Google notebook which showcased the use case and also had reference to the curated dataset: Building AI-powered data-driven applications using pgvector, LangChain and LLMs

During a crisis—whether it’s a pandemic, a natural disaster, or a major supply chain breakdown—swift, informed decision-making can mean the difference between regaining control and facing further escalation. Today’s organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. 

That’s where real-time artificial intelligence (AI) can help. When integrated effectively, AI and machine learning (ML) models can process data streams at near-zero latency, empowering teams to make split-second decisions. In this post, we’ll explore how real-time data and AI-driven analytics reshape crisis management across industries such as healthcare, logistics, and emergency services. We’ll also show how Striim can serve as the backbone for these real-time data pipelines—ensuring that decisions are always based on the most current, accurate information.

The Power of Real-Time Data in Crisis Management

When a crisis unfolds, data moves at lightning speed. Hospitals must juggle incoming patient information, logistics teams track thousands of shipments, and emergency responders monitor multiple channels in parallel. Real-time data is the foundation of effective crisis response; without it, instant updates, continuous monitoring, and timely communication are impossible. 

Here’s how real-time data empowers different facets of crisis management: 

• Instant Updates: Real-time dashboards alert decision-makers to critical events as they happen, rather than hours later.
• Continuous Monitoring: Streaming analytics detect anomalies—such as sudden spikes in patient admissions or unexpected traffic congestion—so you can intervene before a problem grows.
• Timely Communication: Automated alerts and notifications ensure the right teams react immediately, preventing confusion and delays.

By integrating AI/ML models directly into these data streams, organizations gain deeper insights: advanced algorithms can spot emerging patterns, predict cascading effects, and recommend interventions—all in the moment.

Key Challenges in Adopting Real-Time AI 

Despite its transformative potential, implementing real-time AI for crisis management comes with hurdles:

Data Quality and Availability

Inconsistent or incomplete data can severely impact the accuracy of ML models and therefore, your emergency response. Continuous data cleaning and integration are essential to maintain reliable outputs.

Managing AI “Hallucinations”

Certain AI models, including large language models, may produce plausible yet incorrect answers. Validation and monitoring can help reduce this risk. 

Safeguarding Personally Identifiable Information (PII)

Oftentimes, crisis data includes sensitive details (e.g., patient records or geolocation data). Encryption, access controls, and regulatory compliance (HIPAA, GDPR, etc.) are non-negotiable. However, leveraging AI agents like Striim’s Sherlock and Sentinel, which enable encryption and masking for PII, can help ensure that data is safe even in the event a breach occurs. 

Meeting Critical Latency Requirements

In many scenarios—like patient triage or disaster response—latency thresholds are near-zero. Systems must be capable of handling high-velocity data without bottlenecks.

As you can see, there’s a lot to consider in adopting real-time AI. Addressing these challenges demands an end-to-end approach that integrates data ingestion, streaming analytics, AI governance, and security in a cohesive pipeline.

Real-Time AI Use Cases: Healthcare, Logistics, and Emergency Services

There are several real-time AI use cases for crisis management, with three being the most popular. These include: 

Healthcare

Hospitals leverage real-time data to consolidate streaming vital signs, EHR updates, and lab results for in-the-moment patient monitoring.AI models can detect potential complications (like sepsis or respiratory decline) in real time, alerting medical staff before conditions worsen.

Logistics

A supply chain interruption—caused by a factory shutdown or severe weather—can ripple through an entire network.By feeding live shipment data and warehouse updates into an ML model, logistics managers receive instant recommendations on rerouting or inventory reallocation, minimizing costly delays.

Emergency Response Services 

Police and rescue teams often depend on 911 call data, social media information, and geospatial tracking.With real-time AI, dispatchers can prioritize resource allocation where it’s needed most. For example, analyzing social media mentions of flooded areas can guide rescuers to hotspots before formal reports come in.

In each scenario, real-time data plus AI-driven insights create a powerful feedback loop—one that not only accelerates crisis response but also continuously refines itself through ongoing data ingestion and machine learning updates.

How Striim Empowers Real-Time AI Pipelines 

Striim acts as the backbone for your real-time AI initiatives, processing data streams at scale and delivering low-latency insights. Striim enables: 

Real-Time Data Integration

Striim’s distributed, in-memory streaming architecture ingests data from transactional databases, IoT sensors, and application logs in real time. Additionally, parallel processing allows you to handle high-velocity data without sacrificing speed or reliability.

Integration with Inline and External AI/ML Models 

Through Advanced Real-Time ML Analytics, Striim seamlessly integrates with inline and external AI/ML models, so you can embed advanced analytics directly into streaming data flows. This approach delivers meaningful insights the moment data arrives, supported by continuous learning algorithms that adapt models dynamically to evolving conditions.

Retrieval Augmented Generation (RAG) by Creating Vector Embeddings 

Striim also enables RAG by creating instant vector embeddings in enterprise data pipelines and distributing those vectors for next-generation hybrid search. Its AI Insights capabilities further streamline automated PII detection, security, and data preprocessing for prompt engineering, ensuring full compliance without adding complexity.

Leverage Real-Time Data and AI for Crisis Management 

Crisis management in the modern world requires more than just reactive steps—it demands continuous awareness and the ability to pivot on a moment’s notice. By integrating AI/ML models directly into streaming data pipelines, organizations can detect anomalies, predict cascading impacts, and execute real-time interventions. 

Ready to learn how Striim can help your organization leverage real-time AI for crisis management? Register for a demo.

In today’s competitive environment, enterprises need to harness data the instant it’s created. But data teams often face challenges when it comes to capturing, processing, and integrating high-velocity data streams from diverse sources—making it difficult to keep AI applications timely and relevant. Simultaneously, generative AI (GenAI) is becoming indispensable for delivering dynamic, real-time solutions, from chatbots and personalized marketing to adaptive decision-making.

Where these two trends collide—real-time data streaming and GenAI—lies a major opportunity to reshape how businesses operate. However, turning this vision into reality requires more than just powerful AI algorithms. Today’s enterprises are tasked with implementing a robust, flexible data integration layer capable of feeding GenAI models fresh context from multiple systems at scale.

In this post, we’ll explore the synergy between GenAI and streaming data and how this powerful combination is set to shape the next era of enterprise AI.

Key Challenges at the Intersection of GenAI and Streaming Data

While the merging of real-time data with GenAI offers exciting possibilities, the path forward is certainly not without challenges, such as: 

1. Poor Data Quality and Availability

AI’s success is highly dependent upon data quality. To achieve accurate and reliable results, businesses need to ensure their data is clean, consistent, and relevant. This proves especially difficult when dealing with large volumes of high-velocity data from various sources.

To address this, enterprises need robust data validation systems that clean, filter, and process data streams in real time. Consistent monitoring and real-time integration are also necessary to ensure that data remains reliable and relevant for AI models.

2. High Latency 

Real-time applications such as fraud detection, personalized marketing, or anomaly detection require low-latency. If the data infrastructure can’t process and deliver insights in near real-time, the value of streaming data and GenAI models diminishes rapidly.

For businesses using GenAI for customer support, for example, a chatbot must provide responses almost instantaneously, reflecting the latest user inputs. Without low-latency systems, customers may experience delays, leading to reduced satisfaction and engagement.

3. Ensuring the Protection of Personally Identifiable Information (PII) in AI Pipelines 

When processing high-velocity streams of data, particularly in GenAI models, protecting sensitive information is crucial. As AI systems increasingly handle vast amounts of personal and confidential data, ensuring that PII remains secure becomes a major challenge. Without robust safeguards in place, there is a risk of unauthorized access or accidental exposure, which could compromise privacy and violate regulatory requirements, eroding customer confidence. 

4. Unscalable Infrastructure 

As data volume, variety, and velocity increase, organizations must invest in scalable infrastructure that can handle vast and growing datasets. With the rise of IoT devices and increased automation, businesses will generate even larger amounts of data, meaning infrastructure must be able to scale both horizontally and vertically. 

The Future of Enterprise AI: Moving from Vision to Reality

Successfully integrating GenAI with real-time data streaming requires strategic investments across infrastructure, data governance, and AI model development. Here are the critical steps enterprises should take to turn this vision into a tangible, scalable solution.

1. Establish a Solid Data Integration Foundation

To power real-time GenAI models, businesses need a robust data integration infrastructure capable of handling high-velocity streams from multiple sources. It’s also imperative that real-time data streaming platforms are scalable to ensure that data can be ingested, processed, and delivered to AI models in real time.

Key considerations for building this foundation include:

• Unified Data Layer: Integrate data from various sources (cloud, on-premises, IoT devices, social media) into a unified pipeline for seamless AI processing.
• Data Quality Management: Implement data validation, transformation, and normalization techniques to ensure clean, consistent, and relevant data.
• Performance Management: Ensure your infrastructure can handle growing data volumes without sacrificing performance, leveraging cloud-native solutions that dynamically scale as needed.

Example Use Case: Financial institutions can integrate live transaction data, currency exchange rates, and customer behavior patterns into GenAI models for real-time personalized banking services.

2. Prioritize Real-Time Data Governance and Privacy

Real-time data streaming brings significant privacy and governance challenges. Organizations must implement privacy-preserving practices such as encryption, anonymization, and tokenization to protect sensitive data.

Steps for ensuring governance include:

• Real-Time Data Monitoring: Continuously track data integrity and security as it flows through your pipeline to ensure accuracy and protect PII.
• Compliance with Regulations: Ensure that AI models comply with global data privacy regulations, such as GDPR and CCPA, and integrate compliance checks into the data pipeline.

Striim offers AI agents Sentinel and Sherlock, which leverage advanced AI to detect and protect sensitive data in real time. Sherlock monitors your data streams to identify sensitive information. Sentinel applies protection methods including masking or encryption to safeguard your data. 

Example Use Case: A healthcare provider can integrate patient data into real-time AI-driven applications while ensuring compliance with healthcare privacy laws with the help of PII masking.

3. Leverage Continuous Model Training and Fine-Tuning

For GenAI models to stay relevant, they must be continually updated with new data. Real-time data streaming allows for the continuous retraining of AI models, ensuring that they adapt to emerging trends, changes in user behavior, and evolving business needs.

Key steps include:

• Real-Time Model Retraining: Set up processes for automatic model updates as new data arrives, ensuring the AI remains accurate and responsive.
• Feedback Loops: Incorporate real-time feedback from AI models to refine and improve data quality and decision-making.

Example Use Case: E-commerce platforms using real-time browsing data can continuously update product recommendation models, keeping content aligned with current trends.

4. Invest in Scalable Infrastructure

To manage the growing volume of real-time data and the increasing demands of GenAI, businesses need flexible, scalable infrastructure. Cloud-native solutions, edge computing, and distributed frameworks enable companies to process vast amounts of data quickly and efficiently.

Striim Cloud is designed to support these needs by offering fully managed, real-time data streaming pipelines, allowing organizations to build and scale data processing workflows in minutes. With Striim Cloud available on AWS, Google Cloud, and Microsoft Azure, businesses can ensure seamless data integration, rapid decision-making, and low-latency performance across both cloud-native and edge computing environments.

Example Use Case: A logistics company can use Striim’s cloud native infrastructure to stream data from IoT sensors in real time, optimizing fleet operations and reducing maintenance costs.

5. Foster Cross-Functional Collaboration

Realizing the potential of GenAI and streaming data requires collaboration between data teams and business stakeholders. Alignment across departments ensures that AI models meet business goals and deliver measurable value.

Key strategies for fostering collaboration include:

• Unified Business Goals: Ensure that all stakeholders understand the value of real-time data and GenAI models for achieving business outcomes.
• Agile Development: Adopt agile practices to enable rapid prototyping and iteration, allowing teams to test and refine AI solutions quickly.

Example Use Case: Retailers seeking to implement dynamic pricing models based on real-time customer data will benefit from close collaboration between data scientists and business analysts to ensure pricing strategies align with market conditions.

The Future of Enterprise AI

The combination of GenAI and real-time streaming data represents a massive opportunity for businesses to drive innovation, optimize operations, and provide more personalized experiences. However, to fully capitalize on this potential, enterprises must invest in scalable, secure, and efficient infrastructures, maintain continuous learning systems, and foster cross-functional collaboration. Ready to see how Striim can accelerate your data and AI initiatives? Schedule a demo today to explore powerful real-time streaming and data integration solutions tailored to your organization’s needs.

In today’s fast-paced financial landscape, detecting transaction fraud is essential for protecting institutions and their customers. This article explores how to leverage Striim and SGDClassifier to create a robust fraud detection system that utilizes real-time data streaming and machine learning.

Problem

Transaction fraud detection is a critical responsibility for the IT teams of financial institutions. According to the 2024 Global Financial Crime Report from Nasdaq, an estimated $485.6 billion was lost to fraud scams and bank fraud schemes globally in 2023.

AI and ML help detect fraud, while real-time streaming frameworks like Striim play a key role in delivering financial data to reference and train classification models, enhancing customer protection.

Solution

In this article, I will demonstrate how to use Striim to perform key tasks for fraud detection with machine learning:

• Ingest data using a Change Data Capture (CDC) reader in real time, call the model and deliver alerts to a target such as Email, Slack, Teams or any other target supported by Striim
• Train the model using Striim Initial load app and re-train the model if its accuracy score decreases by using automation via REST APIs

Fraud Detection Approach

In typical credit card transactions, a financial institution’s data science team uses supervised learning to label data records as either fraudulent or legitimate. By carefully analyzing the data, engineers can extract key features that define a fraudulent user profile and behavior, such as personal information, number of orders, order content, payment history, geolocation, and network activity.

For this example, I’m using a dataset from Kaggle, which contains credit card transactions collected from EU retailers approximately 10 years ago. The dataset is already labeled with two classes representing fraudulent and normal transactions. Although the dataset is imbalanced, it serves well for this demonstration. Key fields include purchase value, age, browser type, source, and the class parameter, which indicates normal versus fraudulent transactions.

Picking Classification Model

There are many possibilities for classification using ML. In this example, I evaluated logistic regression and SGDClassifier: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html. The main difference is that SGDClassifier uses stochastic gradient descent optimization whereas logistic regression uses the logistic function to model binary classification. Many experts consider SGD to be a more optimal approach for larger datasets, which is why it was selected for this application.

Accuracy Measurement

The accuracy score is a metric that measures how often a model correctly predicts the desired outcome. It is calculated by dividing the total number of correct predictions by the total number of predictions. In an ideal scenario, the best possible accuracy is 100% (or 1). However, due to the challenges of obtaining and diagnosing a high-quality dataset, data scientists typically aim for an accuracy greater than 90% (or 0.9).

Training Step

Striim provides the ability to read historical data from various sources including databases, messaging systems, files, and more. In this case, we have historical data stored in the MySQL database, which is a highly popular data source in the FinTech industry. Here’s what architecture with real-time data streaming augmented with training of the ML model looks like:

You can achieve this in Striim with an Initial Load application that has a Database reader pointed to the transactions table in MySQL and file target. With Striim’s flexible adapters, data can be loaded virtually from any database of choice and loaded into a local file system, ADLS, S3 or GCS.

Once the data load is completed, the application will change its status from RUNNING to COMPLETED. A script, or in this case, a PS made Open Processor (OP), can capture the status change and call the training Python script.

Additionally, I added a step with CQ (Continuous Query) that allows data scientists to add any transformation to the data in order to prepare the form satisfactory for the training process. This step can be easily implemented using Striim’s Flow Designer, which features a drag and drop interface along with the ability to code data modifications using a combination of SQL-like language and utility function calls.

Model Reference Step

Once the model is trained, we can deploy it in a real-time data CDC application that streams user financial transactions from an operational database. The application calls the model’s predict method, and if fraud is detected, it generates and sends an alert. Additionally, it will check the model accuracy and, if needed, initiate the retraining step described above.

Model Reference App Structure

Flow begins with Striim’s CDC reader that streams financial transactions directly from database binary log. It then invokes our classification model that was trained in the previous step via a REST CALL. In this case, I am using an OP that executes REST POST calls containing parsed transaction values needed for predictions. The model service returns the prediction to be parsed by a query. If fraud is detected, it generates an alert. At the same time, if the model accuracy dips below 90 percent, the Application Manager function can restart a training application called IL MySQL App using an internal management REST API.

Final Thoughts on Leveraging SGDClassifier and Striim for Financial Fraud Detection

This example illustrates how a real-world data streaming application can detect fraud by interacting with a classification model. The application sends alerts when fraud is detected using various Striim alert adapters, including email, web, Slack, or database. Furthermore, if the model’s quality deteriorates, it can retain the model for further evaluation.

For reference TQL sources: