Month: November 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases.  

It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake. This fully managed service leverages Striim Cloud’s integration with the Microsoft Fabric stack for seamless data mirroring to Fabric Data Warehouse and Lake House.

Microsoft Azure Fabric is an end-to-end analytics and data platform designed for enterprises that require a unified solution. It offers a comprehensive suite of services including Data Engineering, Data Factory, Data Science, Real-Time Analytics, Data Warehouse, and Databases. Operating on a Software as a Service (SaaS) model, Fabric brings simplicity and integration to your solutions. Striim Cloud is fully integrated with the Fabric stack and runs natively in Azure next to Fabric services making it a simple, secure, fast solution to replicate data to enable Data engineering, Data science, Real-time Intelligence and Data insights through PowerBI etc.

Key Benefits

• Simplified Setup: Quick and easy deployment with a user-friendly interface.
• Real-Time Data Replication: Seamlessly transfer data from SQL Server to Fabric for immediate insights.
• Automated Data Pipelines: Benefit from automated initial load and real-time CDC pipelines, ensuring efficient data transfer.
• Automated Schema Management: Simplify data migration with automatic schema mapping and evolution.
• Robust Security: Protect your data with advanced security measures and compliance standards.
• High Performance and Scalability: Handle large volumes of data with ease and ensure optimal performance.

Getting Started

Striim’s SQL2Fabric-Mirroring is a purpose-built solution for replicating on-premises SQL Server data to Microsoft Fabric. With a few simple steps, you can automate the entire process, from initial data load to continuous replication in a few minutes.

Step 1: Signup to SQL2Fabric-Mirroring service 30-day trial

Signup for SQL2 Fabric-Mirroring service through Azure Marketplace or Striim’s product page to avail 30 days free trial. Lets you create an Striim Cloud account and login

Step 2: Create a Striim Cloud service

To get started, create a Striim Cloud service in a region close to your SQL Server. Establish a secure connection (e.g., VPN or Azure Private Link) between your on-premises environment and Striim Cloud. Then, use the intuitive interface to configure your data pipeline, specifying source and target details. Striim automates the rest.

Step 3: Create a Data pipeline

Striim’s user-friendly interface allows you to easily connect your SQL Server source and Microsoft Fabric target. The platform automatically creates an optimized data pipeline with smart defaults for SQL Server and handling initial load, continuous replication, and schema mapping using SQL server reader and Fabric Mirroring writer that is integrated with Fabric Mirroring service to offer end to end experience. This streamlined approach eliminates the need for managing multiple solutions, complex configurations and enables you to quickly start leveraging your data for real-time analytics and insights.

Step 4: Monitor high throughput data streaming 

Simply monitor real-time and continuous streaming of data from SQL Server to Fabric on Striim’s intuitive monitor screen or directly on Fabric Mirroring monitor. While Striim does the heavy lifting of ingesting Initial load (Historical data) and seamlessly switching to CDC mode (Change data capture) with low latency and high throughput data performance. Striim enables users to just focus on building business decisions by consuming real-time PowerBI dashboard insights

Easily access and create Power BI Reports on the data mirrored by Striim in Fabric.

Organizations across various industries require real-time access to data to drive decisions, enhance customer experiences, and streamline operations. A leading home improvement retailer recognized the need to modernize its data infrastructure in order to move data from legacy systems to the cloud and improve operational efficiency. To achieve these goals, the retailer partnered with Striim to support its data modernization and real-time integration efforts.

About the Retailer 

A leading home improvement retailer with thousands of stores across North America generates annual revenue exceeding $150 billion. Serving both DIY customers and professional contractors, the retailer offers a vast range of products for home improvement, construction, and gardening. Known for its customer-centric approach and expansive product offerings, the company has maintained its leadership position in the industry for decades.

Challenges 

The retailer’s legacy data infrastructure presented significant hurdles, preventing the company from achieving its modernization goals. These challenges stemmed from a complex and fragmented data environment, which included:

• Siloed Data Sources: The retailer’s on-premise databases were spread across various locations, creating silos that made it difficult to consolidate and manage data effectively.
• In-House and Third-Party Solutions: The retailer relied on a combination of in-house developed tools and third-party software. This patchwork of solutions led to inefficiencies, as different systems were not always compatible or easy to integrate.
• Complexity in Data Replication: Moving data between platforms, particularly from legacy systems to newer ones, was a time-consuming and resource-intensive process. This made it difficult for the company to support critical initiatives like supply chain optimization and migration to the cloud.
• Real-Time Data Limitations: The existing infrastructure lacked the ability to ingest and process data in real-time, making it hard for the retailer to stay agile and responsive to market demands.
• Scalability Challenges: As the company grew, its data volumes increased dramatically. The legacy systems were not built to handle this scale, creating bottlenecks and limiting the company’s ability to manage data efficiently.
• Multiple Teams Using Different Tools: Various departments, including migration and supply chain teams, used different tools and processes to manage their data. This lack of standardization added complexity and slowed down decision-making processes.

These issues underscored the need for a more efficient, scalable, and unified approach to managing the retailer’s data infrastructure.

Solution

To address the complexity and inefficiencies of its legacy data infrastructure, the retailer sought a robust platform that could simplify the migration process and provide real-time data integration across its operations. 

The goal was to consolidate data replication efforts and improve supply chain efficiency by utilizing modern cloud infrastructure. After evaluating options, the retailer partnered with Striim to leverage its real-time data streaming and low-code/no-code integration capabilities.

• Striim’s platform enabled the migration of data from legacy Oracle and PostgreSQL databases to Google BigQuery.
• Using Striim’s low-code/no-code capabilities, the retailer streamlined the migration process, reducing the burden on internal resources and cutting costs.
• Striim’s real-time data integration capabilities played a vital role in optimizing supply chain operations.
• Timely, pre-processed data delivered by Striim ensured that reporting and logistics systems could optimize operations, such as configuring truckloads based on store orders.
• The platform met real-time SLAs and performed data transformations and validations on the fly, further simplifying processes.

Outcome

After facing significant challenges with its legacy data infrastructure, the retailer partnered with Striim to completely transform its approach to data management and integration. By migrating critical on-premise databases to Google Cloud and unifying its replication and migration efforts into a single platform, the retailer achieved substantial improvements in operational efficiency, scalability, and agility. These enhancements have allowed the company to optimize its data infrastructure, enabling it to better respond to evolving market demands and maintain a competitive edge in the retail industry.

Key results include:

• Unified Data Platform: Through Striim, the retailer successfully consolidated its fragmented migration and replication processes into a single, unified platform. This eliminated the need for multiple tools and reduced the complexity of managing data across various systems, improving overall operational efficiency.
• Migration to Google Cloud: Critical on-premise databases were seamlessly migrated to Google Cloud, enhancing the retailer’s ability to scale operations and support large volumes of data with greater ease. The migration to the cloud infrastructure enabled the retailer to benefit from more flexible, scalable computing resources.
• Improved Scalability: The modernization effort significantly enhanced the retailer’s ability to handle growing data volumes. With improved scalability, the company can now manage and process vast amounts of data more efficiently, which is essential for its expanding operations and growing customer base.
• Real-Time Data Integration: Striim’s real-time data streaming capabilities allowed the retailer to ingest and process data in real time. This empowered the company to make quicker, data-driven decisions, enabling faster responses to market dynamics and customer demands.
• Operational Efficiency: By modernizing its data infrastructure and integrating real-time data streaming, the retailer was able to reduce operational costs. The transition to a microservices architecture also improved system performance and reliability, resulting in smoother workflows and a more streamlined supply chain.
• Cost-Effectiveness: By moving to a cloud-based infrastructure and consolidating its migration efforts, the retailer reduced its reliance on legacy systems and lowered resource allocation for maintenance, which resulted in significant cost savings.
• Positioned for Future Success: The retailer’s newly modernized, agile, and cost-effective data infrastructure positions the company for continued growth and success. With its scalable cloud environment and real-time data capabilities, the company is well-prepared to adapt to future industry changes and remain competitive.