Ananda Venkatesha

Introducing Striim Cloud for Application Integration: A fully managed, simple, and scalable SaaS service for application connectors. With this new application integration service, users can stream real-time CRM, ERP, Billing, and Payment data from their cloud applications to data warehouses in minutes with zero coding. Instantly unlock the value of your application data through real-time insights, reports, and dashboards for your businesses. Data integration users can now take advantage of a single service that can join the application and transactional data to generate business-critical insights.

The number of cloud applications has exploded; research says enterprises, on average, deploy 500 applications, and the adoption of new applications is growing. Businesses that continuously deploy these applications are facing inevitable challenges in controlling data integration and presenting insightful data to their management and customers. As the leader in change data capture (CDC) from databases, Striim is introducing the new service Striim Cloud for Application Integration, which is built on a proven real-time streaming, scalable, and highly available Striim Cloud platform.

As a Google Cloud native, fully managed service, Striim removes the complexity of data integration, allowing businesses to focus on deriving valuable insights without worrying about the underlying technical challenges. This combination of ease of use, exceptional performance, and comprehensive management makes applications like HubSpot, Stripe, Zendesk, and more an ideal choice for organizations aiming to leverage their data for strategic advantage.

Key features:

• Offers dedicated single-tenant architecture & modern network security features to ensure the highest level of data security 
• Automated schema creation, initial load of historical data, and continuous syncs in real-time to BigQuery
• Secure, OAuth connectivity and SAML 2.0 Authentication
• Ability to transform data in-flight, in real-time to deliver business-ready application data to BigQuery
• Real-time monitoring of data delivery and data quality SLAs

Getting Started:

This blog covers getting started on Striim Cloud for Application integration solutions with an example of our new HubSpot connector. With just a few clicks steps and without coding, anyone in the organization with access to their cloud application and BigQuery can set up the pipeline and show the value of application data to the management in minutes.

Simply follow these easy steps to build your first data streaming pipeline between HubSpot and BigQuery:

1. Login to Google Marketplace, search for Striim or HubSpot
2. Subscribe to a 10-day Trial or purchase the plan
3. Signup with Striim Cloud
4. Create the first integration service (Infra)
5. Create your first pipeline (Requires HubSpot and BigQuery access) 

Step 1: Log in to Google Marketplace

Go to Google Marketplace, search for Striim or HubSpot, and select the solution HubSpot connector by Striim.

Step 2: Choose & subscribe to the plan

Striim offers a 10-day trial through the marketplace, if you want first to see the value, simply select the trial plan. Provide your billing account information to Google, read the Striim Cloud SLA, and agree to proceed. In this step, Google redirects users to the Striim Cloud signup page. Go ahead and sign up and activate your account.

Step 3: Signup with Striim Cloud

After the subscription step in the marketplace, Google will redirect users to the Striim Cloud Signup page as shown below. Sign up with the email address and unique domain name, typically a department or company name, to generate a Striim Cloud tenant with a url to access the service. You may need to activate your account from your email inbox and sign in before going to the next step

Step 4: Create Application Adapter service (Infrastructure to create pipelines)

Select the region and create a service. Striim Cloud automatically creates the infrastructure required to run Striim adapter data pipelines, including K8s cluster, networking, and storage services, and configures Striim software with all smart defaults.

After the service is in a running state, simply Launch the service to get started.

 

Step 5: Create the first data pipeline to stream HubSpot data to BigQuery

After launching the service, users will land on the Application Connectors homepage. Simply select the HubSpot to Bigquery wizard.

Configure HubSpot and BigQuery pipeline using the wizard, by default Striim creates the schema on the target (BigQuery) for your selected objects in HubSpot and the wizard automatically validates connections, permissions, and necessary requirements. 

Configure BigQuery with access; the service account key can be stored securely in the key vault. Check Striim documentation on how to use key vaults to store keys.

Wizard validates both the selected objects on the source and the dataset selected on the BigQuery and summarizes for users to confirm before starting to stream the data.

With that, the user is all set to stream data, and the  Striim for Application Adapter service starts moving data from Hubspot to BigQuery.

Businesses of all scales and industries have access to increasingly large amounts of data, which need to be harnessed effectively. According to an IDG Market Pulse survey, companies collect data from 400 sources on average. Companies that can’t process and analyze it to glean useful insights for their operations are falling behind.

Thousands of companies are centralizing their analytics and applications on the AWS ecosystem. However, fragmented data can slow down the delivery of great product experiences and internal operations.

We are excited to launch Striim Cloud on AWS: a real-time data integration and streaming platform that connects clouds, data and applications with unprecedented speed and simplicity.

With a serverless experience to build smart data pipelines in minutes, Striim Cloud on AWS helps you unify your data in real time with out-of-the box support for the following targets: 

• AWS S3
• AWS Databases on RDS and Aurora 
• AWS Kinesis
• AWS Redshift
• AWS MSK
• Snowflake
• Databricks with Delta Lake on S3

along with over 100 additional connectors available at your fingertips as a fully managed service.

Striim Cloud runs natively on AWS services like EKS, VPC, EBS, Cloudwatch, and S3 enabling it to offer infinite large-scale, high performance, and reliable data streaming. 

How does Striim Cloud bring value to the AWS ecosystem?

Striim enables you to ingest and process real-time data from over one hundred streaming sources. This includes enterprise databases via Change Data Capture, transactional data, and AWS Cloud environments. When you run Striim on AWS, it lets you create real-time data pipelines for Redshift, S3, Kinesis, Databricks, Snowflake and RDS for enterprise workloads. 

Sources and targets

Striim supports more than 120+ sources and targets. It comes with pre-built data connectors that can automate your data movement from any source to AWS Redshift or S3 within a few minutes. 

With Striim, all your team needs to do is to hit a few clicks for configuration, and an automated pipeline will be created between your source and AWS targets. Some of the sources Striim supports include:

• Databases: Oracle, Microsoft SQL Server, MySQL, PostgreSQL, etc.
• Data Streams: Kafka, JMS, IBM MQ, Rabbit MQ, IoT data over MQTT
• Data formats: JSON, XML, Parquet, Free Form Text, CSV, and XML
• AWS targets: RDS for Oracle, RDS for MySQL, RDS for SQL Server, Amazon S3, Databricks via Delta Lake on S3, Snowflake, Redshift, and Kinesis
• Additional targets: Over 100 additional connectors including custom Kafka endpoints with Striim’s full-blown schema registry support

Change data capture

Change data capture (CDC) is a process in ETL used to track changes to data in databases (e.g., insert, update, delete) and stream those changes to target systems like Redshift. However, CDC approaches like trigger-based CDC or timestamps can affect the performance of the source system. 

Striim supports the latest form of CDC — log-based CDC — which can reduce overhead on source systems by retrieving transaction logs from databases. It also helps move data continuously in real time in a non-intrusive manner. Learn about log-based CDC in detail here.                   

Streaming SQL

Standard SQL can only work with bounded data that are stored in a system. While dealing with streaming data in Redshift, you can’t use standard SQL because you are dealing with unbounded data, i.e., data that keep coming in. Striim provides a Streaming SQL engine that helps your data engineers and business analysts write SQL-style declarative queries over streaming data. These queries never stop running and can continuously produce outputs as streams.  

Data transformation and enrichment 

Data transformation and enrichment are critical steps to creating operational data products in the form of tables and materialized views with minimal cost and duplication of data. To organize these data into a compatible format for the target system, Striim helps you perform data transformation with Streaming SQL. This can include operations such as joining, cleaning, correlating, filtering, and enriching. For example, enriching helps you to add context to your data (e.g., by adding geographical information to customer data to understand their behavior). 

What makes Striim unique in this regard is that it not only supports data transformation for batch data, but it also supports in-flight transformations for real-time streams with a full blown Streaming SQL engine called Tungsten. 

Use case: How can an apparel business analyze data with Striim? 

Suppose there’s a hypothetical company, Acme Corporation, which sells apparel across the country. The management wants to make timely business decisions that can help them to increase sales and minimize the number of lost opportunities due to delays in decision-making. Some of the questions that can help them to make the right decisions include the following: 

• Which product is trending at the moment?
• Which store and location received the highest traffic last month?
• What’s the inventory status across warehouses?

Currently, all store data is stored in a transaction database (Oracle). Imagine you’re Acme Corporation’s data architect. You can generate and visualize answers to the above questions by building a data pipeline in two steps:

1. Use Striim Cloud Enterprise to stream data from Oracle to Amazon Redshift. 
2. After data is loaded into Redshift, use Amazon QuickSight service to show data insights and create dashboards. 

Here’s how the flow will look: 

In this blog, we will show you how you can configure and manage Striim Cloud Enterprise on AWS to create this pipeline for your apparel business within a few minutes. 

Sign up for Striim Cloud

Signing up for Striim Cloud Enterprise is simple: just visit striim.com, get a free trial and sign up for the AWS solution. Activate your account by following the instructions. 

 

Once you are signed in, create a Striim Cloud service, which essentially runs in the background and creates a dedicated Kubernetes cluster (EKS service on AWS) to host your pipeline, as you can see in the picture below. 

 

 

Once the cluster is ready and before launching your service, configure secure connections using the secure SSH connection configuration, as seen below.

 

Create a pipeline for Oracle to Amazon Redshift

To create a pipeline, simply type Source: Oracle and target Amazon to see all the supported targets. In our example, we are selecting Amazon S3 as our target. This could be Amazon Redshift, Kinesis, etc.

The wizard will help you walk through the simple process with source and target credentials. The service automatically validates the credentials, connects to the sources, and fetches the list of schemas and tables available on the sources for your selection, as shown below.

 

On the target side, enter Amazon Redshift Access Key and secret key with the appropriate S3 bucket name and Object names to write Oracle data into, as depicted in the image below.

Follow the wizard to finish the configuration, which creates a data pipeline that collects historical data from the Oracle database and moves them to Amazon Redshift. For example, you can see the total number of sales across all branches during the last week.  

In the next step, you can create an Oracle CDC pipeline via Striim to stream real-time change data coming in from different stores into Oracle to Redshift. Now, you can see real-time store data.

A data pipeline streaming data from the source (Oracle) to Amazon RedShift

Your data engineers can use streaming SQL to join, filter, cleanse, and enrich data on the real-time data stream before it’s written to the target system (S3). A monitoring feature offers real-time stream views for further low-latency insights. 

Once data becomes available on Redshift, your data engineer can create dashboards and set up metrics for the relevant business use cases such as:

• Current product trending
• Store and location with the highest traffic last month
• Inventory status dashboard across warehouses; quantity sold by apparel, historic graph vs. latest (last 24 hours)

Data like current trending products can be easily shared with management for real-time decision-making and the creation of business strategies.

For example, here’s a real-time view of the apparel trends by city:

And below are insights on the overall business, where you can see the top-selling and bottom-selling locations. The management can use this information to try out new strategies to increase sales in the bottom-selling locations, such as by introducing discounts or running a more aggressive social media campaign in those locations.

Striim is available for other cloud environments, too

Like AWS, Striim Cloud is available on other leading cloud ecosystems like Google Cloud and Microsoft Azure. You can use Striim with Azure to move data between on-premises and cloud enterprise sources while using Azure analytics tools like Power BI and Synapse. Similarly, you can use Striim with Google Cloud to move real-time data to analytics systems, such as Google BigQuery, without putting any significant load on your data sources. 

Learn more about them here and here.

 

Introducing Striim Cloud on Google Cloud: a fully managed and unified cloud solution offering real time data streaming and integration

Insights-driven organizations grow an average of 30% per year, but with ever-increasing data sources, formats, and volumes, it’s a huge undertaking to integrate and unify it all. While homegrown tools, scripts, and third party utilities may offer temporary relief, it can become unwieldy to manage them across multiple teams and environments. And then you add in the need for low latency — because who wants stale data? — and the struggles with scalability to keep up with company growth.

With the release of Striim Cloud on Google. Remove data silos: Connect your sources and targets and manage your data pipelines within one console. Cloud, we’re excited to offer a solution for data scientists, database admins, and businesses that rely on data.

Starting today, Striim Cloud can be purchased on the Google Cloud marketplace. Striim Cloud on Google Cloud delivers five key benefits:

1. Get started quickly: Launch smart data pipelines within ten minutes of sign up.
2. Remove data silos: Connect your sources and targets and manage your data pipelines within one console.
3. Reduce total cost of ownership: Replace multiple tools with a single platform. Pay as you go based on consumption and quickly scale as needed.
4. Ensure business continuity: Protect your business with daily backups, disaster recovery, uptime SLA of 99.5% and high availability.
5. Rest easy with enterprise-grade features: Proven at enterprise scale with petabytes of data securely and reliably moved every day to the cloud.

Striim Cloud is built on our popular Striim Enterprise platform – proven at enterprise scale. Even though Striim Cloud is designed with simplicity in mind, it is also secure, reliable, and comprehensive.

Striim Cloud gives you extensive options to control and customize your data pipelines. Services come with daily backups, built-in disaster recovery and an uptime SLA of 99.5%. This blog will take you through a sample use case, but Striim Cloud is capable of much more than this specific use case.

Striim Cloud offers great return-on-investment and delivers immediate value to cloud customers as shown below:

Striim Cloud Example Use Case: Build a Ticketing Application on Google BigQuery

To give you a quick tour of Striim Cloud, we’re going to walk through a use case for a ticketing application used to sell tickets for football and baseball games. The app is running an on-premise Oracle database. Our objective is to move data to BigQuery with millisecond latency so we can analyze the data and glean insights — like the number of tickets sold by game, by state, or by stadium — to facilitate real-time business decisions. The same flow is shown in the architecture diagram below, along with other capabilities of Striim Cloud on Google Cloud.

Start by going to the Striim Cloud Enterprise solution on the Google Cloud Marketplace. Go through the standard marketplace SaaS solution purchase flow and sign up with Striim Cloud as shown in the image below. Alternatively, you can also sign up for the trial from Striim.com.

 

 

Once you sign up for Striim Cloud, it takes less than ten minutes to get your first data pipeline up and running through a simple and intuitive user flow. It’s a three step process:

1. Create a cloud service
2. Create a Striim app for your data pipeline
3. Set up content and speed

Create a cloud service: 

In this step you only need to provide the cluster name — Striim Cloud applies smart defaults for everything else. However, if desired you can change the default cluster size, modify security options, sign-in options, user roles, and more.

Create an app for your smart data pipeline:

Next, you create a Striim app — essentially a data pipeline — using drag-and-drop elements or a wizard-based flow. Once again, Striim Cloud automatically applies smart defaults in the app. In our example, we’re creating an Oracle to BigQuery pipeline with source and target credentials for Striim Cloud to connect securely. Striim Cloud connects and validates the connection in this step for a better user experience.

Set up content and speed: 

In the third and final configuration steps, select content like schemas, collections, and tables on the source and map to the corresponding schemas, collections, and tables on the target. Striim Cloud automatically does most of the heavy lifting including auto-schema conversions and data-type conversions.

Striim Cloud offers many advanced features such as data transform, enrich, mask, encrypt, and correlate in the pipeline.

As your data is ingested and delivered, you can monitor its progress and watch real-time ticket data landing in BigQuery. With Striim Cloud, you can easily create actionable data insights and a dashboard for a real-time view of ticket sales data.

Striim Cloud offers many more features and capabilities for real-time data streaming and analytics. Learn more about Striim Cloud here and contact us for a trial or demo.