Tag: Striim

 

 

We are pleased to announce the general availability of Striim 3.9.8 with a rich set of features that span multiple areas, including advanced data security, enhanced development productivity, data accountability, performance and scalability, and extensibility with new data targets.

The new release brings many new features that are summarized here:

Let’s review the key themes and features of the new release starting with the security topic.

Advanced Platform and Adapter Security:

With a sharp focus on business-critical systems and use cases, the Striim team has been boosting the platform’s security features for the last several years. However, in version 3.9.8, we introduced a broad range of advanced security features to both the platform and its adapters to provide users with robust security for the end-to-end solution, and higher control for managing data security.

The new platform security features include the following components:

• Striim KeyStore, which is a secured, centralized repository based on Java Keystore, for storing passwords and encryption keys, streamlines security management across the platform.
• Ultra-secure algorithms for user password encryption across all parts of the platform reducing platform’s vulnerabilities to external or internal breaches.
• Stronger encryption support for inter-node cluster communication with internally generated, long string password and unified security management for all nodes and agents.
• Multi-layered application security via advanced support for exporting and importing pipeline applications within the platform. In Striim, all password properties of an application are encrypted using their own keys. When exporting applications containing passwords or other encrypted property values, you can now add a second level of encryption with a passphrase that will be required at the time of import, to strengthen the application security.
• Encryption support using customer provided key for securing permanent files, via the File Writer, and for the intermediate temporary files via the Google Cloud Storage Writer. Supported encryption algorithm types include RSA, AES and PGP. You can generate keys for encrypting by multiple tools available online or using in house Java program and easily configure the encryption settings of the adapters via the Encryption policy property on the UI.

Overall, these new security features enable:

• Enhanced platform and adapter security for hybrid cloud deployments and mission-critical environments
• Strengthened end-to-end data protection from ingestion to file delivery
• Enhanced compliance with strict security policies and regulations
• Secured application sharing between platform users

Improved Data Accountability:

Striim version 3.9.8 includes an application-specific exception store for storing events discarded by the application, including discarded records. The feature allows viewing discarded records and their details in real time. You can configure this feature with a simple on/off option when building an application. With this feature, Striim improves its accountability for all data passing through the platform and allows users to build applications for replaying and processing discarded records.

Enhanced Application Development Support and Ease of Use

The new release also includes features that accelerate and ease developing integration applications, especially in high-volume data environments.

• A New Enrichment Transformer: Expanding the existing library of out-of-the-box transformers, the new enrichment transformer function allows you to enrich your streaming data in-flight without any manual coding step. You only need Striim’s drag and drop UI to create a real-time data pipeline that performs in-memory data lookups. With this transformer, you can, for example, add City Name and County Name fields to an event containing Zip Code.

• External Lookups: Striim provides an in-memory data cache to enrich data in-flight at very high speeds. With the new release, Striim gives you the option to enrich data with lookups from external data stores. The platform can now execute a database query to fetch data from an external database and return the data as a batch. The external lookup option helps users avoid preloading data in the Striim cache. This is especially beneficial for lookups from or joining with large data sets. External lookups also eliminate the need for a cache refresh since the data is fetched from the external database. The external lookups are supported for all major databases, including Oracle, SQL Server, MySQL, PostgreSQL, HPE NonStop.
• The Option to Use Sample Data for Continuous Queries: With this ability, Striim reduces the data required for computation or displaying results via the dashboards. You can select to use only a portion of your streaming data for the application, if your use case can benefit from this approach. As a result, it increases the speed for computation and displaying the results, especially when working with very large data volumes.
• Dynamic Output Names for Writers: The Striim platform makes it now easy to organize and consume the files and objects on the target system by giving flexible options for naming them. Striim file and object output names can include data, metadata, and user data field values from the source event. This dynamic output naming feature is available for the following targets: Azure Data Lake Store Gen 1 and Gen 2, Azure Blob Storage, Azure File Storage, Google Cloud Storage, Apache HDFS, Amazon S3.
• Event-Augmented Kafka Message Header: Starting with Apache Kafka v11, Striim 3.9.8 introduced a new property called MessageHeader that enriches the Kafka message header with a mix of the event’s dynamic and static values before delivering with sub-second latency. With the help of the additional contextual information, downstream consumer application can rapidly determine how to use the messages arriving via Striim.
• Simplified User Experience: The new UI for configuring complex adapter properties, such as rollover policy, flush policy, encryption policy, speeds new application development.

• New sample application for real-time dashboards: Striim version 3.9.8 added a new sample dashboarding application that uses real-time data from meetup-website and displays in details of the meet-up events happening around the globe using demonstrates the Vector Map visualization.

Other platform improvements for ease of use and manageability include:

• The Open Processor component, which allows bringing external code into the Striim platform, can be loaded and unloaded dynamically without having to restart Striim.
• The Striim REST API allows safely deleting or post-processing the files processed by the Striim File Reader.
• The Striim REST API for application monitoring reports consolidated statistics of various application components within a specified time range.

Increased Performance and Scalability:

For further improving performance and scalability, we have multiple features, including dynamic partitioning and performance fine-tuning for writers:

• Dynamic Partitioning with Higher-Level of Control: Partitions allow parallel processing of the events in the stream by splitting them across multiple servers in the deployment. Striim’s partitioning distributes events dynamically at run-time across server nodes in a cluster and enables high performance and easy scalability. In prior releases, Striim used one or more fields of the events in the stream as key for partitioning. In the new release, users have additional, flexible options for distributing and processing large data volumes in streams or windows. Striim 3.9.8 allows partitioning key to be one or more expressions composed with the fields of the events in the stream. Striim’s flexible partitioning enables load-balancing applications that are deployed on multi-node clusters and process large data volumes. Windows-based partitioning enables grouping the events in windows that can, for example, be consumed by specific downstream writers. As a result, you can perform load-balancing across multiple writers to improve writing performance.

• Writer Fine-Tuning Options: Striim 3.9.8 now offers the ability to configure the number of parallel threads for writing into the target system and simplifies writer configuration for achieving even higher throughput from the platform. The fine-tuning option is available for the following list of writers at this time: Azure Synapse Analytics and Azure SQL Data Warehouse, Google BigQuery, Google Cloud Spanner, Azure Cosmos DB, Apache HBase, Apache Kudu, MapR Database, Amazon Redshift, and Snowflake.

Increased Extensibility with New Data Targets

• The Striim platform now supports SAP Hana as a target with direct integration. SAP Hana customers can now stream real-time data from a diverse set of sources into the platform with in-flight, in-memory data processing. With the availability of real-time data pipelines to SAP Hana, deployed on-premises or in the cloud, customers can rapidly develop time-sensitive analytics applications that transform their business operations.
• Expanding the HTTP Reader capabilities to send custom responses back to the requestor. The HTTP Reader can now defer responding until events reach a corresponding HTTP Writer. This feature enables users to build REST services using Striim.

Other extensibility improvements are:

• Improved support for handling special characters for table names in Oracle and SQL Server databases
• Hazelcast Writer supports multi-column primary keys to enable more complex Hot Cache use cases
• Performance improvement options for the SQL Server CDC Reader

These are only a portion of the new features of Striim 3.9.8. There is more to discover. If you would like to learn more about the new release, please reach out to schedule a demo with a Striim expert.

We are proud to announce that Striim has received two 2019 best places to work awards in the Bay Area by three highly regarded local publications: the San Francisco Business Times, the Silicon Valley Business Journal, and the Bay Area News Group (publisher of The Mercury News in San Jose). This is the third year in a row that Striim was among the top companies on both lists.

This past week, Striim ranked #1 in the Small Companies category of the Bay Area News Group’s Top Workplaces award. This is the second time in three years that Striim has received the top ranking.

In late April, the San Francisco Business Times and the Silicon Valley Business Journal recognized Striim as the #7 best place to work in the Bay Area, up 3 spots from its #10 ranking in 2018.

Striim is honored to consistently rank among the top 10, and even more so to achieve Bay Area News Groups #1 spot. These rankings are a reflection of Striim’s ability to attract amazing employees in the Silicon Valley, and showcase the positive experience of the Striim team members currently working at the company.

What’s great is that both awards were 100% driven by employee feedback. Employees were asked a number of multiple choice and open-ended questions pertaining to a variety of workplace considerations: culture, pay, benefits, work-life balance, team collaboration, etc. Striim employees ranked the company extremely high in all categories.

Striim does not take these 2019 best places to work awards lightly. As a tech startup, it’s difficult to attract and retain top talent that Silicon Valley. Striim, like many other small companies in the Valley, needs to compete with big tech organizations and well-funded start-ups alike.

Along with its own unique perks and offerings, Striim offers a close-knit environment that promotes respect, hard work, and collaboration. Also, every day, employees are given the opportunity to work on emerging technology that is changing the way enterprise companies interact and handle its data.

It’s our belief that this combination is why Striim has done so well with these best places to work awards over the years.

If you’re interested in learning more about why Striim has been recognized as one of the top 2019 best places to work in the Bay Area, please read our San Francisco Business Times/Silicon Valley Business Journal and Bay Area News Group Top Workplaces press releases. And please check our Careers page if you think Striim might be a fit for you!

Now that you have a high-level overview of the Striim platform, let’s discuss how you can build data iPaaS applications with Striim.

You can deploy the entire platform in the cloud either by bringing your own license or as a metered iPaaS service. This gives you everything – it gives you all the sources, all the targets, and all the capabilities of the platform. There are also specific versions that you can deploy for particular solutions. So, for example, if you have on-premises Oracle databases and you want to push that data, as it’s changing, say to Azure SQL Data Warehouse, you can use that specific solution. You can still work with all of the sources, but you’re limited to delivering the data into Azure SQL Data Warehouse. There are dozens of specific cloud service solutions. They also are metered; they run as iPaaS in the cloud.

There are also a lot of different flavors of iPaaS. People usually bring up the multi-tenant type of iPaaS where the vendor hosts the service for you, allowing you to login and have access within an environment to be able to build data flows, etc. Striim chose not to go that route because customers are not typically that happy with the notion of being in a joint, multi-tenant environment where they are worried about data security and being guaranteed use of resources so that their applications will run at the right speed, etc.

Instead, Striim went with the ability to purchase the platform on Azure, Google Cloud, or Amazon as a metered service. With this approach, it’s running in your cloud environments, so you control the security, data, and everything else. Customers are more comfortable with this than the notion of a multi-tenant solution for iPaaS. As you can see in this video, we have metered iPaaS solutions for data in the marketplace for all three major cloud environments – Azure, AWS, and Google Cloud.

When you are working with the platform, on-premises or in the cloud, you interact with it through our intuitive web-based UI. This provides access to existing applications, as well as being able to import and create new applications.

You can start by building or importing applications, so, for example, if you’ve already built something in development, you can import it into production. If you are starting from scratch, you begin with an empty application and drag and drop components into the flow designer. But the easier way to get going is through the wizards which provide a large number of application templates. A lot of users start with a template because it enables you to rapidly build simple data flows, and check everything is correct as you go along.

For example, if you wanted to read from a MySQL database on-premises and deliver into Azure Cosmos DB, you could name the application, “MySQLtoCosmos,” and put it in a namespace. Namespaces keep things separate, and the way our security works, you can lock things down so that only certain people have access to certain namespaces. You can do much finer-grain things than that. You can give users access to the data that’s produced as the end result of the data pipeline, but not the raw data because that may have personally identifiable information in it. In our example, we will filter all that out before we push it into the cloud.

So you create a new namespace and save it. And then you can actually build data iPaaS applications, letting the wizards walk you through setting up the connection. Once all properties are configured, it will test everything to make sure that the connection is correct. This is an important step. One of the reasons Striim introduced its many wizards and templates was to make the development process as easy, intuitive, and fast as possible.

So in these steps, we check to make sure that not only does the connection to the database work, but also that connection has the right privileges, and that change data capture (CDC) is turned on. CDC collects all the inserts, updates, and deletes as they happen in a database (this is enabled at the database level). It also checks that you can get to the database metadata so you can actually see what tables and columns there are. If any of these steps don’t work, then the wizards will tell you what to do. Basically the instructions in the manual are mirrored by steps in the wizards so people know exactly what to do. In certain cases, the wizards can even do it for you. Once the connection is verified, you get to choose your data and go on to the next step. And then finally you’ll configure your target.

To learn more about how to build data iPaaS applications with Striim, read our Striim Platform Overview data sheet, set up a quick demo with a Striim technologist, or provision the Striim platform as an iPaaS solution on Microsoft Azure, Google Cloud Platform, or Amazon Web Services.

If you missed it or would like to catch up on this iPaaS blog series, please read part 1, “The Striim Platform as a Data Integration Platform as a Service.”

 

We are excited to announce that Striim has been recognized as a “Best Workplace” on FORTUNE’s “2019 Best Workplaces in the Bay Area” list.

Striim was selected based on a survey that was created, launched, and evaluated by Great Place to Work, a global people analytics and consulting firm.

The rankings took into account more than 30,000 surveys by employees across the Bay Area, designed to evaluate more than 60 elements of an employee’s job and work environment, including trust in leadership, camaraderie in a team setting, and respect among colleagues. Employee perks and benefits were also factored into the rankings.

This Best Workplaces in the Bay Area recognition is very important to Striim because the rankings were completely driven by employee feedback that Great Place to Work collected and evaluated. Additionally, given the fierce competition of not only attracting, but also retaining the best talent in the Bay Area, having our employees thrive in a culture that the Striim team Striim works so hard to foster is extremely rewarding and indicative that we’re on the right track for employee satisfaction.

Striim scored high across the board in many categories including Justice (100 %), Camaraderie (98%), Integrity (96%), Credibility (96%), and Innovation (96%), just to name a few.

Additionally, according to the survey, the overall Striim employee experience was rated 96%. Other great indications that our employees noted include:

• “Managers avoid playing favorites.” – 100%
• “I can be myself around here.” – 100%
• “When you join the company, you re made to feel welcome.” – 100%
• “Management is approachable, easy to talk with.” – 98%
• “People here are given a lot of responsibility.” – 98%

Learn more about what employees had to say about Striim, as well as further information on the company, by reading the full Great Place to Work review.

To learn more about why Striim was included on FORTUNE’s Best Workplaces in the Bay Area, as well as to see the full list of winners, please read our press release, “Striim Named One of the 2019 Best Workplaces in the Bay Area by FORTUNE and Great Place to Work.”

F. Clark, SOC analyst, Striim

Domo arigato, Mr. Roboto,
Mata ah-oo hima de
Domo arigato, Mr. Roboto,
Himitsu wo shiri tai

Every once in a while an analyst is called upon to look at things that are unique, that pique the interest and make the day go by faster. (For other days, there is coffee.) As the web server admin approached my desk, I was just about to put on another pot of coffee. I decided to hold off.

Our admin came to me with a bit of a puzzle. Although there are a myriad of well-known robots spidering our website daily, there was one that he was not happy with. It was called Yandex, a Russian search engine. Given the amount of malware and other less-than-wanted things coming out of Russian networks, the admin was concerned about this indexer accessing our website. With a grin a mile wide, I set aside the coffee and reached for my green tea with honey, and responded with a heartfelt, “I’m on it.”

FIRST STEPS

The first thing I needed to do was to get our web logs into a place where I could use Striim to analyze them. At my request, the admins had implemented SYSLOG-NG and were using a central repository for all of our logs which made it far easier to access them using Striim. Our primary and backup web servers in production resided in /var/log/www-prod-1 and /var/log/www-prod-2 on the central logging system. From there all I had to do was get them into Striim and we could start having fun. From the UI I whipped up a pair of text readers and configured them to take data from the access logs from both production servers.

The next step was to parse the log files so that the information from the log files was organized into fields that could be processed. A little wave of the REGEX wand and we had both logs parsed, and combined into a single flow for Striim to analyze.

From there, I next created a dashboard to show me just the information related to the web requests from Yandex. This would give me a clean and up-to-date view of the data I needed in real time. A quick TQL query combined with a table and I was on my way!

 

Immediately the information started flowing in. At first, it looked like just normal traffic one would expect from a indexing spider bot. Sure enough, however, my keen eyes spotted something that was not quite right.

As Dorothy Parker would say, “What fresh hell is this?” The spider was making a GET request of the search function of our website! This is not normal behavior for a spider if all it was doing was indexing our site. A little analyst magic performed on that request revealed it was using our own site search feature to look for “Fun HB Slot Machine” and the domain qpyl18.com. A quick check of this domain showed that it was [protected by Cloufflare ( Hi Otto! )], but that the origin server was having issues. A quick check of the IP addresses involved and my spidey-analyst senses were tingling.

The next burning question was how often was this happening, and at what volume? Back to the dashboard I went! I altered my query to show me the requests that were performing the internal searches, and was quickly rewarded with the information I was looking for:

 

Not only was this happening, but it was happening frequently.

 

I configured Striim to keep a watch on this, making it part of my overall security application, and using customized queries to create indicators of how many instances in a day, average number of instances over a week, and a special dashboard page with alerts to let me know if it got out-of-hand.

Like any good analyst, I gathered data for 60 days and then presented it to the web admin, and we both decided this was not something we wanted on our network. A few adjustments to the web server, firewall, and IDS, and we were off for a celebratory lunch.

The ease of use, speed, and myriad of tools along with the flexibility of Striim allowed me and the web admin to quickly and efficiently acquire, process, enrich and report the data on the unusual traffic, and create an environment where any of the shift analysts could keep an eye on the activity, both streaming in real time and stored for historical purposes.

So you want to empower your analysts with tools like this? Request a demo today. We will be happy to guide you through all of the features of Striim and help you improve your security footprint.

Today, we are thrilled to announce the availability of Hazelcast Striim Hot Cache. This joint solution with Hazelcast’s in-memory data grid uses Striim’s Change Data Capture to solve the cache consistency problem.

With Hazelcast Striim Hot Cache, you can reduce the latency of propagation of data from your backend database into your Hazelcast cache to milliseconds. Now you have the flexibility to run multiple applications off a single database, keeping Hazelcast cache refreshes up-to-date while adhering to low latency SLAs.

 

Check out this 5-minute Introduction and Demo of Hazelcast Striim Hot Cache:

https://www.youtube.com/watch?v=B1PYcIQmya4

 

Imagine that you have an application that works by retrieving and storing information in a database. To get faster response times, you utilize a Hazelcast in-memory cache for rapid access to data.

However, other applications also make database updates which leads to inconsistent data in the cache. When this happens, suddenly the application is showing out-of-date or invalid information.

Hazelcast Striim Hot Cache solves this by using streaming change data capture to synchronize the cache with the database in real time. This ensures that both the cache and associated application always have the most up-to-date data.

Through CDC, Striim is able to recognize which tables and key values have changed. Striim immediately captures these changes with their table and key, and, using the Hazelcast Striim writer, pushes those changes into the cache.

We make it easy to leverage Striim’s change data capture functionality by providing CDC Wizards. These Wizards help you quickly configure the capture of change data from enterprise databases – including Oracle, MS SQL Server, MySQL and HPE NonStop – and propagate that data to a Hazelcast cache.

You can also use Striim to facilitate the initial load of the cache.

To learn more, please read the full press release, visit the Hazelcast Striim Hot Cache product page, or jump right in and download a fully loaded evaluation copy of Striim for Hazelcast Hot Cache.

Striim is a proud sponsor at Big Data Innovation Summit SF, April 21-22 at the San Francisco Marriott Marquis Hotel. Join Striim’s Co-founder and CTO, Steve Wilkes, Thursday, April 21 at 9:30 a.m., as he discusses how to innovate your data strategy through streaming in his 30-minute presentation:

The Big One’s Coming!

Modernize Your Data Strategy Before It Hits

April 21 at 9:30 a.m.

San Francisco Marriott Marquis Hotel

in the Yerba Buena Ballroom

Security breaches and fraud. Increased customer churn. Halted manufacturing lines. Lost data. These are just some of the “data earthquakes” that signal a need to modernize your data strategy. But, unlike San Francisco’s imminent mega-quake, these outcomes can be prevented.

The key? Streaming integration. Join this session to learn why no innovative data strategy is complete without streaming integration, and how this can be applied to solve the critical use cases of today and tomorrow.

Exhibition Hours:

• Thursday, April 21, 8:30 a.m. – 5:30 p.m.
• Friday, April 22, 8:30 a.m. – 4:45 p.m.

We look forward to seeing you at Big Data Innovation Summit!