Category: Blog

4 Major Components for Mission-Critical IMC Processing

This is the first blog in a six-part series on making In-Memory Computing Enterprise Grade. Read the entire series:

1. Part 1: overview
2. Part 2: data architecture
3. Part 3: scalability
4. Part 4: reliability
5. Part 5: security
6. Part 6: integration

If you are looking to create an end-to-end in-memory streaming platform that is used by Enterprises for mission critical applications, it is essential that the platform is Enterprise Grade. In a recent presentation at the In-Memory Computing Summit, I was asked to explain exactly what this means, and divulge the best practices to achieving an enterprise-grade, in-memory computing architecture based on what we have learned in building the Striim platform.

There are four major components to an enterprise-grade, in-memory computing platform: namely scalability, reliability, security and integration.

Scalability is not just about being able to add additional boxes, or spin up additional VMs in Amazon. It is about being able increase the overall throughput of a system to be able to deal with an expanded workload. This needs to take into account not just an increase in the amount of data being ingested, but also additional processing load (more queries on the same data) without slowing down the ingest. You also need to take into account scaling the volume of data you need to hold in-memory and any persistent storage you may need. All of this should happen as transparently as possible without impacting running data flows.

For mission-critical enterprise applications, Reliability is an absolutely requirement. In-memory processing and data-flows should never stop, and should guarantee processing of all data. In many cases, it is also imperative that results are generated once-and-only-once, even in the case of failure and recovery. If you are doing distributed in-memory processing, data will be partitioned over many nodes. If a single node fails, the system not only needs to pick up from where the failed node left off, it also needs to repartition over remaining nodes, recover state, and know what results have been written where.

Another key requirement is Security. The overall system needs an end-to-end authentication and authorization mechanism to protect data flow components and any external touch points. For example, a user who is able to see the end results of processing in a dashboard may not have the authority to query an initial data stream that contains personally identifiable information. Additionally any data in-flight should be encrypted. In-memory computing, and the Striim platform specifically, generally does not write intermediate data to disk, but does transmit data between nodes for scalability purposes. This inter-node data should be encrypted, especially over standard messaging frameworks such as Kafka that could easily be tapped into.

The final Enterprise Grade requirement is Integration. You can have the most amazing in-memory computing platform, but if it does not integrate with you existing IT infrastructure it is a barren data-less island. There are a number of different things to consider from an integration perspective. Most importantly, you need to get data in and out. You need to be able to harness existing sources, such as databases, log files, messaging systems and devices, in the form of streaming data, and write the results of processing to existing stores such as a data warehouse, data lake, cloud storage or messaging systems. You also need to consider any data you may need to load into memory from external systems for context or enrichment purposes, and existing code or algorithms you may have that may form part of your in-memory processing.

You can build an in-memory streaming platform without taking into account any of these requirements, but it would only be suitable for research or proof-of-concept purposes. If software is going to be used to run mission-critical enterprise data flows, it must address these criteria and follow best practices to play nicely with the rest of the enterprise.

Striim has been designed from the ground-up to be Enterprise Grade, and not only meets these requirements, but does so in an easy-to-use and robust fashion.

In subsequent blogs I will expand upon these ideas, and provide a framework for ensuring your streaming integration and analytics use cases make the grade.

Here at Striim, we have been living and breathing Big Data Streaming Analytics for four years now. We believe that no Enterprise Data Strategy is complete without Streaming Integration AND Streaming Analytics. In fact, we are successfully helping organizations of all sizes discover the benefits of leveraging streaming integration and intelligence (the two i’s of Striim) to deliver the real-time insights they need.

I therefore find it very encouraging that some of the world’s most respected analysts are also seeing value in this space. Recently Forrester Research published, “The Forrester Wave™: Big Data Streaming Analytics, Q1 2016.” 15 vendors were covered in this report, and it is encouraging to see how thought around this space has matured.

An example of this is the importance of Context. In the latest report, there are a dozen mentions of “context,” including in the subtitle of the report: “Streaming Analytics Are Critical To Building Contextual Insights For Internet-of-Things, Mobile, Web, and Enterprise Applications.”

We started Striim with Context as one of our most critical objectives, and the importance of Context cannot be over-emphasized. Most often the raw data feeds derived from enterprise databases via change data capture (CDC), log files, or IoT do not contain sufficient information to make decisions. In order to ready the data for querying, or to deliver relevant insights, it is almost always necessary to join the raw data with reference or historical information to add context. Striim has been architected from day one to perform this task without slowing down your data flow.

As a relative newcomer to the space, we were very pleased to be considered a Strong Performer in this report, and were impressed by the authors’ keen understanding of what we believe to be our top differentiators.

The only reference to Change Data Capture (CDC) in the entire report relates to Striim. In any streaming architecture, the most effective way to extract real-time information from enterprise applications is to capture the change in their underlying databases as it happens. Whether the application is an in-house CRM solution, Billing System, Point of Sale, or ATM Transactions Processor, the end result of the application is to update a database.

Most DBAs strictly forbid running SQL against a production database, so if you want to know what’s happening in these applications, without having to intrusively modify them, you need CDC. Striim is the only streaming analytics platform to provide CDC as a fully integrated component of the platform.

We believe that Streaming Integration is a pre-requisite for Streaming Analytics, and a platform isn’t complete without it. As such, we have ensured that we provide a great number of data collectors (including CDC and IoT) and targets (including Kafka and Cloud), and we made the internal processing of the data easy through our SQL-like language.

We found it extremely astute that the Forrester report cited Complex Event Processing (CEP) capabilities.. This is the ability to spot patterns of events over time across one or more streams; patterns that may indicate something important is happening. We believe that CEP won’t survive as a standalone technology, and is instead a key component of any streaming analytics platform.

There is one aspect of our product that wasn’t highlighted, and that is Streaming Visualization. Anyone who has tried it knows that it is extremely difficult to build dashboards and reports to truly analyze your streaming data in real time, unless that capability is integrated into the platform.

Striim’s real-time dashboards can be built easily using a drag-and-drop interface, and rapidly deliver insights into your analysis. You don’t even need full-blown analytics to use our visualizations. We have customers, for example, who are performing streaming integration from enterprise databases via CDC to Kafka, who simply want to monitor this integration and drill down into specifics through our dashboards.

If you are thinking about Big Data Streaming Analytics, it is important to consider the entire eco-system. The actual analysis part is, in fact, a small piece of the puzzle, and requires that you can first collect, process, enrich and correlate the data in a real-time fashion. Once you have analyzed it, you most likely also need to visualize and report on it, and send alerts for critical events. It’s hard to piece together multiple technologies to achieve this, or to focus all of your efforts on coding when you would rather empower your analysts. Instead, please consider a single end-to-end streaming analytics platform, like Striim, that enables all of this, and more.

Building complex, financial transaction monitoring applications used to be a time-consuming task. Once you had the business case worked out, you needed to work with a team of analysts, DBAs and engineers to design the system, source the data, build, test, and rollout the software. Typically it wouldn’t be correct the first time, so rinse and repeat.

Not so with Striim. In this video you will see a financial transaction monitoring application that was built and deployed in four days. The main use case is to spot increases in the rate at which customer transactions are declined, and alert on that. But a whole host of additional monitoring capabilities were also built into the application. Increasing decline rates often indicate issues with the underlying ATM and Point of Sale networks, and need to be resolved quickly to prevent potential penalties and decline in customer satisfaction.

The application consists of a real-time streaming dashboard, with multiple drill-downs, coupled with a continuous back-end dataflow that is performing the analytics, driving the dashboard and generating alerts. Streaming data is sourced in real time from a SQL Server database using Change Data Capture (CDC), and used to drive a number of analytics pipelines.

The processing logic is all implemented using in-memory continuous queries written in our easy to work with SQL-like language, and the entire application was built using our UI and dashboard builder. The initial CDC data collection goes through some initial data preparation, and is then fed into parallel processing flows. Each flow is analyzing the data in different ways, and storing the results of the processing in our built-in results store to facilitate deeper analysis later.

If you want to learn how to build complex monitoring and analytics applications quickly, take 6 minutes to watch this video.

The time for speculation is long gone. Predictions about the likely future challenges facing IT have become a reality, at least in one aspect. Simply put, we are facing a wall of data coming, rushing towards us and there are few places to hide. The pundits tell us that wisely used it is this data that will separate winners in business from losers. But is it possible to extol the virtue of water to a drowning man?

In Sydney’s famed Opera House, spread across a long curved wall in the vestibule at the rear of the concert hall, facing an amazing view of the harbor is a mural called Five Bells. However, to the locals it’s simply called the last thoughts of a drowning man as it drew inspiration from a poem of many years before by another artist that fell into the harbor and drowned. A nearby war ship evidently sounded five bells at the time the artist was drowning – hence the mural’s official name.

This past week I was driving to the ATMIA US Conference in New Orleans when the skies east of the city opened unleashing a deluge of water, I have not experienced anything like that in many years. Visibility evaporated and traffic was reduced to a crawl and it was a stark reminder of just how great the volume of data heading towards us has become. And the land quickly became water-logged!

Take just a few industries where the increase volume of data is becoming very apparent. This past holiday season ecommerce really took it to bricks and mortar retailers and from the data deluge there’s rising water where great waves are forming that show no signs of breaking any time soon. To the contrary, it is continuing to climb in height and the shoreline is nowhere to be seen. In the January 20, 2016, article Why brick-and-mortar retail faces a shakeout to the publication, Retail Dive, came the hews that, “‘We are right now in the middle of the biggest, most profound transformation in the history of retail,’ Robin Lewis, CEO of the Robin Report and a former executive at VF Corp. and Women’s Wear Daily.”

Furthermore, according to Retail Dive, “‘We’ve now gone to a business where your best customer can be standing in your best store and with three touches of their thumb to a piece of glass, they can buy from your biggest competitor,’ Fred Argir, Chief Digital Officer for Barnes & Noble, told Retail Dive in an interview. ‘That’s changed everything.’” To be fair, the publication also quoted Steve Barr, a partner and the US Retail and Consumer Sector leader at PricewaterhouseCoopers. “The great news is the retail store is not dead,” Barr said. “But the retail store that does not have a meaningful relationship with the consumer is dead.”

Meaningful relationships? Yes, a much better understanding of the behavior of consumers even as we determine the trends these consumers will likely embrace. This means tapping into data as its being generated and picking up the information gems we need as they pass by. And one sure data source clearly is the byproduct of the transactions customers initiate particularly should it involve your competition. I admit, I did hit “Purchase” button on amazon.com while looking at a book at Barnes & Noble, not only because the price was a tad lower but, more importantly, because the site provided the book’s reviews and more information!

When it comes to thumbing a piece of glass, the change under way from simply having a device from which to make a phone call to where communication is visual and global, is sapping resources from all over the digital footprint surrounding each and every smartphone. But the usage patterns that can be derived not only helps mobile phone operators to better tailor messages to individual subscribers but allows them to accumulate metadata on an unprecedented scale and it’s becoming extremely valuable to not just retailers or even bankers, but to industry as a whole.

Whether it’s the purchase of a new car or just an old pair of shoes, ensuring the right product is presented to a consumer motivated to buy it is paramount, but even so, making sense of so much data coming from each and every one of us makes the data deluge all the more difficult to process. Clearly, the former model of capture, store and process antiquated when the challenge lies in integration with the real time world of transaction processing.

With so much being written about the Internet of Things (IoT) and even the Internet of Everything (IoE) there is recognition that as more and more sensors come online the data deluge is going to become even greater. However, much closer to home for all of us – the human body – as we are beginning to hear about, is one giant collection of almost infinite sensors. Imagine the plight of the medical profession, including researchers of every discipline, as steps are taken to mandate real time processing of everything our body generates from its vast array of sensors?

The new model involving capture, process and store offers the only real way to make sense of it all. This is not to say that there remains little of value for data scientists to exploit when data is finally stored but rather, by industry and markets, data by necessity needs to be parsed with only pertinent data ever making it to storage. For the HPE NonStop community this is of growing importance, as NonStop systems are home to many of the most important real time transaction processing business applications on the planet.

Striim is beginning to gain a small foothold within the NonStop community, with its first deployments taking hold. And this is good news for everyone in the NonStop community as even NonStop users are finding it necessary to fight back to keep consumers engaged with their products and services. The data deluge is upon us and the waters are rising fast – bigger and bigger waves are breaking over companies old and new.

We may be in the biggest and most profound transformation in history and it’s not going away and it’s not a time to simply redirect the waters to lakes hidden from view. To survive we need to know our customers and we need to be sensitive to their ever changing behavior and doing so, even as they instigate a transaction, is critical. Drowning is simply not a viable business option!

“…Canny with its design criteria. [Striim’s] technology platform doesn’t only offer streaming data integration, analytics or alerting: it offers all three.” That’s what Jason Stamper concludes about the Striim platform in his recent 451 Research Impact Report. In his 451 Take, he states:

“Since we expect streaming technologies to get a shot in the arm from the emergence of Internet of Things (IoT) use cases, we think Striim has been canny –­­­­­–­integration, analytics or alerting: it offers all three. Equally wisely, the company is incorporating open source tools such as Apache Kafka, Hive and ElasticSearch to save on development costs, and also allow companies to take advantage of existing infrastructure.”

Mr. Stamper points out how the company has rebranded from WebAction to Striim (pronounced “stream”) because the new name better reflects what the company is trying to do: ‘streaming integration and intelligence.’ (The two ‘i’s in Striim stand for integration and intelligence.)

The report proceeds to detail the unique features of Striim’s end-to-end, real-time data integration and intelligence platform. Everything from real-time, high velocity data collection the instant data is born, to real-time enrichment, correlation and analysis on streaming data, to real-time visualizations and alerts.

There are other vendors in the streaming analytics space trying to assemble an end-to-end platform with open source technologies. However, 451 Research agrees that this approach is ultimately time-consuming and costly. It simply takes too many high-priced developers to wire all of the various technologies together, and in the end, there’s no guarantee it will work at an enterprise scale.

Download the complete 451 Research Impact Report.

The Internet of Things Means You Now Have Less Time to ID Threats

In a July 12 Cyber-security brief “The internet of things (to be hacked)” The Economist discussed the coming explosion of connected devices sharing data in what has commonly become called the Internet of Things (IoT), or as some are now calling the Internet of Everything (IoE). Either term gets you to a place where 18 months from now you have way too much data coming at you to store it all now to process “later”. “Later” will never come and every one of those devices introduces a new potential security threat to your enterprise. The Economist notes:

“There have already been instances of nefarious types taking control of webcams, televisions and even a fridge, which was roped into a network of computers pumping out e-mail spam.”

In this hyper-connected world you need to be continuously monitoring all of devices and traffic on your networks to note interesting correlated events across your infrastructure. WebAction Security Event Processing Data Driven Apps give you a unique insight across your data streams.

Wireless Networks Will Become Saturated

By the nature of IoT devices, wireless is the preferred method of communication. As the number of devices grows, so does the wireless chatter and noise. The chatter is building over all wireless communication channels: WiFi, cellular, Bluetooth, near field communications (NFC), and others. There is no end in sight to the expected growth of wireless connected devices. Networks will need to be fortified and new methods of managing wireless traffic are being considered. Enriching wireless network traffic with rich context and history allows for dynamic network traffic prioritization based on the profiles of your customers. Always make sure that your most important customers always get the best Quality of Service, and know immediately when quality degrades.

Insights Available in Your IoT Streams

The Economist fears that with the loose regulation of connected devices we will see more incidents of hackers working their way into your refrigerator and thermostat. The brief concludes with: “Who needs a smart fridge anyway?” I suppose that is an interesting question, but rather than resisting progress and change (which we know doesn’t work in the long-run), we suggest finding novel ways to immediately identify and neutralize security threats arising from the Internet of Things.

All of those connect devices are reporting their streams back to home base, and home base needs to make some snappy decisions about what to do with the data streams flowing in. At the same time every  stream rides on your wireless networks and contains potential threats and useful data signatures. That’s where the WebAction Real-time App Platform shines, monitoring streams to identify patterns in-memory enabling immediate (and informed) action downstream. On the fly your real-time data is correlated across streams, filtered, and enriched with history and context to create highly actionable Big Data Records.

The Internet of Things comes with some very significant blue sky ahead of it and the WebAction Real-time App Platform enables you to take advantage of this new frontier. Request a demo of the WebAction Platform.