Why modern businesses are based on data streaming
Data is moved. Almost all data sources have an element of dynamism and movement. Even data at rest in some form of archival storage tier has previously lived a more fluid life as it moved between applications, devices and network backbones, and inevitably has also been moved to its resting place via some transport mechanism.
But while almost all data moves, not all data moves at the same speed, at the same cadence, and with the same type of system circularity, size, or value.
Continuous data in streams
In the always-on world of cloud computing, the ubiquity of mobile devices, and with the new population of intelligent “edge” machines in the Internet of Things (IoT), there is a continuous flow of data that we naturally refer to as a stream.
So what is data streaming and how should we understand and work with it?
Data streaming is a computing principle and operational system reality in which (usually small) data items move through an IT system in a time-ordered sequence. The smaller pieces of data are often referred to in the same breath as IT “events” (everything from a user pressing a button on a mouse or a keystroke on the keyboard… and on to asynchronous changes that occur when applications run code and their perform work). This form of data streams can consist of log files (small records that track every step of behavior of applications and services), financial transaction logs, web browser activity records, IoT smart machine sensor readings, in-game geospatial or telemetry information, video game movement and action information…and everything down to the tiniest device instrumentation record, which like everything else here creates a droplet of data that is part of a continuous flow that ultimately forms a stream.
A company operating with a data-driven, data-centric, and data-derived approach can take practical steps to analyze its data-streaming pipelines in real-time to get a detailed and accurate view of what’s happening in the business. By using a data streaming platform to perform sequentially processed analysis of each data set in the stream, an organization can sample, filter, correlate and aggregate its data streaming pipeline and begin to create a new layer of business insight and control.
Real-time data streaming apps
A business starting out simply may choose to build simple, real-time, streaming alert applications that label minimum and maximum values to trigger alarms and alerts when selected metrics fall below or exceed predefined thresholds. Moving forward, the same company might then consider applying machine learning (ML) algorithms to its data-streaming pipeline to look for deeper trends that might surface over the long-term (and eventually near-term, too).
If data streaming is for (pretty smart) dummies and tech savvy business tech folks then please feel comfortable taking this basic presentation and explanation i.e. this is a technology that is currently being applied to applications in every conceivable industry .
Data streaming player and open source
The shape of the data streaming market is typical of the enterprise cloud space in general. There are offerings from all the major hyperscalers from cloud service providers (AWS, Google Cloud Platform, and Microsoft Azure), IBM has its finger in the pie, and there’s a group of IT vendors traditionally known for their enterprise data management and integration platforms (Tibco is a good example) who also enjoy Share of Voice.
Then there’s open source, which in this case revolves around Apache Kafka, an open source data stream processing platform written in the Java and Scala languages. Confluent supports enterprise-level use of Kafka.
confluent is a comprehensive data streaming platform that enables users to access, store and manage data as continuous real-time streams. Developed by the original developers of Apache Kafka, Confluent extends the benefits of Kafka with enterprise-class features while removing the burden of Kafka administration or monitoring. Originally created by software engineers working at LinkedIn in 2011, today Kafka has evolved from a “simple” messaging queue to a technology that functions as a full data streaming platform. It can handle over 1 million messages per second, or trillions of messages per day.
Confluent provides enterprises with cloud-native, simple, and scalable data streaming pipelines, providing more than 120 pre-built connectors for real-time integration between source and target systems, in-flight processing of data streams, and a range of security, governance, and Resiliency Features Designed to meet enterprise regulations governing use cases across distributed mission-critical workloads.
According to the company, Confluent also enables customers to go beyond pure real-time integration between data systems and support real-time stream processing and analysis to support real-time decisions and modern applications. The company’s technology offering revolves around the idea that companies love Apache Kafka but may hate managing it. As a result, Confluent offers a cloud-native, fully managed service that promises to go beyond Kafka.
What this transcending factor means operationally is data streaming without the need to perform tasks such as cluster sizing (to determine the size of the data backbone required for a specific task in the data development lifecycle) and the ability to over-provision the cloud to avoid taking over data systems (purchasing and paying for more computing power, analysis and storage than needed) before the data streaming is brought online. The company also offers failover design, infrastructure management, security, data governance and global availability.
Removing the underlying mechanics
By positioning its core technology offering as a way to extract value from business data without incurring the administrative burden associated with the “underlying mechanisms” (such as how data is transported or integrated between various disparate systems), the company says, that it simplifies connecting data sources with Kafka, which is clearly a key factor in creating streaming applications. In addition, this connectivity factor helps to secure, monitor and manage a Kafka infrastructure.
The organization’s core product pages state, “Today, Confluent is used for a variety of use cases across numerous industries, from financial services, omni-channel retail, and autonomous cars to fraud detection, microservices, and IoT.”
The process here is to integrate both historical and real-time data into the platform, all of which (Confluent claims) creates a new category of software applications, i.e. data-driven, capable of accessing a single source of data truth within the enterprise a universal data pipeline.
For its 2022 State of Data in Motion market analysis, Confluent surveyed around 1,950 IT and engineering executives in six countries. One user eloquently expressed the trend movement in software engineering. “Real-time data streams are becoming a core part of our customer service and business,” said Yaël Gomez, vice president, global IT, integration and intelligent automation, Walgreens Boots Alliance.
Gomez explained that his department (and company) has used data on the move to manage customer engagement, ensure patient accessibility of vaccines and tests, while working to deliver a differentiated online retail offering in what he calls seamless omni- enable channel experience.
Modern business and data streaming
Hopefully, if this picture of data movement says anything, it conveys just how much data dynamics have changed over the past quarter century.
As Jay Kreps, Confluent co-founder and CEO, explained, we used to live in a business and data world that was very batch-centric. Businesses would close up shop for the day, week, or month and take stock of where their product inventories were, how staff were working, and maybe take a look at what customer sentiment might be like. Operational adjustments by the company might come quarterly, sometimes.
This business world no longer exists. In the age of cloud, web, mobile ubiquity, and the wider world of connected systems, businesses must enable the continuous movement and processing of data for better workflows, increased automation, real-time analytics, and differentiated digital customer experiences.
The term “modern” is already being overused in tech circles, with vendors claiming to have modern programming tools (low-code), modern databases (with intelligent big data analytics), modern automation systems (with massive AI power), and everything else you offer, offer today up to modern user interfaces (capable of offering the same experience from desktop to tablet to smartphone or even kiosk and beyond).
But for all the modernization babble, the only thing that could truly become modern is the business itself. A modern enterprise runs hundreds of applications, cloud layers, and services… all of which have thousands (if not hundreds of thousands) of user and machine endpoints in countless workflows serve.
The modern enterprise will now seek to leverage event-driven applications with real-time data, and that means data streaming. Don’t forget to paddle.