Learning Journey - getting started (101 content)
This chapter is to get you understanding what is Event-Driven architecture, what is Kafka, how to consider Messaging as a service as a foundation for event-driven solution, and getting started on IBM Event Streams and IBM MQ.
- Important concepts around Event-driven solution and Event-driven architecture
- Understand Kafka technology
- So what is Event Streams?
- Use Cases
- Why microservices are becoming event-driven and why we should care?
- Messaging as a service?
- Fit for purpose
- Event Streams environment sizing
- Getting Started With Event Streams
- Methodology: getting started on a good path
- Frequently asked questions
First to get an agreement on the terms and definitions used in all
this body of knowledge content, with a clear definitions for
The main up-to-date reference architecture presentation is in this section with all the component descriptions.
This architecture is built after a lot of customer engagements, projects, and review, and it is driven to support the main technical most common use cases we observed for EDA adoption so far.
You may want to read from the [Kafka] documentation](https://kafka.apache.org/documentation/#intro_nutshell) to understand the main concepts, but we have also summarized those into this chapter and if you like quick video, here is a seven minutes review of what Kafka is:
The major building blocks are summarized in this diagram:
To learn more about:
- Kafka Cluster see this 101 content
- For Producer introduction see this section
- For Consumer introduction see this section
- For Kafka Connector framework introduction
Event Streams is IBM packaging of Kafka with some enhanced features, and available as Managed Service or as part of Cloud Pak for integration. Here is a developer.ibm.com article for Event Streams on Cloud.
The product documentation as part of cloud pak for integration is here.
See the detailed tables in the product documentation.
A lot of information is available for sizing:
- Cloud Pak for integration system requirements
- Foundational service specifics
- Event Streams resource requirements
- Kafka has a sizing tool (eventsizer) that may be questionable but some people are using it.
But in general starts small and then increase the number of nodes over time.
Minimum for production is 5 brokers and 3 zookeepers.
We have different level of use cases in this repository:
The generic use cases which present the main drivers for EDA adoption are summarized here.
But architects need to search and identify a set of key requirements to be supported by thier own EDA:
- get visibility of the new data created by existing applications in near real-time to get better business insights and propose new business opportunities to the end-users / customers. This means adopting data streaming, and intelligent query solutions.
- integrate with existing transactional systems, and continue to do transaction processing while adopting eventual data consistency, characteristics of distributed systems. (Cloud-native serverless or microservices are distributed systems)
- move to asynchronous communication between serverless and microservices, so they become more responsive, resilient, and scalable (characteristics of ‘Reactive applications’).
- get loosely coupled but still understand the API contracts. API being synchronous (RES, SOAP) or async (queue and topic).
- get clear visibility of data governance, with cataloging applications to be able to continuously assess which apps securely access what, with what intent.
We have done some reference implementations to illustrate the major EDA design patterns:
|Shipping fresh food over sea (external)||The EDA solution implementation using event driven microservices in different language, and demonstrating different design patterns.||EDA reference implementation solution|
|Vaccine delivery at scale (external)||An EDA and cross cloud pak solution||Vaccine delivery at scale|
|Real time anomaly detection (external)||Develop and apply machine learning predictive model on event streams||Refrigerator container anomaly detection solution|
This article explains why microservices are becoming event-driven and relates to some design patterns and potential microservice code structure.
Yes EDA is not just Kafka, IBM MQ and Kafka should be part of any serious EDA deployment. The main reasons are explained in this fit for purpose section and in the messaging versus eventing presentation.
We have a general fit for purpose document that can help you reviewing messaging versus eventing, MQ versus Kafka, but also Kafka Streams versus Apache Flink.
The most easy way to start is to create one
Event Streams on IBM Cloud service instance,
and then connect a starter application. See this service at this URL.
As a developer, you may want to start quickly playing with a local Kafka Broker and MQ Broker
on you own laptop. We have developed such compose files in this gitops project under
# Kafka based on strimzi image and MQ containerdocker-compose -f maas-compose.yaml up &# Kafka onlydocker-compose -f kafka-compose.yaml up &
The kafka images are based on Strimzi images, but act as Event Streams one. So any code developed and configured on this image will run the same when deployed on OpenShift using Event Streams.
Finally if you have access to an OpenShift Cluster, version 4.7 +, you can deploy Event Streams as part of IBM Cloud Pak for Integration very easily using the OpenShift Admin console.
See this simple step by step tutorial here which covers
how to deployment and configuration an Event Streams cluster using the OpenShift Admin
Console or the
oc CLI and our manifest from our EDA gitops catalog repository.
Once you have deployed a cluster, access the administration console and use the very good getting started application.
If you want to adopt a pure GitOps approach we have demonstrated how to use ArgoCD (OpenShift GitOps) to deploy an Event Streams cluster and maintain it states in a simple real time inventory demonstration in this repository.
We have adopted the Event Storming workshop to discover the business process from an event point of view, and then apply domain-driven design practices to discover aggregate, business entities, commands, actors and bounded contexts. Bounded context helps to define microservice and business services scope. Below is a figure to illustrate the DDD elements used during those workshops:
Bounded context map define the solution. Using a lean startup approach, we focus on defining a minimum viable product, and preselect a set of services to implement.
A separate FAQ document groups the most common questions around Kafka.