Event Streams on Cloud hands on lab
This documentation aims to be a introductory hands-on lab on IBM Event Streams on Cloud with topic creation.
- Create a Event Streams service instance
- Creating Event Streams instance with IBM Cloud CLI
- Create topic
- Create topic with CLI
This lab requires the following components to work against:
- An IBM Cloud account. Get a IBM Cloud Account by using the register link in https://cloud.ibm.com/login Create a new account is free of charge.
On your development workstation you will need:
- IBM Cloud CLI (https://cloud.ibm.com/docs/cli?topic=cloud-cli-getting-started)
- IBM CLoud CLI Event Streams plugin (
ibmcloud plugin install event-streams)
As any other IBM Cloud services, Event Streams can be part of a resources group, is controlled by user roles, and is accessible via API keys. To get familiar with those concepts, it is recommended to study the concepts of IBM account and how it is related to resource group and services. The following diagram is a summary of the objects managed in IBM Cloud:
- Account represents the billable entity, and can have multiple users.
- Users are given access to resource groups.
- Applications are identified with a service ID.
- To restrict permissions for using specific services, you can assign specific access policies to the service ID and user ID
- Resource groups are here to organize any type of resources (services, clusters, VMs…) that are managed by Identity and Access Management (IAM).
- Resource groups are not scoped by location
- Access group are used to organize a set of users and service IDs into a single entity and easily assign permissions
From the IBM Cloud Dashboard page, you can create a new resource, using the right top button
which leads to the service and feature catalog. From there in the
services view, select the
category and then the
Event Streams tile:
You can access this screen from this URL: https://cloud.ibm.com/catalog/event-streams.
Within the first page for the Event Streams creation, you need to select the region, the pricing plan, a service name and the resource group.
For the region, it is important to note that the ‘lite’ plan is available only in Dallas, and it used to do some proof of concept. It is recommended to select a region close to your on-premise data center. For the Plan description, the product documentation goes over the different plans in details.
The ‘multi-tenancy’ means the Kafka cluster is shared with other people. The cluster topology is covering multi availability zones inside the same data center. The following diagram illustrates a simple view of this topology with the different network zones and availability zones:
We will address fine-grained access control in the security lab.
As described in the Kafka concept introduction, topic may have partitions. Partitions are used to improve throughput as consumer can run in parallel, and producer can publish to multiple partitions.
The plan set a limit on the total number of partitions.
Each partition records, are persisted in the file system and so the maximum time records are kept on disks is controlled by the maximum retention period and total size. Those Kafka configurations are described in the topic and broker documentation.
Fro the Standard plan, the first page has also a price estimator. The two important concepts used for pricing are the number of partition instances and the number of GB consumed: each consumer reading from a topic/partition will increase the number of byte consumed. The cost is per month.
Go to IBM Cloud and click on the user avatar on the top right corner. Then, click on Log in to CLI and API option:
IBM Cloud CLIlogin command
Open a terminal window, paste and execute the command:$ ibmcloud login -a https://cloud.ibm.com -u passcode -p XsgEKGb84ZAPI endpoint: https://cloud.ibm.comAuthenticating...OKTargeted account bill s Account (b63...) <-> 195...Select a region (or press enter to skip):1. au-syd2. in-che
List your services with
ibmcloud resource service-instancesand make sure your IBM Event Streams instance is listed:$ ibmcloud resource service-instancesRetrieving instances with type service_instance in all resource groups in all locations under account Kedar Kulkarni's Account as ALMARAZJ@ie.ibm.com...OKName Location State TypeIBM Cloud Monitoring with Sysdig-rgd us-south active service_instanceapikey for simple toolchain us-east active service_instanceaapoc-event-streams us-south active service_instanceEvent Streams-wn eu-de active service_instance
We can see our instance called: Event Streams-wn
Create an Event Streams instance using CLIibmcloud resource service-instance-create EventStreamsEDA2 messagehub standard us-south
List your IBM Event Streams instance details with
ibmcloud resource service-instance <instance_name>:$ ibmcloud resource service-instance Event\ Streams-wnRetrieving service instance Event Streams-wn in all resource groups under account Kedar Kulkarni's Account as ALMARAZJ@ie.ibm.com...OKName: Event Streams-wnID: crn:v1:bluemix:public:messagehub:eu-de:a/b636d1d8...cfa:b05be9...2e687a::GUID: b05be932...e687aLocation: eu-deService Name: messagehub
\character in your IBM Event Streams instance.
Initialize your IBM Event Streams plugin for the IBM Cloud CLI with
ibmcloud es init:$ ibmcloud es initSelect an Event Streams instance:1. Event Streams-2t2. Event Streams-wn3. aapoc-event-streams4. tutorialEnter a number> 2API Endpoint: https://mh-tcqsppdpzlrkdmkb.....175-0000.eu-de.containers.appdomain.cloud
Check all the CLI commands available to you to manage and interact with your IBM Event Streams instance with
$ ibmcloud es:$ ibmcloud esNAME:ibmcloud es - Plugin for IBM Event Streams (build 1908221834)USAGE:ibmcloud es command [arguments...] [command options]COMMANDS:broker Display details of a broker.
List your cluster configuration with
$ ibmcloud es cluster:$ ibmcloud es clusterDetails for clusterCluster ID Controllermh-tcqsppdpzlrkdmkbgmgl-4c20...361c6f175-0000 0Details for brokersID Host Port Rack0 kafka-0.mh-tcqsppdpzlrkdmkbgmgl-4c201a12d......22e361c6f175-0000.eu-de.containers.appdomain.cloud 9093 fra051 kafka-1.mh-tcqsppdpzlrkdmkbgmgl-4c201a12d......22e361c6f175-0000.eu-de.containers.appdomain.cloud 9093 fra02
Looking at broker details:
ibmcloud es broker 0:ibmcloud es broker 0Details for brokerID Host Port Rack0 broker-0-t19zgvnykgdqy1zl.kafka.svc02.us-south.eventstreams.cloud.ibm.com 9093 dal10Details for broker configurationName Value Sensitive?broker.id 0 falsebroker.rack dal10 false
Get detail view of a broker configuration:
ibmcloud es broker-config 0
We will see other CLI commands in future labs.
When coming back to the IBM Cloud dashboard the simplest way to find the Event Streams service is to go
- Click on your IBM Event Streams instance:
- Click on Launch Dashboard button to open the IBM Event Streams dashboard
Once the instance is created, or when you come back to the service, you reach the
managepanel, as illustrated in previous figure.
From the Dashboard we can access the Topics and Consumer groups panels.
In this section we are going to see how to create, list and delete topics both using the User Interface and then the IBM Event Streams CLI.
Open the IBM Event Streams user interface (go into your IBM Event Streams service within your IBM Cloud portal and click on the launch dashboard button). Once there, click on the Topics tab from the top menu:
Let create a
demo-topic-ui topic. If you need to revisit the topic concepts, you can read this note. When you go to the topics view you get the list of existing topics.
From this list an administrator can delete an existing topic or create new one.
- The ‘create topic’ button leads to the step by step process.
- Switch to the Advanced mode to get access to the complete set of parameters. The first panel is here to define the core configuration
Some parameters to understand:
- Number of partitions: the default value should be 1. If the data can be partitioned without loosing semantic, you can increase the number of partitions.
- Retention time: This is how long messages are retained before they are deleted to free up space. If your messages are not read by a consumer within this time, they will be missed. It is mapped to the retention.ms kafka topic configuration.
The bottom part of the configuration page, includes logs, cleanup and indexing.
The partition’s log parameter section includes a cleanup policy that could be:
- delete: discard old segments when their retention time or size limit has been reached
- compact: retain at least the last known value for each message key within the log of data for a single topic partition. The topic looks like a table in DB.
- compact, delete: compact the log and remove old records
retention bytes: represents the maximum size a partition (which consists of log segments) can grow to, before old log segments will be discarded to free up space.
log segment size is the maximum size in bytes of a single log file.
Cleanup segment time - segment.ms controls the period of time after which Kafka will force the log to roll, even if the segment file isn’t full, this is to ensure that retention can delete or compact old data.
Index - segment.index.bytescontrols the size of the index that maps offsets to file positions.
The log cleaner policy is supported by a log cleaner, which are threads that recopy log segment files, removing records whose key appears in the head of the log.
The number of replications is set to three with a
min-in-sync replicas of two.
We can now see our new topic:
To delete a topic, click on the topic options button at the right end of a topic, click on Delete this topic and then on the Delete button in the confirmation pop-up window:
The topic should now be deleted:
List your topics with
$ ibmcloud es topics:$ ibmcloud es topicsOKNo topics found.
Create a topic: (Default 1 partition - 3 replicas)$ ibmcloud es topic-create --name demo-topicCreated topic demo-topicOK
$ ibmcloud es topic-create --helpfor more further configuration of your topic creation
List topics:$ ibmcloud es topicsTopic namedemo-topicOK
Display details of a topic:$ ibmcloud es topic demo-topicDetails for topic demo-topicTopic name Internal? Partition count Replication factordemo-topic false 1 3Partition details for topic demo-topicPartition ID Leader Replicas In-sync0 2 [2 1 0] [2 1 0]
Delete records in a topic (in the command below, we want to delete record on a partition 0 offset 5 and partition 1 from offset 0):$ ibmcloud es topic-delete-records --name demo-topic --partition-offset 1:0;0:5 --force
Add partitions to an existing topic, by setting the new target number of partition:$ ibmcloud es topic-partition-set --name demo-topic --partitions 30
Delete a topic:$ ibmcloud es topic-delete demo-topicReally delete topic 'demo-topic'? [y/N]> yTopic demo-topic deleted successfullyOK
List topics:$ ibmcloud es topicsOKNo topics found.
For the last list of commands see the CLI Reference manual.
From the manage dashboard we can download a getting started application that has two processes: one consumer and one producer, or we can use a second application that we have in this repository
To be able to build the code you need to get gradle installed or use the docker image:
docker run --rm -u gradle -v "$PWD":/home/gradle/project -w /home/gradle/project gradle gradle <gradle-task>
The instructions are in this documentation and can be summarized as:
- Clone the github repository:
git clone https://github.com/ibm-messaging/event-streams-samples.git
- Build the code:
Using the gradle CLI
cd kafka-java-console-samplegradle clean && gradle build
or the gradle docker image
docker run --rm -u gradle -v "$PWD":/home/gradle/project -w /home/gradle/project gradle gradle build
- Start consumer
java -jar ./build/libs/kafka-java-console-sample-2.0.jar broker-0-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-1-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-2-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093 am_rbb9e794mMwhE-KGPYo0hhW3h91e28OhT8IlruFe5 -consumer
- Start producer
java -jar ./build/libs/kafka-java-console-sample-2.0.jar broker-0-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-1-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-2-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093 am_rbb9e794mMwhE-KGPYo0hhW3h91e28OhT8IlruFe5 -producer