Skip to main contentIBM Automation Event-Driven Solution - Sharing knowledge

Event Streams on Cloud hands on lab

This documentation aims to be a introductory hands-on lab on IBM Event Streams on Cloud with topic creation.

Index

Pre-requisites

This lab requires the following components to work against:

  1. An IBM Cloud account. Get a IBM Cloud Account by using the register link in https://cloud.ibm.com/login Create a new account is free of charge.

On your development workstation you will need:

  1. IBM Cloud CLI (https://cloud.ibm.com/docs/cli?topic=cloud-cli-getting-started)
  2. IBM CLoud CLI Event Streams plugin (ibmcloud plugin install event-streams)

Login CLI

ibmcoud login

Account and resource group concepts

As any other IBM Cloud services, Event Streams can be part of a resources group, is controlled by user roles, and is accessible via API keys. To get familiar with those concepts, it is recommended to study the concepts of IBM account and how it is related to resource group and services. The following diagram is a summary of the objects managed in IBM Cloud:

To summarize:

  • Account represents the billable entity, and can have multiple users.
  • Users are given access to resource groups.
  • Applications are identified with a service ID.
  • To restrict permissions for using specific services, you can assign specific access policies to the service ID and user ID
  • Resource groups are here to organize any type of resources (services, clusters, VMs…) that are managed by Identity and Access Management (IAM).
  • Resource groups are not scoped by location
  • Access group are used to organize a set of users and service IDs into a single entity and easily assign permissions

Create a Event Streams service instance

From the IBM Cloud Dashboard page, you can create a new resource, using the right top button Create resource.

D

which leads to the service and feature catalog. From there in the services view, select the integration category and then the Event Streams tile:

C

You can access this screen from this URL: https://cloud.ibm.com/catalog/event-streams.

Plan characteristics

Within the first page for the Event Streams creation, you need to select the region, the pricing plan, a service name and the resource group.

C2

For the region, it is important to note that the ‘lite’ plan is available only in Dallas, and it used to do some proof of concept. It is recommended to select a region close to your on-premise data center. For the Plan description, the product documentation goes over the different plans in details.

The ‘multi-tenancy’ means the Kafka cluster is shared with other people. The cluster topology is covering multi availability zones inside the same data center. The following diagram illustrates a simple view of this topology with the different network zones and availability zones:

T

We will address fine-grained access control in the security lab.

As described in the Kafka concept introduction, topic may have partitions. Partitions are used to improve throughput as consumer can run in parallel, and producer can publish to multiple partitions.

The plan set a limit on the total number of partitions.

Each partition records, are persisted in the file system and so the maximum time records are kept on disks is controlled by the maximum retention period and total size. Those Kafka configurations are described in the topic and broker documentation.

Fro the Standard plan, the first page has also a price estimator. The two important concepts used for pricing are the number of partition instances and the number of GB consumed: each consumer reading from a topic/partition will increase the number of byte consumed. The cost is per month.

Creating Event Streams instance with IBM Cloud CLI

  1. Go to IBM Cloud and click on the user avatar on the top right corner. Then, click on Log in to CLI and API option:

    1
  2. Copy the IBM Cloud CLI login command

    2
  3. Open a terminal window, paste and execute the command:

    $ ibmcloud login -a https://cloud.ibm.com -u passcode -p XsgEKGb84Z
    API endpoint: https://cloud.ibm.com
    Authenticating...
    OK
    Targeted account bill s Account (b63...) <-> 195...
    Select a region (or press enter to skip):
    1. au-syd
    2. in-che
  4. List your services with ibmcloud resource service-instances and make sure your IBM Event Streams instance is listed:

    $ ibmcloud resource service-instances
    Retrieving instances with type service_instance in all resource groups in all locations under account Kedar Kulkarni's Account as ALMARAZJ@ie.ibm.com...
    OK
    Name Location State Type
    IBM Cloud Monitoring with Sysdig-rgd us-south active service_instance
    apikey for simple toolchain us-east active service_instance
    aapoc-event-streams us-south active service_instance
    Event Streams-wn eu-de active service_instance

    We can see our instance called: Event Streams-wn

  5. Create an Event Streams instance using CLI

    ibmcloud resource service-instance-create EventStreamsEDA2 messagehub standard us-south
  6. List your IBM Event Streams instance details with ibmcloud resource service-instance <instance_name>:

    $ ibmcloud resource service-instance Event\ Streams-wn
    Retrieving service instance Event Streams-wn in all resource groups under account Kedar Kulkarni's Account as ALMARAZJ@ie.ibm.com...
    OK
    Name: Event Streams-wn
    ID: crn:v1:bluemix:public:messagehub:eu-de:a/b636d1d8...cfa:b05be9...2e687a::
    GUID: b05be932...e687a
    Location: eu-de
    Service Name: messagehub

    Mind the \ character in your IBM Event Streams instance.

  7. Initialize your IBM Event Streams plugin for the IBM Cloud CLI with ibmcloud es init:

    $ ibmcloud es init
    Select an Event Streams instance:
    1. Event Streams-2t
    2. Event Streams-wn
    3. aapoc-event-streams
    4. tutorial
    Enter a number> 2
    API Endpoint: https://mh-tcqsppdpzlrkdmkb.....175-0000.eu-de.containers.appdomain.cloud
  8. Check all the CLI commands available to you to manage and interact with your IBM Event Streams instance with $ ibmcloud es:

    $ ibmcloud es
    NAME:
    ibmcloud es - Plugin for IBM Event Streams (build 1908221834)
    USAGE:
    ibmcloud es command [arguments...] [command options]
    COMMANDS:
    broker Display details of a broker.
  9. List your cluster configuration with $ ibmcloud es cluster:

    $ ibmcloud es cluster
    Details for cluster
    Cluster ID Controller
    mh-tcqsppdpzlrkdmkbgmgl-4c20...361c6f175-0000 0
    Details for brokers
    ID Host Port Rack
    0 kafka-0.mh-tcqsppdpzlrkdmkbgmgl-4c201a12d......22e361c6f175-0000.eu-de.containers.appdomain.cloud 9093 fra05
    1 kafka-1.mh-tcqsppdpzlrkdmkbgmgl-4c201a12d......22e361c6f175-0000.eu-de.containers.appdomain.cloud 9093 fra02
  10. Looking at broker details: ibmcloud es broker 0:

    ibmcloud es broker 0
    Details for broker
    ID Host Port Rack
    0 broker-0-t19zgvnykgdqy1zl.kafka.svc02.us-south.eventstreams.cloud.ibm.com 9093 dal10
    Details for broker configuration
    Name Value Sensitive?
    broker.id 0 false
    broker.rack dal10 false
  11. Get detail view of a broker configuration: ibmcloud es broker-config 0

We will see other CLI commands in future labs.

Coming back another time

When coming back to the IBM Cloud dashboard the simplest way to find the Event Streams service is to go to the Services:

3
  • Click on your IBM Event Streams instance:
4
  • Click on Launch Dashboard button to open the IBM Event Streams dashboard
5

Main Event Streams Dashboard page

Once the instance is created, or when you come back to the service, you reach the managepanel, as illustrated in previous figure.

From the Dashboard we can access the Topics and Consumer groups panels.

Create topic

In this section we are going to see how to create, list and delete topics both using the User Interface and then the IBM Event Streams CLI.

  1. Open the IBM Event Streams user interface (go into your IBM Event Streams service within your IBM Cloud portal and click on the launch dashboard button). Once there, click on the Topics tab from the top menu:

    12-0

Let create a demo-topic-ui topic. If you need to revisit the topic concepts, you can read this note. When you go to the topics view you get the list of existing topics.

From this list an administrator can delete an existing topic or create new one.

  1. The ‘create topic’ button leads to the step by step process.
  2. Switch to the Advanced mode to get access to the complete set of parameters. The first panel is here to define the core configuration

Some parameters to understand:

  • Number of partitions: the default value should be 1. If the data can be partitioned without loosing semantic, you can increase the number of partitions.
  • Retention time: This is how long messages are retained before they are deleted to free up space. If your messages are not read by a consumer within this time, they will be missed. It is mapped to the retention.ms kafka topic configuration.

The bottom part of the configuration page, includes logs, cleanup and indexing.

  • The partition’s log parameter section includes a cleanup policy that could be:

    • delete: discard old segments when their retention time or size limit has been reached
    • compact: retain at least the last known value for each message key within the log of data for a single topic partition. The topic looks like a table in DB.
    • compact, delete: compact the log and remove old records
  • retention bytes: represents the maximum size a partition (which consists of log segments) can grow to, before old log segments will be discarded to free up space.

  • log segment size is the maximum size in bytes of a single log file.

  • Cleanup segment time - segment.ms controls the period of time after which Kafka will force the log to roll, even if the segment file isn’t full, this is to ensure that retention can delete or compact old data.

  • Index - segment.index.bytescontrols the size of the index that maps offsets to file positions.

The log cleaner policy is supported by a log cleaner, which are threads that recopy log segment files, removing records whose key appears in the head of the log.

The number of replications is set to three with a min-in-sync replicas of two.

142
  1. We can now see our new topic:

    15
  2. To delete a topic, click on the topic options button at the right end of a topic, click on Delete this topic and then on the Delete button in the confirmation pop-up window:

    16
  3. The topic should now be deleted:

    16-1

Create topic with CLI

  1. List your topics with $ ibmcloud es topics:

    $ ibmcloud es topics
    OK
    No topics found.
  2. Create a topic: (Default 1 partition - 3 replicas)

    $ ibmcloud es topic-create --name demo-topic
    Created topic demo-topic
    OK

    * Execute $ ibmcloud es topic-create --help for more further configuration of your topic creation

  3. List topics:

    $ ibmcloud es topics
    Topic name
    demo-topic
    OK
  4. Display details of a topic:

    $ ibmcloud es topic demo-topic
    Details for topic demo-topic
    Topic name Internal? Partition count Replication factor
    demo-topic false 1 3
    Partition details for topic demo-topic
    Partition ID Leader Replicas In-sync
    0 2 [2 1 0] [2 1 0]
  5. Delete records in a topic (in the command below, we want to delete record on a partition 0 offset 5 and partition 1 from offset 0):

    $ ibmcloud es topic-delete-records --name demo-topic --partition-offset 1:0;0:5 --force
  6. Add partitions to an existing topic, by setting the new target number of partition:

    $ ibmcloud es topic-partition-set --name demo-topic --partitions 30
  7. Delete a topic:

    $ ibmcloud es topic-delete demo-topic
    Really delete topic 'demo-topic'? [y/N]> y
    Topic demo-topic deleted successfully
    OK
  8. List topics:

    $ ibmcloud es topics
    OK
    No topics found.

For the last list of commands see the CLI Reference manual.

Getting started applications

From the manage dashboard we can download a getting started application that has two processes: one consumer and one producer, or we can use a second application that we have in this repository

Using the Event Streams on cloud getting started app

To be able to build the code you need to get gradle installed or use the docker image:

docker run --rm -u gradle -v "$PWD":/home/gradle/project -w /home/gradle/project gradle gradle <gradle-task>

The instructions are in this documentation and can be summarized as:

  • Clone the github repository:
git clone https://github.com/ibm-messaging/event-streams-samples.git
  • Build the code:

Using the gradle CLI

cd kafka-java-console-sample
gradle clean && gradle build

or the gradle docker image

docker run --rm -u gradle -v "$PWD":/home/gradle/project -w /home/gradle/project gradle gradle build
  • Start consumer
java -jar ./build/libs/kafka-java-console-sample-2.0.jar broker-0-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-1-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-2-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093 am_rbb9e794mMwhE-KGPYo0hhW3h91e28OhT8IlruFe5 -consumer
  • Start producer
java -jar ./build/libs/kafka-java-console-sample-2.0.jar broker-0-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-1-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093,broker-2-qnprtqnp7hnkssdz.kafka.svc01.us-east.eventstreams.cloud.ibm.com:9093 am_rbb9e794mMwhE-KGPYo0hhW3h91e28OhT8IlruFe5 -producer