Messaging platform with extreme scaling, fault-tolerance, replication, parallelism, real-time streaming and load balancing - Apache Kafka is arguably the most commonly used distributed messaging platform today.
A few weeks ago, I was working with one of my customers on their enterprise cloud strategy. They are one of the largest retail brands in the US. As part of their rationalization exercise and "Pivot to the Cloud" strategy, their Kafka event hub had to be containerized and deployed on cloud infrastructure.
The idea of this blog post is to walk you through running a full-stack Docker based Apache Kafka + Zookeeper cluster on Oracle Container Cloud in a matter of minutes without having to deal with the complex infrastructure/network setup, Docker toolset installs, upgrades and maintenance.
If you are new to Oracle Container Cloud, please refer to my earlier blog here.
If you would like to get a feel of the Oracle Container Cloud service, head out to https://cloud.oracle.com/en_US/tryit and request a fully-featured instance.
Once you are logged-in as a cloud administrator, click on Container Cloud Service from the list of services available on the cloud dashboard.
In the Oracle Container Cloud Service console, click "Create Service" to create a new Container Cloud Service instance.
Define the service details on the "Create Service" page. Click Next and Confirm.
Give it a few minutes and you will find a Container Manager Node and Worker Nodes provisioned for use. Click on the service to explore the service details.
Click on the hamburger menu on the container service and choose "Container Console" to open the service administrator console. Login using the administrator user (provided during the service creation).
For users of Apache Kafka on Docker, you would be aware of tens of publicly available containers.
If you are looking to have a simple single container Kafka service where Zookeeper and Kafka brokers co-exist on a single container, I have found spotify/kafka easy to setup & use.
For the more complex multi-tier setup, where Zookeeper and Kafka brokers run on dedicated container nodes, wurstmeister/kafka is the most popular option.
Since, I want to demonstrate how to provision a production grade Kafka stack on Oracle Container Cloud, we will go with the wurstmeister/kafka docker image
Go to the Services section and click "New Service" button. You can see that OCCS offers multiple options to define a service container;
The cool thing is that, you can use any / either / a combination of these options to define and create your container service. Any changes you make on any of these will reflect immediately on the other automagically.
Let's use the "Builder" tab to define our first Kafka service.
Service Name: Provide a name for the Kafka service. Notice that the service ID is automatically generated which will be used to uniquely identify our service.
Service Description: Describe the service. Eg., My Kafka Event Hub.
Notice that this would automatically create a environment variable "occs:description"
Scheduler: Determines how & where containers will be provisioned across hosts.
Availability: Define availability of the service based on pool, host or tags.
Image: Enter "wurstmeister/kafka" (without quotes).
Since this is a public image available on docker hub, OCCS can pull this automatically. Remember you can also pull from private docker registries that you might have. If so, head over to "Registries" section on the main console to add your docker registry.
Command: If you want to run some commands on container startup that goes here.
In the "Available Options" panel, choose "Ports". This will add a new "Ports" section to your Builder panel. Click "Add". This will define on what port our Kafka service would run.
Leave the IP field empty (this would default to the container IP based on the host it would run - determined dynamically). Enter host port as 9092, container port as 9092 and choose TCP for protocol.
Your first Kafka service should look like below.
Now, head over to the "Docker Run" and "YAML" tabs and notice the service definition created automagically in the background while we were defining the service. Click "Save" to exit.
Let's now create another definition for our Zookeeper service. Kafka uses Zookeeper for cluster and member management.
Follow the same steps as earlier to create the new Zookeeper service which would run on port 2181.
Your Zookeeper service should look like below. Save & Exit.
Now that we have our Kafka and Zookeeper services ready, time to link them up together for our full-fledged Kafka stack on cloud.
Go to "Stacks" section and click "New Stack".
Note: This is the Docker Compose feature. If you already had your Docker Compose YAML files, you can simply copy-paste here to stack up your services.
Provide the new stack a name: MyKafkaStack (This would create a stack id automatically to uniquely identify the stack).
Notice all the services displayed on the right under the "Available Services" section.
Similar to the "Service" definition, "Stacks" offer 2 modes to define and create stacks. Either drag & drop services on UI (or) click on "Advanced Editor" to wire services using YAML constructs. Even better, use a combination of both.
Let's use a combination.
Let's drag & drop MyKafka and MyZookeeper services on to the "Stacks" screen.
Click on "Advanced Editor" to open the YAML composer. Immediately notice that the YAML script is generated based on the service we composed on the UI (drag & drop).
The Kafka service requires a few environment variables to be set to expose itself for external connectivity. Add the following environment variables to the YAML editor under the "MyKafka" service.
- "KAFKA_ADVERTISED_HOST_NAME={{hostip_for_interface .HostIPs \"public_ip\"}}"
- KAFKA_ADVERTISED_PORT=9092
- "KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181"
Note: We want the stack to run on "Any Host" irrespective of IP address changes. The expression above will fetch the IP address of the host dynamically at run-time. You can leverage the "Tips & Tricks" option in the editor to see some tips & examples.
Define the links under the "MyKafka" service to link it to our Zookeeper container, using the YAML construct below;
links:
-"MyZookeeper:zookeeper"
Ensure that your stack editor looks like below and click Done to exit. Save to exit the stack editor.
Note: A link is shown on the UI indicating the Kafka service is "linked" to the Zookeeper.
OCCS offers a convenient single-click deployment of stacks. Click "Deploy" next to the "MyKafkaStack" to deploy both Kafka and Zookeeper services.
On successful deployment, you should see 2 healthy services running. Note that OCCS allows you to define health checks on containers.
You can also define "webhooks" for your Continuous Integration (CI) / Continuous Delivery (CD) capabilities.
Let's quickly test our new Kafka stack. I am using my local Kafka command line client to test my Kafka service. First create a new topic, start the producer and consumer scripts in your terminals.
We just deployed a Apache Kafka Zookeeper Docker stack on Oracle Cloud. You can now start scaling your Kafka cluster, add more container/hosts and dynamically scale up/down and use this as your cloud-based event hub.
In my next blog, we will see how to rapidly deploy a LAMP stack application on Oracle Container Cloud. Stay Tuned!
A few weeks ago, I was working with one of my customers on their enterprise cloud strategy. They are one of the largest retail brands in the US. As part of their rationalization exercise and "Pivot to the Cloud" strategy, their Kafka event hub had to be containerized and deployed on cloud infrastructure.
The idea of this blog post is to walk you through running a full-stack Docker based Apache Kafka + Zookeeper cluster on Oracle Container Cloud in a matter of minutes without having to deal with the complex infrastructure/network setup, Docker toolset installs, upgrades and maintenance.
If you are new to Oracle Container Cloud, please refer to my earlier blog here.
If you would like to get a feel of the Oracle Container Cloud service, head out to https://cloud.oracle.com/en_US/tryit and request a fully-featured instance.
Once you are logged-in as a cloud administrator, click on Container Cloud Service from the list of services available on the cloud dashboard.
In the Oracle Container Cloud Service console, click "Create Service" to create a new Container Cloud Service instance.
Define the service details on the "Create Service" page. Click Next and Confirm.
Give it a few minutes and you will find a Container Manager Node and Worker Nodes provisioned for use. Click on the service to explore the service details.
Click on the hamburger menu on the container service and choose "Container Console" to open the service administrator console. Login using the administrator user (provided during the service creation).
For users of Apache Kafka on Docker, you would be aware of tens of publicly available containers.
If you are looking to have a simple single container Kafka service where Zookeeper and Kafka brokers co-exist on a single container, I have found spotify/kafka easy to setup & use.
For the more complex multi-tier setup, where Zookeeper and Kafka brokers run on dedicated container nodes, wurstmeister/kafka is the most popular option.
Since, I want to demonstrate how to provision a production grade Kafka stack on Oracle Container Cloud, we will go with the wurstmeister/kafka docker image
Go to the Services section and click "New Service" button. You can see that OCCS offers multiple options to define a service container;
- Builder: For the not-so-tech-savvy users where you can simply enter service details and OCCS takes care of building the docker commands for you
- Docker Run: If you are a Docker pro, you can simply head out to this tab and enter your Docker Run commands directly. This is also a great option, if you already have existing Docker setup which allows you to simply copy paste your Docker Run command
- YAML: For the YAML lovers, you can also define your service using YAML constructs
The cool thing is that, you can use any / either / a combination of these options to define and create your container service. Any changes you make on any of these will reflect immediately on the other automagically.
Let's use the "Builder" tab to define our first Kafka service.
Service Description: Describe the service. Eg., My Kafka Event Hub.
Notice that this would automatically create a environment variable "occs:description"
Scheduler: Determines how & where containers will be provisioned across hosts.
Availability: Define availability of the service based on pool, host or tags.
Image: Enter "wurstmeister/kafka" (without quotes).
Since this is a public image available on docker hub, OCCS can pull this automatically. Remember you can also pull from private docker registries that you might have. If so, head over to "Registries" section on the main console to add your docker registry.
Command: If you want to run some commands on container startup that goes here.
In the "Available Options" panel, choose "Ports". This will add a new "Ports" section to your Builder panel. Click "Add". This will define on what port our Kafka service would run.
Leave the IP field empty (this would default to the container IP based on the host it would run - determined dynamically). Enter host port as 9092, container port as 9092 and choose TCP for protocol.
Now, head over to the "Docker Run" and "YAML" tabs and notice the service definition created automagically in the background while we were defining the service. Click "Save" to exit.
 |
 |
Let's now create another definition for our Zookeeper service. Kafka uses Zookeeper for cluster and member management.
Follow the same steps as earlier to create the new Zookeeper service which would run on port 2181.
Your Zookeeper service should look like below. Save & Exit.
Now that we have our Kafka and Zookeeper services ready, time to link them up together for our full-fledged Kafka stack on cloud.
Go to "Stacks" section and click "New Stack".
Note: This is the Docker Compose feature. If you already had your Docker Compose YAML files, you can simply copy-paste here to stack up your services.
Provide the new stack a name: MyKafkaStack (This would create a stack id automatically to uniquely identify the stack).
Notice all the services displayed on the right under the "Available Services" section.
Similar to the "Service" definition, "Stacks" offer 2 modes to define and create stacks. Either drag & drop services on UI (or) click on "Advanced Editor" to wire services using YAML constructs. Even better, use a combination of both.
Let's use a combination.
Let's drag & drop MyKafka and MyZookeeper services on to the "Stacks" screen.
Click on "Advanced Editor" to open the YAML composer. Immediately notice that the YAML script is generated based on the service we composed on the UI (drag & drop).
The Kafka service requires a few environment variables to be set to expose itself for external connectivity. Add the following environment variables to the YAML editor under the "MyKafka" service.
- "KAFKA_ADVERTISED_HOST_NAME={{hostip_for_interface .HostIPs \"public_ip\"}}"
- KAFKA_ADVERTISED_PORT=9092
- "KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181"
Note: We want the stack to run on "Any Host" irrespective of IP address changes. The expression above will fetch the IP address of the host dynamically at run-time. You can leverage the "Tips & Tricks" option in the editor to see some tips & examples.
Define the links under the "MyKafka" service to link it to our Zookeeper container, using the YAML construct below;
links:
-"MyZookeeper:zookeeper"
Ensure that your stack editor looks like below and click Done to exit. Save to exit the stack editor.
Note: A link is shown on the UI indicating the Kafka service is "linked" to the Zookeeper.
OCCS offers a convenient single-click deployment of stacks. Click "Deploy" next to the "MyKafkaStack" to deploy both Kafka and Zookeeper services.
On successful deployment, you should see 2 healthy services running. Note that OCCS allows you to define health checks on containers.
You can also define "webhooks" for your Continuous Integration (CI) / Continuous Delivery (CD) capabilities.
Let's quickly test our new Kafka stack. I am using my local Kafka command line client to test my Kafka service. First create a new topic, start the producer and consumer scripts in your terminals.
We just deployed a Apache Kafka Zookeeper Docker stack on Oracle Cloud. You can now start scaling your Kafka cluster, add more container/hosts and dynamically scale up/down and use this as your cloud-based event hub.
In my next blog, we will see how to rapidly deploy a LAMP stack application on Oracle Container Cloud. Stay Tuned!
