This document assumes knowledge of Docker and Dockerfiles. To review best practices for writing Dockerfiles, see Docker’s best practices guide.
If you want to contribute back to the project, review the contributing guidelines.
Confluent Platform image bootup process
Upon startup of Docker, the entry point /etc/confluent/docker/run runs three executable scripts found in the /etc/confluent/docker. They are run in the following sequence:
Configure script
The /etc/confluent/docker/configure script does all the necessary con…
This document assumes knowledge of Docker and Dockerfiles. To review best practices for writing Dockerfiles, see Docker’s best practices guide.
If you want to contribute back to the project, review the contributing guidelines.
Confluent Platform image bootup process
Upon startup of Docker, the entry point /etc/confluent/docker/run runs three executable scripts found in the /etc/confluent/docker. They are run in the following sequence:
Configure script
The /etc/confluent/docker/configure script does all the necessary configuration for each image. This includes the following:
Create all configuration files and copying them to their proper location.
Ensure that mandatory configuration properties are present.
If required, handle service discovery. 1.
Ensure script
The /etc/confluent/docker/ensure scripts makes sure that all the prerequisites for launching the service are in place. This includes:
Ensure the configuration files are present and readable.
Ensure that you can write/read to the data directory. The directories must be world writable.
Ensure that supporting services are in the READY state. For example, ensure that the Kafka controller is ready before launching a Kafka broker.
Ensure supporting systems are configured properly. 1.
Launch
The /etc/confluent/docker/launch script runs the actual process. The script should ensure that:
The process is run with process id 1. Your script should use exec so the program takes over the shell process rather than running as a child process. This is so that your program will receive signals like SIGTERM directly rather than its parent shell process receiving them.
Log to stdout.
Prerequisites
Install Docker:
Docker version 1.11 or later is installed and running.
Docker Compose is installed. Docker Compose is installed by default with Docker for Mac.
Docker memory is allocated minimally at 8 GB. When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. You can change the default allocation to 8 GB in Docker. Navigate to Preferences > Resources > Advanced. 1.
Install Maven:
brew install maven
Create or edit the ~/.m2/settings.xml file to add the following Confluent Maven repository content to your Maven settings:
<settings>
<profiles>
<profile>
<id>confluent</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<repositories>
<repository>
<id>confluent</id>
<url>https://packages.confluent.io/maven/</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>confluent</id>
<url>https://packages.confluent.io/maven/</url>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
</settings>
Build the Confluent Platform images
Refer to Docker Image Reference for Confluent Platform for a list of GitHub repos for the Confluent Platform components.
For each Confluent Platform image you want to build:
Clone the repo. 1.
Checkout the release branch. 1.
Get the values for the required and optional arguments for the build command.
For the list of supported arguments, see this README file.
The following are required arguments:
CONFLUENT_PACKAGES_REPO: Specify the location of the Confluent Platform packages repository. Depending on the type of OS for the image you are building, you might need to provide a Debian or RPM repository.
CONFLUENT_VERSION: Specify the full Confluent Platform release version, e.g., 8.1.1.
docker.upstream-registry: Registry to pull base images from. Trailing / is required. Used as DOCKER_UPSTREAM_REGISTRY during docker build.
docker.upstream-tag: Use the given tag when pulling base images. Used as DOCKER_UPSTREAM_TAG during docker build.
1.
Optionally, you can choose an operating system you want your docker image based on, specifically Debian or RHEL UBI. To build a RHEL UBI image pass the following argument to the Maven command:
-Ddocker.os_type=ubi9
From the root folder of the repo, build the Confluent Platform images using Maven.
For example:
mvn clean package \
-DskipTests -Pdocker \
-DCONFLUENT_PACKAGES_REPO='https://packages.confluent.io/rpm/8.1' \
-DCONFLUENT_VERSION=8.1.1 \
-Ddocker.upstream-registry=docker.io/ \
-Ddocker.upstream-tag=8.1.1
Extend Confluent Platform images
You can extend the images to add or customize connectors, add new software, change the configuration management, and set up external service discovery.
The following sections provide examples of extending Confluent Platform images.
Note
You can apply the similar steps to extend custom Docker images that you built.
However, if you run into an issue with a custom Docker image, you need to work with Confluent Support to narrow down the issue. Confluent Support can only assist with the issues involving Confluent Platform software.
The issues with Docker or any 3rd party software cannot be further debugged by Confluent Support.
Add Connectors or Software
The Kafka Connect Base image contains Kafka Connect and all of its dependencies. When started, it will run the Connect framework in distributed mode. To add new connectors to this image, you need to build a new Docker image that has the new connectors installed. The following examples show how to add connectors.
Create a Docker Image containing Confluent Marketplace Connectors
This example shows how to use the Confluent Marketplace Client to create a Docker image that extends from one of Confluent’s Kafka Connect images but which contains a custom set of connectors. This may be useful if you’d like to use a connector that isn’t contained in the cp-server-connect image, or if you’d like to keep the custom image lightweight and not include any connectors that you don’t plan to use.
Add connectors from Confluent Marketplace. 1.
Choose an image to extend. 1.
Choose the connectors from Confluent Marketplace that you’d like to include in your custom image. Note that the remaining steps result in a custom image containing a MongoDB connector, a Microsoft Azure IoT Hub connector, and a Google BigQuery connector. 1.
Write a Dockerfile.
FROM confluentinc/cp-server-connect-base:8.1.1
RUN confluent-hub install --no-prompt hpgrahsl/kafka-connect-mongodb:1.1.0 \
&& confluent-hub install --no-prompt microsoft/kafka-connect-iothub:0.6 \
&& confluent-hub install --no-prompt wepay/kafka-connect-bigquery:1.1.0
Build the Dockerfile.
docker build . -t my-custom-image:1.0.0
The output from that command should resemble:
Step 1/2 : FROM confluentinc/cp-server-connect-base
---> e0d92da57dc3
...
Running in a "--no-prompt" mode
Implicit acceptance of the license below:
Apache 2.0
https://github.com/wepay/kafka-connect-bigquery/blob/master/LICENSE.md
Implicit confirmation of the question: You are about to install 'kafka-connect-bigquery' from WePay, as published on Confluent Hub.
Downloading component BigQuery Sink Connector 1.1.0, provided by WePay from Confluent Hub and installing into /usr/share/confluent-hub-components
Adding installation directory to plugin path in the following files:
/etc/kafka/connect-distributed.properties
/etc/kafka/connect-standalone.properties
/etc/schema-registry/connect-avro-distributed.properties
/etc/schema-registry/connect-avro-standalone.properties
Completed
Removing intermediate container 48d4506b8a83
---> 496befc3d3f7
Successfully built 496befc3d3f7
Successfully tagged my-custom-image:1.0.0
This results in an image named my-custom-image that contains the MongoDB, Azure IoT Hub, and BigQuery connectors, and which is capable of running any or all of the connectors using the Kafka Connect framework.
If you are using a docker-compose.yml file and the Confluent Marketplace Client to build your Kafka environment, use the following properties to enable a connector. (The example shows the Datagen development and testing connector.)
connect:
image: confluentinc/kafka-connect-datagen:latest
build:
context: .
dockerfile: Dockerfile-confluenthub
Create a Docker Image containing Local Connectors
This example shows how to create a Docker image that extends the cp-server-connect-base image to contain one or more local connectors. This is useful if you want to use your connectors instead of pulling connectors from Confluent Marketplace.
Package your local connector in a zip file. 1.
Set up the Dockerfile as shown in the example below.
FROM confluentinc/cp-server-connect-base:8.1.1
COPY target/components/packages/my-connector-8.1.1.zip /tmp/my-connector-8.1.1.zip
RUN confluent-hub install --no-prompt /tmp/my-connector-8.1.1.zip
Build the Dockerfile.
docker build . -t my-custom-image:1.0.0
Add Additional Software
This example shows how to add new software to an image. For example, you might want to extend the Kafka Connect client to include the MySQL JDBC driver. If this approach is used to add new connectors to an image, the connector JARs must be on the plugin.path or the CLASSPATH for the Connect framework.
Write the Dockerfile.
FROM confluentinc/cp-server-connect
ENV MYSQL_DRIVER_VERSION 5.1.39
RUN curl -k -SL "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-${MYSQL_DRIVER_VERSION}.tar.gz" \
| tar -xzf - -C /usr/share/java/kafka/ --strip-components=1 mysql-connector-java-5.1.39/mysql-connector-java-${MYSQL_DRIVER_VERSION}-bin.jar
Build the image.
docker build -t foo/mysql-connect:latest .
Note
This approach can also be used to create images with your own Kafka Connect Plugins.
Change configuration management
This example describes how to change the configuration management. To accomplish this, you override the configure script to download the scripts from a URL.
For example, with the Schema Registry image, you need the following Dockerfile and configure script. This example assumes that each property file has a URL.
Dockerfile:
FROM confluentinc/cp-schema-registry
COPY include/etc/confluent/docker/configure /etc/confluent/docker/configure
Example Configure dub Script:
Location: include/etc/confluent/docker/configure
. /etc/confluent/docker/bash-config
dub ensure SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
dub ensure SCHEMA_REGISTRY_HOST_NAME
dub path /etc/"${COMPONENT}"/ writable
if [[ -n "${SCHEMA_REGISTRY_PORT-}" ]]
then
echo "PORT is deprecated. Please use SCHEMA_REGISTRY_LISTENERS instead."
exit 1
fi
if [[ -n "${SCHEMA_REGISTRY_JMX_OPTS-}" ]]
then
if [[ ! $SCHEMA_REGISTRY_JMX_OPTS == *"com.sun.management.jmxremote.rmi.port"* ]]
then
echo "SCHEMA_REGISTRY_OPTS should contain 'com.sun.management.jmxremote.rmi.port' property. It is required for accessing the JMX metrics externally."
fi
fi
dub template "/etc/confluent/docker/${COMPONENT}.properties.template" "/etc/${COMPONENT}/${COMPONENT}.properties"
dub template "/etc/confluent/docker/log4j2.yaml.template" "/etc/${COMPONENT}/log4j2.yaml"
dub template "/etc/confluent/docker/admin.properties.template" "/etc/${COMPONENT}/admin.properties"
Example Configure ub Script:
Location: include/etc/confluent/docker/configure
. /etc/confluent/docker/bash-config
ub ensure SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
ub ensure SCHEMA_REGISTRY_HOST_NAME
ub path /etc/"${COMPONENT}"/ writable
if [[ -n "${SCHEMA_REGISTRY_PORT-}" ]]
then
echo "PORT is deprecated. Please use SCHEMA_REGISTRY_LISTENERS instead."
exit 1
fi
if [[ -n "${SCHEMA_REGISTRY_JMX_OPTS-}" ]]
then
if [[ ! $SCHEMA_REGISTRY_JMX_OPTS == *"com.sun.management.jmxremote.rmi.port"* ]]
then
echo "SCHEMA_REGISTRY_OPTS should contain 'com.sun.management.jmxremote.rmi.port' property. It is required for accessing the JMX metrics externally."
fi
fi
ub render-template "/etc/confluent/docker/${COMPONENT}.properties.template" > "/etc/${COMPONENT}/${COMPONENT}.properties"
ub render-template "/etc/confluent/docker/log4j2.yaml.template" > "/etc/${COMPONENT}/log4j2.yaml"
ub render-template "/etc/confluent/docker/admin.properties.template" > "/etc/${COMPONENT}/admin.properties"
The build the image:
docker build -t foo/schema-registry:latest
Run the image, like the following:
docker run \
-e SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=http://foo.com:9092 \
-e SCHEMA_REGISTRY_HOST_NAME=foo.com \
-e SCHEMA_REGISTRY_LISTENERS=http://foo.com:8081 \
-e SCHEMA_REGISTRY_LOG4J_ROOT_LOGLEVEL=INFO \
foo/schema-registry:latest
Log to external volumes
The images only expose volumes for data and security configuration. But you might want to write to external storage for some use cases. The following example shows how to write the Kafka authorizer logs to a volume for auditing.
Dockerfile:
FROM confluentinc/cp-server
# Make sure the log directory is world-writable
RUN echo "===> Creating authorizer logs dir ..." \\
&& mkdir -p /var/log/kafka-auth-logs \\
&& chmod -R ag+w /var/log/kafka-auth-logs
VOLUME \["/var/lib/$\{COMPONENT}/data", "/etc/$\{COMPONENT}/secrets", "/var/log/kafka-auth-logs"]
COPY include/etc/confluent/log4j2.yaml.template /etc/confluent/docker/log4j2.yaml.template
log4j2.yaml.template:
Location: include/etc/confluent/docker/log4j2.yaml.template
Configuration:
name: "Log4j2"
Appenders:
Console:
name: STDOUT
target: SYSTEM_OUT
PatternLayout:
Pattern: "[%d] %p %m (%c)%n"
Loggers:
Root:
level: "{{ env['KAFKA_LOG4J_ROOT_LOGLEVEL'] | default('INFO') }}"
AppenderRef:
- ref: STDOUT
Logger:
{% set loggers = {
'kafka': 'INFO',
'kafka.network.RequestChannel$': 'WARN',
'kafka.producer.async.DefaultEventHandler': 'DEBUG',
'kafka.request.logger': 'WARN',
'kafka.controller': 'TRACE',
'kafka.log.LogCleaner': 'INFO',
'state.change.logger': 'TRACE',
'kafka.authorizer.logger': 'WARN'
} -%}
{% if env['KAFKA_LOG4J_LOGGERS'] %}
{% set loggers = parse_log4j_loggers(env['KAFKA_LOG4J_LOGGERS'], loggers) %}
{% endif %}
{% for logger,loglevel in loggers.items() %}
- name: "{{ logger }}"
level: "{{ loglevel }}"
AppenderRef:
ref: STDOUT
{% endfor %}
Then, build the image.
docker build -t foo/kafka-auditable:latest .
Write garbage collection logs to an external volume
The following example shows how to log heap dumps and GC logs to an external volume. This is useful for debugging the Kafka image.
Dockerfile:
FROM confluentinc/cp-server
# Make sure the jvm log directory is world-writable
RUN echo "===> Creating jvm logs dir ..." \
&& mkdir -p /var/log/jvm-logs
&& chmod -R ag+w /var/log/jvm-logs
VOLUME ["/var/lib/${COMPONENT}/data", "/etc/${COMPONENT}/secrets", "/var/log/jvm-logs"]
Build the image.
docker build -t foo/kafka-verbose-jvm:latest .
Enter the command.
docker run \
-e KAFKA_HEAP_OPTS="-Xmx256M -Xloggc:/var/log/jvm-logs/verbose-gc.log -verbose:gc -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/jvm-logs" \
foo/kafka-verbose-jvm:latest
Use external service discovery
You can extend the images to support for any service discovery mechanism either by overriding relevant properties or by overriding the configure script as shown in Change Configuration Management.
The Docker images provide Mesos support by overriding relevant properties for Mesos service discovery. See debian/kafka-connect/includes/etc/confluent/docker/mesos-overrides for examples.
Use the Oracle JDK
The Confluent Platform images ship with Temurin open source JDK, based on the OpenJDK. If you are required to use Oracle’s version, follow the steps below to modify the base image to include the Oracle JDK instead of the Temurin JDK.
Obtain a download URL from Oracle’s website. The URL must be accessible by your build machine.
Important
Older versions of the Oracle JDK are not recommended for production use as they no longer receive security patches. 1.
In the base directory of the Confluent Base image repo, edit the Dockerfile.ubi9 file in the correct version branch, 8.1.1-post.
1.
Comment out the following lines for the Temurin JDK and add an Oracle URL.
# RUN printf "[temurin-jdk] \n\
# name=temurin-jdk \n\
# baseurl=https://packages.adoptium.net/artifactory/rpm/rhel/\$releasever/\$basearch \n\
# enabled=1 \n\
# gpgcheck=1 \n\
# gpgkey=https://packages.adoptium.net/artifactory/api/gpg/key/public \n\
# " > /etc/yum.repos.d/adoptium.repo
OracleUrl=<url-from-step-1>
Then, replace the Temurin JDK line (temurin-17-jdk${TEMURIN_JDK_VERSION}" "temurin-17-jre${TEMURIN_JDK_VERSION}") with the Oracle URL in the following code:
1RUN microdnf --nodocs install yum \
2&& yum --nodocs -q update -y \
3&& yum --nodocs -q install -y wget tar gzip make perl gcc \
4&& wget -q https://github.com/openssl/openssl/releases/download/openssl${OPENSSL_VERSION}/openssl${OPENSSL_VERSION}.tar.gz \
5&& tar -xzf openssl${OPENSSL_VERSION}.tar.gz \
6&& cd openssl${OPENSSL_VERSION} \
7&& echo "installing FIPS compliant openssl" \
8&& ./Configure enable-fips \
9&& make > /dev/null 2>&1 \
10&& make install > /dev/null 2>&1 \
11&& echo "successfully installed FIPS compliant openssl" \
12&& cd .. \
13&& rm -rf openssl${OPENSSL_VERSION} openssl${OPENSSL_VERSION}.tar.gz \
14&& yum remove -y wget tar make perl gcc glibc-gconv-extra --setopt=clean_requirements_on_remove=1 \
15&& yum --nodocs install -y --setopt=install_weak_deps=False \
16 git \
17 "wget${WGET_VERSION}" \
18 "nmap-ncat${NETCAT_VERSION}" \
19 "python39${PYTHON39_VERSION}" \
20 "python39-pip${PYTHON_PIP_VERSION}" \
21 "tar${TAR_VERSION}" \
22 "procps-ng${PROCPS_VERSION}" \
23 "krb5-workstation${KRB5_WORKSTATION_VERSION}" \
24 "iputils${IPUTILS_VERSION}" \
25 "hostname${HOSTNAME_VERSION}" \
26 "xz-libs${XZ_LIBS_VERSION}" \
27 "glibc${GLIBC_VERSION}" \
28 "glibc-common${GLIBC_VERSION}" \
29 "glibc-minimal-langpack${GLIBC_VERSION}" \
30 "curl${CURL_VERSION}" \
31 "libcurl${CURL_VERSION}" \
32 "findutils${FINDUTILS_VERSION}" \
33 "crypto-policies-scripts${CRYPTO_POLICIES_SCRIPTS_VERSION}" \
34 "Oracle17Url" \
35&& alternatives --set python /usr/bin/python3 \
36&& python3 -m pip install --upgrade "setuptools${PYTHON_SETUPTOOLS_VERSION}" \
37&& python3 -m pip install --prefer-binary --prefix=/usr/local --upgrade "${PYTHON_CONFLUENT_DOCKER_UTILS_INSTALL_SPEC}" \
38&& yum remove -y git \
39&& yum clean all \
40&& rm -rf /tmp/* \
41&& mkdir -p /etc/confluent/docker /usr/logs \
42&& useradd --no-log-init --create-home --shell /bin/bash appuser \
43&& chown appuser:appuser -R /etc/confluent/ /usr/logs
Rebuild the base image. Refer to Build the Confluent Platform images for the steps to do this.
Utility scripts
Because there are dependencies between the various Confluent Platform components, you might need to check the status of different services. The following utilities are used during the bootup sequence of the images and in the testing framework.
Note
The ub script is available only with Docker images based on cp-base-java. For images based on cp-base-new, use the cub or dub scripts instead.
Utility Belt (ub)
render-template
usage: ub render-template [-h] input
Renders a template file's content to standard output (stdout).
Positional arguments:
input The path to the template file.
ensure
usage: ub ensure [-h] name
Checks if an environment variable exists.
Positional arguments:
name The name of the environment variable to check.
path
usage: ub path <path-to-file> <operation> [flags]
Checks if a specific operation is permitted on a file.
Positional arguments:
path The full path to the file.
operation The specific operation to check, which can be one of the following: writable, readable, executable, or exists.
kafka-ready
usage: ub kafka-ready <min-num-brokers> <timeout-secs> [flags]
Checks if Kafka brokers are up and running.
Positional arguments:
min-num-brokers Minimum number of brokers to wait for.
timeout Time in seconds to wait for service to be ready.
Flags:
-b, --bootstrap-servers string Comma-separated list of Kafka brokers.
-c, --config string Path to the config file.
-h, --help Help for Kafka-ready.
-s, --security string Security protocol to use when multiple listeners are enabled.
http-ready
usage: ub http-ready <url> <timeout>
Waits for an HTTP/HTTPS URL to become accessible and retrievable.
Positional arguments:
url The URL to the HTTP service to check.
timeout The maximum time (in seconds) to wait for the URL to become retrievable.
render-properties
usage: ub render-properties <path-to-config-spec>
Creates and renders properties to stdout using the JSON configuration specification.
Positional arguments:
path-to-config-spec. Path to the JSON configuration file that defines the properties to be rendered.
wait
usage: ub wait <host> <port> <timeout-in-secs>
Waits for a service to start listening on a specified host and port.
Positional arguments:
host The hostname of the service to wait for.
port The port number on which the service is expected to start listening.
timeout The maximum time (in seconds) to wait for the service to become available.
kr-ready
usage: ub kr-ready <host> <port> <timeout-secs> [flags]
Check if Kafka REST Proxy service is ready to accept client requests.
Positional arguments:
host The hostname of the Kafka REST Proxy service.
port The port number on which the Kafka REST Proxy is running.
timeout The time in seconds to wait for the service to be ready.
Flags:
-b, --bootstrap-servers string Comma-separated list of Kafka brokers.
-c, --config string Path to the config file.
-h, --help Help for kr-ready.
--ignore-cert Ignore TLS certificate errors.
--password string Password to authenticate to the Kafka REST Proxy.
--secure Use TLS to secure the connection.
-s, --security string Security protocol to use when multiple listeners are enabled.
--username string Username to authenticate to the Kafka REST Proxy.
connect-ready
usage: ub connect-ready <host> <port> <timeout-secs> [flags]
Check if Connect service is ready to accept connector tasks.
Positional arguments:
host The hostname of the Connect service.
port The port number on which Connect is running.
timeout The time in seconds to wait for the service to be ready.
Flags:
-h, --help Help for connect-ready.
--ignore-cert Ignore TLS certificate errors.
--password string Password to authenticate to Connect.
--secure Use TLS to secure the connection.
--username string Username to authenticate to Connect.
ksql-server-ready
usage: ub ksql-server-ready <host> <port> <timeout-secs> [flags]
Check if KSQL server is ready to accept client requests.
Positional arguments:
host The hostname of the KSQL server.
port The port number on which the KSQL server is running.
timeout The time in seconds to wait for the service to be ready.
Flags:
-h, --help Help for ksql-server-ready.
--ignore-cert Ignore TLS certificate errors.
--password string Password to authenticate to the KSQL server.
--secure Use TLS to secure the connection.
--username string Username to authenticate to the KSQL server.
control-center-ready
usage: ub control-center-ready <host> <port> <timeout-secs> [flags]
Check if Confluent Control Center is ready to accept client requests.
Positional arguments:
host The hostname of the Control Center.
port The port number on which Control Center is running.
timeout The time in seconds to wait for the service to be ready.
Flags:
-h, --help Help for control-center-ready.
--ignore-cert Ignore TLS certificate errors.
--password string Password to authenticate to Control Center.
--secure Use TLS to secure the connection.
--username string Username to authenticate to Control Center.
sr-ready
usage: ub sr-ready <host> <port> <timeout-secs> [flags]
Check if Schema Registry is ready to accept client requests.
Positional arguments:
host The hostname of the Schema Registry.
port The port number on which Schema Registry is running.
timeout The time in seconds to wait for the service to be ready.
Flags:
-h, --help Help for sr-ready.
--ignore-cert Ignore TLS certificate errors.
--password string Password to authenticate to Schema Registry.
--secure Use TLS to secure the connection.
--username string Username to authenticate to Schema Registry.
Docker Utility Belt (dub)
template
usage: dub template [-h] input output
Generate template from env vars.
positional arguments:
input Path to template file.
output Path of output file.
ensure
usage: dub ensure [-h] name
Check if env var exists.
positional arguments:
name Name of env var.
wait
usage: dub wait [-h] host port timeout
wait for network service to appear.
positional arguments:
host Host.
port Host.
timeout timeout in secs.
path
usage: dub path [-h] path {writable,readable,executable,exists}
Check for path permissions and existence.
positional arguments:
path Full path.
{writable,readable,executable,exists} One of [writable, readable, executable, exists].
path-wait
usage: dub path-wait [-h] path timeout
Wait for a path to exist.
positional arguments:
path Full path.
timeout Time in secs to wait for the path to exist.
optional arguments:
-h, --help show this help message and exit
Confluent Platform Utility Belt (cub)
The following utilities are used to check the status of various Confluent Platform components.
kafka-ready
Used to check if Kafka is ready.
usage: cub kafka-ready [-h] (-b BOOTSTRAP_BROKER_LIST)
[-c CONFIG] [-s SECURITY_PROTOCOL]
expected_brokers timeout
Check if Kafka is ready.
positional arguments:
expected_brokers Minimum number of brokers to wait for
timeout Time in secs to wait for service to be ready.
optional arguments:
-h, --help show this help message and exit
-b BOOTSTRAP_BROKER_LIST, --bootstrap_broker_list BOOTSTRAP_BROKER_LIST
List of bootstrap brokers.
-c CONFIG, --config CONFIG
Path to config properties file (required when security
is enabled).
-s SECURITY_PROTOCOL, --security-protocol SECURITY_PROTOCOL
Security protocol to use when multiple listeners are
enabled.
sr-ready
Used to check if Schema Registry is ready. If you have multiple Schema Registry nodes, you may need to check their availability individually.
usage: cub sr-ready [-h] host port timeout
positional arguments:
host Hostname for Schema Registry.
port Port for Schema Registry.
timeout Time in secs to wait for service to be ready.
kr-ready
Used to check if the REST Proxy is ready. If you have multiple REST Proxy nodes, you may need to check their availability individually.
usage: cub kr-ready [-h] host port timeout
positional arguments:
host Hostname for REST Proxy.
port Port for REST Proxy.
timeout Time in secs to wait for service to be ready.
connect-ready
Used to check if Kafka Connect is ready.
usage: cub connect-ready [-h] host port timeout
positional arguments:
host Hostname for Connect worker.
port Port for Connect worker.
timeout Time in secs to wait for service to be ready.
ksql-server-ready
Used to check if ksqlDB is ready.
usage: cub ksql-server-ready [-h] host port timeout
positional arguments:
host Hostname for KSQL server.
port Port for KSQL server.
timeout Time in secs to wait for service to be ready.
control-center-ready
Used to check if Confluent Control Center is ready.
usage: cub control-center-ready [-h] host port timeout
positional arguments:
host Hostname for Control Center.
port Port for Control Center.
timeout Time in secs to wait for service to be ready.
Client properties
The following properties may be configured when using the kafka-ready utility described above.
bootstrap.servers
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
Type: list
Default:
Importance: high
ssl.key.password
The password of the private key in the key store file. This is optional for client.
Type: password
Importance: high
ssl.keystore.location
The location of the key store file. This is optional for client and can be used for bidirectional client authentication.
Type: string
Importance: high
ssl.keystore.password
The store password for the key store file.This is optional for client and only needed if ssl.keystore.location is configured.
Type: password
Importance: high
ssl.truststore.location
The location of the trust store file.
Type: string
Importance: high
ssl.truststore.password
The password for the trust store file.
Type: password
Importance: high
sasl.kerberos.service.name
The Kerberos principal name that Kafka runs as. This can be defined either in Kafka’s JAAS config or in Kafka’s config.
Type: string
Importance: medium
sasl.mechanism
SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism.
Type: string
Default: “GSSAPI”
Importance: medium
security.protocol
Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
Type: string
Default: “PLAINTEXT”
Importance: medium
ssl.enabled.protocols
The comma-separated list of protocols enabled for TLS connections. The default value is TLSv1.2,TLSv1.3 when running with Java 11 or later, TLSv1.2 otherwise. With the default value for Java 11 (TLSv1.2,TLSv1.3), Kafka clients and brokers prefer TLSv1.3 if both support it, and falls back to TLSv1.2 otherwise (assuming both support at least TLSv1.2).
Type: list
Default: TLSv1.3,TLSv1.3
Importance: medium
ssl.keystore.type
The file format of the key store file. This is optional for client.
Type: string
Default: “JKS”
Importance: medium
ssl.protocol
The TLS protocol used to generate the SSLContext. The default is TLSv1.3 when running with Java 11 or newer, TLSv1.2 otherwise. This value should be fine for most use cases. Allowed values in recent JVMs are TLSv1.2 and TLSv1.3. TLS, TLSv1.1, SSL, SSLv2 and SSLv3 might be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. With the default value for this configuration and ssl.enabled.protocols, clients downgrade to TLSv1.2 if the server does not support TLSv1.3. If this configuration is set to TLSv1.2, clients do not use TLSv1.3, even if it is one of the values in ssl.enabled.protocols and the server only supports TLSv1.3.
Type: string
Default: TLSv1.3
Importance: medium
ssl.provider
The name of the security provider used for TLS connections. Default value is the default security provider of the JVM.
Type: string
Importance: medium
ssl.truststore.type
The file format of the trust store file.
Type: string
Default: “JKS”
Importance: medium
sasl.kerberos.kinit.cmd
Kerberos kinit command path.
Type: string
Default: “/usr/bin/kinit”
Importance: low
sasl.kerberos.min.time.before.relogin
Login thread sleep time between refresh attempts.
Type: long
Default: 60000
Importance: low
sasl.kerberos.ticket.renew.jitter
Percentage of random jitter added to the renewal time.
Type: double
Default: 0.05
Importance: low
sasl.kerberos.ticket.renew.window.factor
Login thread will sleep until the specified window factor of time from last refresh to ticket’s expiry has been reached, at which time it will try to renew the ticket.
Type: double
Default: 0.8
Importance: low
ssl.cipher.suites
A list of cipher suites. This is a named combination of authentication, encryption, MAC, and key exchange algorithms used to negotiate the security settings for a network connection using TLS. By default, all the available cipher suites are supported.
Type: list
Importance: low
ssl.endpoint.identification.algorithm
The endpoint identification algorithm to validate server hostname using server certificate.
Type: string
Importance: low
ssl.keymanager.algorithm
The algorithm used by key manager factory for TLS connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine.
Type: string
Default: “SunX509”
Importance: low
ssl.trustmanager.algorithm
The algorithm used by trust manager factory for TLS connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.
Type: string
Default: “PKIX”
Importance: low