tchiotludo / kafkahq, Hacker News

********************: /app/truststore.jks********************************** (ssl-key-store-password) **************************************: key-store-password

**********************************************************************************************

************************************************************ SSL Kafka Cluster with basic auth

Configuration example for kafka cluster secured by ssl for saas provider like aiven (full https & basic auth):

You need to generate a jks & p (file from pem, cert files give by saas provider.)

openssl pkcs -export -inkey service.key -in service.cert -out client.keystore.p -name service_key keytool -import -file ca.pem -alias CA -keystore client.truststore.jks

Configurations will look like this example:

(kafkahq) ************************************************************:    (connections) ************************************************************:      (ssl-dev) ************************************************************:        (properties) **********************************************************:          (bootstrap.servers) ************************************************************:" {{host}}. aivencloud.com:"

         (security.protocol) ************************************************************: SSL         

ssl.truststore.location (********************************************************: {{path} } /avnadmin.truststore.jks

        

ssl.truststore.password (**************************************************************: {{password} }         (ssl.keystore.type) ********************************************************: " (PKCS) **************************************************************************************************************************************************************************************************************************” (************************************************************         ssl.keystore.location (**********************************************************: {{path} } /avnadmin.keystore.p (************************************************************         ssl.keystore.password (**********************************************************: {{password} }         (ssl.key.password) ************************************************************: {{password} }       (schema-registry) **************************************************************:          (url) ************************************************************:" (https: // {{host}} .aivencloud.com:******************************************************************************************************************************************" (**************************************************          (basic-auth-username) ************************************************************: avnadmin         (basic-auth-password) ************************************************************: {{password} }       (connect) ************************************************************:          (url) ************************************************************:" (https: // {{host}} .aivencloud.com: {{port}}"

         (basic-auth-username) ************************************************************: avnadmin

         (basic-auth-password) ************************************************************: {{password} } (******************************** (************************************************************************************************************************************ KafkaHQ configuration**************************************Pagination****************kafkahq.pagination.page-sizenumber of topics per page (default:

********************************************************************************************Topic List****************kafkahq.topic.default-viewis default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM)********************************** kafkahq.topic.internal-regexps **************************************** is a list of regexp to be considered as internal (internal topic can't be deleted or updated)********************************** kafkahq.topic.stream-regexps ************************************ is a list of regexp to be considered as an internal stream topic ********************************** kafkahq.topic.skip-consumer-groups disable loading of consumer group information when showing topics ( (true) , default is to load the information********************************************************************************Topic creation default values ​​

These parameters are the default values ​​used in the topic creation page.****************kafkahq.topic.retentionDefault retention in ms

********************************** (kafkahq.topic.replication) ************************************** Default number of replica to use********************************** kafkahq.topic.partition (Default number of partition

********************************************************************

(Topic Data) ****************kafkahq.topic-data.sort: default sort order (OLDEST, NEWEST) (default: OLDEST)****************************************************** kafkahq.topic-data.size (**************************************: max record per page (default:) ********************************** (kafkahq.topic-data.poll-timeout) ****************************************: The time, in milliseconds, spent waiting in poll if data is not available in the buffer (default:).********************************

************************************************** Security ****************kafkahq.security.default-roles: Roles available for all the user even unlogged user, roles available are:****************topic / read

topic / insert

topic / delete

topic / config / update

********************************** (node ​​/ read) **************************************

********************************** node / config / update

topic / data / read

topic / data / insert

topic / data / delete

********************************** (group / read) **************************************

********************************** (group / delete) **************************************

********************************** (group / offsets / update) **************************************

********************************** (acls / read) **************************************

********************************** (registry / read) **************************************

********************************** (registry / insert******************

********************************** (registry / update******************

********************************** (registry / delete) **************************************

********************************** registry / version / delete

************************

By default, security & roles. is enabled by default but anonymous user have full access. You can completely disabled security withmicronaut.security.enabled: false

If you need a read-only application, simply add this to your configuration files:

(kafkahq) ************************************************************:    (security) ************************************************************:      (default-roles) ************************************************************:       - topic / read      - (node ​​/ read)       - (topic / data / read       - (group / read)       - (registry / read)       - (connect / read) ************************************************************ (************************************************

******************************** (Auth) ********************************************************** (************************************************************

**********************Groups

**********************************************

Groups allow you to limit user

Define groups with specific roles for your users****************kafkahq.security.groups: Groups list definition****************group-name: Group identifier****************roles: Roles list for the group********************************** (attributes.topics-filter-regexp) **************************************: Regexp to filter topics available for current group (************************************************************************************

2 defaults group are available:****************adminwith all right********************************** (reader) **************************************** (with only read acces on all KafkaHQ (************************************************************************kafkahq.security.basic-auth: List user & password with affected roles****************actual-username: Login of the current user as a yaml key (may be anything email, login, ...)**************** (password) : Password in sha (**************************************************************************************************************************************************************************************, can be converted with command echo -n "password" | sha (sum) ***************************************

******************************** (groups) **************************************: Groups for current user

************************************************

Take care that basic auth will use session store in servermemory

If your instance is behind a reverse proxy or a loadbalancer, you will need to forward the session cookie namedSESSIONand / or use sesssion stickiness

(******************************************************************** () **************************

Configure how the ldap groups will be matched in KafkaHQ groups****************kafkahq.security.ldap.group: Ldap groups list****************ldap-group-name: Ldap group name (same name as in ldap)****************groups: KafkaHQ group list to be used for current ldap group

************************************************

Example using online ldap test server

Configure ldap connection in micronaut

(micronaut) ************************************************************:    (security) ************************************************************:      (ldap) **************************************************************:        (default) ************************************************************:          (enabled) ************************************************************: (true)          (context) ************************************************************:            (server) ************************************************************:' ldap: //ldap.forumsys.com: (******************************************** '           (managerDn) ************************************************************:' cn=read-only-admin, dc=example, dc=com (') **************************************************************            (managerPassword) ************************************************************:' (password) ****************************************************************************************************** ()          (search) ************************************************************:            (base) ************************************************************:" (dc=example, dc=com

**************************************

         (groups) ************************************************************:            (enabled) ************************************************************: (true)            (base) ************************************************************:" (dc=example, dc=com

**************************************

Configure KafkaHQ groups and Ldap groups

(kafkahq) ************************************************************:    (security) ************************************************************:      (groups) ************************************************************:        (topic-reader) *************************************************************: (**************************************************************** (#) ************************************************************ (Group name) ************************************************************          (roles) ************************************************************:#roles for the group           - topic / read         (attributes) **********************************************************:            (******************************************** (#) ********************************************************** (Regexp to filter topic available for group           (topics-filter-regexp) ************************************************************: " (test) **********************************************************. reader. ***********************************       topic-writer (**************************************************************:          (roles) ************************************************************:           - topic / read          - topic / insert          - topic / delete

          - topic / config / update

         (attributes) **********************************************************:            (topics-filter-regexp) ************************************************************: " test. *

     (ldap) **************************************************************:        (group) ************************************************************:          (mathematicians) *************************************************************:            (groups) ************************************************************:             - (topic-reader)          (scientists) ************************************************************:            (groups) ************************************************************:             - (topic-reader)             - (topic-writer) ************************************************************ (************************************************Server****************kafkahq.server.base-path: if behind a reverse proxy, path to kafkahq with trailing slash (optional). Example: kafkahq is behind a reverse proxy with url http: // my-server / kafkahq, set base-path: "/ kafkahq /". Not needed if you're behind a reverse proxy with subdomain (http: //kafkahq.my-server/(**********************************************************************************************************************kafkahq.clients-defaults. {{admin | producer | consumer}}. properties: default configuration for admin producer or consumer. All properties from Kafka documentationis available . (****************************************************************************************************Micronaut configuration

Since KafkaHQ is based on (Micronaut) , you can customize configurations (server port, ssl, ...) withMicronaut configuration. More information can be found on (Micronaut documentation)

(****************************************************Docker

KafkaHQ docker image support 3 environment variables to handle configuraiton:****************KAFKAHQ_CONFIGURATION: a string that contains the full configuration in yml that will be written on /app/configuration.yml on container. ********************************** (MICRONAUT_APPLICATION_JSON) ************************************: a string that contains the full configuration in JSON format ********************************** (MICRONAUT_CONFIG_FILES) **********************************: a path to a configuration file on container. Default path is / app / application.yml (*********************************************************************************************************************************How to mount configuration file

Take care when you mount configuration files to not remove kafkahq files located on / app. You need to explicitely mount the / app / application.ymland not mount the / app directory. This will remove the KafkaHQ binnaries and give you this error:/usr/local/bin/docker-entrypoint.sh: 9: exec: ./kafkahq: not found

(volumeMounts) ************************************************************: - (mountPath) ************************************************************: **************************************************** (/ app / application.yml)    (subPath) **************************************************************: the application.yml

   (name) ************************************************************: config   (readOnly) ************************************************************: (true)

********************************************************************************************************************

Several monitoring endpoint is enabled by default. You can disabled it or restrict access only for authenticated users following micronaut configuration below.****************/ info(Info Endpoint) with git status informations.

/ health

(Health Endpoint) ******************************************** / loggers Loggers Endpoint / metrics

(Metrics Endpoint) ******************************************** / prometheusPrometheus Endpoint**************

************************ () ****************************************

Development Environment********************

Early dev image

You can have access to last feature / bug fix with docker dev image automatically build on tag (dev)

docker pull tchiotludo / kafkahq: dev

The dev jar is not publishing on GitHub, you have 2 solutions to have the (dev) jar:

Get it from docker image

docker pull tchiotludo / kafkahq: dev docker run --rm --name=kafkahq -it tchiotludo / kafkahq: dev docker cp kafkahq: /app/kafkahq.jar************************************ (**************************************************

Or build it with a./ gradlew shadowJar, the jar will be located here build / libs / kafkahq - *. jar(**************************************************************************************************************************************

************

Development Server

A docker-compose is provide to start a development environnement. Just install docker & docker-compose, clone the repository and issue a simpledocker-compose -f docker-compose-dev.yml upto start a dev server. Dev server is a java server & webpack-dev-server with live reload.

******************************************************************************************Who's using KafkaHQ**************** Adeo

******************************************************************************************** (Auchan Retail******************************************************************************************** La Redoute

************************************************************************************** Leroy Merlin **************************************************************************************************************************************** (Nuxeo) ****************************************************************************************************************************** GetYourGuide (**************************************************

Many thanks to:

(****************************************************************************************

(JetBrains) ******** for their free OpenSource license.

************************************************************************************************************************************

Apache 2.0 ©tchiotludo

   (************************************************************************************************************  (**************************************************************************************************************

(**************************************************************************************************************** (Read More) ************************************************************************************************************** (****************************************************************************************************************