Cloud provider's fixed topic settings

[AlexanderZagaynov]

I'm trying to run bundle exec karafka-web install while using "Serverless" Redpanda's Cloud (link).

But getting a (vague) error: karafka-rdkafka-0.17.6/lib/rdkafka/error.rb:86:in `validate!': Broker: Configuration is invalid (invalid_config) (Rdkafka::RdkafkaError)

After some puts debugging (outputting error_string and result_name here), I saw that broker returns 'retention.ms 3600000 is too small, must be at least 24h' and 'cannot set "segment.ms"' kind of errors, while trying to create topics like 'karafka_consumers_metrics'.

---

Expected behavior

Karafka-Web should be able to create needed topics on the cloud service.

Actual behavior

Obscure Broker Configuration error(s).

Steps to reproduce the problem

karafka.rb connection lines:

class KarafkaApp < Karafka::App
  setup do |config|
    config.kafka = {
      'bootstrap.servers': ENV['KARAFKA_SERVERS'] || '127.0.0.1:9092',
      'security.protocol': 'SASL_SSL',
      'sasl.mechanism': 'SCRAM-SHA-256',
      'sasl.username': ENV.fetch('KARAFKA_USERNAME'),
      'sasl.password': ENV.fetch('KARAFKA_PASSWORD'),
    }
    # config.client_id = ENV.fetch('KARAFKA_CLUSTER')
    config.concurrency = (ENV['KARAFKA_CONCURRENCY'].presence || 5).to_i
  end

Your setup details

Please provide kafka version and the output of karafka info or bundle exec karafka info if using Bundler.

❯ bundle exec karafka info
I, [2024-09-22T15:04:54.255195 #819318]  INFO -- : 
@@@                                             @@@@@  @@@
@@@                                            @@@     @@@
@@@  @@@    @@@@@@@@@   @@@ @@@   @@@@@@@@@  @@@@@@@@  @@@  @@@@   @@@@@@@@@
@@@@@@     @@@    @@@   @@@@@    @@@    @@@    @@@     @@@@@@@    @@@    @@@
@@@@@@@    @@@    @@@   @@@     @@@@    @@@    @@@     @@@@@@@    @@@    @@@
@@@  @@@@  @@@@@@@@@@   @@@      @@@@@@@@@@    @@@     @@@  @@@@   @@@@@@@@@@


I, [2024-09-22T15:04:54.255482 #819318]  INFO -- : Karafka version: 2.4.12
Ruby version: ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-linux]
Rdkafka version: 0.17.6
Consumer groups count: 2
Subscription groups count: 2
Workers count: 5
Application client id: karafka
Boot file: <...>/karafka.rb
Environment: development
License: LGPL-3.0
I, [2024-09-22T15:04:54.255600 #819318]  INFO -- : [waterdrop-806ff2150f53] Closing producer
I, [2024-09-22T15:04:54.255681 #819318]  INFO -- : [waterdrop-806ff2150f53] Closing producer took 0.06 ms

Workaround

# before the application class

module Karafka::Admin
  module ClassMethodPatches
    def create_topic(*args)
      options = args.extract_options!
      retention_ms = options.fetch(:'retention.ms', nil).to_i
      options[:'retention.ms'] = [retention_ms, 24.hours.in_seconds * 1_000].max
      options = options.except(*%i[segment.ms segment.bytes])
      super(*args, options)
    end
  end
  class << self
    prepend ClassMethodPatches
  end
end

[mensfeld]

redpanda has different limitations than kafka. I will reconsider the defaults to have them work in both cases. Your workaround is pointless if you use declarative topics (https://karafka.io/docs/Declarative-Topics/) where you can define those topics accordingly to the documentation: https://karafka.io/docs/Web-UI-Getting-Started/#manual-web-ui-topics-management

just align this to the minimum redpanda accepts and bundle exec karafka topics migrate before bunde exec karafka-web install.

I will keep the issue open for now not to forget to revisit defaults.