{"id":3715,"date":"2025-11-24T10:00:13","date_gmt":"2025-11-24T15:00:13","guid":{"rendered":"https:\/\/www.mymiller.name\/wordpress\/?p=3715"},"modified":"2025-11-24T10:00:13","modified_gmt":"2025-11-24T15:00:13","slug":"optimizing-spring-kafka-message-delivery-compression-batching-and-delays","status":"publish","type":"post","link":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/optimizing-spring-kafka-message-delivery-compression-batching-and-delays\/","title":{"rendered":"Optimizing Spring Kafka Message Delivery: Compression, Batching, and Delays"},"content":{"rendered":"\n<div class=\"wp-block-jetpack-markdown\"><p>Spring Kafka provides a powerful framework for interacting with Apache Kafka, but efficient message delivery requires some fine-tuning. Here\u2019s how to optimize your Spring Kafka producer using compression, batching, and small delays.<\/p>\n<p><strong>1. Compression<\/strong><\/p>\n<p>Compressing messages before sending them to Kafka significantly reduces the overall data size, leading to:<\/p>\n<ul>\n<li><strong>Lower bandwidth usage:<\/strong> Less data to transfer means reduced network costs and faster transmission.<\/li>\n<li><strong>Improved throughput:<\/strong> Smaller messages are processed faster by both producers and consumers.<\/li>\n<li><strong>Increased storage capacity:<\/strong>  Compressed messages take up less space on Kafka brokers.<\/li>\n<\/ul>\n<p>Spring Kafka supports several compression codecs like <code>gzip<\/code>, <code>snappy<\/code>, and <code>lz4<\/code>. You can configure the desired codec using the <code>compression.type<\/code> property in your producer configuration:<\/p>\n<pre><code class=\"language-properties\">spring.kafka.producer.compression-type=snappy\n<\/code><\/pre>\n<p><strong>2. Batching<\/strong><\/p>\n<p>Instead of sending individual messages, batching combines multiple messages into a single batch, reducing overhead and improving efficiency. This leads to:<\/p>\n<ul>\n<li><strong>Fewer network requests:<\/strong>  Sending one large batch is more efficient than multiple small requests.<\/li>\n<li><strong>Improved throughput:<\/strong> Kafka brokers can handle batches more efficiently than individual messages.<\/li>\n<\/ul>\n<p>Configure batching using the following producer properties:<\/p>\n<ul>\n<li><code>batch.size<\/code>:  Maximum size of a batch in bytes.<\/li>\n<li><code>linger.ms<\/code>:  Maximum time to wait for additional messages to be added to a batch before sending it.<\/li>\n<\/ul>\n<pre><code class=\"language-properties\">spring.kafka.producer.batch-size=16384 \nspring.kafka.producer.linger-ms=10\n<\/code><\/pre>\n<p><strong>3. Small Delays (linger.ms)<\/strong><\/p>\n<p>The <code>linger.ms<\/code> property introduces a small delay before sending a batch. This allows more messages to be added to the batch, increasing its size and improving efficiency. However, be aware that increasing <code>linger.ms<\/code> too much can introduce latency.<\/p>\n<p><strong>4. Programmatic Configuration<\/strong><\/p>\n<p>While configuring through properties is convenient, you can also achieve the same programmatically. This gives you more dynamic control over your producer settings. Here\u2019s an example:<\/p>\n<pre><code class=\"language-java\">import org.apache.kafka.clients.producer.ProducerConfig;\nimport org.springframework.kafka.core.DefaultKafkaProducerFactory;\nimport org.springframework.kafka.core.KafkaTemplate;\nimport org.springframework.kafka.core.ProducerFactory;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\npublic class KafkaProducerConfig {\n\n    public KafkaTemplate&lt;String, String&gt; kafkaTemplate() {\n        Map&lt;String, Object&gt; configProps = new HashMap&lt;&gt;();\n        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, &quot;localhost:9092&quot;);\n        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);\n        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);\n        configProps.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, &quot;snappy&quot;);\n        configProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);\n        configProps.put(ProducerConfig.LINGER_MS_CONFIG, 10);\n\n        ProducerFactory&lt;String, String&gt; producerFactory = new DefaultKafkaProducerFactory&lt;&gt;(configProps);\n        return new KafkaTemplate&lt;&gt;(producerFactory);\n    }\n}\n<\/code><\/pre>\n<p>This code snippet demonstrates how to programmatically set the bootstrap servers, serializers, compression type, batch size, and linger time. You can further customize this based on your application\u2019s requirements.<\/p>\n<p><strong>Finding the Right Balance<\/strong><\/p>\n<p>The optimal configuration for compression, batching, and delays depends on your specific needs and message characteristics. Experiment with different values to find the sweet spot between throughput and latency.<\/p>\n<p><strong>Monitoring and Fine-tuning<\/strong><\/p>\n<p>Continuously monitor your Kafka producer performance using tools like Burrow or Kafka Manager to identify bottlenecks and fine-tune your configuration. Pay attention to metrics like:<\/p>\n<ul>\n<li><strong>Produce latency:<\/strong> Time taken to send a message.<\/li>\n<li><strong>Throughput:<\/strong> Number of messages sent per second.<\/li>\n<li><strong>Broker disk usage:<\/strong> Monitor storage space on your Kafka brokers.<\/li>\n<\/ul>\n<p><strong>Beyond the Basics<\/strong><\/p>\n<p>For further optimization, consider:<\/p>\n<ul>\n<li><strong>Increasing the number of partitions:<\/strong> This allows for parallel processing of messages.<\/li>\n<li><strong>Using acks=all:<\/strong> Ensures messages are replicated to all in-sync replicas for higher durability.<\/li>\n<li><strong>Tuning broker settings:<\/strong> Optimize Kafka broker configurations for your specific hardware and workload.<\/li>\n<\/ul>\n<p>By implementing these optimization techniques, you can significantly enhance the performance and efficiency of your Spring Kafka message delivery, leading to a more robust and scalable messaging infrastructure.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":3716,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[438],"tags":[69,319],"series":[],"class_list":["post-3715","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spring_messaging","tag-java-2","tag-spring"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/09\/management-1137648_1280-jpg.avif","jetpack-related-posts":[{"id":3842,"url":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/taming-the-stream-effective-unit-testing-with-kafka-in-spring-boot\/","url_meta":{"origin":3715,"position":0},"title":"Taming the Stream: Effective Unit Testing with Kafka in Spring Boot","author":"Jeffery Miller","date":"December 24, 2025","format":false,"excerpt":"Kafka\u2019s asynchronous, distributed nature introduces unique challenges to testing. Unlike traditional synchronous systems, testing Kafka interactions requires verifying message production, consumption, and handling potential asynchronous delays. This article explores strategies for robust unit testing of Kafka components within a Spring Boot application. Understanding the Testing Landscape Before diving into specifics,\u2026","rel":"","context":"In &quot;Spring Messaging&quot;","block_context":{"text":"Spring Messaging","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring_messaging\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/intro-7400243_640.jpg?fit=640%2C334&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/intro-7400243_640.jpg?fit=640%2C334&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/intro-7400243_640.jpg?fit=640%2C334&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":3890,"url":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/spring-cloud-stream\/","url_meta":{"origin":3715,"position":1},"title":"Spring Cloud Stream","author":"Jeffery Miller","date":"December 24, 2025","format":false,"excerpt":"Spring Cloud Stream is a framework for building highly scalable, event-driven microservices that are connected by a shared messaging system. In simple terms, it's a powerful tool that takes away the complexity of communicating with message brokers like RabbitMQ or Apache Kafka, allowing you to focus purely on your application's\u2026","rel":"","context":"In &quot;Spring Messaging&quot;","block_context":{"text":"Spring Messaging","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring_messaging\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2023\/11\/network-3152677_640.jpg?fit=640%2C427&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2023\/11\/network-3152677_640.jpg?fit=640%2C427&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2023\/11\/network-3152677_640.jpg?fit=640%2C427&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":3878,"url":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/building-robust-kafka-applications-with-spring-boot-and-avro-schema-registry\/","url_meta":{"origin":3715,"position":2},"title":"Building Robust Kafka Applications with Spring Boot, and Avro Schema Registry","author":"Jeffery Miller","date":"November 24, 2025","format":false,"excerpt":"As a software architect, designing solutions that are scalable, maintainable, and resilient is paramount. In the world of event-driven architectures, Apache Kafka has become a cornerstone for high-throughput, low-latency data streaming. However, simply sending raw bytes over Kafka topics can lead to data inconsistency and make future evolution a nightmare.\u2026","rel":"","context":"In &quot;Spring Messaging&quot;","block_context":{"text":"Spring Messaging","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring_messaging\/"},"img":{"alt_text":"","src":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/ai-generated-7947638_1280.avif","width":350,"height":200,"srcset":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/ai-generated-7947638_1280.avif 1x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/ai-generated-7947638_1280.avif 1.5x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/ai-generated-7947638_1280.avif 2x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/ai-generated-7947638_1280.avif 3x"},"classes":[]},{"id":3884,"url":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/mastering-polymorphic-data-in-spring-kafka-with-avro-union-types\/","url_meta":{"origin":3715,"position":3},"title":"Mastering Polymorphic Data in Spring Kafka with Avro Union Types","author":"Jeffery Miller","date":"November 24, 2025","format":false,"excerpt":"As a software architect, designing robust, scalable, and adaptable distributed systems is a constant pursuit. When working with Apache Kafka, a common challenge arises: how do you send messages that, while adhering to a generic wrapper, can carry different types of payloads based on the specific event or context? Imagine\u2026","rel":"","context":"In &quot;Spring Messaging&quot;","block_context":{"text":"Spring Messaging","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring_messaging\/"},"img":{"alt_text":"","src":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/plastic-5527530_1280.avif","width":350,"height":200,"srcset":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/plastic-5527530_1280.avif 1x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/plastic-5527530_1280.avif 1.5x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/plastic-5527530_1280.avif 2x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/06\/plastic-5527530_1280.avif 3x"},"classes":[]},{"id":3844,"url":"https:\/\/www.mymiller.name\/wordpress\/spring_messaging\/the-power-of-kafka-connect\/","url_meta":{"origin":3715,"position":4},"title":"The Power of Kafka Connect","author":"Jeffery Miller","date":"December 24, 2025","format":false,"excerpt":"Kafka Connect is a powerful framework for streaming data between Kafka and other systems in a scalable and reliable way. Connectors handle the complexities of data integration, allowing you to focus on your core application logic. Sink Connectors are used to export data from Kafka to other systems, and in\u2026","rel":"","context":"In &quot;Spring Messaging&quot;","block_context":{"text":"Spring Messaging","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring_messaging\/"},"img":{"alt_text":"","src":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/04\/ai-generated-8131434_1280-png.avif","width":350,"height":200,"srcset":"https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/04\/ai-generated-8131434_1280-png.avif 1x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/04\/ai-generated-8131434_1280-png.avif 1.5x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/04\/ai-generated-8131434_1280-png.avif 2x, https:\/\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2025\/04\/ai-generated-8131434_1280-png.avif 3x"},"classes":[]},{"id":3548,"url":"https:\/\/www.mymiller.name\/wordpress\/spring\/beyond-the-basics-optimizing-your-spring-boot-applications-for-performance-fine-tune-your-application-for-speed-and-efficiency\/","url_meta":{"origin":3715,"position":5},"title":"Beyond the Basics: Optimizing Your Spring Boot Applications for Performance &#8211; Fine-tune your application for speed and efficiency.","author":"Jeffery Miller","date":"November 25, 2025","format":false,"excerpt":"Absolutely! Here\u2019s a blog article on optimizing Spring Boot applications, aimed at those who already have some experience with the framework: Beyond the Basics: Optimizing Your Spring Boot Applications for Performance Spring Boot is a fantastic framework for rapidly building production-ready applications. However, as your application grows and handles more\u2026","rel":"","context":"In &quot;Spring&quot;","block_context":{"text":"Spring","link":"https:\/\/www.mymiller.name\/wordpress\/category\/spring\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/ai-generated-8619544_1280.jpg?fit=1200%2C685&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/ai-generated-8619544_1280.jpg?fit=1200%2C685&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/ai-generated-8619544_1280.jpg?fit=1200%2C685&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/ai-generated-8619544_1280.jpg?fit=1200%2C685&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.mymiller.name\/wordpress\/wp-content\/uploads\/2024\/06\/ai-generated-8619544_1280.jpg?fit=1200%2C685&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/posts\/3715","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/comments?post=3715"}],"version-history":[{"count":1,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/posts\/3715\/revisions"}],"predecessor-version":[{"id":3717,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/posts\/3715\/revisions\/3717"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/media\/3716"}],"wp:attachment":[{"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/media?parent=3715"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/categories?post=3715"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/tags?post=3715"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/www.mymiller.name\/wordpress\/wp-json\/wp\/v2\/series?post=3715"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}