Home › Guides › What Is Apache Kafka
Tech Explained · 2026What Is Apache Kafka? How Topics, Partitions and Consumer Groups Actually Work in 2026
Apache Kafka is an open source event streaming platform that stores records in an append only log split into partitions, so many producers can write while many consumer groups read the same data independently. Kafka keeps every record for a configured window, 168 hours by default, instead of deleting it once somebody has read it.
- Kafka is a log, not a queue. Reading a record does not remove it, so two teams consume the same stream without coordinating.
- Partitions carry parallelism and ordering. Order holds inside a partition and nowhere else, so your message key matters more than broker tuning.
- A consumer group tops out at the partition count. Each partition goes to one member, so extra consumers idle.
- ZooKeeper is gone. Kafka 4.x is KRaft only, and the project blog announced 4.3.1 on 25 June 2026.
Your nightly batch finished at 3am. The claims team heard about a suspicious motor claim at 9:15am, when the dashboard refreshed. Six hours of latency is fine for a monthly report and worthless for a decision somebody makes before a payout clears. Kafka is what teams run when they close that gap.
What Is Apache Kafka, and What Problem Does It Actually Solve
Picture a mid-size Indian insurer with a two person data team. The claims application writes to Oracle, a fraud model reads a nightly extract, a Power BI dataset refreshes off a view, and an email service hangs off a database trigger nobody wants to touch. Four consumers, four bespoke integrations, and each new one means another pipeline negotiated with the application team.
Kafka inverts that. The claims application publishes one record per event to a topic and stops caring who reads it. Fraud model, warehouse loader and notifier each read that topic at their own pace.
The mechanism is boring and important. Kafka stores records in an append only log and gives each a permanent sequential number called an offset. A message queue deletes a message when a consumer acknowledges it. Kafka keeps it until retention expires, which defaults to 168 hours per the documented broker setting log.retention.hours, and each consumer tracks its own position. That is why replaying yesterday afternoon after a bad transformation is a normal Tuesday, not a backup restore.
Three numbers for your first Kafka project
Version, default retention and the realistic small-scale bill
Kafka release announcements, documented broker defaults and Confluent's pricing page. Checked 1 October 2026.
Kafka Topics and Partitions: What Is Apache Kafka Doing Under the Hood
A topic is a name, like claims. It holds nothing. Its partitions hold the data, and a partition is a file on disk that only ever gets appended to. Your producer sends a record, the client hashes the key, and that hash picks the partition. Same key, same partition, every time.
From this falls the rule that governs Kafka data modelling: records are ordered within a partition and have no defined order across partitions. Key every claim event by policy number and all events for POL-4471 land in one partition in the order they happened. Key by a random UUID and you have thrown that away permanently.
One topic, three partitions, two independent consumer groups
Offsets belong to the consumer groups, not the topic, so neither reader blocks the other
Offsets illustrative; structure follows the documented Kafka consumer group model. Checked 1 October 2026.
Where brokers and KRaft fit
Partitions live on brokers and are replicated across them, so losing a machine does not lose data. One replica leads and takes all reads and writes for its partition. Deciding who leads was ZooKeeper's job until Kafka 4.0 removed it entirely; metadata now lives in Kafka itself through KRaft, its Raft based controller quorum. Every ZooKeeper tutorial you find describes software you cannot run on a current broker.
Apache Kafka Tutorial for Beginners: Produce and Consume in 10 Minutes
Reading about offsets does less for you than watching one move. The official quickstart image runs broker and controller in one container, no config file:
docker run -d --name kafka -p 9092:9092 apache/kafka:4.3.1
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create --topic claims --partitions 3 --replication-factor 1Note that you had to ask for three partitions. The broker default num.partitions is 1, so a topic auto created by a careless producer gets one partition and a hard ceiling of one consumer. Now write and read with the Python client:
from confluent_kafka import Producer, Consumer
p = Producer({"bootstrap.servers": "localhost:9092"})
for claim_id in (9912, 9913, 9914):
p.produce("claims", key="POL-4471",
value='{"claim_id": %d, "amount": 48000}' % claim_id)
p.flush()
c = Consumer({"bootstrap.servers": "localhost:9092",
"group.id": "fraud-scorer",
"auto.offset.reset": "earliest"})
c.subscribe(["claims"])
while True:
msg = c.poll(1.0)
if msg is None:
continue
if msg.error():
print("error:", msg.error())
continue
print(msg.partition(), msg.offset(), msg.key(), msg.value())All three records print the same partition number because they share a key. Change the key to the claim id and they scatter. That is the entire ordering lesson in one edit.
Stop the consumer and run it again: it resumes instead of replaying, because the group committed its offsets. Run kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group fraud-scorer for a row per partition with CURRENT-OFFSET, LOG-END-OFFSET and LAG. Lag is the end of the log minus where your group committed, and it is the most useful number in Kafka operations. Lag growing while traffic is flat means your consumer is dying and restarting.
Also read: Docker Tutorial for Beginners 2026 if docker run is the unfamiliar part rather than the Kafka line.
How a Kafka Consumer Group Divides the Work
A Kafka consumer group is a set of processes sharing one group.id, and Kafka assigns each partition to exactly one member. Three partitions and two consumers means one handles two. Three partitions and five means two sit idle forever, holding a connection and doing nothing.
That assignment is the knob you turn when a stream falls behind, and the partition count is its ceiling. You cannot scale past it, and you cannot reduce partitions on an existing topic. This is the one Kafka decision genuinely expensive to get wrong, so over provision modestly at creation. Kafka 4.2 added share groups (KIP-932) for work that is not a stream, trading per partition ordering for per record acknowledgement, but learn classic consumer groups first.
The Settings That Decide Whether You Lose Messages
Durability is earned by configuration, and the defaults moved in your favour. Since Kafka 3.0 and KIP-679 the producer enables idempotence by default and acks defaults to all, so the leader waits for the in sync replicas before confirming a write. If you learned on a 2.x cluster where acks=1 was the default, that removed a common data loss path.
| Setting | Default | What it controls | Production choice |
|---|---|---|---|
acks |
all since Kafka 3.0 |
Replicas confirming a write before the producer returns | Leave at all
|
enable.idempotence |
On since Kafka 3.0 | Prevents duplicates from producer retries | Leave it on |
min.insync.replicas |
1 | Replicas alive for an acks=all write |
2 on a 3 broker cluster |
replication.factor |
Set per topic | Copies of each partition across brokers | 3 in production, 1 locally |
log.retention.hours |
168 | How long records stay readable | Your worst case replay window |
num.partitions |
1 | Partitions for an auto created topic | Create topics explicitly |
The combination that catches people is acks=all with min.insync.replicas=1. It reads as safe and is not: one surviving replica satisfies the write, so a badly timed broker loss still costs you records. Set it to 2 on a three broker cluster. Replica placement is cloud platform work, and the part most self taught engineers skip; both the AWS Solutions Architect and DevOps track and its Azure AZ-305 and AZ-400 equivalent cover availability zones and quorum.
What Kafka Actually Costs: A Worked Monthly Bill
Kafka is free. Running it is not, so here is the arithmetic for our insurer on Confluent Cloud's Basic tier at the unit prices published today: 2 GB of claim events a day, seven day retention, two consumer groups each reading everything. That is 60 GB written a month, 120 GB read, roughly 14 GB in retention.
Ingress is 60 GB at $0.05/GB, so $3.00. Egress is 120 GB at the same rate, so $6.00. Storage is about 14 GB at $0.08/GB-month, so $1.12. Basic starts at $0 a month with the first eCKU free, then $0.14/eCKU-hour. Total: around ten dollars a month. Confluent bills in binary gigabytes, so an invoice will not match a decimal calculation exactly, and Basic caps out at 1,500 partitions and 5 TB.
Where the money goes on a 2 GB a day stream
Reads cost more than writes, so a new consumer group is a bigger budget event than a traffic increase
From Confluent Cloud Basic tier unit prices on confluent.com/pricing, checked 1 October 2026. Bars proportional to values.
When Kafka Is the Wrong Answer
Most teams that install Kafka do not need Kafka. With one producer, one consumer and a few thousand messages an hour, a Postgres table with a status column will serve you for years and anybody can debug it at 2am. Kafka earns its operational cost when you have several independent consumers, replay requirements, or throughput a database cannot absorb.
It is also harder to operate than its tutorials suggest. Partition rebalances, consumer group stalls, a disk filling because somebody set infinite retention on a chatty topic: for a two person team, a managed cluster beats becoming Kafka administrators.
Kafka vs RabbitMQ and the managed alternatives
| What you need | Kafka | RabbitMQ | Managed endpoint |
|---|---|---|---|
| Replay last week's records | Built in, within retention | No, they leave on acknowledgement | Yes, within retention |
| Many unrelated consumers of one stream | Natural fit | Via fan out exchanges | Natural fit |
| Per message routing and priorities | Weak, filter in the consumer | Strong, its design centre | Weak |
| Strict ordering per entity | Yes, by key within a partition | Only within a single queue | Yes, by partition key |
| Who operates it | You or a vendor | You or a vendor | The platform |
That third column matters more in India than the arguments allow, because most enterprise data teams here already stand on a cloud data platform. Microsoft Fabric is the common case: a Fabric Eventstream custom endpoint exposes an Apache Kafka endpoint, documented on Microsoft Learn with bootstrap.servers, security.protocol=SASL_SSL and sasl.mechanism=PLAIN. Your client connects unchanged.
What Usually Goes Wrong in Your First Month
The four mistakes that cost the most to undo
Cheap to avoid at topic creation, expensive once data is flowing
Letting a topic auto create
It appears with one partition because num.partitions defaults to 1, so the consumer group never scales past one process. Adding partitions later reshuffles key placement.
No key, then wanting order
Null keys spread across partitions, so events for one policy land anywhere and sequence is gone. There is no consumer side fix: you republish with a proper key and backfill.
Key by entity idTreating lag as a dashboard
Lag tells you a consumer is dead, so it belongs in an alert with a threshold, not a weekly review. Growing lag on flat traffic means a crash loop, not slow code.
Alert on lag, not CPUInfinite retention on a chatty topic
Somebody sets retention to minus one on a debug topic to be safe, the disk fills, and the broker stops accepting writes for every topic on it.
Set retention per topicDefaults from the documented Kafka broker configuration. Checked 1 October 2026.
Your consumer processes a record, does its work, then crashes before committing its offset. On restart it reads that record again. Kafka is at least once by default, so your consumer must tolerate seeing a record twice. The fix is not a setting: make the write idempotent, upserting on the claim id instead of inserting. Teams that skip this find out through duplicated rows in a report somebody trusts.
Where Kafka Shows Up in an Indian Data Engineering Job
Streaming is a line item inside data engineering and platform roles, not a specialist title. Feeding the warehouse in near real time, where an Eventstream or Kafka topic lands claim events into a lakehouse table, is squarely the DP-700 skill set, and 360DT's live Microsoft Fabric data engineer course builds these pipelines in class rather than describing them. Keeping a dashboard current makes refresh schedules matter more than visual design, which puts the PL-300 Power BI and SQL track closer. Serving models and agents is why streaming now appears in both MLOps Engineer (AI-300) and AI engineering and RAG syllabi; the certifications overview maps which covers which.
Also read: Apache Airflow Tutorial 2026 for the batch half of this picture, and What Is Apache Iceberg in 2026 for where streamed records land.
If I were you I would run the commands above tonight, watch an offset move, then key the same records two ways and prove to myself what breaks. Partitioning and ordering are the only two Kafka ideas hard to absorb by reading. Then I would not build a self managed cluster: use your platform's Kafka endpoint and spend the learning time on pipelines and modelling, which is what interviewers ask about. Broker administration is a real skill and not the one getting data engineers hired in India in 2026. For that work taught in order, the Microsoft Fabric data engineer course is the direct route, and a free webinar tells you in an hour whether the pace suits you.
Build streaming and batch pipelines in Microsoft Fabric, live with a mentor
A live weekend program over 8 weeks covering Microsoft Fabric data engineering for the DP-700 certification, with DP-900 fundamentals included. Hands on projects, mentor support and placement guidance are part of the course.
Explore the course
Related guides
- Data Engineer Roadmap 2026 sequences streaming with the skills that come before it.
- Microsoft Fabric vs Databricks in 2026 settles which platform, and so which Kafka endpoint, you use.
- Data Engineering Projects for 2026 has a streaming build with a dataset and cost estimate.
- Data Engineer Salary in India 2026 shows what these skills are advertised at by experience band.
- DP-700 Certification Guide 2026 covers the exam that formally tests Fabric real time intelligence.
Frequently asked questions
What is Apache Kafka in simple terms?
Apache Kafka is a shared, append only log that applications write events into and others read from at their own pace. It is a notebook, not a postbox: writing a page does not remove it. Records stay readable until retention expires, 168 hours by default.
Is Kafka a message queue or a database?
Neither. A queue deletes a message once acknowledged; Kafka keeps it until retention expires, so you can replay. A database queries by any field; Kafka reads sequentially from an offset. Current state belongs in a database your consumer writes into.
How many partitions should a Kafka topic have?
Enough for the consumer processes you might realistically run, since the partition count is the hard ceiling on a consumer group. Six is a sane start. You can add partitions later but never remove them, and adding them changes which partition a key lands in.
What is a Kafka consumer group and how does it balance load?
A set of processes sharing one group.id. Kafka assigns every partition to one member, so work divides by partition. Add more consumers than partitions and the extras idle. Check the split with kafka-consumer-groups.sh --describe --group your-group.
Is Kafka free to use, and what does managed Kafka cost?
Kafka is open source and free to run. Confluent Cloud's Basic tier publishes $0.05/GB in and out and $0.08/GB-month of storage, the first eCKU free then $0.14/eCKU-hour, and new accounts get $400 of credit valid 30 days. The example above lands near ten dollars a month at 2 GB a day.
What is the difference between Kafka and RabbitMQ?
Kafka retains records so independent consumers can read and replay the same stream. RabbitMQ routes a message to one consumer and removes it, with strong priority and routing support. Pick Kafka when several systems need the same events; RabbitMQ for dispatching discrete tasks.
Do I still need Kafka if my company already uses Microsoft Fabric?
Usually not as a cluster you operate. A Fabric Eventstream custom endpoint exposes an Apache Kafka endpoint, documented on Microsoft Learn with bootstrap.servers, security.protocol=SASL_SSL and sasl.mechanism=PLAIN, so your client code works unchanged.
About this guide. 360 Digital Transformation is an Anthropic Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 1 October 2026.
Microsoft Fabric
Python
