Kafka Deep Dive w/ a Ex-Meta Staff Engineer

Evan, a former Meta staff engineer and co-founder of Hello Interview, provides a comprehensive guide to using Kafka in system design interviews, emphasizing its significance as a top technology for event streaming and message queuing, used by 80% of Fortune 100 companies. He explains Kafka's architecture, including producers, consumers, brokers, partitions, and topics, and illustrates its application through a World Cup example. Evan highlights Kafka's scalability, fault tolerance, error handling, and performance optimization strategies, advising on effective partitioning and retention policies. He also promotes Hello Interview's resources for further learning and mock interviews.

• CFA is an event streaming platform used as a message queue or stream processing system.
• It is widely adopted by Fortune 100 companies and is considered essential for system design interviews.
• The goal is to understand when to use CFA and discuss trade-offs in interviews.

"CFA is an event streaming platform and it can be used either as a message queue or as a stream processing system."

• CFA's dual functionality as a message queue and stream processing system makes it versatile for different use cases.

• A hypothetical website provides real-time updates for World Cup games.
• Events like goals, bookings, and substitutions are placed on a queue by a producer.
• A consumer reads events from the queue and updates the website.

"We can imagine that we're running a website that's going to provide real-time event updates on each of the games."

• The example illustrates the basic producer-consumer model in event streaming.

• When events increase due to more games, the solution is to add more servers.
• This introduces the challenge of maintaining event order across multiple queues.

"The solution is one that you've probably read plenty about as you've been studying for system design, and that's to scale horizontally."

• Horizontal scaling involves adding more servers to handle increased load but requires careful management of event order.

• Events are distributed based on the game, ensuring order within each game.
• This strategy prevents disorder when distributing events across multiple servers.

"We can distribute the items in our queue based on the game that they're associated with."

• Distributing events by game ensures that events remain in order within each game's context.

• When a single consumer can't keep up, more consumers are added.
• Consumer groups ensure each event is processed by only one consumer, preventing duplicate processing.

"With a consumer group, each event is guaranteed to only be processed by one consumer in the group."

• Consumer groups allow multiple consumers to process events without duplication, enhancing scalability.

• Topics segregate events by type (e.g., soccer vs. basketball).
• Producers and consumers specify topics to ensure relevant events are processed.

"Each event is going to be associated with a topic and consumers will subscribe to a specific topic."

• Topics provide logical separation of event streams, allowing targeted processing by consumers.

• Brokers are servers (physical or virtual) that hold queues, known as partitions in Kafka.

"A broker is simply a server; Kafka clusters are made up of these brokers."

• Brokers form the backbone of a Kafka cluster, managing event storage and retrieval.

• Partitions are ordered, immutable sequences of messages, functioning like log files.

"Partitions are an ordered immutable sequence of messages that we append to."

• Partitions ensure message order and are essential for scaling within Kafka.

• Topics are logical groupings of partitions, organizing data, while partitions handle scaling.

"A topic is just a logical grouping of messages, where a partition is a physical grouping."

• Understanding the distinction helps in designing efficient Kafka systems.

• Producers write messages to topics, and consumers read them.

"Producers write the messages or records to topics, consumers read them."

• The producer-consumer model is central to Kafka's operation, facilitating data flow.

• A message or record consists of a key, value, timestamp, and headers.
• Producers publish messages, which are then consumed by consumers.

"A message or a record in Kafka is made up of four attributes: key, value, timestamp, and headers."

• The structure of a message ensures it carries essential data and metadata for processing.

These notes encapsulate the core ideas and functionalities of CFA (Kafka) in the context of system design interviews, providing a comprehensive understanding of its components, challenges, and solutions.

• Kafka enables efficient message processing by utilizing keys and partitions.
• Messages are assigned keys, which determine the partition they go to.
• If no key is specified, messages are randomly assigned to partitions.
• The hashing of keys using a hash function ensures consistent partition assignment.

"The first with a key of one or key1 excuse me and a value of hello cfco with key and the second key is key2 and another message with a different key so now both of those message messages are on our queue."

• Messages are assigned keys, and each key directs the message to a specific partition in the queue.

"In our motivating example, this partition key was like the ID or the name of the game Argentina versus Brazil for example."

• The partition key, like a game ID, determines which partition the data is stored in, ensuring consistent data placement.

• Kafka uses a hashing mechanism to assign messages to partitions.
• The hash of the key, modulated by the number of partitions, determines the partition number.
• The partitioning process is deterministic, ensuring consistent message routing.

"We take that key we hash it this is actually using I believe a murmur hash which is just a a fast hash function and then we take the modulo of that hash over the number of partitions that we have here say n and this is going to give us a number and that number is going to correspond to the partition."

• The hashing and modulo operation ensure that each message is consistently routed to the correct partition.

"This is deterministic so the next event that comes in here with a key of Argentina versus Brazil will also need to go to partition five."

• Deterministic partitioning ensures that messages with the same key are consistently routed to the same partition.

• Brokers store messages in append-only logs, maintaining message order.
• Consumers read messages based on offsets, ensuring no message is read twice.
• Kafka maintains offsets to allow consumers to resume from the last read position after a failure.

"The broker is going to receive it and append it to that append Only log file and so everything that happens here is kind of uh behind the scenes it happens native to the CFA cluster."

• Brokers append messages to logs, maintaining the sequence in which messages are received.

"We read the message from Kafka we get that message back we maintain the current offset locally in the consumer periodically we commit those offsets."

• Consumers maintain offsets locally and commit them to Kafka to track reading progress and resume after failures.

• Kafka ensures durability and availability through replication.
• Each partition has a leader replica and follower replicas.
• Followers replicate data from the leader and can take over if the leader fails.

"Each partition is designed a leader replica like this one here and it resigns on one of the Brokers and that leader replica is responsible for handling all the read and write requests to the partition."

• The leader replica handles all read and write requests, ensuring efficient data processing.

"Followers don't handle direct client requests instead they just passively replicate the data from the leader."

• Followers act as backups by replicating data from the leader, ensuring data redundancy and fault tolerance.

• Kafka is suitable for asynchronous processing, such as video transcoding.
• It supports ordered message processing, ideal for event booking systems.
• Kafka decouples producers and consumers, allowing independent scaling.

"With YouTube when you upload a new video you upload that full video and you store that video in S3 but then you need to transcode that video basically take that video and put it into 480p 720p 1080p Etc."

• Kafka is used to buffer video transcoding tasks, allowing asynchronous processing without impacting user experience.

"In an event booking service like Ticket Master for a really popular event we might want to put people under a waiting queue and only let people off of this waiting queue in groups."

• Kafka manages waiting queues for event booking, ensuring orderly processing and reducing contention.

"You want to decouple the producer and the consumer and the reason that you typically want to do that is because you want to scale them independently."

• Kafka allows producers and consumers to scale independently, optimizing resource allocation and cost-efficiency.

• Real-time data processing involves consuming and processing data streams quickly for immediate updates or statistics.
• Pub/Sub (Publish/Subscribe) systems allow multiple consumers to process message streams simultaneously, useful in applications like live video comments.
• Kafka is often used for real-time data processing and Pub/Sub systems due to its ability to handle high-throughput data streams.

"When a user clicks an ad, we put that click on our Kafka queue, and then we have a consumer like Flink reading off this stream in real time to aggregate clicks."

• Kafka queues facilitate real-time data aggregation for immediate feedback to advertisers.

"If we have a commenter who leaves a comment on a live video, we can put that comment on Kafka as a pub/sub, and services connected to users watching the video can deliver the comment in real time."

• Pub/Sub systems ensure real-time communication and updates for live applications, enhancing user experience.

• Understanding Kafka's role in system design interviews involves demonstrating technical depth in specific areas.
• Key areas include scalability, fault tolerance and durability, errors and retries, performance optimizations, and retention policies.

"Our recommended framework ends with documenting a high-level design and then expanding upon it by going deep in a handful of areas to show technical depth."

• Demonstrating technical depth in Kafka involves exploring specific areas that highlight expertise in system design.

• Scalability is a common focus in interviews, questioning how Kafka can scale within a system.
• Important considerations include message size limits and hardware constraints, such as broker storage and message handling capacity.
• Effective scaling involves adding brokers and choosing appropriate partition keys to distribute load evenly.

"There's no limit on the maximum size of a message in Kafka beyond hardware limits, but it's advised to keep messages under 1 Megabyte for optimal performance."

• Message size management is crucial to prevent network or memory overload and ensure optimal Kafka performance.

"A single broker can store about a terabyte of data and handle about 10,000 messages per second, depending on hardware and message size."

• Understanding broker capacity helps in estimating system requirements and planning for scalability.

"If you need to scale, introduce more brokers and choose your partition key carefully to avoid hot partitions."

• Scaling involves increasing broker numbers and selecting effective partition keys to balance data distribution.

• Hot partitions occur when data is unevenly distributed, leading to overload on specific partitions.
• Solutions include removing keys for random distribution, using compound keys, and implementing back pressure to manage production rates.

"A hot partition is when everything goes to a single partition, overwhelming it with traffic."

• Hot partitions result from poor key choices, causing traffic congestion on specific partitions.

"You can remove the key to randomly distribute messages or use a compound key to spread data across multiple partitions."

• Strategies for handling hot partitions involve redistributing data to prevent overload and ensure balanced processing.

• Kafka offers strong durability guarantees, with leader and follower partitions ensuring data availability.
• Configuration settings like AXS and replication factor are essential for setting up fault-tolerant Kafka clusters.

"Kafka has strong guarantees on durability with leader and follower partitions to take over if the leader goes down."

• Kafka's architecture supports data durability and fault tolerance through leader-follower partitioning.

"Two relevant settings are AXS and replication factor, determining how many followers need to acknowledge a message."

• Configuring AXS and replication factor settings is crucial for ensuring message durability and system reliability.

• Durability is crucial in Kafka, requiring all followers to acknowledge receipt of a message before moving to the next.
• Trade-off exists between durability and performance; fewer acknowledgments can speed up processes but risk data loss.
• Replication factor determines the number of followers and impacts durability and storage efficiency.

"All is maximum durability; this means that every single follower needs to acknowledge that they also got the message, and then we can say that we got this message."

• Ensures all followers have received a message, enhancing durability.

"The trade-off here is, of course, durability versus performance."

• Balancing act between ensuring message safety and processing speed.

• Kafka is designed to be always available; consumer failures are more realistic than cluster failures.
• Consumers commit offsets to Kafka, allowing them to resume from the last processed message after a failure.
• Consumer groups handle partition ranges; rebalancing occurs if a consumer goes down.

"Kafka is usually thought of as always available."

• Kafka's design minimizes the likelihood of system-wide failures.

"It's really important when you decide to commit your offset."

• Timing of offset commits is crucial to ensure no data is lost or processed twice.

• Kafka handles most reliability, but systems may fail to send or receive messages.
• Producer retries are supported with configurations for retries and waiting periods.
• Consumer retries are not natively supported; custom solutions or alternative systems like AWS SQS may be needed.

"The CFA producer API supports a couple of configurations that allow us to retry gracefully."

• Kafka provides retry configurations to enhance message delivery reliability.

"Kafka actually does not support consumer retries out of the box."

• Consumer retries require custom implementation or alternative solutions.

• Batching and compressing messages can significantly improve performance by reducing requests and data size.
• Partitioning strategy is crucial for maximizing parallelism and performance.
• Retention policy determines how long messages are stored, impacting storage and performance.

"We can batch messages in the producer so that we have fewer requests."

• Batching reduces the number of requests, enhancing throughput.

"Arguably the biggest impact you can have on performance comes back to the choice of that partition key."

• Correct partitioning is fundamental for optimal performance.

• Retention policies in Kafka determine how long messages are kept before purging.
• Configured per topic, using retention.ms for time-based and retention.bytes for size-based retention.
• Longer retention periods can increase storage costs and affect performance.

"Kafka topics have a retention policy that determines how long messages are retained in those logs."

• Retention policies are critical for managing data lifecycle and storage costs.

"You can configure the retention policy to keep those messages for a longer duration."

• Adjusting retention settings allows for longer data availability but requires careful consideration of resource impact.