Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. The design is heavily influenced by transaction logs.
Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. In November 2014, several engineers who built Kafka at LinkedIn created a new company named Confluent with a focus on Kafka.
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.
- A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
- Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
- Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
- Distributed by Design
- Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.