cert-bank

Guides

이 글을 한국어로 읽기

Kinesis Data Streams vs Amazon MSK: How to Tell Which One an Exam Question Wants

Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (MSK) both move a continuous stream of records from producers to consumers, and both appear as options in the same exam questions. They are not interchangeable, though, and the stem nearly always contains one detail that decides between them. This comparison explains the difference and the clues to look for.

Concept comparison · SAA-C03, DEA-C01 and other AWS data exams

Kinesis Data Streams vs Amazon MSK: How to Tell Which One an Exam Question Wants

The short answer

Choose MSK when the scenario already speaks Kafka: existing Kafka producers or consumers, Kafka Connect, Kafka Streams, or a requirement to keep the open-source Kafka APIs. Choose Kinesis Data Streams when the scenario wants an AWS-native stream with the least setup: no brokers to size, direct integration with Lambda and Amazon Data Firehose, and capacity that can scale on demand.

Side by side

Kinesis Data StreamsAmazon MSK
What it isAn AWS streaming service with its own APIManaged Apache Kafka: the open-source Kafka APIs and tools
Unit of scaleShards (or on-demand mode, where AWS manages them)Brokers and topic partitions (provisioned), or MSK Serverless
OrderingPer shard, by partition keyPer partition, by message key
Retention24 hours by default, extendable up to 365 daysSet per topic; tiered storage keeps older data cheaply for long periods
ConsumersLambda, Firehose, Managed Service for Apache Flink, KCL applications; enhanced fan-out gives each consumer its own read throughputAny Kafka client or consumer group, Kafka Connect, Flink, Lambda
What you operateLittle: shard count, or nothing in on-demand modeCluster sizing, broker storage and Kafka configuration (less with Express brokers or MSK Serverless)
Moving off it laterCode uses the Kinesis API, so it is tied to AWSStandard Kafka, so applications move between Kafka clusters with few changes
Which streaming service does the question want?
Does the scenario already use Kafka clients, Kafka Connect or Kafka Streams?
YES →
Amazon MSK
↓ no
Is the only job to deliver records into S3, Redshift or OpenSearch?
YES →
Amazon Data Firehose
↓ no
Must several consumers read every record, or replay it later?
YES →
Kinesis Data Streams (enhanced fan-out for many readers)
↓ no
Is each message handled once by one worker, with no replay?
YES →
Amazon SQS (a queue, not a stream)
↓ no
OTHERWISE →
Kinesis Data Streams in on-demand mode: the least to operate
Read the stem from the top: the first question answered yes decides.

Clues in the question

Points to MSK

  • "The company runs Apache Kafka on premises" or "on EC2" and wants to reduce the operational work: the answer migrates to MSK, because the applications keep their Kafka clients.
  • Kafka Connect connectors, Kafka Streams applications or a schema registry already in use.
  • A requirement to avoid vendor-specific APIs or to stay portable between clouds.
  • Very long retention of the raw stream for replay, beyond what a Kinesis stream keeps.

Points to Kinesis Data Streams

  • A new application with no streaming system yet, and "least operational overhead" in the stem.
  • Traffic that is hard to predict, where on-demand capacity removes shard planning.
  • Records that must trigger Lambda functions, or flow on to S3, Redshift or OpenSearch through Firehose with no custom code.
  • Several applications reading the same stream at full speed: enhanced fan-out.
Going deeper: why ordering only holds within a shard or partition

Both services scale by splitting the stream: Kinesis into shards, Kafka topics into partitions. Each split is an ordered log read by one consumer of a group at a time, and the splits are read in parallel. Ordering across splits would force all consumers to coordinate on every record, which would remove the parallelism the split was made for. So the record key decides the split (a hash of the partition key in Kinesis, of the message key in Kafka), and order is kept per key.

The cost is the hot shard or hot partition: if one key carries most of the traffic, its split hits its throughput limit while others sit idle. Exam stems describe this as throttling on one shard while the stream as a whole is far below its capacity; the fix is a key with more distinct values, not simply more shards.

The third option people forget: Amazon Data Firehose

If the stem only asks to deliver streaming data into S3, Redshift, OpenSearch or an HTTP endpoint, possibly with light transformation and batching, neither Kinesis Data Streams nor MSK is the best answer on its own. Amazon Data Firehose (formerly Kinesis Data Firehose) does the delivery with no consumers to write and no capacity to manage. It is not a stream you replay: data is buffered for a short time and delivered, which is why it fits "load into the data lake" but not "let three teams read the same events independently".

Beginner tip: before answering a streaming question, sketch it in three boxes: who writes (producers), the stream, and who reads (consumers). Then count the consumers and ask whether each must see every record. One consumer that only loads data → Firehose. Several independent readers or replay → a stream. Existing Kafka code → MSK.

Common wrong turns

  • Choosing SQS for a stream. In SQS the consumer deletes each message after processing it, so several consumers cannot each read every message, and there is no replay. If the stem says "replay" or "multiple independent consumers", it wants a stream.
  • Running Kafka on EC2 yourself. It works, but it is almost never the answer when MSK is offered and the stem mentions operational effort.
  • Assuming ordering across the whole stream. Both services order records only within a shard or partition. If all records for one customer must stay in order, they need the same partition key.

Practise it

Streaming choices appear in the Design Resilient Architectures and Design High-Performing Architectures domains of SAA-C03 and mostly in the Data Ingestion and Transformation domain of DEA-C01:

Service limits and features change over time; check the current Kinesis and MSK documentation for exact quotas before relying on a number.