This is just a short one about something that happened recently at work, that I thought was worth sharing in case others come across it.

We were preparing for our busy period of the year and part of that involves scaling up/out some services that don’t automatically scale and that included some AWS Kinesis Data Streams. Now that the busiest part of the year is done, we needed to scale in some of those resources, which lead us to find a very strange thing. When we tried to shard in (remove shards) from a couple of AWS Kinesis Data Streams, we were left with an extra shard, or as we liked to call it, a ‘Dangling Shard’. Having performed shard in and out actions a fair few times on many different Kinesis Data Streams, this was a definite first.

So why do I have an extra shard?

When you action a shard in or out, either using the AWS Kinesis Scaling Utility or via the Kinesis console, it has to work out how to place the hash keys that contain the data. This involves creating new shards, and either splitting or merging the data, depending on if you are adding or removing shards to the stream.

So why do I need to shard???

  1. Throughput is the name of the game. If you want to be able to have more data flowing through your Kinesis Data Stream, you will require more shards to handle the increase.
  2. Well, the cost is always something to keep in mind when using Kinesis Data Streams. If you are at a low point of data processing, then it makes sense to have as little shards as possible to save some pennies.
How do I get rid of this dangling shard?

To resolve this odd occurrence, the first thing is to identify the ‘Dangling Shard’. The easiest way is to use the AWS CLI and list the shards for the stream that has the issue:

aws kinesis list-shards --stream-name {{stream name}} --region {{region}} --profile {{account profile}}

This will output something like the below AWS example:

{
    "Shards": [
        {
            "ShardId": "shardId-000000000001",
            "HashKeyRange": {
                "StartingHashKey": "113427455640312821154458202477256070485",
                "EndingHashKey": "226854911280625642308916404954512140969"
            },
            "SequenceNumberRange": {
                "StartingSequenceNumber": "49600871682979337187563555549332609155523708941634633746"
            }
        },
        {
            "ShardId": "shardId-000000000002",
            "HashKeyRange": {
                "StartingHashKey": "226854911280625642308916404954512140970",
                "EndingHashKey": "340282366920938463463374607431768211455"
            },
            "SequenceNumberRange": {
                "StartingSequenceNumber": "49600871683001637932762086172474144873796357303140614178"
            }
        }
    ]
}

As the above example doesn’t show an actual ‘Dangling Shard’, we will just have to pretend that shardId-000000000002 is the ‘Dangling Shard’ in this use case. We would want to then merge this shard into the adjacent shard, in this case, shardId-000000000001. This will remove the ‘Dangling Shard’ and allow the Kinesis consumers to see all the shards and process them as normal. Job done!

aws kinesis merge-shards --stream-name {{stream name}} --shard-to-merge {{destination shard}} --adjacent-shard-to-merge {{source shard}} --region {{region}} --profile {{account profile}}

Hopefully, this will be useful to others, as it took us by complete surprise as we had never seen it before. AWS is actively working on this issue, so hopefully, soon it will be a thing of the past.