Amazon MSK Serverless now helps Kafka shoppers written in all programming languages

Amazon MSK Serverless is a cluster sort for Amazon Managed Streaming for Apache Kafka (Amazon MSK) that’s the most simple approach to run Apache Kafka clusters with out having to handle compute and storage capability. With MSK Serverless, you possibly can run your purposes with out having to provision, configure, or optimize clusters, and also you pay for the information quantity you stream and retain. MSK Serverless absolutely manages partitions, together with monitoring and shifting them to even load throughout a cluster.

With as we speak’s launch, MSK Serverless now helps writes and reads from Kafka shoppers written in all programming languages. Directors can simplify and standardize entry management to Kafka sources utilizing AWS Id and Entry Administration (IAM). This help for IAM in Amazon MSK is predicated on SASL/OUATHBEARER, an open normal for authorization and authentication.

On this submit, we present how one can join your purposes to MSK Serverless with minimal code modifications utilizing the open-sourced consumer helper libraries and code samples for widespread languages, together with Java, Python, Go, JavaScript, and .NET. Utilizing IAM authentication and authorization is the popular alternative of many shoppers as a result of you possibly can safe Kafka sources similar to you do with all different AWS providers. Moreover, you get all the opposite advantages of IAM, akin to short-term role-based credentials, exactly scoped permission insurance policies, and extra. Now you should use MSK Serverless with IAM-based authentication extra broadly with the help for a number of languages.

Resolution overview

You may get began by utilizing IAM principals as identities in your Apache Kafka shoppers and outline id insurance policies to offer them exactly scoped entry permissions. For instance, you possibly can create an IAM consumer and a coverage that permits the consumer to write down to a particular Kafka subject however restricts entry to different sources with out worrying about managing Kafka ACLs. After you present the id insurance policies with the mandatory permissions, you possibly can configure consumer purposes to make use of the IAM authentication with minimal code modifications.

The code modifications permit your shoppers to make use of SASL/OAUTHBEARER, a Kafka supported token-based entry mechanism, to move the credentials required for IAM authentication. With OAUTHBEARER help, you possibly can construct shoppers that may work throughout each Amazon MSK and different Kafka environments. On this submit, we present how one can make these code modifications by utilizing the offered code libraries and examples.

With this launch, Amazon MSK gives new code libraries for the next programming languages within the AWS GitHub repo:

The next diagram reveals the conceptual course of circulate of utilizing SASL/OAUTHBEARER with IAM entry management for non-Java shoppers.

The workflow comprises the next steps:

1. The consumer generates an OAUTHBEARER token with the assistance of the offered library. The token comprises a signed base64 encoded transformation of your IAM id credentials.
2. The consumer sends this to Amazon MSK utilizing the bootstrap deal with together with its request to entry Apache Kafka sources.
3. The MSK Serverless cluster decodes the OATHBEARER token, validates the credentials, and checks if the consumer is allowed to carry out the requested motion in line with the coverage connected to the IAM id.
4. When the token expires, the consumer Kafka library robotically refreshes the token by making one other name to the required token supplier.

Create IAM identities and insurance policies

IAM entry management for non-Java shoppers is supported for MSK Serverless clusters with no extra price. Earlier than you begin, it’s worthwhile to configure the IAM identities and insurance policies that outline the consumer’s permissions to entry sources on the cluster. The next is an instance authorization coverage for a cluster named MyTestCluster. To know the semantics of the motion and useful resource components, see Semantics of actions and sources.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/*"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:group/MyTestCluster/*"
            ]
        }
    ]
}

Configure the consumer

You need to make code modifications to your software that permit the shoppers to make use of SASL/OAUTHBEARER to move the credentials required for IAM authentication. You additionally want to ensure the safety group related along with your MSK Serverless cluster has an inbound rule permitting the visitors from the consumer purposes within the related VPCs to the TCP port quantity 9098.

You need to use a Kafka consumer library that gives help for SASL with OAUTHBRARER authentication.

For this submit, we use the Python programming language. We additionally use https://github.com/dpkp/kafka-python as our Kafka consumer library.

Amazon MSK gives you with a brand new code library per every language that generates the OAUTHBEARER token.

1. To get began working with the Amazon MSK IAM SASL signer for Python along with your Kafka consumer library, run the next command:

   $ pip set up aws-msk-iam-sasl-signer-python

2. Import the put in Amazon MSK IAM SASL signer library in your code:

   from kafka import KafkaProducer
   from kafka.errors import KafkaError
   import socket
   import time
   from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

3. Subsequent, your software code must outline a token supplier that wraps the perform that generates new tokens:

   class MSKTokenProvider():
           def token(self):
               token, _ = MSKAuthTokenProvider.generate_auth_token('<your aws area>')
               return token

4. Specify security_protocol as SASL_SSL and sasl_mechanism as oauthbearer in your Python Kafka consumer properties, and move the token supplier within the configuration object:

   tp = MSKTokenProvider()
   
       producer = KafkaProducer(
           bootstrap_servers="<Amazon MSK Serverless bootstrap string>",
           security_protocol="SASL_SSL",
           sasl_mechanism='OAUTHBEARER',
           sasl_oauth_token_provider=tp,
           client_id=”my.kafka.consumer.distinctive.id”,
       )

You at the moment are completed with all of the code modifications. For extra examples of producing auth tokens or for extra troubleshooting ideas, confer with the next GitHub repo.

Conclusion

MSK Serverless now helps writes and reads from Kafka shoppers written in all programming languages. You’ll be able to run your purposes with out having to configure and handle the infrastructure or optimize clusters, and also you pay for the information quantity you stream and retain. MSK Serverless absolutely manages partitions, together with monitoring, and ensures a fair stability of partition distribution throughout brokers within the cluster (auto-balancing).

For additional studying on Amazon MSK, go to the official product web page.

In regards to the creator

Ali Alemi is a Streaming Specialist Options Architect at AWS. Ali advises AWS clients with architectural greatest practices and helps them design real-time analytics knowledge methods which are dependable, safe, environment friendly, and cost-effective. He works backward from buyer’s use circumstances and designs knowledge options to resolve their enterprise issues. Previous to becoming a member of AWS, Ali supported a number of public sector clients and AWS consulting companions of their software modernization journey and migration to the Cloud.