Amazon MSK IAM authentication now helps all programming languages

The AWS Id and Entry Administration (IAM) authentication characteristic in Amazon Managed Streaming for Apache Kafka (Amazon MSK) now helps all programming languages. Directors can simplify and standardize entry management to Kafka assets utilizing IAM. This assist relies on SASL/OUATHBEARER, an open commonplace for authorization and authentication. Each Amazon MSK provisioned and serverless cluster varieties assist the brand new Amazon MSK IAM enlargement to all programming languages.

On this put up, we present how one can join your functions to MSK clusters with minimal code modifications utilizing the open-sourced shopper helper libraries and code samples for widespread languages, together with Java, Python, Go, JavaScript, and .NET. You may also use commonplace IAM entry controls similar to short-term role-based credentials and exactly scoped permission insurance policies extra broadly with the a number of language assist on Amazon MSK.

For shoppers that want to attach from different VPCs to an MSK cluster, whether or not in a identical or a special AWS account, you’ll be able to allow multi-VPC non-public connectivity and cluster coverage assist. IAM entry management through cluster coverage helps you handle all entry to the cluster and matters in a single place. For instance, you’ll be able to management which IAM principals have write entry to sure matters, and which principals can solely learn from them. Customers who’re utilizing IAM shopper authentication may also add permissions for required kafka-cluster actions within the cluster useful resource coverage.

Answer overview

You will get began by utilizing IAM principals as identities on your Apache Kafka shoppers and outline id insurance policies to offer them exactly scoped entry permissions. After IAM authentication is enabled on your cluster, you’ll be able to configure shopper functions to make use of the IAM authentication with minimal code modifications.

The code modifications enable your shoppers to make use of SASL/OAUTHBEARER, a Kafka supported token-based entry mechanism, to go the credentials required for IAM authentication. On this put up, we present how one can make these code modifications by utilizing the supplied code libraries and examples.

With this launch, new code libraries for the next programming languages can be found within the AWS GitHub repo:

The next diagram reveals the conceptual course of circulation of utilizing SASL/OAUTHBEARER with IAM entry management for non-Java shoppers.

The workflow comprises the next steps:

1. The shopper generates an OAUTHBEARER token with the assistance of the supplied library. The token comprises a signed base64 encoded transformation of your IAM id credentials.
2. The shopper sends this to Amazon MSK utilizing the IAM bootstrap dealer addresses together with its request to entry Apache Kafka assets.
3. The MSK dealer decodes the OATHBEARER token, validates the credentials, and checks if the shopper is allowed to carry out the requested motion in accordance with the coverage hooked up to the IAM id.
4. When the token expires, the shopper Kafka library mechanically refreshes the token by making one other name to the desired token supplier.

Create IAM identities and insurance policies

IAM entry management for non-Java shoppers is supported for MSK clusters with Kafka model 2.7.1 and above. Earlier than you begin, you want to configure the IAM identities and insurance policies that outline the shopper’s permissions to entry assets on the cluster. The next is an instance authorization coverage for a cluster named MyTestCluster. To know the semantics of the motion and useful resource parts, see Semantics of actions and assets.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/*"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:group/MyTestCluster/*"
            ]
        }
    ]
}

Arrange the MSK cluster

It is advisable to allow the IAM entry management authentication scheme on your MSK provisioned cluster and wait till the cluster finishes updating and turns to the Lively state. It’s because SASL/OAUTHBEARER makes use of the identical dealer addresses for IAM authentication.

Configure the shopper

It’s best to make code modifications to your utility that enable the shoppers to make use of SASL/OAUTHBEARER to go the credentials required for IAM authentication. Subsequent, replace your utility to make use of the bootstrap server addresses for IAM authentication. You additionally want to verify the safety group related together with your MSK cluster has an inbound rule permitting the site visitors from the shopper functions in the identical VPC because the cluster to the TCP port 9098.

You could use a Kafka shopper library that gives assist for SASL with OAUTHBRARER authentication.

On this put up, we use the JavaScript programming language. We additionally use https://github.com/tulios/kafkajs as our Kafka shopper library.

Amazon MSK gives you with a brand new code library per every language that generates the OAUTHBEARER token.

1. To get began working with the Amazon MSK IAM SASL signer for JavaScript together with your Kafka shopper library, run the next command:

   npm set up https://github.com/aws/aws-msk-iam-sasl-signer-js

2. It is advisable to import the put in Amazon MSK IAM SASL signer library in your code:

   const { Kafka } = require('kafkajs')
   
   const { generateAuthToken } = require('aws-msk-iam-sasl-signer-js')

3. Subsequent, your utility code must outline a token supplier that wraps the operate that generates new tokens:

   async operate oauthBearerTokenProvider(area) {
       // Makes use of AWS Default Credentials Supplier Chain to fetch credentials
       const authTokenResponse = await generateAuthToken({ area });
       return {
           worth: authTokenResponse.token
       }
   }

4. Specify security_protocol as SASL_SSL and sasl_mechanism as oauthbearer in your JavaScript Kafka shopper properties, and go the token supplier within the configuration object:

   const run = async () => {
       const kafka = new Kafka({
           clientId: 'my-app',
           brokers: [bootstrap server addresses for IAM],
           ssl: true,
           sasl: {
               mechanism: 'oauthbearer',
               oauthBearerProvider: () => oauthBearerTokenProvider('us-east-1')
           }
       })
   
       const producer = kafka.producer()
       const client = kafka.client({ groupId: 'test-group' })
   
       // Producing
       await producer.join()
       await producer.ship({
           matter: 'test-topic',
           messages: [
               { value: 'Hello KafkaJS user!' },
           ],
       })
   
       // Consuming
       await client.join()
       await client.subscribe({ matter: 'test-topic', fromBeginning: true })
   
       await client.run({
           eachMessage: async ({ matter, partition, message }) => {
               console.log({
                   partition,
                   offset: message.offset,
                   worth: message.worth.toString(),
               })
           },
       })
   }
   
   run().catch(console.error)

You at the moment are completed with all of the code modifications. For extra examples of producing auth tokens or for extra troubleshooting ideas, seek advice from the next GitHub repo.

Conclusion

IAM entry management for Amazon MSK allows you to deal with each authentication and authorization on your MSK cluster. This eliminates the necessity to use one mechanism for authentication and one other for authorization. For instance, when a shopper tries to write down to your cluster, Amazon MSK makes use of IAM to verify whether or not that shopper is an authenticated id and in addition whether or not it’s licensed to supply to your cluster.

With right now’s launch, Amazon MSK IAM authentication now helps all programming languages. This implies you’ll be able to join your functions in all languages with out worrying about implementing separate authentication and authorization mechanisms. For workloads that require Amazon MSK multi-VPC non-public connectivity and cluster coverage assist, now you can simplify connectivity to your MSK brokers and handle all entry to the cluster and matters in a single place that’s your cluster coverage.

For additional studying on Amazon MSK, go to the official product web page.

In regards to the creator

Ali Alemi is a Streaming Specialist Options Architect at AWS. Ali advises AWS prospects with architectural finest practices and helps them design real-time analytics information programs which can be dependable, safe, environment friendly, and cost-effective. He works backward from buyer’s use instances and designs information options to unravel their enterprise issues. Previous to becoming a member of AWS, Ali supported a number of public sector prospects and AWS consulting companions of their utility modernization journey and migration to the cloud.