Summary
We recently pushed OAuth2 authentication for Kafka into Production for a fortune 500 client. KIP-255 allows one to implement OAuth2 authentication from Java clients to brokers and for inter-broker authentication. In addition to these, we also implemented OAuth2 authentication for REST Proxy clients. In this post, we give enough details for anyone interested to do these on their own (short of handing over the code). If you’re interested, we would love to get engaged with you to implement these for your Kafka clusters. Keep in mind that there’s no official implementation of this available anywhere yet, including confluent distribution of Kafka.
Once the client is authenticated, we can use simpleAclAuthorizer that ships with Kafka to authorize operations on Kafka resources (topics, for example). This helps implement security in Kafka clusters which is very important in many customer installations, especially multi-tenant ones.
Architecture
Following picture depicts how a Java client will interact with the brokers using credentials configured in JAAS file (credentials can be configured programmatically as well).
Following diagram illustrates on how brokers authenticate between themselves for inter-broker communication. In reality, this is very similar to how Java clients authenticate with brokers.
Finally, following diagram shows how a REST client uses OAuth2 authentication mechanism.
Implementation Notes
- AuthenticateCallbackHandler interface is implemented for Java authentication (inter-broker is a Java client really)
- RestResourceExtension interface is implemented for REST Client authentication
- Useful JMX Metrics are provided to let Operators know how security is working (Successful and failed request count at cluster level as well as topic level)
- Security can be rolled into an existing cluster in a phased manner. Eg: cluster can be setup to serve authenticated as well as unauthenticated traffic in the beginning. You can work with tenants to move the applications to use OAuth2 authentication. When all tenants (or Applications) are using authentication, you can turn off unauthenticated traffic completely.
- You can add additional scope check (shown in the diagram), in case you want additional security.
Feel free to leave comments. If you’re really interested in implementing this, drop me a line at manoj@upala.com