eventbridge-kafka-connector v1.3.1+ breaks authentication with kube2iam (InstanceProfileCredentialsProvider) #396
Comments
sugarcrm-jgminder
changed the title
sugarcrm-jgminder
changed the title
|
@sugarcrm-jgminder thx for flagging this and glad you could revert to the previous version for now. I'll discuss this with the team and see how it can be fixed. |
|
@sugarcrm-jgminder it sounds though that the initial credential load that we do here succeeds and the exception is thrown after a token refresh? |
|
@embano1 Below is a full message after a Kafka Connect pod starts with the failure stacktrace. The credential load fails and it fails to find any suitable credential provider within the default chain: |
|
Hi @embano1, @maschnetwork, I had a look how the code behaves if the The Finally it's used in and Ec2MetadataConfigProvider: private ProfileFile resolveProfileFile() {
if (profileFile != null) {
return profileFile.get();
}
return ProfileFile.defaultProfileFile();
}ProfileFile: public static ProfileFile defaultProfileFile() {
return ProfileFile.aggregator()
.applyMutation(ProfileFile::addCredentialsFile)
.applyMutation(ProfileFile::addConfigFile)
.build();
}
private static void addCredentialsFile(ProfileFile.Aggregator builder) {
ProfileFileLocation.credentialsFileLocation()
.ifPresent(l -> builder.addFile(ProfileFile.builder()
.content(l)
.type(ProfileFile.Type.CREDENTIALS)
.build()));
}
private static void addConfigFile(ProfileFile.Aggregator builder) {
ProfileFileLocation.configurationFileLocation()
.ifPresent(l -> builder.addFile(ProfileFile.builder()
.content(l)
.type(ProfileFile.Type.CONFIGURATION)
.build()));
}ProfileFileSupplier: static ProfileFileSupplier defaultSupplier() {
Optional<ProfileFileSupplier> credentialsSupplierOptional
= ProfileFileLocation.credentialsFileLocation()
.map(path -> reloadWhenModified(path, ProfileFile.Type.CREDENTIALS));
Optional<ProfileFileSupplier> configurationSupplierOptional
= ProfileFileLocation.configurationFileLocation()
.map(path -> reloadWhenModified(path, ProfileFile.Type.CONFIGURATION));
ProfileFileSupplier supplier = () -> ProfileFile.builder().build();
if (credentialsSupplierOptional.isPresent() && configurationSupplierOptional.isPresent()) {
supplier = aggregate(credentialsSupplierOptional.get(), configurationSupplierOptional.get());
} else if (credentialsSupplierOptional.isPresent()) {
supplier = credentialsSupplierOptional.get();
} else if (configurationSupplierOptional.isPresent()) {
supplier = configurationSupplierOptional.get();
}
return supplier;
}It looks like they behave the same The log message for successfully resolved credentials/exception is done here: InstanceProfileCredentialsProvider: private RefreshResult<AwsCredentials> refreshCredentials() {
if (isLocalCredentialLoadingDisabled()) {
throw SdkClientException.create("IMDS credentials have been disabled by environment variable or system property.");
}
try {
LoadedCredentials credentials = httpCredentialsLoader.loadCredentials(createEndpointProvider());
Instant expiration = credentials.getExpiration().orElse(null);
log.debug(() -> "Loaded credentials from IMDS with expiration time of " + expiration);
return RefreshResult.builder(credentials.getAwsCredentials())
.staleTime(staleTime(expiration))
.prefetchTime(prefetchTime(expiration))
.build();
} catch (RuntimeException e) {
throw SdkClientException.create("Failed to load credentials from IMDS.", e);
}
} |
|
I tried the following locally: package software.amazon.event.kafkaconnector;
import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import software.amazon.awssdk.profiles.ProfileFile;
public class Test {
public static void main(String[] args) {
var provider = InstanceProfileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(false)
.profileFile(ProfileFile.defaultProfileFile())
.profileName(null)
.build();
var creds = provider.resolveCredentials();
System.out.println(creds);
}
}It fails with the exception: The cause of Instead of using the default credentials provider we should test/instantiate the |
|
@sugarcrm-jgminder would you be open to try a test build to debug this further as per @agebhar1's recommendation? @agebhar1 thx a ton for jumping in here! |
|
@embano1 Sure no problem. I'd be happy to trying test builds out for debugging this issue. |
|
The different behavior of package software.amazon.event.kafkaconnector;
import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import software.amazon.awssdk.profiles.ProfileFile;
import software.amazon.awssdk.profiles.ProfileFileSupplier;
public class Test {
public static void main(String[] args) {
var provider = InstanceProfileCredentialsProvider.builder()
//.profileFile(ProfileFile.defaultProfileFile()) // w/ < v1.3.1
.profileFile(ProfileFileSupplier.defaultSupplier()) // w >= v1.3.1
.profileName(null)
.build();
var creds = provider.resolveCredentials();
System.out.println(creds);
}
}This yields to the following error, if no AWS profile exists in the filesystem: |
|
@sugarcrm-jgminder forgot to ask whether IRSA would be an option for you? https://aws.github.io/aws-eks-best-practices/security/docs/iam/#iam-roles-for-service-accounts-irsa |
|
Regarding your question in your test PR whether it can be considered a bug on the SDK behavior, I'm inclined to agree here
Reading this in the SDK docs I would expect the convenience method to work in any of the supported cases. @maschnetwork WDYT? |
|
A temporary fix for us would be to implement a custom supplier using the default behavior but catching NPEs from |
|
A workaround could be: Which works in both cases, presence and absence of |
|
Thanks. Indeed this looks like a bug in the SDK. Trying to summarize my debugging process below for reference and documentation. ProfileFileSupplier.defaultSupplier() effectively returns @Override
public ProfileFile build() {
InputStream stream = content != null ? content :
FunctionalUtils.invokeSafely(() -> Files.newInputStream(contentLocation));Results in: This will happen first in the This should be ultimately fixed in the SDK and I'll cut a GIthub issue tomorrow outlining our findings. As a quick workaround I agree with @agebhar1 last proposal and provide something like a |
|
Great work team! @sugarcrm-jgminder is this currently blocking for you? |
|
Created issue here: aws/aws-sdk-java-v2#5635 |
|
@embano1 This is not a blocker and I can wait for a resolution on aws/aws-sdk-java-v2#5635. |
|
Perfect, thx a ton for your investigation here and raising the issue! |
Describe the bug
Commit
1bc23b83f65659ba2cac3c637974cddc04b63dc9introduced a change in v1.3.1 to Leverage DefaultSupplier to automatically reload credentials on file refresh. This change causes authentication to fail when usingkube2iamwhich in turn usesInstanceProfileCredentialsProvider.To Reproduce
Steps to reproduce the behavior:
kube2iamin a k8s clusterkube2iamannotations for a role:Expected behavior
The connector should be able to authenticate using the
InstancProfileCredentialsProvider. Versions prior to v1.3.1 would have an authentication log like the following:Environment:
Additional context
I reverted commit
1bc23b83f65659ba2cac3c637974cddc04b63dc9and redeployed the connector and the expected behavior was observed:The text was updated successfully, but these errors were encountered: