eventbridge-kafka-connector v1.3.1+ breaks authentication with kube2iam (InstanceProfileCredentialsProvider)

[sugarcrm-jgminder]

Describe the bug

Commit 1bc23b83f65659ba2cac3c637974cddc04b63dc9 introduced a change in v1.3.1 to Leverage DefaultSupplier to automatically reload credentials on file refresh. This change causes authentication to fail when using kube2iam which in turn uses InstanceProfileCredentialsProvider.

To Reproduce

Steps to reproduce the behavior:

1. Deploy kube2iam in a k8s cluster
2. Deploy a KafkaConnect deployment with kube2iam annotations for a role:

iam.amazonaws.com/role: <role>

3. Deploy an eventbridge-kafka-connector sink to the KafkaConnect cluster in Step 2.
4. Watch authentication fail with the following message:

{
  "@timestamp": "2024-09-16T19:40:38.334Z",
  "source_host": "cxp-connect-eventbridge-connect-0",
  "file": "EventBridgeSinkTask.java",
  "method": "handleFailedEntries",
  "level": "WARN",
  "line_number": "145",
  "thread_name": "task-thread-eventbridge-sink-0",
  "@version": 1,
  "logger_name": "software.amazon.event.kafkaconnector.EventBridgeSinkTask",
  "message": "Retrying failed putItems call: attempts=1 maxRetries=2 errorMessage=software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]",
  "class": "software.amazon.event.kafkaconnector.EventBridgeSinkTask",
  "mdc": {
    "connector.context": "[eventbridge-sink|task-0] "
  }
}

Expected behavior

The connector should be able to authenticate using the InstancProfileCredentialsProvider. Versions prior to v1.3.1 would have an authentication log like the following:

cxp-connect-eventbridge-connect-0 cxp-connect-eventbridge-connect {"@timestamp":"2024-09-18T18:12:32.669Z","source_host":"cxp-connect-eventbridge-connect-0","file":"EventBridgeWriter.java","method":"<init>","level":"INFO","line_number":"137","thread_name":"task-thread-eventbridge-sink-0","@version":1,"logger_name":"software.amazon.event.kafkaconnector.EventBridgeWriter","message":"[@9d66194-dirty] Resolving iam credentials","class":"software.amazon.event.kafkaconnector.EventBridgeWriter","mdc":{"connector.context":"[eventbridge-sink|task-0] "}}
cxp-connect-eventbridge-connect-0 cxp-connect-eventbridge-connect {"@timestamp":"2024-09-18T18:12:32.791Z","source_host":"cxp-connect-eventbridge-connect-0","file":"Logger.java","method":"debug","level":"DEBUG","line_number":"85","thread_name":"task-thread-eventbridge-sink-0","@version":1,"logger_name":"software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider","message":"Loaded credentials from IMDS with expiration time of 2024-09-18T18:42:32Z","class":"software.amazon.awssdk.utils.Logger","mdc":{"connector.context":"[eventbridge-sink|task-0] "}}

Environment:

Client Version: v1.29.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3
Java 11

Additional context

I reverted commit 1bc23b83f65659ba2cac3c637974cddc04b63dc9 and redeployed the connector and the expected behavior was observed:

cxp-connect-eventbridge-connect-0 cxp-connect-eventbridge-connect {"@timestamp":"2024-09-18T18:12:32.669Z","source_host":"cxp-connect-eventbridge-connect-0","file":"EventBridgeWriter.java","method":"<init>","level":"INFO","line_number":"137","thread_name":"task-thread-eventbridge-sink-0","@version":1,"logger_name":"software.amazon.event.kafkaconnector.EventBridgeWriter","message":"[@9d66194-dirty] Resolving iam credentials","class":"software.amazon.event.kafkaconnector.EventBridgeWriter","mdc":{"connector.context":"[eventbridge-sink|task-0] "}}
cxp-connect-eventbridge-connect-0 cxp-connect-eventbridge-connect {"@timestamp":"2024-09-18T18:12:32.791Z","source_host":"cxp-connect-eventbridge-connect-0","file":"Logger.java","method":"debug","level":"DEBUG","line_number":"85","thread_name":"task-thread-eventbridge-sink-0","@version":1,"logger_name":"software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider","message":"Loaded credentials from IMDS with expiration time of 2024-09-18T18:42:32Z","class":"software.amazon.awssdk.utils.Logger","mdc":{"connector.context":"[eventbridge-sink|task-0] "}}

[embano1]

@sugarcrm-jgminder thx for flagging this and glad you could revert to the previous version for now. I'll discuss this with the team and see how it can be fixed.

cc/ @maschnetwork @agebhar1

[embano1]

@sugarcrm-jgminder it sounds though that the initial credential load that we do here succeeds and the exception is thrown after a token refresh?

[sugarcrm-jgminder]

@embano1 Below is a full message after a Kafka Connect pod starts with the failure stacktrace. The credential load fails and it fails to find any suitable credential provider within the default chain:

{
  "exception": {
    "stacktrace": "org.apache.kafka.connect.errors.ConnectException: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default): Cannot invoke \"java.nio.file.Path.getFileSystem()\" because \"path\" is null, ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]\n\tat software.amazon.event.kafkaconnector.EventBridgeWriter.<init>(EventBridgeWriter.java:141)\n\tat software.amazon.event.kafkaconnector.EventBridgeSinkTask.start(EventBridgeSinkTask.java:43)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:329)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default): Cannot invoke \"java.nio.file.Path.getFileSystem()\" because \"path\" is null, ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]\n\tat software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)\n\tat software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)\n\tat software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)\n\tat software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:129)\n\tat software.amazon.event.kafkaconnector.EventBridgeWriter.<init>(EventBridgeWriter.java:139)\n\t... 10 more",
    "exception_class": "org.apache.kafka.connect.errors.ConnectException",
    "exception_message": "software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default): Cannot invoke \"java.nio.file.Path.getFileSystem()\" because \"path\" is null, ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]"
  },
  "source_host": "cxp-connect-eventbridge-connect-0",
  "method": "doRun",
  "level": "ERROR",
  "message": "WorkerSinkTask{id=eventbridge-sink-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted",
  "mdc": {
    "connector.context": "[eventbridge-sink|task-0] "
  },
  "@timestamp": "2024-09-17T13:33:44.789Z",
  "file": "WorkerTask.java",
  "line_number": "212",
  "thread_name": "task-thread-eventbridge-sink-0",
  "@version": 1,
  "logger_name": "org.apache.kafka.connect.runtime.WorkerTask",
  "class": "org.apache.kafka.connect.runtime.WorkerTask"
}

[agebhar1]

Hi @embano1, @maschnetwork,

I had a look how the code behaves if the DefaultCredentialsProvider is build w/ or w/o providing ProfileFileSupplier.defaultSupplier() 1bc23b8. As far as I can see, the build of the DefaultCrednetialsProvider propagates the ProfileFileSupplier to the InstanceProfileCredentialsProvider:

https://github.com/aws/aws-sdk-java-v2/blob/832c1b93a2c2712e65d03f687d11fde8a8a6faa2/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.java#L106-L110

The InstanceProfileCredentialsProvider propagates it further:

https://github.com/aws/aws-sdk-java-v2/blob/832c1b93a2c2712e65d03f687d11fde8a8a6faa2/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/InstanceProfileCredentialsProvider.java#L107-L112

Finally it's used in

https://github.com/aws/aws-sdk-java-v2/blob/2.28.6/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/internal/Ec2MetadataConfigProvider.java#L86-L105

and

https://github.com/aws/aws-sdk-java-v2/blob/2.28.6/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/internal/Ec2MetadataDisableV1Resolver.java#L47-L51

Ec2MetadataConfigProvider:

private ProfileFile resolveProfileFile() {
    if (profileFile != null) {
        return profileFile.get();
    }

    return ProfileFile.defaultProfileFile();
}

ProfileFile:

public static ProfileFile defaultProfileFile() {
    return ProfileFile.aggregator()
                      .applyMutation(ProfileFile::addCredentialsFile)
                      .applyMutation(ProfileFile::addConfigFile)
                      .build();
}

private static void addCredentialsFile(ProfileFile.Aggregator builder) {
    ProfileFileLocation.credentialsFileLocation()
                       .ifPresent(l -> builder.addFile(ProfileFile.builder()
                                                                  .content(l)
                                                                  .type(ProfileFile.Type.CREDENTIALS)
                                                                  .build()));
}

private static void addConfigFile(ProfileFile.Aggregator builder) {
    ProfileFileLocation.configurationFileLocation()
                       .ifPresent(l -> builder.addFile(ProfileFile.builder()
                                                                  .content(l)
                                                                  .type(ProfileFile.Type.CONFIGURATION)
                                                                  .build()));
}

ProfileFileSupplier:

static ProfileFileSupplier defaultSupplier() {
    Optional<ProfileFileSupplier> credentialsSupplierOptional
        = ProfileFileLocation.credentialsFileLocation()
                             .map(path -> reloadWhenModified(path, ProfileFile.Type.CREDENTIALS));

    Optional<ProfileFileSupplier> configurationSupplierOptional
        = ProfileFileLocation.configurationFileLocation()
                             .map(path -> reloadWhenModified(path, ProfileFile.Type.CONFIGURATION));

    ProfileFileSupplier supplier = () -> ProfileFile.builder().build();
    if (credentialsSupplierOptional.isPresent() && configurationSupplierOptional.isPresent()) {
        supplier = aggregate(credentialsSupplierOptional.get(), configurationSupplierOptional.get());
    } else if (credentialsSupplierOptional.isPresent()) {
        supplier = credentialsSupplierOptional.get();
    } else if (configurationSupplierOptional.isPresent()) {
        supplier = configurationSupplierOptional.get();
    }

    return supplier;
}

It looks like they behave the same ProfileFileSupplier#defaultSupplier and ProfileFile#defaultProfileFile.

The log message for successfully resolved credentials/exception is done here:

InstanceProfileCredentialsProvider:

private RefreshResult<AwsCredentials> refreshCredentials() {
    if (isLocalCredentialLoadingDisabled()) {
        throw SdkClientException.create("IMDS credentials have been disabled by environment variable or system property.");
    }

    try {
        LoadedCredentials credentials = httpCredentialsLoader.loadCredentials(createEndpointProvider());
        Instant expiration = credentials.getExpiration().orElse(null);
        log.debug(() -> "Loaded credentials from IMDS with expiration time of " + expiration);

        return RefreshResult.builder(credentials.getAwsCredentials())
                            .staleTime(staleTime(expiration))
                            .prefetchTime(prefetchTime(expiration))
                            .build();
    } catch (RuntimeException e) {
        throw SdkClientException.create("Failed to load credentials from IMDS.", e);
    }
}

[agebhar1]

I tried the following locally:

package software.amazon.event.kafkaconnector;

import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import software.amazon.awssdk.profiles.ProfileFile;

public class Test {

    public static void main(String[] args) {
        var provider = InstanceProfileCredentialsProvider.builder()
                .asyncCredentialUpdateEnabled(false)
                .profileFile(ProfileFile.defaultProfileFile())
                .profileName(null)
                .build();
        var creds = provider.resolveCredentials();

        System.out.println(creds);
    }

}

It fails with the exception:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Failed to load credentials from IMDS.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:167)
	at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
	at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448)
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:149)
	at software.amazon.event.kafkaconnector.Test.main(Test.java:14)
Caused by: java.io.UncheckedIOException: java.net.SocketTimeoutException: connect timed out
	at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:180)
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110)
	at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:283)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:212)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:158)
	... 6 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:509)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:604)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:277)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:376)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:397)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1252)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1231)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1080)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1014)
	at software.amazon.awssdk.regions.internal.util.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:45)
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:112)
	at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:283)
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
	... 10 more

The cause of SdkClientException: Failed to load credentials from IMDS. is in the socket timeout to 169.254.169.254.

Instead of using the default credentials provider we should test/instantiate the InstanceProfileCredentialsProvider directly in a debug-build to see the root cause @embano1, @maschnetwork.

[embano1]

@sugarcrm-jgminder would you be open to try a test build to debug this further as per @agebhar1's recommendation?

@agebhar1 thx a ton for jumping in here!

[sugarcrm-jgminder]

@embano1 Sure no problem. I'd be happy to trying test builds out for debugging this issue.

[agebhar1]

The different behavior of InstanceProfileCredentialsProvider is due to the (default) profile.

package software.amazon.event.kafkaconnector;

import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import software.amazon.awssdk.profiles.ProfileFile;
import software.amazon.awssdk.profiles.ProfileFileSupplier;

public class Test {

    public static void main(String[] args) {
        var provider = InstanceProfileCredentialsProvider.builder()
                //.profileFile(ProfileFile.defaultProfileFile()) // w/ < v1.3.1
                .profileFile(ProfileFileSupplier.defaultSupplier()) // w >= v1.3.1
                .profileName(null)
                .build();
        var creds = provider.resolveCredentials();

        System.out.println(creds);
    }

}

This yields to the following error, if no AWS profile exists in the filesystem:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Failed to load credentials from IMDS.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:167)
	at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
	at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448)
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:149)
	at software.amazon.event.kafkaconnector.Test.main(Test.java:15)
Caused by: java.lang.NullPointerException
	at java.base/java.nio.file.Files.provider(Files.java:101)
	at java.base/java.nio.file.Files.newInputStream(Files.java:156)
	at software.amazon.awssdk.profiles.ProfileFile$BuilderImpl.lambda$build$0(ProfileFile.java:314)
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
	at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
	at software.amazon.awssdk.profiles.ProfileFile$BuilderImpl.build(ProfileFile.java:314)
	at software.amazon.awssdk.profiles.ProfileFileSupplier.lambda$defaultSupplier$2(ProfileFileSupplier.java:58)
	at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.resolveProfileFile(Ec2MetadataConfigProvider.java:129)
	at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.resolveProfile(Ec2MetadataConfigProvider.java:121)
	at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.configFileEndpointOverride(Ec2MetadataConfigProvider.java:117)
	at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.getEndpointOverride(Ec2MetadataConfigProvider.java:102)
	at software.amazon.awssdk.auth.credentials.internal.Ec2MetadataConfigProvider.getEndpoint(Ec2MetadataConfigProvider.java:70)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getImdsEndpoint(InstanceProfileCredentialsProvider.java:224)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:210)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:158)
	... 6 more

[embano1]

@sugarcrm-jgminder forgot to ask whether IRSA would be an option for you? https://aws.github.io/aws-eks-best-practices/security/docs/iam/#iam-roles-for-service-accounts-irsa

[embano1]

Regarding your question in your test PR whether it can be considered a bug on the SDK behavior, I'm inclined to agree here

You can configure any credentials provider that has a profileFile() method on its builder to reload profile credentials. These credentials profile classes are: ProfileCredentialsProvider, DefaultCredentialsProvider, InstanceProfileCredentialsProvider, and ProfileTokenProvider.

When ProfileCredentialsProvider.resolveCredentials() is called, the SDK for Java reloads the settings. ProfileFileSupplier.defaultSupplier() is one of several convenience implementations of ProfileFileSupplier provided by the SDK.

Reading this in the SDK docs I would expect the convenience method to work in any of the supported cases. @maschnetwork WDYT?

[embano1]

A temporary fix for us would be to implement a custom supplier using the default behavior but catching NPEs from InstanceProfileCredentialsProvider, right?