Retry when getting the pod_environment_secret

[snarlysodboxer]

See #1776.

Not sure if this is how you would want this implemented, but it didn't take long so I thought I'd get the conversation going. It's unclear to me how retries here might play into retries at higher levels.

Should the interval or timeout be configurable? Should it retry indefinitely?

[snarlysodboxer]

I see the e2e failure. Wondering if it's flakiness? The error doesn't seem related to my changes, but I could be wrong...

[FxKu]

LGTM. Thanks. Yeah e2e tests on github are very flaky. You can ignore them.

[FxKu]

Ok, one unit test is failing because the intervals must be set for it. Please add the following two lines (I mark then with +):

				Resources: config.Resources{
					PodEnvironmentSecret:  "idonotexist",
+					ResourceCheckInterval: time.Duration(3),
+					ResourceCheckTimeout:  time.Duration(10),
				},

here and here.

[snarlysodboxer]

Ok, one unit test is failing because the intervals must be set for it. Please add the following two lines

Done.

I had a lot of trouble trying to run the tests. I couldn't get them to pass even in the master branch. GO111MODULE=on go test ./... returns undefined: mocks.NewMockVolumeResizer errors.

[snarlysodboxer]

I was able to get the unit tests running after some trouble. - I added to the docs/developers.md to hopefully help others avoid the problems I ran into.

[FxKu]

Suggested change

	err: fmt.Errorf(`could not read Secret PodEnvironmentSecretName: still failing after 3 retries: secret.core "idonotexist" not found`),
	err: fmt.Errorf("could not read Secret PodEnvironmentSecretName: still failing after 3 retries: secret.core %q not found", "idonotexist"),

[FxKu]

Would make sense to have a constant for the name, check #1794

[dmvolod]

Also would be nice to don't hardcode 3 as its 10/3, i.e. ResourceCheckTimeout/ResourceCheckInterval

[snarlysodboxer]

I've added this commit, is that what you were looking for?

[FxKu]

👍

[jopadi]

👍

[FxKu]

Thanks @snarlysodboxer for your contribution!