feat(nimbus): Always fetch newer results

[RJAK11]

Because

• the fetch task used experiment status and a fixed time window to decide when to fetch results

This commit

• adds a field to track when an experiment's results were last loaded
• finds the newest modified time among each experiment's statistics, metadata, and errors files
• fetches results when that timestamp is newer than the stored one and saves the new timestamp

Fixes #15737

[mikewilli]

Will this work with our error checks? Right now we inject an error inside the client code if we expect results but the file doesn't exist. I think with this new feature we'd just skip over them.

[RJAK11]

Will this work with our error checks? Right now we inject an error inside the client code if we expect results but the file doesn't exist. I think with this new feature we'd just skip over them.

Thank you @mikewilli for the review! You are right, the changes would've skipped the error checks and I didn’t notice that. I added a new function so that the task also re-fetches when we expect a window whose file isn't in the bucket, so now the client should still inject the error. It's bounded by the analysis in the exact way we originally used so a result that never arrives doesn't just keep refetching forever. This might be a messy solution but I couldn’t think of another approach. Let me know what you think!

[jaredlockhart]

Okay I understand better, yes it makes sense to compute whether we 'expect' results to compute errors but if there's anything newer we always fetch it anyways so it still catches the 'long since ended but forced reran' catches so we don't have to manually reload. This is good, great test coverage. One idea about using the stored results date instead of adding a new field 🙏