version: every will skip versions if a parallel upstream job's latest build finishes before an older one

[jpalermo]

Bug Report

Follow up to #666 and #563

We're running 2.4.0.

What I saw this morning, Job starts multiple copies of itself; A, B, C, and D. In that order.

The jobs finished in A, C, D, B order. The next job that depends on that resource (which is version: every) did not run with resource version B.

[joonas]

I'm also seeing this behavior on my end on the pullrequest-resource, currently running Concourse 2.4.0

[vito]

So this is a tricky one. The semantics for version: every coming from a job whose builds run in parallel are complicated by how the passed constraints work with version: every. See #358 (comment)

tl;dr: passed constraints allow it to skip versions, otherwise when a bad version enters the mix, all downstream jobs can no longer run.

The scheduling for the second job would have to know that a build is in-flight for the ideal next version, and know to wait for it to finish before determining the inputs.

[oppegard]

Our team also makes heavy use of the pullrequest-resource, and reliably running all passed versions of a PR in downstream jobs is important. As I currently understand it, we need to set serial: true or max_in_flight: 1 to ensure all PRs make it through a pipeline?

[vito]

@oppegard That's right. Until we figure out whether there's something better we could do. I'm a little wary of the changes this kind of implies; having scheduling so dependent on ephemeral job state feels wrong.

[vito]

Possible workaround:

Have each job do a put to an otherwise-unused time resource. Then thread that through the subsequent jobs with trigger: true and a passed constraint. Do this for each job. This would make it so there's always a newer version once the build completes.

e.g.:

resources:
- name: my-pr
  type: pr
  source: {...}

- name: dummy-time
  type: time
  source: {interval: 24h} # doesn't matter, but something has to be there

jobs:
- name: job-1
  plan:
  - get: my-pr
    version: every
    trigger: true
  - task: test-pr
    # ...
  - put: dummy-time

- name: job-2
  plan:
  - get: my-pr
    trigger: true
    passed: [job-1]
  - get: dummy-time
    version: every
    trigger: true
    passed: [job-1]
  - # ...
  - put: dummy-time

- name: job-3
  plan:
  - get: my-pr
    trigger: true
    passed: [job-2]
  - get: dummy-time
    version: every
    trigger: true
    passed: [job-2]
  - # ...
  - put: dummy-time

Note that version: every is configured on the dummy-time resource in subsequent jobs.

[thewoolleyman]

Hi @vito,

Is still an issue (and is the above workaround is still required) when using the pullrequest-resource, even though the mentioned problems with using passed and every together have been (I believe?) resolved by #358 (comment) ?

If so, could you please explain the details, and why the problem seems to be specific to pullrequest-resource?

My guess is that it is because the pullrequest-resource conflates two independent "versions" (PR numbers and SHAs) into a single "version", which breaks some of the concourse assumptions about how resources and versioning work. It would be great to have you confirm or deny that, and if so, possibly explain why from your perspective.

Thanks!
-- Chad

[vito]

The problem is not specific to the PR resource. This issue is still open. The linked comment explains why this issue exists; it's a fundamental issue with passed and every together, where it's either broken this way or broken in the way that comment mentioned, and I think this way is the less broken of the two (as the alternative is a deadlocked pipeline).

[thewoolleyman]

@vito OK, thanks.

EDIT
I think understand now. Actually, I'm still confused. I thought when you said above "the second job", you meant a downstream job, but now I realize you probably meant a concurrently executing instance of the same job. So, I'm still trying to understand the crux of the problem, and why it only occurs with the combination of version: every, passed:, and non-serial or max-in-flight > 1 (and others on my team are also having trouble understanding it). So, it would be great if you could expand and clarify exactly what problem this causes, and why:

tl;dr: passed constraints allow it to skip versions, otherwise when a bad version enters the mix, all downstream jobs can no longer run.

The scheduling for the second job would have to know that a build is in-flight for the ideal next version, and know to wait for it to finish before determining the inputs.

END EDIT

But, with respect to the pullrequest-resource specifically:

• The only reason we need to use "every" is because the pullrequest-resource conflates two independent "versions" (PR# and SHA) into one emitted "version". If we want the latest (not every) SHA to be built for every PR#.
• If we only used the standard concourse git resource, which only emits one "version" (a SHA), in conjunction with a pipeline-per-pr# (i.e. pipeline-per-branch) approach as you discuss here, we would not need to use "every", because the separate pipeline-per-pr# (each with separate standard git resources defined for the PRs' branches) would ensure the latest SHA for the per-branch git resource would always be run (not every SHA, which is fine because we only care about running the latest commit on the PR's branch).
• Thus, since we wouldn't need to use "every", we would not be exposed to this issue.

Correct?

If not, then please explain in more detail, because I'm still confused :)

Thanks,
-- Chad

[thewoolleyman]

@vito Also, I tried out the workaround above.

It doesn't appear to work if you ever have more than one entry in a passed: array.

E.g., if you try this, job-3 hang and never start with the message "waiting for a suitable set of input versions\n dummy-time - no versions satisfy passed constraints"

resources:
- name: my-pr
  type: pr
  source: {...}

- name: dummy-time
  type: time
  source: {interval: 24h} # doesn't matter, but something has to be there

jobs:
- name: job-1
  plan:
  - get: my-pr
    version: every
    trigger: true
  - task: test-pr
    # ...
  - put: dummy-time

- name: job-2a
  plan:
  - get: my-pr
    trigger: true
    passed: [job-1]
  - get: dummy-time
    version: every
    trigger: true
    passed: [job-1]
  - # ...
  - put: dummy-time

- name: job-2b
  plan:
  - get: my-pr
    trigger: true
    passed: [job-1]
  - get: dummy-time
    version: every
    trigger: true
    passed: [job-1]
  - # ...
  - put: dummy-time

- name: job-3
  plan:
  - get: my-pr
    trigger: true
    passed: [job-2a, job-2b]
  - get: dummy-time
    version: every
    trigger: true
    passed: [job-2a, job-2b]
  - # ...
  - put: dummy-time

Is this expected (and thus the workaround doesn't work in this situation), or do I have something wrong?

Thanks,
-- Chad