PSIJ stress test

[wilke]

Define and implement stress test scenarios, e.g.

• throughput
• concurrent jobs

GanttStart: 2022-08-29
GanttDue: 2022-09-14

[andre-merzky]

This script was used to stress-test PSIJ:

#!/usr/bin/env python3

import sys
import psij

import radical.utils as ru


class StateCB(psij.JobStatusCallback):

    def __init__(self, prof):
        self._cnt = 0
        self._prof = prof

    def job_status_changed(self, job, status):
        self._prof.prof('state', uid=job.id, state=status.state)


if __name__ == '__main__':

    name = 'local'
    url  = None

    if len(sys.argv) > 1: name = sys.argv[1]
    if len(sys.argv) > 2: url = sys.argv[2]

    for e in range(16):

        jobs = list()
        n = 2 ** e
        print('=== %d' % n)

        prof = ru.Profiler('psij.stress.%s.%06d' % (name, n))

        jex = psij.JobExecutor.get_instance(name=name, url=url)
        spec = psij.JobSpec(executable='/bin/sleep', arguments=['1'])

        prof.prof('create_start', 'jex', msg=str(n))
        for _ in range(n):
            job = psij.Job(spec)
            prof.prof('create', job.id)
            prof.prof('state', job.id, state=job.status.state)
            jobs.append(job)
        prof.prof('create_stop', 'jex', msg=str(n))

        prof.prof('addcb_start', 'jex', msg=str(n))
        for job in jobs:
            job.status_callback = StateCB(prof)
            prof.prof('addcb', job.id)
        prof.prof('addcb_stop', 'jex', msg=str(n))

        prof.prof('submit_start', 'jex', msg=str(n))
        for job in jobs:
            jex.submit(job)
            prof.prof('submit', job.id)
        prof.prof('submit_stop', 'jex', msg=str(n))

        jobs[0].wait()
        prof.prof('wait_start', 'jex', msg=str(n))
        prof.prof('wait', jobs[0].id)
        for j in jobs[1:]:
            j.wait()
            prof.prof('wait', job.id)
        prof.prof('wait_stop', 'jex', msg=str(n))

The prof() calls record the respective events in a profile. radical.analytics was used to create the following plots:

For the local executor running 1024 jobs (sleep 1), plot time of state transition:

Conclusion:

• job creation takes basically no time
• job submission is mostly linear
• job collection on completion shows some (negligible) bulking behavior. Those could be system related - I did not dig deeper.

The same plot type for the local executor running 32k sleep 1 jobs:

Conclusion:

• same as above, performance is steady up to that scale

Also: a proper scaling plot for total job creation time, time needed to set job state callbacks, job submission time, and job collection time (see *_start and *_stop events in the script above) for 1 .. 32k jobs:

Conclusion:

• almost constant time for the first 3 metrics
• basically linear scaling for job collection - possibly again caused by system load, but does not matter, it's 'good enough' (TM) by far.

When running a similar test against a proper batch system(I used Frontera / slurm), the metrics are either (a) the same as for the local runs (creating jobs, register callbacks, etc), or (b) dominated by the batch system itself (job throughout, job collection time, job submission time). It might be useful to micro-benchmark the time used to create job scripts - but unless the shared FS acts up, that should be negligible compared to batch submission overhead.

[andre-merzky]

For completeness, some slurm plots:

[hategan]

Thanks @andre-merzky. These are very nice.

Just want to be clear: the times are total times? Aka, the linear scaling of wait() is saying, roughly, that the total time for all wait() calls for 20 jobs is twice as much as the total time for all wait() calls for 10 jobs?

[andre-merzky]

Just want to be clear: the times are total times? Aka, the linear scaling of wait() is saying, roughly, that the total time for all wait() calls for 20 jobs is twice as much as the total time for all wait() calls for 10 jobs?

That is correct.

I just realized though that I misrepresented the scaling figure: the submit time is what shows linear scaling, the wait time is mostly constant. That is likely because almost all jobs have finished once the last submit is done and the wait is just collecting state... Sorry for any confusion, should have been more careful...

[hategan]

the submit time is what shows linear scaling

That's somewhat to be expected since it's a synchronous process.

[wilke]

Thanks - closing this task for now.

[hategan]

We should probably save this info. I have a feeling it will be useful in the near future.

[andre-merzky]

We could add a page about performance / scale in the user guide. I don't want to commit the code to the psi/j repo though as it pulls in radical.utils which I think we want to avoid right now. Any thoughts where to save it? Should we open an experiments repo or something like that?

[hategan]

Maybe use the wiki for pointers we want to keep but not distribute with the code?

In this case, we would only link to the closed issue, so that our heads don't garbage collect it.

[andre-merzky]

Yeah, that would also work.