CI setup on OLCF Crusher

[ketancmaheshwari]

I have been trying to run a cron driven CI setup for psi-j on OLCF Crusher. Looking at the logs, there seems to be a few problems. First, as indicated on the comments, I added a module load statement in the psij-ci-load file. However, I am seeing in the testing.log that the module command is throwing error: psij-ci-load: line 5: module: command not found.

Next, I am seeing that the setup is trying to ssh into localhost and failing because it requires MFA:
DEBUG Output from ['ssh', '-oBatchMode=yes', '-oStrictHostKeyChecking=no', 'localhost', 'true']

Which leads to a credential prompt and error:
ketan2@localhost: Permission denied (publickey,keyboard-interactive,hostbased).

I am seeing more errors further down on tests for slurm but unable to find the cause and suspect that these above issues might be the reason for those.

[hategan]

Thanks Ketan.

I have been trying to run a cron driven CI setup for psi-j on OLCF Crusher. Looking at the logs, there seems to be a few problems. First, as indicated on the comments, I added a module load statement in the psij-ci-load file. However, I am seeing in the testing.log that the module command is throwing error: psij-ci-load: line 5: module: command not found.

I'm a bit confused here. What comments are you referring to? Do you need to load a module in order for tests to work? Is the module system installed on crusher?

Next, I am seeing that the setup is trying to ssh into localhost and failing because it requires MFA: DEBUG Output from ['ssh', '-oBatchMode=yes', '-oStrictHostKeyChecking=no', 'localhost', 'true']

One test does need SSH, so the setup tries to run that in order to add the site key. We could probably add a timeout to that part.

Which leads to a credential prompt and error: ketan2@localhost: Permission denied (publickey,keyboard-interactive,hostbased).

I am seeing more errors further down on tests for slurm but unable to find the cause and suspect that these above issues might be the reason for those.

Possibly, but we can't tell without more information. Perhaps paste some of those errors here.

[ketancmaheshwari]

By comments, I am referring to the ones in the psij-ci-load file:

$ cat psij-ci-load 
# This file can be used to set up an environment before tests are run.
# For example, one can issue relevant "module load" commands here. The
# file will then be sourced by both psij-ci-setup and psij-ci-run.

module load cray-python/3.9.12.1

The other errors further down are as follows:

            f"{self.status_code} Server Error: {reason} for url: {self.url}"
        )

    if http_error_msg:

      raise HTTPError(http_error_msg, response=self)

E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://psij.testing.exaworks.org/result

______________ ERROR at teardown of test_simple_job[slurm:single] ______________

capsys = <_pytest.capture.CaptureFixture object at 0x7fffcfb52fd0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x7fffcfb52fa0>
request = <SubRequest 'report' for <Function test_simple_job[slurm:single]>>
pytestconfig = <_pytest.config.Config object at 0x7fffe8c0dca0>

@pytest.fixture(autouse=True)
def report(capsys, caplog, request, pytestconfig):
    save = pytestconfig.getoption('save_results')
    upload = pytestconfig.getoption('upload_results')
    if save or upload:
        caplog.set_level(logging.DEBUG)
        start = _now()

And many others that are spread over the 60K+ lines of the testing.log file.

[hategan]

I can't quite make out what the error is. I suspect there's more relevant info in the log. Can you do me a favor and email me two files:
psij-ci-env
testing.log

[ketancmaheshwari]

Sent. Thank you.

[hategan]

So I looked at the log. It seems like some python is available to run things, so the fact that the module load fails is not critical. It's puzzling though. So maybe we can work more to figure that out next week.

I'll take a look at the server logs. For some reason, the results fail to upload.

[andre-merzky]

Module load failing or the module command not being available may hint at incomplete shell initialization. Is the shell used to run the crontab entry the same as your login shell? If so, are some parts of your shell initialization (~/.bashrc) shielded for non-interactive logins?

If everything else fails, a source ~/.bashrc (for bash, your mileage may vary) before the module load might help, or maybe you can trigger the use of an interactive login shell (bash -il)?

[hategan]

@andre-merzky, to add some info to the discussion, the environment is saved in psij-ci-env. However, I noticed now that psij-ci-load is invoked before psij-ci-env. Perhaps not the right choice. Thoughts?

[ketancmaheshwari]

Module is a bash function on Crusher. I am not setting it up. I believe it is being set somewhere as part of /etc/rc:

[ketan2@login2.crusher ~]$ type module
module is a function
module () 
{ 
    eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}

[hategan]

Module is a bash function

I was suspecting this to be the case. And I don't think functions are saved in psij-ci-env. However, the result of calling module load x is. So you can remove it from psij-ci-load.

[andre-merzky]

@andre-merzky, to add some info to the discussion, the environment is saved in psij-ci-env. However, I noticed now that psij-ci-load is invoked before psij-ci-env. Perhaps not the right choice. Thoughts

Yes, I agree - switching those around would seem like a reasonable choice IMHO.

PS.: I just checked the env capture, and the problem might be here actually (in addition to the proposed switch): module is declared as a bash function, and bash has the (non-POSIX compliant) way to export functions as env variables:

$ foo(){ 
> echo bar;
> }
$ foo
bar
$ export -f foo
$ env | grep foo
BASH_FUNC_foo%%=() {  echo bar
$ declare -x | grep foo
declare -x foo

Our env capture uses declare -x, but as seen above that does not capture the function body, and thus the module function will not be made available in the new shell.

We have been struggling with bash for quite a bit in the past - if you are interested, radical.utils has an env_dump and env_prep which addresses this issue. Feel free to use or copy that code if you think it's useful:

$ . ~/j/ru/bin/radical-utils-env.sh 
$ env_dump | grep foo > tmp.env
$ cat tmp.env
BASH_FUNC_foo%%=() {  echo bar\n}
$ env_prep -s tmp.env -t prep.env
$ cat prep.env

# blacklist
unset 'PS1'
unset 'LS_COLORS'
unset '_'

# export

# functions
foo () {
 echo bar
}
export -f foo

# end
$ unset -f foo
$ foo
command not found
$ . prep.env
$ foo
bar

[hategan]

Our env capture uses declare -x, but as seen above that does not capture the function body, and thus the module function will not be made available in the new shell.

That was intentional. The environment saved in psij-ci-env is not meant to be an interactive one. In other words, it is meant to capture what happens after module load rather than enable one to invoke module load. I think the problem is the comment in psij-ci-load which seems to suggest, a bit too quickly, that some kind of module load should be added there. I believe the solution is to make it clear that this should only be done if things fail to work without. Perhaps a quick test that starts with an empty environment, loads psij-ci-env, and invokes a simple python tool during setup may be useful to inform the user that things are likely to be ok.

I would perhaps caution against a too-complex solution for this problem if the problem isn't even a problem.

[ketancmaheshwari]

I did a git pull yesterday and resubmitted the tests but now the log shows the following error:

From https://github.com/ExaWorks/psi-j-python
 * branch            HEAD       -> FETCH_HEAD
Reading configuration... OK
Cloning repository... FAILED
Traceback (most recent call last):
  File "tests/ci_runner.py", line 195, in <module>
    run_tests(conf, site_ids, branches, clone)
  File "tests/ci_runner.py", line 149, in run_tests
    do_clone(conf, tmpp)
  File "tests/ci_runner.py", line 80, in do_clone
    run('git', 'clone', 'https://github.com/%s.git' % repo, 'code', cwd=str(dir))
  File "tests/ci_runner.py", line 55, in run 
    cwd=cwd, text=True)
  File "/usr/lib64/python3.6/subprocess.py", line 423, in run 
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'text'

[hategan]

We claim Python 3.6 compatibility, but the text parameter appears to have been added in Python 3.7.

We could fix based on this statement in the python docs:

The universal_newlines argument is equivalent to text and is provided for backwards compatibility.

(from https://docs.python.org/3/library/subprocess.html#subprocess.run)