Quality assurance of the MONAI/examples folder

[wyli]

Is your feature request related to a problem? Please describe.
As the size and content scope of the MONAI/examples folder increase,
it's necessary to figure out the hardware/software requirements for running the examples,
and also provide some forms of quality assurance of the example codes.

Describe the solution you'd like
could automatically run the examples as a part of the automated CI/CD pipeline?

Describe alternatives you've considered
manually verifying all the examples regularly (tedious and error-prone)

Additional context
see also https://github.com/Project-MONAI/MONAI/issues/296

[arp95]

@wyli Regarding the python files, we can setup running them during CI/CD right, but wont executing each file take time and increase build time?
Also, is there a way for running jupyter notebooks automatically, like through a command?

[wyli]

yes, with this feature implemented, the CI/CD is likely to take significantly more time. but I think we should address it with a separate ticket (e.g. refactoring the CI job configures so that they could run in parallel and take less time).

for this ticket we need to implement some initial pipelines (welcome discussions) for:

• checking the coding style
• checking that the examples work fine:
  • no syntax/type hint error
  • can finish in expected execution time
  • can generate correct outcomes -- model files, model quality measurements, tensorboard files

the notebooks are trickier (see also Project-MONAI/MONAI#874), there are some discussions in https://github.com/Project-MONAI/MONAI/issues/296 of the relevant tools but we still need to check their feasibility for our use case

cc @IsaacYangSLA @Nic-Ma @ericspod @benjamin-gorman

[arp95]

Thanks for the explanation. Regarding checking the syntax error, we can simply execute the python file during the CI/CD "python filename.py" and then give it some upper execution time limit? Also, if I can be of help here, I would like to contribute to some of the points mentioned above.

[wyli]

thanks @arp95, would be great to have a simple run_example_tests.sh for those python filename.py commands with a timer (and perhaps some checks of the commandline output logs with grep?), then we take another iteration to integrate the commands into the ci configure for example:
https://github.com/Project-MONAI/MONAI/blob/8f1c290b58ddfb4f9a9f5810916c3a7d48e26643/.github/workflows/setupapp.yml#L171-L181

[arp95]

sure. i will work on the shell script and then raise a pull request.

[arp95]

@wyli i was running the file "densenet_training_array.py". It requires the data to be downloaded from this website, https://brain-development.org/ixi-dataset/. Should we save the data in some MONAI google drive for online running or is there another way around this? Because we cant expect to be downloading data everytime during the CI/CD build right?

[wyli]

@wyli i was running the file "densenet_training_array.py". It requires the data to be downloaded from this website, https://brain-development.org/ixi-dataset/. Should we save the data in some MONAI google drive for online running or is there another way around this? Because we cant expect to be downloading data everytime during the CI/CD build right?

I put those demo data here: https://www.dropbox.com/s/y890gb6axzzqff5/testing_ixi_t1.tar.gz?dl=1 please let me know if there's further issue

[arp95]

thanks for the help. will check it out and let you know if I face any issues.