transitioning to python 3

[goldmanm]

Python3 is likely where new developing for python is occurring. After some discussion of different ideas and issues we thought there are kind of two parts to the transition.

1. writing test cases and other changes that improve current RMG functionality (which also aid in transitioning to python 3)
2. applying other changes to RMG that do not provide direct benefit to current RMG functioning, like 2to3 or upgrading dependencies.

@rspeth, we were wondering how much effort was involved in upgrading Cantera to python3 (and maintaining a python2 version) and if you have any other wisdom.

[goldmanm]

Here is information from our discussion:

many people think python 3 'is' the future of python

more intuitive feel

• 1 / 2 = 0.5 not 1 / 2 = 0
• Unicode support is default
• print('this is now a method')
• input/output classes streamlined
• generally more coherant programing experience

no more bug fixes provided for 2.X after 2020

any new updates applied only to 3.X since 2013

converting is difficult

version 3 not backwards compatible

• some programs exist to 'help' but still take training

need all dependencies compatible

no direct research value

the conversion process has many helpful tools

1. ensure good test coverage of code with coverage.py
2. ensure dependency compatibility with caniusepython3
3. rewrite ambiguous parts of code. __future__ imports help
4. use porting converters like 2to3 for the rest
5. touch up failing test cases

much easier than rewriting from java

RMG-Py currently has 44% of its code tested

You can run coverage.py with command line make test. Then open testing/coverage/*.html to see methods aren't tested.

coverage looks at how many lines of executable code are called when all the test methods are run.

Recent master branch has 44% coverage. 'good' coverage is around 80%.

The amount of code called when running minimal is 21%,

Warning: Website has little to no tests.

areas of RMG with lower testing

• uncertainty analysis ~ 10%
• pressure dependence ~ 10%
• non-canterm stat mech ~ 15%
• quantum mechanics ~ 20%
• diffusion limited kinetics ~ 25%
• kinetics groups and libraries ~ 20%
• the main RMG class ~ 30%
• the main RMG database class ~ 40%

examples of critical, untested methods

• rmg.main.RMG
  • initializeLog()
  • execute()
  • register_listeners()
  • saveInput()
  • checkInput()
  • loadThermoInput()
• rmg.data.Database
  • getEntriesToSave()
  • saveDictionary()
  • decendents()
  • matchNodeToChild()
  • areSiblings()
  • matchToStructure()

most methods related to loading or saving 'old' or new versions are also untested

most packages pass the caniusepython3 test

• numpy
• cantera
• cython == 0.21 (version restrictions might need changing)
• matplotlib
• pydas
• scipy
• xlwt
• xlrd
• markupsafe
• jinja2
• pyparsing
• pydot == 1.2.2 (version restrictions might need changing)
• nose
• coverage
• cairocffi
• psutil
• graphviz
• pandas
• rdkit

other packages also list python 3 versions

• coolprop
• gprof2dot
• openbabel
• quantities
• scoop
• openbabel - used for the website, display of images (without file writing), not supported.
• cantera works with Python 3 but compiling requires Python 2 (though maybe a binary would avoid this issue)

4 dependencies do not have python 3 capabilities

• guppy - not ever used in the code
• symmetry - used only in molecule.py, RMG specific
• pydas - sort of RMG specific
• pydqed - sort of RMG specific

What options are available for switching?

• fix dependencies
• remove dependencies
• change to alternative that is compatible with python 3

non-python modules should not be affected significantly

• libgcc
• libgfortran
• ffmpeg - used only for flux diagram creation
• mopac - used for on the fly quantum
• cairo - backup to cairocffi. Draws pdep diagrams and molecules for RMG.

discussion questions

What would make it worth the time to upgrade?

• when we can't find python 2.7 anymore
• when python 3 has a new-amazing feature we should utilize
• users demand it
• if we can all convert it in less than X work days

Should we keep support for python 2.7 or move completely to python 3?

• Only maintain one version.

Do we have any concerns or thoughts?

• Are there any benefits to a newer Cython?
• From Bill's experience, the risk of upgrading to Python 3 is that we may end up changing languages in the future anyways.
• A downside of converting is that python 2.7 is a long term release, so python 3 might require more upgrading afterwards
• Another downside is that the transition might introduce bugs which aren't well tested for.
• Can Python 3 improve speed or memory performance? - possibly, but it would need a rewrite of the code.

What pieces of code would be most troublesome to upgrade?

• not sure
• Since we have significant cython code, we should see how much effort is involved in converting it.

resources for conversion information

official python conversion guide

unofficial book about conversion

many python packages 2to3, coverage, caniusepython3

conclusions

1. improving testing of the code would aid in improving reliability and help the transition python 3
2. actually transitioning may be dependent on the need and benefit to transitioning.

[mliu49]

I just noticed that coverage.py does not analyze cythonized code at all. So our reported 44% could be completely misrepresented.

[goldmanm]

That is a really great observation, especially since the cythonized code is harder to read and debug.

Two blog posts, here and here, discuss some ways that coverage.py can cover cythonized code. It seems like we might have to integrate cythonized code into the building of the cython code in our MakeFile.

[mliu49]

I'm documenting the results of playing around with cython coverage. Basically, I couldn't get it to work. However, @KEHANG pointed out that most of the cythonized code is pretty well tested, so maybe we don't need to worry about it too much.

It seems that cython v0.23+ is compatible with coverage v4.0+. We're currently using the latest version of coverage, but we restrict cython to v0.21. I tested the latest version, v0.24.1, without encountering any apparent issues.

To allow coverage to analyze cython code, the cython plugin needs to be enabled in .coveragerc using plugins = Cython.Coverage, and line tracing needs to be enabled. One option is to add the following to the top of every file that will be cythonized:

# cython: linetrace=True
# distutils: define_macros=CYTHON_TRACE=1

Apparently, enabling CYTHON_TRACE slows down the program during runtime, so it's not something that we would want to leave there. The code also needs to be compiled in place for coverage to be able to find the compiled modules.

After adding those lines to every cythonized file and changing the make command to use --inplace, I ran make test, but apparently the coverage plugin to nose doesn't really work with the cython plugin to coverage. The coverage report printed at the end was identical to before. Some online threads mentioned that it's better to run nose from coverage, but I couldn't figure out how to do that.

I did try to directly run some unit tests using coverage, by using the unittest module.

coverage run -m unittest [some test module, eg. rmgpy.molecule.moleculeTest]

Then generated the report using coverage report. In this case, the cythonized modules were listed in the report, but they all had zero coverage, which is clearly wrong. I didn't go much further from this point.

[rwest]

Apparently Django 2.0 (just hit their master branch) will no longer support Python 2.

[goldmanm]

@mliu49, thanks for keeping such detailed notes. I recreated some of the changes to try to see if I could fix the problem of not testing cythonized code, and I placed them in branch coverage_update.

I made a few adaptations to your methodology:

1. I placed the options for CYTHON_TRACE and linetrace in setup.py instead of at the header of individual files. This enables us to easily switch from the coverage method to the quicker method.
2. They updated cython, so I am using Cython 0.25.2 with coverage 4.3.4.
3. I ran the coverage with nosetests with:
   coverage run -m nose --nocapture --nologcapture --all-modules --verbose --exe rmgpy

Using this slower method, the code is taking at least 10x as long as typical to run make test. Running one test with this method did show the coverage of cythonized code, so I anticipate this will too. It is still running, so I will let you know the result when it finishes (and maybe post it somewhere as well).

Note: to get the test result on this branch of RMG, run the following commands:

conda env update environment_linux.yml
make clean
make
make test

The coverage with cythonized code was 58%. You can view and download the output: here

[goldmanm]

In similar lines to our coverage conversation, codecov is a utility that informs how much the coverage changes with each pull request. It runs just like our unittests on Travis, and submits a fail if substantial coverage is decreased. It may be helpful to push people to improve coverage.

[goldmanm]

To add to the list of python2 deprecations, Ubuntu's new LTS release 18.04 will be the last LTS to include python2 in its main depository (Source).

[mliu49]

We did it!! 🎉 🍾 #1724 🎊