Use pytest for (unit)testing?

[mielvds]

RDFLib and many other libraries use pytest to run unittest. Currently, we only run the spec's test-suite, but we probably want to test specific parts of the code as well.
Introducing pytest will simplify this for developers, or do we stick to using unittest? In case we go for the former: do we migrate https://github.com/digitalbazaar/pyld/blob/master/tests/runtests.py as well.

@anatoly-scherbakov @BigBlueHat @davidlehn what are your thoughts about this?

[ioggstream]

I suggest the following approach:

1. add pytest dependency
2. add new pytest unittests
3. use test spec files (e.g., YAML files containing the input and the expected output)
4. measure code coverage
5. progressively remove tests from runtest.py

[mielvds]

Thanks for the input @ioggstream ! I agree with 1, 2 and 4, although I think 5 can also be included in this process. Most of the code is about running the tests, rather than actual tests (which are in de JSON-LD 1.1 test-suite).

3 seems complimentary to non-spec based tests and therefore the last step in the process.

[anatoly-scherbakov]

Agreed with a migration to pytest, and with a need of non-spec tests.

[davidlehn]

1. add pytest dependency
2. add new pytest unittests

Not knowing much about pytest, that's fine if it's what modern python users expect.

3. use test spec files (e.g., YAML files containing the input and the expected output)

All the test suites are using JSON-LD spec files now. Probably find to continue that vs adding YAML deps to all the implementations tests suite runners.

4. measure code coverage

Yeah, I've been hoping more implementations would do that and add more test suite tests for edge cases that were not covered.

5. progressively remove tests from runtest.py

That file has no tests, it only runs them. Are you saying remove the whole file if something else is doing the same thing?

Agreed with a migration to pytest, and with a need of non-spec tests.

Please please please add any generic tests to the main test suites, or at a minimum file spec issues to move new tests over. It will help all implementations be compliant with each other. I'm not even sure what non-spec tests are needed? There's the IRI ones in other PRs, but I also think those should be moved to a shared spec test space.

[anatoly-scherbakov]

IMHO adding non-spec tests makes the development much, much faster because it enables a TDD-esque workflow. I use it more often than not. Instead of a test you could also write a custom pseudo test script but that would be a reinvention of the wheel. If you write a test, developers doing the review of your PR will know what you did and how to interpret it. It is also a method of communication.

Spec-based tests are more interoperable but they are also slower. I think migration of tests towards the specs could be an asynchronous process if that makes sense.

[mielvds]

What I would propose is a best of both worlds: unittests test particular parts of the code (probably calling them non-spec tests is not right), like individual functions, so they need to be added to ensure coverage and good maintainability. That's what the tests in the other PR does; it only tests resolving and unresolving IRIs; nothing more and it's not distracted by JSON-LD specific things. They are not meant to replace or augment the test-suites.
That said, some of these tests might contain useful test cases that could be included in the generic tests. Some older PR's already contain some of those. As maintainers, we should make sure that we communicate those to the official repo at all times.

[anatoly-scherbakov]

Yes, "non-spec tests" probably is a bad term.

• Unit tests are pytest test cases in the repo,
• Integration, or acceptance tests are specification based, they're testing behavior of the library as a black box, which warrants the name.

Normally, people use both types of tests in projects.

[davidlehn]

IMHO adding non-spec tests makes the development much, much faster because it enables a TDD-esque workflow. I use it more often than not. Instead of a test you could also write a custom pseudo test script but that would be a reinvention of the wheel. If you write a test, developers doing the review of your PR will know what you did and how to interpret it. It is also a method of communication.

Spec-based tests are more interoperable but they are also slower. I think migration of tests towards the specs could be an asynchronous process if that makes sense.

Huh? What is slower? I have to admit I'm forgetting how this test suite works. In the js code I added hacks to allow "skip" and "only" properties in the manifest files. It's fast and great when working on a few selected tests. I'm not sure the pyld runner has that? JS processes everything to build the tests to run, so works well there with "only" set. Might not be as easy if test manifests are run direct. Also not too hard to copy & paste tests into a new temp manifest to run tests you want. All that has worked well for me so far.

It seems we have different opinions on this. So all I can say is please ensure any generic test is in the main test suite. And I think it's better to not duplicate those locally to avoid confusion and syncing problems.