Metadata API: test invalid input values for attributes

[MVrachev]

Fixes #1434

Description of the changes being introduced by the pull request:

A while ago we decided that it's best to research each of the individuals
attributes one by one and identify what level of validation it needs
compared to how we use it:
#1366 (comment).

This work is ongoing and there are a couple of commits already merged
for this:

• 6c5d970
• f20664d
• 41afb1e

We want to be able to test the validation we add for attributes against known bad
values. The way we want to do that is with table testing we have added
using decorators for our metadata classes defined in New API:
#1416.
This gives us an easy way to add new cases for each of the attributes and
not depend on external files.

It's important to note that I haven't added invalid tests for all attributes, because
we haven't finished with the metadata research.

When new validation for a certain attribute is added we should make sure to add
invalid tests in TestInvalidSerialization as well.

Please verify and check that the pull request fulfills the following
requirements:

• The code follows the Code Style Guidelines
• Tests have been added for the bug fix or new feature
• Docs have been added for the bug fix or new feature

[MVrachev]

This pr is ready for review after the merge of #1416.

[jku]

This seems very fine grained. There's no benefit from having a separate test for each attribute (when tests are practically identical) but it makes the patch massive. I would expect less granularity would mean at least 50% reduction in commit size?

Also, currently it's hard to see what is being tested: I need to read deep into the test code to find what is the actual class that is being tested by an individual test.

I don't see why we would not do the exact same thing we already do for valid data tests: one invalid-data dataset and one new test per class-under-test. I also think the invalid test+data for a class should be next to the valid test+data for the same class -- If the tests need to be split by for some reason, I expect it to be more useful to split by class-under-test (so e.g. TestKeySerialization) than splitting by TestSerialization/TestInvalidSerialization: I often want to run tests for a specific class, but I have never wanted to run tests based whether success or failure is expected.

[MVrachev]

This seems very fine grained. There's no benefit from having a separate test for each attribute (when tests are practically identical) but it makes the patch massive. I would expect less granularity would mean at least 50% reduction in commit size?

Also, currently it's hard to see what is being tested: I need to read deep into the test code to find what is the actual class that is being tested by an individual test.

I don't see why we would not do the exact same thing we already do for valid data tests: one invalid-data dataset and one new test per class-under-test. I also think the invalid test+data for a class should be next to the valid test+data for the same class -- If the tests need to be split by for some reason, I expect it to be more useful to split by class-under-test (so e.g. TestKeySerialization) than splitting by TestSerialization/TestInvalidSerialization: I often want to run tests for a specific class, but I have never wanted to run tests based whether success or failure is expected.

I did it more granularly because we have more cases per attribute.
Will create a commit making test cases per class and we will comment it again.

[MVrachev]

In a discussion with Jussi, he proposed that I should group invalid and valid tests close to each other in the same test class.
Added a commit doing that.

[jku]

I would still really like it if the variable and function names were consistent

invalid_key vs valid_keys
invalid_role vs valid_roles
invalid_metafile_attr vs valid_metafiles
invalid_targetfile_hashes_length vs valid_targetfiles

With some of those I really need to squint to figure out that they are supposed to be tests for the same class

Other comments are just nitpicks I think

[jku]

there's a lot of test cases here that are unlikely to be useful (e.g. 5 invalid spec versions is just overkill) but since the tests are so fast... I guess no harm done?

[MVrachev]

I removed one of the cases one digit spec_version, the others seem relevant to the checks we are making.

[jku]

ok, but do keep in mind that the idea isn't to test the lines of code that we happen to have in there right now: the idea is to have test data that covers as much of the possible inputs as practically useful -- since we can't cover 100% of the possibilities we inevitably end up making decisions on what's important: we test the inputs we think could be interesting.

[MVrachev]

@jku squashed my commits and addressed all of your comments (including the inconsistencies) except for the ones
about keyval which are addressed in another pr.

[jku]

LGTM, left a couple of comments that are up to you

[jku]

that's an invalid hashes value type

[jku]

Snapshot meta isn't required to contain anything: "meta": {} is enough if you're going for the minimal serialised form

nitpick on the style: I think this would be easier to read if you just barely need two lines (also allows names to be as long as you want):

        "empty str spec_version":
            '{"_type": "signed", "spec_version": "", "version": 1, "expires": "2030-01-01T00:00:00Z", "meta": {}}',

[MVrachev]

Okay, will do that, but will add a comment that per spec having a snapshot object like this in a repository is considered invalid.

[jku]

ok, but do keep in mind that the idea isn't to test the lines of code that we happen to have in there right now: the idea is to have test data that covers as much of the possible inputs as practically useful -- since we can't cover 100% of the possibilities we inevitably end up making decisions on what's important: we test the inputs we think could be interesting.

[jku]

(that last comment is a reply to earlier discussion on spec_version test cases, for some reason github shows it independently here)

[MVrachev]

@jku I addressed your comments.

[MVrachev]

Rebased on top of the latest changes in develop branch and fixed conflicts.