Introduce pydantic schema

[skshetry]

From the dvcyamlschema/gen.py, added some 2.0 stuff and improved schema in the hopes that this will replace the Voluptuous schema we have for dvc.yaml at least.

I'm still hesitant to go full-blown with Pydantic yet, as making both of these schemas compatible is harder than I thought previously, which means there might be some bugs lurking around. The tests do pass with this new schema, but they don't tests the negative cases well. And, we also need to consider performance (voluptuous is faster, but is it worth to continue using it as it's DX is very bad?).

This pydantic schema can be used to provide jsonschema as well, you can just use it as:

from dvc.schema.dvc_yaml import Schema

print(Schema.schema_json(indent=4))

The dvcyaml_schema repo will simply use this API if/when this gets merged, which will make it closer to where we work.
See also #5371 (comment), pydantic has a bit better error messages than Voluptuous.

Related: treeverse/dvcyaml-schema#7

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[skshetry]

To test new schema, you can toggle this to True, and it should start using the new schema for validation.

[skshetry]

Closing it for now, will come back to it after UI is done. Unfortunately, I haven't been able to work more on this right now, so this needs more research.

As this PR stands, it is only meant to change the validation schema, so I can't provide any compelling argument other than to say that we are maintaining this schema already on dvcyaml-schema for generating schema.json and with this PR, it provides benefits of both generating schema.json and validating dvc.yaml file with a single schema definition.

It feels like cheating if I make other arguments just to make a point on its potential benefits that we might get in the future, which I am quite not sure of myself. I do believe we'll come to this again soon. 🙂

[daavoo]

Closing it for now, will come back to it after UI is done. Unfortunately, I haven't been able to work more on this right now, so this needs more research.

As this PR stands, it is only meant to change the validation schema, so I can't provide any compelling argument other than to say that we are maintaining this schema already on dvcyaml-schema for generating schema.json and with this PR, it provides benefits of both generating schema.json and validating dvc.yaml file with a single schema definition.

It feels like cheating if I make other arguments just to make a point on its potential benefits that we might get in the future, which I am quite not sure of myself. I do believe we'll come to this again soon. slightly_smiling_face

I'm not aware of the internal implications of this change but I think that having the schema inside the core repo makes a lot of sense. Issues related with having dvcyaml-schema outdated have been raised at least a few times. (And some discord messages, i.e. recently https://discord.com/channels/485586884165107732/485596304961962003/869072338778468373)

Probably not enough to be considered an argument at this point but I think that adding this feature (and introducing internal usage of pydantic) could open new possibilities regarding params validation (#5477) and enhancements to the overall params UI. I was working on a proposal about having internal params validation and potentially allowing users to provide it's own params schemas. Having this P.R. as foundation would make it way easier to implement.

[SoyGema]

Sorry. Better in #9606 (comment)