docs: add schema specification

[wjones127]

As we add new data types, we'll want a spec we can modify in proposed PRs. This adds a spec for the schema and data types, as well as the high-level encoding strategy.

[github-actions]

PR Review: docs: add schema specification

This PR adds comprehensive documentation for the Lance schema specification. The documentation is well-structured and accurate based on code verification.

Summary

The PR introduces a new schema.md file documenting:

• Data type mappings (logical types to Arrow types)
• Field structure and properties
• Field ID assignment and tombstoning semantics
• Schema evolution operations
• Blob column formats (v1 and v2)

Verification

I verified the following against the codebase:

• Logical type mappings: Match implementation in rust/lance-core/src/datatypes.rs:139-231
• Blob v2 descriptor fields: Match BLOB_V2_DESC_FIELDS in rust/lance-core/src/datatypes.rs:49-57
• Field ID semantics: Consistent with existing documentation in index.md

No Issues Found

The documentation accurately reflects the implementation. No P0/P1 issues identified.

LGTM ✓

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!