Skip to content

Harden Python evaluation responses for pytest collection/import errors #20

@NickK21

Description

@NickK21

Summary

Python test runs can currently be reported incorrectly when pytest fails before normal test assertions run.

A common failure mode is:

  • state: "passed"
  • tests_run: 0
  • non-empty runtime_error

This happens because the current Python result handling relies on regex parsing of pytest terminal output and does not model collection/import/syntax errors cleanly.

Current Problem

The Python execution path in codeval/executor.js does not distinguish well between:

  • assertion failures
  • import/collection errors
  • syntax errors
  • missing dependencies
  • no tests collected

The response object is also missing structured fields that would make these cases easier to handle.

Planned Changes

  • Add structured response fields for Python test execution:
    • errors
    • exit_code
    • no_tests_collected
  • Ensure Python collection/import/syntax errors are reported as failed, not passed.
  • Attach the subprocess exit code to Python execution errors in the executor.
  • Add a synthetic failure_details entry for collection/import/syntax failures when no normal assertion failure is available.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions