Save pulse shape parameters directly in json (and not in a table file)

[GernotMaier]

This is a larger PR to allow the storage of tabled date directly in the model parameter json and not in externally linked files. Implemented here for the fadc_pulse_shape parameter but will be extended to all tabled data (and some items working for for fadc_pulse_shape will be generalized.

This pull request introduces a unified and more robust export flow for model parameter tables, improves schema selection and validation, and enhances documentation and CLI usability. The changes ensure stricter validation and normalization of table content, support for schema versioning, and clearer export logic for both file-backed and dict-backed parameters. The documentation and command-line interfaces are updated to reflect these improvements and provide better guidance to users.

Parameter export and validation improvements:

• Implemented a unified export flow for both file-backed and dict-backed table parameters, with stricter validation and normalization (including required column_units for fadc_pulse_shape). The export logic now ensures consistent handling and output formats for all table-type parameters. [1] [2] [3]
• Added support for selecting the schema document by model_parameter_schema_version (with fallback to the latest version), and exposed schema version selection in CLI and application logic. [1] [2] [3] [4] [5]

Database and model parameter handling:

• Refactored the database handler to delegate export logic to a new parameter_exporter module, aligning sim_telarray/model/database handling and expanding unit test coverage. [1] [2] [3]
• Improved error handling and logging in database file export applications, raising clear exceptions and providing informative log messages. [1] [2]

Documentation and CLI usability:

• Updated CLI documentation and argument parsing for parameter export applications to clarify output modes, file naming, and schema versioning. Added detailed usage examples for all export scenarios. [1] [2] [3] [4] [5] [6]
• Added API documentation entries for the new parameter_exporter and simtel_table_writer modules. [1] [2]

Model data writer enhancements:

• Added a method to retrieve parameter types from a selected schema version, and ensured metadata is written only when present during parameter submission. [1] [2] [3]

Typical applications / steps:

1. Submit an existing simtel_array style table and store it in the new dict-backed table parameter:

simtools-submit-model-parameter-from-external --parameter fadc_pulse_shape --value LST-1_spe_pulse_shape_pix3.dat --instrument LSTN-01 --parameter_version 4.0.0 --data_path ../simulation-model-parameter-setting/input/LSTN-01/fadc_pulse_shape/20260316T110922

(Requires to have the the corresponding simulation model setting branch checked out)

2. Getting the new dict-type parameter as astropy table

python src/simtools/applications/db_get_parameter_from_db.py --parameter fadc_pulse_shape --parameter_version 2.0.0 --telescope LSTN-01 --site North --export_model_file  --export_model_file_as_table --output_file fadc_pulse_shape.ecsv\

3. Testing this branch
   Requires to run with a test branch of simulation models: fadc-pulse-shape-format-change

[Copilot]

Pull request overview

Enable storing table-like model parameter values directly inside model-parameter JSON (initially for fadc_pulse_shape), while unifying export/validation logic across file-backed and dict-backed parameters and selecting schema entries by model_parameter_schema_version.

Changes:

• Add row-oriented table serialization + table-file writer/reader utilities and wire them into sim_telarray config generation and validation flows.
• Introduce parameter_exporter module and delegate DB export logic to it (including dict-backed → ECSV export support).
• Update schemas, CLI apps, docs, reference sim_telarray configs, and unit tests for schema-version selection and dict-backed table handling.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit_tests/visualization/test_plot_tables.py	Update plotting tests for schema-version selection and parameter-dict lookup.
tests/unit_tests/simtel/test_simulator_light_emission.py	Update patches to use new simtel_table_writer functions.
tests/unit_tests/simtel/test_simtel_table_writer.py	Add unit tests for the new sim_telarray table writer.
tests/unit_tests/simtel/test_simtel_table_reader.py	Add tests for row-data serialization and dict-value resolution helpers.
tests/unit_tests/simtel/test_simtel_output_validator.py	Add tests ensuring dict-typed metadata values can be resolved from table files.
tests/unit_tests/simtel/test_simtel_config_writer.py	Update tests to use new writer functions; add tests for new conversion helpers.
tests/unit_tests/model/test_model_parameter.py	Add coverage for legacy table-value resolution during schema upgrades.
tests/unit_tests/model/test_legacy_model_parameter.py	Add/extend legacy migration handlers for fadc_pulse_shape and resolver plumbing.
tests/unit_tests/db/test_parameter_exporter.py	Add unit tests for the new DB parameter export orchestration module.
tests/unit_tests/db/test_db_handler.py	Update DB handler tests for delegation and dict-backed export behavior.
tests/unit_tests/data_model/test_model_data_writer.py	Extend writer tests for embedded fadc_pulse_shape and schema-version type lookup.
tests/resources/sim_telarray_configurations/6.0.2/CTA-South-SSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/6.0.2/CTA-South-MSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/6.0.2/CTA-South-LSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/6.0.2/CTA-North-MSTN-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/6.0.2/CTA-North-LSTN-02_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/6.0.2/CTA-North-LSTN-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-South-SSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-South-MSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-South-LSTS-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-North-MSTN-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-North-LSTN-02_test.cfg	Update expected fadc_pulse_shape filename and config_release.
tests/resources/sim_telarray_configurations/5.0.0/CTA-North-LSTN-01_test.cfg	Update expected fadc_pulse_shape filename and config_release.
src/simtools/visualization/plot_tables.py	Select schema entry by parameter’s schema version before generating plot configs.
src/simtools/simtel/simulator_light_emission.py	Switch pulse/angular-distribution table writing to simtel_table_writer.
src/simtools/simtel/simtel_table_writer.py	New module for writing sim_telarray table files (pulse/angle/generic tables).
src/simtools/simtel/simtel_table_reader.py	Add row-data serialization, unit normalization, and dict-value resolution helpers.
src/simtools/simtel/simtel_output_validator.py	Resolve dict-typed metadata values from table files prior to comparison.
src/simtools/simtel/simtel_config_writer.py	Remove embedded table-writer methods; write dict table parameters via new writer.
src/simtools/schemas/model_parameters/fadc_pulse_shape.schema.yml	Add schema v0.2.0 for embedded row-data representation and plot configuration.
src/simtools/model/model_parameter.py	Pass legacy value resolver into schema-upgrade path and implement table-value resolver.
src/simtools/model/legacy_model_parameter.py	Add value_resolver plumbing + fadc_pulse_shape migration to embedded row-data.
src/simtools/io/ascii_handler.py	Update JSON writing and add compact encoding for numeric lists (row tables).
src/simtools/db/parameter_exporter.py	New shared export logic for file-backed and dict-backed parameter payloads.
src/simtools/db/db_handler.py	Delegate export operations to parameter_exporter; add export_parameter_data.
src/simtools/data_model/model_data_writer.py	Write metadata only after successful validation; add schema-version type lookup.
src/simtools/applications/submit_model_parameter_from_external.py	Add schema-version arg and pre-resolve dict parameter values for submission.
src/simtools/applications/db_get_parameter_from_db.py	Rework CLI export modes; use export_parameter_data for unified exports.
src/simtools/applications/db_get_file_from_db.py	Improve error handling by raising a clear FileNotFoundError with context.
docs/source/api-reference/sim_telarray.md	Add API docs entry for simtel_table_writer.
docs/source/api-reference/db_handler.md	Add API docs entry for db.parameter_exporter.
docs/changes/2088.feature.md	Add changelog entry describing unified export flow and schema selection changes.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube