diff --git a/docs/examples/instruct_validate_repair/README.md b/docs/examples/instruct_validate_repair/README.md index 003cd6914..013654f19 100644 --- a/docs/examples/instruct_validate_repair/README.md +++ b/docs/examples/instruct_validate_repair/README.md @@ -39,6 +39,17 @@ Shows how to use custom validation functions for complex requirements. - Using `simple_validate()` helper - Combining multiple validation strategies +### qiskit_code_validation/qiskit_code_validation.py +Advanced example demonstrating IVR pattern for Qiskit code generation with external validation. + +**Key Features:** +- Integrating external validation tools (flake8-qiskit-migration) +- Automatic repair of deprecated Qiskit APIs +- Pre-condition validation of input code +- Custom validation functions for linters + +**See:** [qiskit_code_validation/README.md](qiskit_code_validation/README.md) for full documentation and example prompts. + ### multiturn_strategy_example.py Demonstrates MultiTurnStrategy for conversational repair with validation feedback. diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md new file mode 100644 index 000000000..06414ca0d --- /dev/null +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md @@ -0,0 +1,229 @@ +# Qiskit Code Validation with Instruct-Validate-Repair + +This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern to generate Qiskit quantum computing code that automatically passes flake8-qiskit-migration validation rules (QKT rules). + +## What This Example Does + +Takes a prompt containing deprecated Qiskit code and: +1. Detects QKT violations in the input code +2. Passes those violations to the LLM as context +3. Generates corrected code that passes QKT validation +4. Automatically repairs the code if validation fails (up to 5 attempts) + +## Quick Start + +```bash +# Run the example (uses default deprecated code prompt) +uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +``` + +Dependencies (`mellea`, `flake8-qiskit-migration`) are automatically installed. + +## Requirements + +- **Ollama backend** running locally (`ollama serve`) +- **Compatible model**: e.g., `hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest` or `granite4:small-h` +- **flake8-qiskit-migration**: Automatically installed when using `uv run` + +## How It Works + +### The IVR Pipeline + +1. **Pre-condition validation**: Validates the input prompt and any code it contains +2. **Instruction**: LLM generates code following structured requirements +3. **Post-condition validation**: Validates generated code against QKT rules (see [Qiskit Migration Guide](https://docs.quantum.ibm.com/api/migration-guides)) +4. **Repair loop**: Automatically repairs code that fails validation (up to 5 attempts) + +### Code Structure + +``` +qiskit_code_validation/ +├── qiskit_code_validation.py # Main example +├── validation_helpers.py # Validation utilities +└── README.md # This file +``` + +**validation_helpers.py** provides: +- `extract_code_from_markdown()`: Extracts code from markdown blocks +- `validate_qiskit_migration()`: Validates against QKT rules +- `validate_input_code()`: Pre-validates input prompts + +## Trying Different Prompts + +To try different prompts, edit the `prompt` variable in `test_qiskit_code_validation()` function. Here are some examples you can copy/paste: + +### Simple Prompts + +**Bell State Circuit:** +```python +prompt = "create a bell state circuit" +``` + +**List Backends:** +```python +prompt = "use qiskit to list fake backends" +``` + +**Random Circuit:** +```python +prompt = "give me a random qiskit circuit" +``` + +### Code Completion Prompts + +**Toffoli Gate:** +````python +prompt = """Complete this code: +```python +from qiskit import QuantumCircuit + +qc = QuantumCircuit(3) +qc.toffoli(0, 1, 2) + +# draw the circuit +``` +""" +```` + +**Entanglement Circuit:** +```python +prompt = """from qiskit import QuantumCircuit + +# create an entanglement state circuit +""" +``` + +### Deprecated Code (Default) + +The default prompt demonstrates fixing deprecated Qiskit APIs: + +```python +prompt = """from qiskit import BasicAer, QuantumCircuit, execute + +backend = BasicAer.get_backend('qasm_simulator') + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cnot(0, range(1, 5)) +qc.measure_all() + +# run circuit on the simulator""" +``` + +This code uses deprecated APIs (`BasicAer`, `execute`) that the LLM will automatically fix to use modern Qiskit APIs. + +### Complex Prompts + +**Runtime Service with Estimator:** +```python +prompt = """from qiskit.circuit.random import random_circuit +from qiskit.quantum_info import SparsePauliOp +from qiskit_ibm_runtime import Estimator, Options, QiskitRuntimeService, Session + +# create a Qiskit random circuit named "circuit" with 2 qubits, depth 2, seed 1. +# After that, generate an observable type SparsePauliOp("IY"). Run it in the backend "ibm_sherbrooke" using QiskitRuntimeService inside a session +# Instantiate the runtime Estimator primitive using the session and the options optimization level 3 and resilience level 2. Run the estimator +# Conclude the code printing the observable, expectation value and the metadata of the job.""" +``` + +**Bell Circuit with Runtime Service:** +```python +prompt = """from qiskit import QuantumCircuit +from qiskit_ibm_runtime import QiskitRuntimeService + +# define a Bell circuit and run it in ibm_salamanca using QiskitRuntimeService""" +``` + +## Expected Output + +When you run the example with the default deprecated code prompt, you'll see: + +```` +====== Prompt ====== +from qiskit import BasicAer, QuantumCircuit, execute + +backend = BasicAer.get_backend('qasm_simulator') + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cnot(0, range(1, 5)) +qc.measure_all() + +# run circuit on the simulator +====================== + +Validation failed with 1 error(s): +QKT101: QuantumCircuit.cnot() has been removed in Qiskit 1.0; use `.cx()` instead + +====== Result (83.5s) ====== +```python +from qiskit_aer import AerSimulator +from qiskit import QuantumCircuit + +backend = AerSimulator() + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cx(0, range(1, 5)) # Fixed: use .cx() instead of .cnot() +qc.measure_all() + +job = backend.run(qc) +result = job.result() +``` +I fixed the code by replacing `QuantumCircuit.cnot()` with `QuantumCircuit.cx()` as required by Qiskit 1.0. I also replaced the deprecated `BasicAer.get_backend('qasm_simulator')` with `AerSimulator()`. This code should now pass Qiskit migration validation (QKT rules). +====================== + +✓ Code passes Qiskit migration validation +```` + +**Note**: The exact output may vary depending on the model and its interpretation of the prompt. + +## Changing the Model + +To try a different model, edit the `model_id` variable in the `test_qiskit_code_validation()` function: + +```python +# Uncomment one to try different models +# model_id = "granite4:micro-h" +# model_id = "granite4:small-h" +model_id = "hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest" +``` + +**Note**: Smaller models (like `granite4:micro-h`) may not have enough Qiskit knowledge to pass validation consistently. The Qiskit-specific model or `granite4:small-h` work best. + +## Troubleshooting + +### Ollama Connection Refused +``` +Error: Connection refused +``` +**Solution**: Start Ollama with `ollama serve` + +### Model Not Found +``` +Error: model 'hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest' not found +``` +**Solution**: Pull the model first: +```bash +ollama pull hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest +``` + +### Validation Always Fails +If using smaller models (e.g., `granite4:micro-h`), they may not have enough Qiskit knowledge. Try: +- Using a larger model (`granite4:small-h` or the Qiskit-specific model) +- Reducing prompt complexity +- Using simpler prompts + +### Import Error: flake8-qiskit-migration +``` +ModuleNotFoundError: No module named 'flake8_qiskit_migration' +``` +**Solution**: Use `uv run` which auto-installs dependencies + +## Future Work + +The following enhancements are planned for future iterations: + +1. **MultiTurnStrategy Integration** - Try using `MultiTurnStrategy` (see [Sampling Strategies](../README.md#sampling-strategies)) which builds conversation history by adding validation failures as new user messages, to see if this approach improves results over the current `RepairTemplateStrategy` which adds failures directly to the instruction. + +2. **Enable Smaller Models** - Add system prompt or grounding context with Qiskit API documentation to help smaller models perform accurate migrations. This would allow removing the `pytest.mark.skip` marker and make the example run in standard test suites. \ No newline at end of file diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py new file mode 100644 index 000000000..03713f923 --- /dev/null +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py @@ -0,0 +1,165 @@ +# pytest: ollama, llm, qualitative, skip +# /// script +# dependencies = [ +# "mellea", +# "flake8-qiskit-migration", +# ] +# /// +"""Qiskit Code Validation with Instruct-Validate-Repair Pattern. + +This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern +to generate Qiskit quantum computing code that automatically passes +flake8-qiskit-migration validation rules (QKT rules). + +The pipeline follows these steps: +1. **Pre-condition validation**: Validate prompt content and any input code +2. **Instruction**: LLM generates code following structured requirements +3. **Post-condition validation**: Validate generated code against QKT rules +4. **Repair loop**: Automatically repair code that fails validation (up to 5 attempts) + +Requirements: + - flake8-qiskit-migration: Installed automatically when run via `uv run` + - Ollama backend running with a compatible model (e.g., mistral-small-3.2-24b-qiskit-GGUF) + +Example: + Run as a standalone script (dependencies installed automatically): + $ uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +""" + +import time +from typing import Literal + +from validation_helpers import validate_input_code, validate_qiskit_migration + +import mellea +from mellea.backends import ModelOption +from mellea.stdlib.requirements import req, simple_validate +from mellea.stdlib.sampling import RepairTemplateStrategy + + +def generate_validated_qiskit_code( + m: mellea.MelleaSession, prompt: str, max_repair_attempts: int = 5 +) -> str: + """Generate Qiskit code that passes Qiskit migration validation. + + This function implements the Instruct-Validate-Repair pattern: + 1. Pre-validates input code + 2. Instructs the LLM with structured requirements + 3. Validates output against QKT rules + 4. Repairs code if validation fails (up to max_repair_attempts times) + + Args: + m: Mellea session + prompt: User prompt for code generation + max_repair_attempts: Maximum number of repair attempts for validation failures + + Returns: + Generated code that passes validation + + Raises: + ValueError: If prompt validation fails + """ + # Pre-validate input code if present — include violations as context rather than failing + is_valid, error_msg = validate_input_code(prompt) + input_code_errors = None + if not is_valid: + print( + f"Input code has QKT violations, including as context for LLM: {error_msg}" + ) + input_code_errors = error_msg + + # Build the instruction prompt, optionally augmented with input code violations + instruct_prompt = prompt + if input_code_errors is not None: + instruct_prompt = ( + f"{prompt}\n\n" + f"Note: the code above has the following Qiskit migration issues that must be fixed:\n" + f"{input_code_errors}" + ) + + # Generate code with output validation only + code_candidate = m.instruct( + instruct_prompt, + requirements=[ + req( + "Code must pass Qiskit migration validation (QKT rules)", + validation_fn=simple_validate(validate_qiskit_migration), + ) + ], + strategy=RepairTemplateStrategy(loop_budget=max_repair_attempts), + return_sampling_results=True, + ) + + if code_candidate.success: + return str(code_candidate.result) + else: + print("Code generation did not fully succeed, returning best attempt") + # Log detailed validation failure reasons + if code_candidate.result_validations: + for requirement, validation_result in code_candidate.result_validations: + if not validation_result: + print( + f" Failed requirement: {requirement.description} — {validation_result.reason}" + ) + # Return best attempt even if validation failed + if code_candidate.sample_generations: + return str(code_candidate.sample_generations[0].value or "") + print("No code generations available") + return "" + + +def test_qiskit_code_validation() -> None: + """Test Qiskit code validation with deprecated code that needs fixing. + + This test demonstrates the IVR pattern by providing deprecated Qiskit code + that uses old APIs (BasicAer, execute) and having the LLM fix it to use + modern Qiskit APIs that pass QKT validation rules. + """ + # Model selection - uncomment one to try different models + # model_id = "granite4:micro-h" + # model_id = "granite4:small-h" + model_id = "hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest" + + # Prompt - replace with your own or see README.md for examples + prompt = """from qiskit import BasicAer, QuantumCircuit, execute + +backend = BasicAer.get_backend('qasm_simulator') + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cnot(0, range(1, 5)) +qc.measure_all() + +# run circuit on the simulator +""" + + print("\n====== Prompt ======") + print(prompt) + print("======================\n") + + with mellea.start_session( + model_id=model_id, + backend_name="ollama", + model_options={ModelOption.TEMPERATURE: 0.8, ModelOption.MAX_NEW_TOKENS: 2048}, + ) as m: + start_time = time.time() + code = generate_validated_qiskit_code(m, prompt, max_repair_attempts=5) + elapsed = time.time() - start_time + + print(f"\n====== Result ({elapsed:.1f}s) ======") + print(code) + print("======================\n") + + # Validate the generated code + is_valid, error_msg = validate_qiskit_migration(code) + + if is_valid: + print("✓ Code passes Qiskit migration validation") + else: + print("✗ Validation errors:") + print(error_msg) + + +if __name__ == "__main__": + # Run the example when executed as a script + test_qiskit_code_validation() diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/validation_helpers.py b/docs/examples/instruct_validate_repair/qiskit_code_validation/validation_helpers.py new file mode 100644 index 000000000..81b3f5ee1 --- /dev/null +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/validation_helpers.py @@ -0,0 +1,109 @@ +"""Helper functions for Qiskit code validation. + +This module provides utilities for extracting code from markdown and validating +Qiskit code against migration rules using the flake8-qiskit-migration plugin. +""" + +import ast +import re + +try: + from flake8_qiskit_migration.plugin import Plugin +except ImportError: + raise ImportError( + "flake8-qiskit-migration is required for this example. " + "Run with: uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py" + ) + + +def extract_code_from_markdown(text: str) -> str: + """Extract code from markdown code block. + + Handles both fenced code blocks (```python or ```) and returns the code content. + If no code block is found, returns the original text. + + Args: + text: Text potentially containing markdown code blocks + + Returns: + Extracted code or original text if no code block found + """ + # Pattern for fenced code blocks with optional language identifier + # Matches ```python code``` or ``` code ``` + pattern = r"```(?:python|py)?\s*(.*?)```" + + matches = re.findall(pattern, text, re.DOTALL) + + if matches: + # Return the first code block found + return matches[0].strip() + + # If no code block found, return original text stripped + return text.strip() + + +def validate_qiskit_migration(md_code: str) -> tuple[bool, str]: + """Validate code against Qiskit migration rules using flake8-qiskit-migration plugin. + + This function is used as a post-condition validator to check if the generated + code passes all QKT (Qiskit) migration rules. + + Args: + md_code: Python code (potentially in markdown format) to validate + + Returns: + Tuple of (is_valid, error_message) where error_message retains QKT rule codes + for the repair loop. + """ + try: + code = extract_code_from_markdown(md_code) + tree = ast.parse(code) + plugin = Plugin(tree) + errors = list(plugin.run()) + + if not errors: + return True, "" + else: + error_messages = [] + for _line, _col, message, _error_type in errors: + error_messages.append(message) + error_str = "\n".join(error_messages) + print(f"Validation failed with {len(errors)} error(s):\n{error_str}") + return False, error_str + + except SyntaxError as e: + print(f"Syntax error during validation: {e}") + return False, f"Invalid Python syntax: {e}" + except Exception as e: + print(f"Unexpected validation error: {e}") + return False, f"Validation error: {e}" + + +def validate_input_code(prompt: str) -> tuple[bool, str]: + """Validate any Qiskit code contained in the user's prompt. + + This is used as a pre-condition validation to check if the prompt + contains code that needs to be fixed or improved. + + Args: + prompt: User's input prompt (may contain code blocks) + + Returns: + Tuple of (is_valid, error_message) + """ + # Try to extract code from the prompt + extracted_code = extract_code_from_markdown(prompt) + + # If no code block found (extracted == original), skip validation + if extracted_code == prompt.strip(): + return True, "" + + # Code block found, validate it + is_valid, error_msg = validate_qiskit_migration(extracted_code) + + if not is_valid: + # Return the raw fix instructions — no wrapper prefix. + # The caller (generate_validated_qiskit_code) frames these in the instruct prompt. + return False, error_msg + + return True, "" diff --git a/pyproject.toml b/pyproject.toml index d5d7bb24f..27844bc64 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -304,7 +304,7 @@ disable_error_code = [ # ----------------------------- [tool.codespell] -ignore-words-list = 'mellea,hashi,noo,Asai,asai,nd,mot,rouge,Rouge,Strat,Wight' +ignore-words-list = 'mellea,hashi,noo,Asai,asai,nd,mot,rouge,Rouge,Strat,Wight,Aer,aer' check-filenames = true check-hidden = false regex = "(?