Multiple data validation and error handling bugs in production

[phernandez]

Issue Summary

Production tenant logs reveal several bugs causing repeated errors during normal operations. These need better error handling and validation.

Bugs Identified

1. Tags handling: 'dict' object has no attribute 'strip'

Frequency: ~40 occurrences in 12 hours
Affected files: Multiple (L2_regulation/GOV/.md, L3_design/emotion/.md, etc.)

The code expects tags to be strings but receives dict objects. Need to handle both string and dict tag formats gracefully.

Example errors:

• Error writing resource L2_regulation/GOV/rgs.md: 'dict' object has no attribute 'strip'
• Error writing resource L3_design/emotion/biorhythm_and_color_profile.md: 'dict' object has no attribute 'strip'

Fix needed: Add type checking and conversion for tags in frontmatter parsing.

2. Double slashes in file paths causing DB conflicts

Frequency: Multiple occurrences
Affected files: L3_design/emotion//biorhythm_and_color_profile.md

Double slashes in paths cause UNIQUE constraint violations in the database.

Example errors:

(sqlite3.IntegrityError) UNIQUE constraint failed: entity.file_path, entity.project_id
[SQL: INSERT INTO entity ... file_path='L3_design/emotion//biorhythm_and_color_profile.md' ...]

Fix needed:

• Normalize paths before database operations (remove double slashes)
• Better error messages explaining the path issue to users
• Potentially auto-fix double slashes during write operations

3. Invalid YAML frontmatter crashes

Example:

tags:[design,thesis]  # Missing space after colon

Error:

ParseError: Invalid YAML in frontmatter: could not find expected ':'

Fix needed:

• Better YAML validation with helpful error messages
• Suggest corrections (e.g., "Did you mean 'tags: [design,thesis]'?")
• Consider more lenient parsing or auto-correction

4. Edit operation text not found

Frequency: Multiple occurrences

Users attempting find/replace operations that fail because text doesn't match exactly.

Example:

Error editing entity: Text to replace not found: '| 30.10.2025 | **GERICHTSTERMIN...'

Fix needed:

• More helpful error messages showing what was searched for and what was found
• Fuzzy matching suggestions
• Preview mode for edit operations

5. Import failures with missing fields

Error:

Import failed: AttributeError: 'NoneType' object has no attribute 'success'
Failed to import memory.json: KeyError: 'type'

Fix needed:

• Better validation of import data structure
• Clear error messages about required fields
• Graceful degradation (import what's valid, report what failed)

Priority

High - These bugs cause repeated failures in production and degrade user experience.

Affected Users

Currently affecting production tenant e5d1c86a (power user with complex data structure).

Reproduction

These are all visible in Logfire production logs for the last 12 hours.

Proposed Solutions

1. Add robust type checking for all frontmatter fields
2. Normalize file paths consistently
3. Improve YAML parser error messages
4. Add validation layers before database operations
5. Better error messages with actionable suggestions
6. Consider adding --strict vs --lenient modes for parsing

[claude]

Claude finished @phernandez's task —— View job • claude/issue-431-20251113-1619 • Create PR ➔

---

Production Bugs Fixed ✓

I've successfully implemented and committed fixes for all 5 production bugs reported in this issue.

Issue Triage

• Type: Bug report (multiple related bugs)
• Priority: HIGH - Production issues affecting real users
• Components: Database/sync, MCP tools, Import functionality
• Complexity: Medium - Multiple fixes needed across different areas

Fixes Implemented

Bug #1: Tags Dict Handling

File: src/basic_memory/utils.py

Added handling for dict objects in parse_tags() function. When frontmatter contains malformed YAML like tags:{key:value}, PyYAML parses it as a dict instead of failing. The fix:

• Detects dict type before attempting .strip() operations
• Converts dict keys to tag list with warning log
• Provides helpful error message suggesting correct format: tags: [tag1, tag2]

Bug #2: Path Normalization

Files: src/basic_memory/file_utils.py, src/basic_memory/schemas/base.py, src/basic_memory/api/routers/resource_router.py

Created normalize_path_separators() function to prevent double slashes causing DB UNIQUE constraint violations:

• Compresses multiple slashes (folder//file.md → folder/file.md)
• Normalizes backslashes to forward slashes
• Applied in Entity.file_path property (where paths are constructed)
• Applied in resource router endpoint (where paths come from URLs)

Bug #3: YAML Error Messages

File: src/basic_memory/file_utils.py

Enhanced parse_frontmatter() to provide actionable suggestions for common YAML mistakes:

• Detects missing space after colon (key:value vs key: value)
• Shows the problematic line from the YAML
• Suggests the corrected version
• Example output:

  Missing space after colon - YAML requires 'key: value' not 'key:value'
  Problem line: 'tags:[design,thesis]'
  Try: 'tags: [design,thesis]'
  

Bug #4: Edit Operation Error Context

File: src/basic_memory/services/entity_service.py

Improved "Text to replace not found" error in apply_edit_operation():

• Includes 500-character content preview for debugging
• Helps users see what content actually exists
• Suggests using read_note() to verify the exact text
• Makes it easier to diagnose why text wasn't found

Bug #5: Import Validation

File: src/basic_memory/importers/memory_json_importer.py

Added comprehensive validation before accessing data fields:

• Checks if data is a dict (not some other type)
• Checks if "type" field exists before accessing
• Logs helpful messages showing available fields when validation fails
• Continues processing valid entries instead of crashing
• Increments skipped_entities counter for reporting

Technical Details

All fixes include:

• Inline comments referencing issue Multiple data validation and error handling bugs in production #431 for maintainability
• Defensive coding patterns to handle edge cases
• Helpful error messages to guide users
• Logging for debugging and monitoring

The changes are backward compatible and handle both valid and invalid data gracefully.

Files Changed

• src/basic_memory/utils.py - Tags dict handling
• src/basic_memory/file_utils.py - Path normalization + YAML errors
• src/basic_memory/schemas/base.py - Path normalization in Entity
• src/basic_memory/api/routers/resource_router.py - Path normalization in API
• src/basic_memory/services/entity_service.py - Edit error context
• src/basic_memory/importers/memory_json_importer.py - Import validation

Next Steps

The fixes are ready for review. These changes should significantly reduce the error rates reported in production logs.

--- | Branch: claude/issue-431-20251113-1619