Implement Vector Search with sqlite-vec Extension

[renecannao]

Summary

This PR implements comprehensive vector search capabilities in ProxySQL by integrating the sqlite-vec extension.

Changes

sqlite-vec Integration

• Added complete sqlite-vec extension (9,751 lines) to SQLite3 backend
• Implemented vector similarity search with distance calculations
• Added support for JSON vector format
• Extended SQLite3 server with vector match query support

Build System Updates

• Modified build configuration to include sqlite-vec static extension
• Updated dependency management for vector search components
• Ensured proper linking and initialization

Core Integration

• Modified Admin_Bootstrap.cpp to load and initialize sqlite-vec extension
• Added proper error handling for vector operations
• Ensured thread-safe operation of vector search functions

Testing Framework

• Added comprehensive test suite with 4 modular test scripts
• Tests connectivity, table creation, data insertion, and similarity search
• Includes proper error handling and validation
• Provides clear success/failure reporting

Documentation

• Added SQLite3 server documentation with usage examples
• Created vector search testing guide with instructions
• Included troubleshooting and integration guides

Technical Implementation

Vector Search Support

• Virtual Tables: Create vector tables using CREATE VIRTUAL TABLE USING vec0
• Vector Format: JSON array format: [1.0, 0.5, 0.0, ..., 0.0]
• Similarity Queries: Use WHERE vector MATCH json(...) for similarity search
• Distance Metrics: Automatic distance calculation and ordering

Example Usage

-- Create vector table
CREATE VIRTUAL TABLE embeddings USING vec0(vector float[128]);

-- Insert vectors
INSERT INTO embeddings(rowid, vector) 
VALUES (1, "[1.0, 0.0, 0.0, ..., 0.0]");

-- Search for similar vectors
SELECT rowid, distance
FROM embeddings
WHERE vector MATCH json("[0.9, 0.1, 0.0, ..., 0.0]")
ORDER BY distance ASC;

Files Added

• deps/sqlite3/sqlite-vec-source/sqlite-vec.c (9,751 lines)
• deps/sqlite3/sqlite-vec-source/sqlite-vec.h
• deps/sqlite3/sqlite-vec-source/sqlite-vec.h.tmpl
• deps/sqlite3/sqlite-vec-source/README.md
• deps/sqlite3/README.md
• doc/SQLite3-Server.md
• doc/vector-search-test/ (5 test files)

Files Modified

• lib/Admin_Bootstrap.cpp (171 lines added)
• lib/Makefile
• Makefile
• deps/Makefile

Testing

The implementation includes comprehensive testing:

• Basic connectivity validation
• Vector table creation and verification
• Data insertion and integrity checks
• Similarity search functionality validation

Impact

This enables ProxySQL to support:

• AI and ML applications requiring vector similarity
• Recommendation systems
• Search and retrieval applications
• Any use case requiring vector operations

Security

• No new security vulnerabilities introduced
• Extension loading follows existing ProxySQL security patterns
• Proper error handling prevents information leakage

Performance

• sqlite-vec provides efficient vector indexing
• Static extension ensures optimal performance
• Minimal overhead when not using vector operations