# Trend-Scout AI Semantic Search Fix Development Tasks

## Specification Summary
**Original Requirements**: 
- Fix semantic search (ChromaDB) issues: `/latest` command ignores category filters.
- Fix `/search` command returning semantically irrelevant text.
- Follow TDD (Test-Driven Development), SOLID principles, and use `asyncio`.
**Technical Stack**: Python, asyncio, pytest, ChromaDB (Vector Storage), aiogram (Telegram Bot), Ollama (Embeddings/LLM).
**Target Timeline**: ~1 Development Day (6-8 hours)

## Execution Plan & Estimation

### Phase 1: Architectural Review & Setup (1 Hour)
Review the existing `IVectorStore` interface. Ensure that the interface supports passing metadata filters (for categories) and distance thresholds (for semantic relevance). Evaluate the embedding model being used, ensuring it supports Russian context effectively, as the AI Processor outputs summaries in Russian (`summary_ru`).

### Phase 2: TDD & Test Creation (2 Hours)
Strict adherence to TDD. Before touching the ChromaDB implementation, write failing `pytest` cases that mock the database and test that queries with category filters and relevance score thresholds return the expected subsets of data.

### Phase 3: ChromaDB Query Tuning & Implementation (2 Hours)
Implement the actual fixes in the ChromaDB wrapper. Map category arguments to ChromaDB's `where` metadata filter. Adjust the vector space distance metric (e.g., switching to `cosine` similarity via `hnsw:space`) and enforce a maximum distance threshold to drop irrelevant results.

### Phase 4: Bot Integration & E2E Testing (1-2 Hours)
Update the Telegram bot (`aiogram` handlers) to correctly extract category arguments from the `/latest` command and pass them to the Vector Storage Agent. Handle cases where `/search` returns no results due to the new relevance thresholds.

---

## Development Tasks

### [ ] Task 1: Architecture Review & Interface Update
**Description**: Update the `IVectorStore` interface to explicitly support metadata filtering and similarity thresholds.
**Acceptance Criteria**: 
- `IVectorStore.search()` method signature accepts `filters: dict` and `max_distance: float` (or `min_relevance: float`).
- Existing mock classes/stubs are updated to match the new interface.
**Files to Create/Edit**:
- `src/vector_storage/interfaces.py`
**Estimation**: 30-45 minutes
**Reference**: Phase 1: Architectural Review

### [ ] Task 2: TDD Setup for Metadata Filtering
**Description**: Write failing tests for the Vector Storage Agent to verify category filtering.
**Acceptance Criteria**:
- Test verifies that calling `search()` with `filters={"category": "AI"}` only returns records with that exact metadata category.
- Test verifies that omitting the filter returns all categories.
- Tests must fail initially (Red phase of TDD).
**Files to Create/Edit**:
- `tests/vector_storage/test_chroma_filters.py`
**Estimation**: 45 minutes
**Reference**: Phase 2: TDD & Test Creation

### [ ] Task 3: TDD Setup for Semantic Relevance
**Description**: Write failing tests to verify that semantically irrelevant results are dropped based on distance/score thresholds.
**Acceptance Criteria**:
- Test inserts "apple", "banana", and "quantum computing". Searching for "fruit" with a strict threshold should return "apple" and "banana" but exclude "quantum computing".
- Tests must fail initially.
**Files to Create/Edit**:
- `tests/vector_storage/test_chroma_relevance.py`
**Estimation**: 45 minutes
**Reference**: Phase 2: TDD & Test Creation

### [ ] Task 4: Implement ChromaDB Metadata Filtering
**Description**: Fix the ChromaDB implementation to pass the `filters` dictionary into the `where` parameter of the ChromaDB `query` method.
**Acceptance Criteria**:
- The `tests/vector_storage/test_chroma_filters.py` tests pass (Green phase).
- Empty filters gracefully fall back to querying without the `where` clause.
**Files to Create/Edit**:
- `src/vector_storage/chroma_store.py`
**Estimation**: 30-45 minutes
**Reference**: Phase 3: ChromaDB Query Tuning

### [ ] Task 5: Tune Embeddings & Distance Thresholds
**Description**: Fix the ChromaDB implementation to respect `max_distance`. Ensure the collection is initialized with `hnsw:space` set to `cosine` (if applicable) for better semantic separation.
**Acceptance Criteria**:
- The ChromaDB `query` method filters out results where the returned `distances` exceed the `max_distance` threshold.
- The `tests/vector_storage/test_chroma_relevance.py` tests pass.
**Files to Create/Edit**:
- `src/vector_storage/chroma_store.py`
**Estimation**: 60 minutes
**Reference**: Phase 3: ChromaDB Query Tuning

### [ ] Task 6: Update Telegram Bot `/latest` Handler
**Description**: Fix the `/latest` command in the bot to parse category arguments and pass them to the Vector Store.
**Acceptance Criteria**:
- Command `/latest AI` successfully parses "AI" and calls `vector_store.search(filters={"category": "AI"})`.
- Command `/latest` defaults to no filters.
- Unit tests for the aiogram handler pass.
**Files to Create/Edit**:
- `src/bot/handlers/commands.py`
- `tests/bot/test_handlers.py`
**Estimation**: 45 minutes
**Reference**: Phase 4: Bot Integration

### [ ] Task 7: Update Telegram Bot `/search` Handler
**Description**: Fix the `/search` command to utilize the new semantic relevance threshold and handle empty results gracefully.
**Acceptance Criteria**:
- Command `/search [query]` calls `vector_store.search()` with an optimal `max_distance` threshold.
- If no results meet the threshold, the bot replies politely: "No highly relevant news found for your query." instead of showing garbage data.
**Files to Create/Edit**:
- `src/bot/handlers/commands.py`
- `tests/bot/test_handlers.py`
**Estimation**: 45 minutes
**Reference**: Phase 4: Bot Integration

## Quality Requirements
- [ ] 100% of new code must have `pytest` coverage.
- [ ] No blocking I/O calls; all ChromaDB and Telegram API interactions must use `asyncio` or run in executors if synchronous.
- [ ] Follow SOLID: Do not tightly couple the bot handlers directly to the ChromaDB client; route through `IVectorStore`.
- [ ] Ensure the embedding model used for Russian text (`summary_ru`) is correctly configured in the Vector Storage initialization.

## Technical Notes
**Development Stack**: Python, aiogram, ChromaDB, pytest, asyncio.
**Special Instructions**: ChromaDB's default distance function is `l2` (Squared L2). When comparing textual embeddings, `cosine` similarity is often much better at separating irrelevant text. Check the ChromaDB collection creation code to ensure `metadata={"hnsw:space": "cosine"}` is set. If changing this, the ChromaDB collection may need to be recreated/reindexed.
**Timeline Expectations**: ~5.5 to 7 hours.