Agent Observatory & Tool-Calling RAG Workflow Guide
This skill guides agents and engineers on how to build, extend, test, and enhance AI agent services utilizing Tool-Calling RAG, LangChain/LiteLLM architectures, and pgvector embeddings.
1. Local Branch-First Development
main: ``bash git switch -c <developer-or-agent>/main/<feature-name> ` Never develop or modify agent code directly on main`.2. Adding a New Agent Tool
agent/tools/ or data/tools/):```python
from typing import Annotated, Any
from langchain_core.tools import tool
@tool
async def inspect_custom_metric(
metric_name: Annotated[str, "The name of the metric to query"],
days: Annotated[int, "Number of lookback days"] = 7,
) -> dict[str, Any]:
"""Query operational metrics from the database or external API."""
# Perform database query or API call
return {"metric": metric_name, "value": 42}
```
__init__.py.TOOLS list.3. Tool-Calling RAG & Vector Knowledge Base
The AI service operates as a Tool-Calling RAG Agent:
hybrid_search_knowledge_base during reasoning loops for deep evidence gathering:```python
from agent.tools import hybrid_search_knowledge_base
```
* Supported source filters: ['chat_message', 'execution_log', 'investigation', 'task_result', 'incident_fix'].
* Combines Full-Text Search (tsvector), pgvector cosine similarity, and Reciprocal Rank Fusion (RRF).
4. Implementing Human-In-The-Loop (HITL) Actions
For sensitive or destructive actions (e.g., dispatching external emails, modifying records, triggering external deployments):
```python
if tool_name in CONFIRMATION_REQUIRED_TOOLS:
confirm_id = str(uuid.uuid4())
await confirmations.create(confirm_id, username=username, tool_name=tool_name, args=tool_args)
yield json.dumps({
"type": "confirm_required",
"confirm_id": confirm_id,
"name": tool_name,
"args": tool_args,
"tool_call_id": tool_call_id,
})
# Poll this confirmation's own row; on timeout or cancellation mark it expired.
decision = await confirmations.wait_for_decision(confirm_id, timeout=120.0, poll_interval=0.5)
```
5. Multi-Key API Failover
When interacting with external LLM APIs (e.g. OpenRouter, OpenAI, Anthropic):
401, 402 or 429. Other errors, such as 400 for a bad request, are the caller's problem and must be returned, not retried on every key.6. Comprehensive Agent Test Suites
Use pytest with the network blocked, so a test that would call a real LLM, email server or backend fails instead:
# Unit tests (HTTP mocked with respx, LLMs replaced by fakes)
uv run pytest
# Integration tests against a disposable local database
TEST_DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres uv run pytest -m integration