Indexing Repositories

This page covers everything you need to configure, run, and maintain a code index with TeaRAGs.

Environment Variables

All code-vectorization settings are controlled via environment variables passed to the MCP server.

Indexing variables reference

Variable	Description	Default
INGEST_CHUNK_SIZE	Maximum chunk size in characters. Larger = more context per chunk; smaller = more granular search results.	2500
INGEST_CHUNK_OVERLAP	Overlap between adjacent chunks in characters. Prevents loss of context at chunk boundaries.	300
INGEST_ENABLE_AST	Use AST-aware chunking (tree-sitter) instead of line-based splitting. Preserves semantic boundaries like functions, classes, and methods.	true
INGEST_ENABLE_HYBRID	Provision BM25 sparse vectors. Required to use the hybrid_search tool.	true
EMBEDDING_TUNE_BATCH_SIZE	Upper bound on the chunks per embedding batch. The run picks the batch size below it from measured throughput and server failures (Adaptive Embedding).	auto
TRAJECTORY_GIT_ENABLED	Enrich every chunk with git signals: recent committers (recent* family), live-line owners (blame* family), commit count, bug-fix rate, age, task IDs. Required for hotspots / techDebt / ownership / recentActivityConcentration / securityAudit / etc. presets. Silently skipped for non-git directories.	true

Legacy names (CODE_CHUNK_SIZE, CODE_CHUNK_OVERLAP, CODE_ENABLE_AST, CODE_ENABLE_HYBRID, CODE_BATCH_SIZE, CODE_ENABLE_GIT_METADATA) still work as fallbacks but are deprecated.

Per-index overrides (pass to index_codebase directly instead of env):

• extensions: string[] — e.g., [".proto", ".graphql", ".prisma"]
• ignorePatterns: string[] — e.g., ["**/*.generated.ts", "**/dist/**"]

See also the full Configuration Variables page for embedding, Qdrant batch pipeline, and performance variables.

How Chunking Works

TeaRAGs splits source files into chunks — self-contained code fragments that become individual vectors in the search index. The chunking strategy directly affects search quality: well-bounded chunks return precise, meaningful results; poorly split chunks return noisy, incomplete fragments.

AST-Aware Chunking (default)

Enabled by default (INGEST_ENABLE_AST=true). Uses tree-sitter to parse the code into an Abstract Syntax Tree, then splits along semantic boundaries — functions, classes, methods, interfaces.

Why it matters:

• A function stays as one chunk, not split in the middle of a loop
• Class methods are extracted as individual chunks with parent class context preserved (parentName, parentType metadata)
• Comments and decorators stay attached to the code they describe
• Large classes are automatically decomposed: methods become individual chunks, non-method body code is extracted separately

Two-level extraction:

1. Top-level nodes — functions, classes, modules, interfaces are identified as chunk candidates
2. Child extraction — if a container (class/module) exceeds 2x INGEST_CHUNK_SIZE, its methods are extracted as individual chunks while preserving parent context in metadata

Fallback: If a language has no tree-sitter grammar, or a node is still too large after child extraction, TeaRAGs falls back to character-based splitting with overlap — trying to break at empty lines or closing braces.

Top-level module code. Statements at the top of a file that no function or class chunk carries — a const loaders = { … } object beside the file's functions, top-level configuration, a script body — are indexed as a module remainder chunk (chunkType: "block"). Imports, export … from lines and bare export lists are left out; comments directly above a kept statement stay with it. A remainder that is a single named declaration takes its name as symbolId. Before this, such code was dropped whenever the file produced at least one other chunk.

TSX. .tsx files are parsed with the tsx grammar. Under the plain typescript grammar JSX parsed into error nodes and the component itself was not chunked.

Markdown. Documents are split by heading. Small sections share a chunk with their siblings: an h3 joins its h2 while the chunk fits the maximum chunk size, and a small h2 (under 300 characters) joins the chunk before it under the same h1. An h1 boundary is never crossed. Sections shorter than 50 characters are kept by joining a neighbour instead of being dropped; only a document whose whole content is under 50 characters produces no chunk. A heading-less document or an oversized preamble is split under the chunk size.

Test files. Test files (RSpec, and Jest / Vitest / Mocha-style TypeScript and JavaScript tests) are chunked by scope:

• Setup is stored once per scope. let, before / beforeEach and other setup lines of a describe / context go into a test_setup chunk instead of being copied into every example. The setup of consecutive scopes is packed into one chunk up to the chunk size.
• Setup is added back when you read an example. Search results and find_symbol return a test example with the setup of every enclosing scope prepended, outermost first, so the example reads as runnable code. metaOnly answers skip this. Setup that comes from shared definitions elsewhere (include_context, shared_examples, it_behaves_like) is not resolved; the delegating line appears as written.
• Small adjacent examples are packed. Examples of the same scope with no child scope between them share one test chunk up to 1500 characters; an example larger than that stays on its own. Each example in a pack keeps its own symbolId: find_symbol on it returns that example alone, with its own line range, and file and scope outlines list every example.

note

These chunking changes move the chunk set of existing indexes. After upgrading, the drift report asks for tea-rags index-codebase --project <alias> --force; see After upgrading tea-rags.

tip

AST-aware chunking is the recommended mode for all code. Disable it (INGEST_ENABLE_AST=false) only for plain-text or unstructured files where tree-sitter adds no value.

Embedding Dimensions

The embedding dimension is determined automatically based on your model — no configuration needed in most cases.

Provider	Model	Dimensions
Ollama	unclemusclez/jina-embeddings-v2-base-code (default)	768
Ollama	nomic-embed-text	768
Ollama	mxbai-embed-large	1024
OpenAI	text-embedding-3-small	1536
Cohere	embed-english-v3.0	1024
Voyage	voyage-code-2	1536

Override with EMBEDDING_DIMENSIONS only if your model is not in the built-in registry.

warning

Changing the embedding model or dimensions after indexing requires a full reindex — existing vectors are incompatible with a different dimension.

File Filtering

Built-in Exclusions

The indexer automatically skips common non-source directories and files:

• node_modules/, vendor/, .git/, dist/, build/, coverage/
• Binary files, images, lock files, and minified bundles

Using .contextignore

Create a .contextignore file in your project root to define project-specific exclusions. The syntax is identical to .gitignore:

# Exclude test files from the index
**/test/**
**/*.test.ts
**/*.spec.ts

# Exclude generated code
*.generated.ts
**/dist/**

# Exclude fixtures and mocks
**/fixtures/**
**/mocks/**
**/__tests__/**

.contextignore rules are applied in addition to .gitignore. If a file is ignored by either, it will not be indexed.

Custom Extensions & Ignore Patterns

Pass per-index overrides to index_codebase instead of global env vars:

{
  "path": "/project",
  "extensions": [".proto", ".graphql", ".prisma"],
  "ignorePatterns": ["**/*.generated.ts", "**/dist/**"]
}

For team-wide rules use .contextignore (see Ignoring Files).

Indexing Workflow

Full Index

Run a full index the first time you set up a codebase:

/tea-rags:index [path]

The indexer will:

1. Discover files (respecting .gitignore, .contextignore, built-in exclusions)
2. Parse each file with the appropriate language parser (AST or line-based)
3. Split into chunks preserving semantic boundaries
4. Generate vector embeddings in batches
5. Upsert chunks into a Qdrant collection
6. Save a file-hash snapshot for future incremental updates

Register and Index in One Step

From the CLI, index-codebase --name <alias> registers a new project alias and then indexes it in a single command — the first-index shortcut for a project that has never been indexed:

tea-rags index-codebase --name my-project /path/to/repo
tea-rags index-codebase --name my-project --force /path/to/repo   # full rebuild while registering

--name writes the alias into the registry before forking the indexer, so a registration error (e.g. the alias already exists) is reported and the command exits before any indexing work starts — text by default, or a parseable { error: { code, message } } under --json.

--name (write mode — register a new alias) conflicts with --project (read mode — use an existing alias). To index an already-registered project, use --project; to seed a worktree clone from an already-indexed source, use tea-rags worktree create instead — it registers the alias with worktree provenance.

First Index of a Git Worktree

When the path is a git worktree and another working tree of the same repository is already indexed with the same model and settings, the first index seeds from it: the sibling's index is cloned and only the files that differ are embedded. Git signals are then rebuilt against the worktree's own history in the background. The run's result reports it under worktreeSeed — which sibling, how many files were copied and how many embedded, or why no sibling qualified.

tea-rags index-codebase --name my-feature /path/to/worktree       # seeds when a sibling qualifies
tea-rags index-codebase --no-worktree-seed /path/to/worktree      # always index from scratch

Over MCP the opt-out is seedFromWorktree: false on index_codebase. Checks, what is copied and what is recomputed: Automatic Seeding on First Index.

Incremental Reindex

After the initial index, /tea-rags:index auto-detects and runs an incremental reindex — only added, modified, and deleted files are processed:

/tea-rags:index [path]

Significantly faster than a full re-index — only the files that changed since the last snapshot get re-embedded.

Force Reindex (Zero-Downtime)

If you change chunking settings (e.g., INGEST_CHUNK_SIZE) or embedding model, the existing chunks are no longer compatible. Use the force-reindex skill — it builds a new versioned collection in the background while search stays available on the current one, then atomically switches:

/tea-rags:force-reindex [path]

Requires explicit user confirmation. See Skills for details.

Check Index Status

Your agent calls get_index_status directly (no dedicated skill):

Show me stats for the current index

For percentile-based signal thresholds (label maps that map raw git metrics to low / typical / high / extreme in your codebase), use the get_index_metrics tool — see MCP Tools Atlas.

The status endpoint reports three states:

Status	Meaning
not_indexed	No index exists for this path. Run indexing first.
indexing	Indexing is in progress. Shows the number of chunks processed so far.
indexed	Index is ready. Returns collection name, chunk count, and last-updated timestamp.

Example response when indexed:

{
  "isIndexed": true,
  "status": "indexed",
  "collectionName": "code_a3f8d2e1",
  "chunksCount": 1823,
  "lastUpdated": "2025-01-30T10:15:00Z"
}

Best Practices

1. Optimize Chunk Size

The right chunk size depends on your codebase style:

Codebase Type	Recommended INGEST_CHUNK_SIZE	Recommended INGEST_CHUNK_OVERLAP
Small, focused functions	1500 - 2000	200
Large classes / modules	3000 - 4000	400
Documentation (Markdown)	2000 - 2500	300

Smaller chunks produce more granular results but lose surrounding context. Larger chunks preserve context but may dilute relevance if the chunk contains unrelated code. Start with the defaults (2500 / 300) and adjust based on search quality.

2. Filter Aggressively

Exclude noise from the index to improve both speed and search relevance:

# .contextignore
**/vendor/**
**/node_modules/**
**/*.min.js
**/*.bundle.js
**/coverage/**
**/.git/**
**/dist/**
**/build/**

Every excluded file saves embedding time and reduces index size.

3. Choose the Right Embedding Model

Use Case	Recommended Model
General code search	nomic-embed-text (Ollama)
Code-specialized (default)	unclemusclez/jina-embeddings-v2-base-code (Ollama)
Multilingual code	jina-embeddings-v2-base-code
Maximum accuracy	voyage-code-2 (Voyage AI)

For most projects, the default Jina code embeddings via Ollama offer the best balance of quality, speed, and privacy. See the Embedding Providers page for detailed comparisons.

4. Use Incremental Updates

After the initial full index, always prefer incremental reindex. /tea-rags:index handles both cases automatically:

/tea-rags:index [path]

This avoids re-embedding unchanged files and keeps your index fresh with minimal latency.

5. Monitor Index Status

Before searching, verify the index is ready:

Show me stats for the current index

This is especially useful after long-running indexing operations on large codebases (10k+ files).

6. Enable Git Metadata When Needed

Git metadata enrichment (TRAJECTORY_GIT_ENABLED=true) adds significant value for analytics — ownership reports, tech debt detection, hotspot analysis. Enrichment runs concurrently with embedding and does not increase indexing time. Enable it when you plan to use rerank presets like techDebt, hotspots, ownership, or codeReview.

Troubleshooting

Search Returns No Results

1. Check index status: ask your agent to show index stats and confirm the status is indexed (not indexing or not_indexed)
2. Verify files are not ignored: check .gitignore, .contextignore, and the ignorePatterns parameter for overly broad patterns
3. Broaden your query: semantic search works best with natural language; instead of "getUserById", try "user retrieval by ID"
4. Check the collection name: if you moved or renamed the project directory, the collection name hash will change and a new index is needed

Slow Indexing

1. Use local Ollama instead of a cloud provider to avoid network latency and rate limits:

   export EMBEDDING_PROVIDER=ollama

2. Check the batch size the run settled on — infraHealth.embedding.throughputTune in get_index_status. If it sits at EMBEDDING_TUNE_BATCH_SIZE, the ceiling may be holding it back; raise it if your system has enough memory:

   export EMBEDDING_TUNE_BATCH_SIZE=2048

3. Exclude large or binary files that produce low-value chunks (lock files, minified bundles, vendored code)
4. Let pipeline concurrency climb. Left unset, the run climbs embedding concurrency from 1 up to an implicit ceiling of 8 while throughput rises. An explicit value is a hard cap — set one above 8 only for a cloud provider with spare rate-limit headroom:

   export INGEST_PIPELINE_CONCURRENCY=16

Memory Issues

1. Reduce chunk size to lower per-batch memory:

   export INGEST_CHUNK_SIZE=1500

2. Reduce the batch-size ceiling to process fewer chunks at once (the run never goes above it):

   export EMBEDDING_TUNE_BATCH_SIZE=50

3. Index subdirectories separately for very large monorepos — ask your agent to index each service path individually

Files Not Being Indexed

1. Check that the file extension is in the supported list or added via the extensions parameter to index_codebase
2. Verify the file is not matched by .gitignore, .contextignore, or the ignorePatterns parameter
3. Ensure the file is not empty or binary

For more troubleshooting scenarios see Troubleshooting and Recovery & Reindexing.

See Also

• Skills — /tea-rags:index, /tea-rags:force-reindex
• MCP Tools Atlas — index_codebase, get_index_status, get_index_metrics, clear_index parameters
• Collections — multi-project workflow and collection naming
• Ignoring Files — .contextignore and built-in exclusions