Embeddings Configuration

Embeddings are numerical representations of content that capture semantic meaning, enabling powerful similarity search and content understanding. Vertesia supports multiple embedding types to index different aspects of your documents.

Understanding Embeddings

An embedding converts content (text, images, or structured data) into a high-dimensional vector. Documents with similar meanings have vectors that are close together in this space, enabling semantic search that goes beyond keyword matching.

For example, a document about "machine learning algorithms" would have an embedding similar to one about "AI model training" because they share semantic meaning, even though they use different words.

Embedding Types

Vertesia supports three types of embeddings, each capturing different aspects of your documents:

Text Embeddings

Text embeddings capture the semantic meaning of document content. When enabled, Vertesia:

1. Extracts text content from documents (including OCR for images/PDFs)
2. Chunks text to fit within model token limits
3. Generates embeddings using the configured model
4. Stores vectors for similarity search

Use cases:

• Finding documents with similar content
• Natural language queries
• Content recommendations

Image Embeddings

Image embeddings (also called vision embeddings) capture visual features of images and document pages. When enabled:

1. Document pages/images are processed by a vision embedding model
2. Visual features are encoded as vectors
3. Enables similarity search based on visual content

Use cases:

• Finding visually similar documents
• Image-based search
• Visual content analysis

Properties Embeddings

Properties embeddings index the structured metadata of documents. This includes:

• Custom properties defined in your schema
• Document type information
• Tags and categories

Use cases:

• Finding documents with similar metadata
• Structured data similarity
• Schema-aware search

Configuration Options

Each embedding type can be configured independently with these settings:

Environment Selection

Select an environment that provides access to embedding models. Common choices include:

• OpenAI: text-embedding-ada-002 (1536 dimensions), text-embedding-3-small (1536 dimensions), text-embedding-3-large (3072 dimensions)
• Google Vertex AI: Various embedding models
• AWS Bedrock: Titan and other embedding models

Dimension Selection

Embedding dimensions affect both quality and performance:

Most projects work well with 1536 dimensions (OpenAI's default).

Enabling Embeddings via API

Get Embeddings Status

Check the current status of an embedding type:

curl --location --request GET \
  'https://api.vertesia.io/api/v1/commands/embeddings/text/status' \
  --header 'Authorization: Bearer <YOUR_JWT_TOKEN>'

Example Response:

{
  "status": "idle",
  "embeddingRunsInProgress": 0,
  "totalRunsInProgress": 0,
  "embeddingsModels": ["text-embedding-ada-002"],
  "vectorIndex": {
    "status": "READY",
    "name": "vsearch_text",
    "type": "hnsw"
  }
}

Enable Embeddings

Activate embeddings for a specific type:

curl --location --request POST \
  'https://api.vertesia.io/api/v1/commands/embeddings/text/enable' \
  --header 'Authorization: Bearer <YOUR_JWT_TOKEN>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "environment": "<ENVIRONMENT_ID>",
    "max_tokens": 500,
    "dimensions": 1536,
    "model": "text-embedding-ada-002"
  }'

Disable Embeddings

Deactivate embeddings for a specific type:

curl --location --request POST \
  'https://api.vertesia.io/api/v1/commands/embeddings/text/disable' \
  --header 'Authorization: Bearer <YOUR_JWT_TOKEN>'

Recalculating Embeddings

You may need to recalculate embeddings when:

• Switching to a different embedding model
• Changing dimension settings
• Fixing corrupted or missing embeddings
• Upgrading to a newer model version

Trigger Recalculation

curl --location --request POST \
  'https://api.vertesia.io/api/v1/commands/embeddings/text/recalculate' \
  --header 'Authorization: Bearer <YOUR_JWT_TOKEN>'

Recalculation runs as a background process. Monitor progress through the status endpoint.

Vector Index Status

The vector index status indicates the state of your search index:

Best Practices

Model Selection

• Start with standard models: text-embedding-ada-002 or text-embedding-3-small work well for most use cases
• Consider costs: Higher-dimension models cost more to generate and store
• Test with your data: Different models perform better on different content types

Token Limits

• 500-1000 tokens: Good for short documents, faster processing
• 2000-4000 tokens: Better for longer documents, captures more context
• Consider chunking: Very long documents are automatically chunked

Performance Optimization

• Enable only the embedding types you need
• Use appropriate dimensions for your quality requirements
• Monitor index status to ensure embeddings are being generated

Troubleshooting

Embeddings Not Generating

1. Check that the embedding type is enabled in project settings
2. Verify the environment has a valid API key
3. Ensure the model is available in your environment
4. Check for quota limits on your AI provider

Dimension Mismatch Errors

If you change embedding dimensions, you may encounter mismatches:

1. Recalculate all embeddings with the new dimensions
2. Or recreate the vector index (requires reindexing all documents)

Slow Embedding Generation

• Large backlogs process over time
• Check embeddingRunsInProgress in status
• Consider enabling parallel processing in workflows

Next Steps

• Search Configuration - Configure search backends
• Content Overview - Understand the full search architecture
• Commands API - API reference for embedding commands