Skip to content

Amgix Collections

A collection is a grouping of documents in Amgix. All documents live in a collection. Collection defines how the documents are indexed and stored.

You create a collection by POSTing a collection config to /v1/collections/<collection-name> endpoint. Collection name can be any combination of letters, numbers, underscores, and dashes.

A simple collection can be created like this:

curl -X POST http://localhost:8234/v1/collections/my_collection \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": [
      {
        "name": "my_keywords",
        "type": "keyword",
        "index_fields": ["name"]
      }
    ]
  }'

This tells Amgix, index name field (of the documents) using keyword tokenizer and call this vector my_keywords.

Here is an example of the shape of a more complex collection config:

{
  "store_content": true,
  "metadata_indexes": [
    { "key": "category", "type": "string" },
    { "key": "year", "type": "integer" },
    { "key": "published", "type": "datetime" }
  ],
  "vectors": [
    {
      "name": "keywords",
      "type": "wmtr",
      "index_fields": ["name"]
    },
    {
      "name": "dense",
      "type": "dense_model",
      "model": "sentence-transformers/all-MiniLM-L6-v2",
      "index_fields": ["content"]
    },
    {
      "name": "my_embeddings",
      "type": "dense_custom",
      "dimensions": 384,
      "index_fields": ["content"]
    }
  ]
}
There are only 3 top level fields:

  • store_content: an optional boolean value (defaults to true). Because content field of the document can be a rather large text that is often used only to create the embeddings, this option allows you to not store the content text after the document is indexed, resulting in smaller search results payloads and storage requirements.

  • metadata_indexes is an optional field where you can declare which metadata keys needs to be indexed for filtering. Only metadata fields declared here can be used to filter results or lists of documents. The following types of metadata are supported: string, integer, float, boolean, or datetime.

  • vectors is the only required field. It specifies how the collection is embedded/indexed and how it can be searched.

Vectors

Amgix is a vector-first search system. Under the hood, Amgix represents all searchable entities as either dense or sparse vectors. This is true even when you just want a keyword search.

Unique name, type, and a list of index_fields are the only fields required for all vector types. Additional properties may be required or available for different types. Let's first look at what vector types are available.

Vector Types

The following vector types are supported by Amgix:

Lexical Tokenizers

Lexical tokenizers in Amgix are algorithmic (not ML-model based) text processing functions that represent text as sparse vectors. Because they do not use model inference and are written in Rust, they are very fast and lightweight. And because they produce sparse vectors they have some unusual lexical-processing properties. For example, they all require top_k parameter, which means that only top number of the most frequent tokens in the text is stored (and searched). They all implement the TF-IDF-ish semantics, which improves search relevance.

keyword

keyword vector type is just an alias for wmtr. See next.

wmtr

WMTR (Weighted Multilevel Token Representation) is Amgix go-to keyword tokenizer, hence it's aliased as keyword. WMTR is designed to capture lexical information at 3 different levels/views at the same time: surface level (think whitespace), natural language level (think full-text), and character level (think trigrams). It is designed to work well on natural language, identifier-heavy data (part numbers, SKUs, special characters, etc.), or mixed/unknown text. You can learn more about WMTR performance in our benchmarks and blog posts. In addition to the above, WMTR is typo-tolerant and does partial string matching. It works well with all the defaults, but there are some knobs you can tweak. See later.

full_text

Just what you would expect: full text tokenizer that does stopword removal and stemming. Great for natural language text. If you don't need type-tolerance or partial matching of wmtr, this is a faster option.

trigrams

Trigrams representation of the text. Text is broken up in 3-character patterns to capture partial string matches.

whitespace

Text is broken up based on whitespace.

Model-Based Vectors

These vector types do local (on the machine where Amgix is running) ML-model embeddings using models that are auto-downloaded (and cached locally) from Hugging Face. model field is required for these vector types. Other options are available. See below.

dense_model

This type uses transformer-based models (like BERT) do create dense embeddings. Dimensions are auto-detected at collection creation time. Examples of models that can be used are sentence-transformers/all-MiniLM-L6-v2 or BAAI/bge-small-en-v1.5.

sparse_model

This type is used for models generating sparse vector representations of the text (think SPLADE): prithivida/Splade_PP_en_v1 or naver/splade-v3-distilbert.

Custom Vectors

Custom vector types are used for situations where you want to supply your own vectors with documents (you are doing your own embedding work).

dense_custom

This type is for your own dense embeddings. dimensions field must be set to the number of dimensions your model produces.

sparse_custom

Custom sparse vectors require top_k to be set - it defaults to 128.

Miscellaneous

noop

noop is a no-op vector type - it generates no vectors. It can be used to create collections that you are not planning to search, but just want to store the documents (and their metadata) in them. Not searchable collections can be useful for plain storage and for using them in joins to add additional related information to your search results from other collections.

Vector Options

Let's look at all the available vector fields:

  • name: unique name of the vector in this collection.
  • type: type of the vector.
  • model: model used for embeddings (required for dense_model and sparse_model types).
  • revision: optional specific revision (commit id) of the model.
  • query_model: some setups use different models to embed the documents and search queries. This option allows you to specify a different model to be used at query time. For example you may use naver/efficient-splade-V-large-doc as your document embedding model and naver/efficient-splade-V-large-query for query_model to use in search queries.
  • query_revision: specific revision of query model to be used.
  • dimensions: required for dense_custom vector type, auto-detected for dense_model type. Ignored by the rest of the types.
  • top_k: number of top-scoring terms for sparse vectors (including Lexical Tokenizers). Defaults to 128. Ignored for dense vectors.
  • wmtr_word_ratio: the percentage of WMTR vector's top_k to be used for words (vs trigrams): 1 to 100. Default is 80. Setting this 100, effectively disables the trigram component of WMTR. Setting it to lower numbers increases the number of trigrams captured while reducing the number of words indexed.
  • index_fields: an array of fields to be indexed by this vector. Allowed values are name, description, and content. For the discussion on why only those fields, see our Documents Guide.
  • language_default_code: Two-letter ISO 639-1 language code for language-based vector types (e.g., 'en', 'es', 'fr'). Defaults to en. This code is used by wmtr and full_text tokenizers for selecting language specific stopwords and stemming (where available).
  • language_detect: boolean instructing Amgix if it should attempt to auto-detect language (for stopwords and stemming). Defaults to false. Enabling this option will negatively affect indexing and query performance, but can be useful in the collections where documents are in different languages.
  • language_confidence: float (0-1). Minimum confidence threshold for language detection. If detection confidence is below this value, language_default_code will be used instead. Default is 0.9.
  • normalization: (true|false). Defaults to false. Whether to normalize vectors. Only supported for dense vectors. Sparse vectors do not support normalization.
  • dense_distance: Distance metric for dense vectors (cosine, dot, euclid). Defaults to cosine.
  • keep_case: Whether to keep original case for text preprocessing. Only applies to model-based vectors (dense_model, sparse_model). Defaults to false (lowercase).

Supported Languages

Here is the list of currently supported languages for stopwords and stemming:

Code Language Stopwords Stemming
ar Arabic yes yes
da Danish yes yes
de German yes yes
el Greek yes yes
en English yes yes
es Spanish yes yes
fi Finnish yes no
fr French yes yes
hu Hungarian yes yes
it Italian yes yes
nl Dutch yes yes
no Norwegian yes yes
pt Portuguese yes yes
ro Romanian yes yes
ru Russian yes yes
sv Swedish yes yes
ta Tamil no yes
tr Turkish yes yes

Any other valid ISO 639-1 code is accepted, but tokenization falls back to Unicode word segmentation without stopword filtering or stemming.