Skip to content

Amgix Search Query

To search an Amgix collection you POST to /v1/collections/<collection-name>/search endpoint.

A simple query may look like this:

curl -X POST http://localhost:8234/v1/collections/my_collection/search \
  -H "Content-Type: application/json" \
  -d '{"query": "search text"}'

This will return top 10 search results from the collection using all the vectors specified in the collection config with default weight of 1.0 and results fused/ranked with default RRF (Reciprocal Rank Fusion) algorithm.

But there are obviously more options that you can specify. Let's look at all of them.

Search Query Options

  • query: the search query text.
  • vector_weights: list of vectors, fields, and weights to use for searching. If empty, equal weights will be auto-generated for all available vectors. Details below.
  • custom_vectors: pre-generated custom vectors for this search query. For collections that use dense_custom or sparse_custom vector types, this is where you provide the vectors to search with.
  • limit: max number of results to return (1-100).
  • score_threshold: optional minimum score threshold. Results below this score will be excluded.
  • document_tags: optional filter to include only documents with specific tags.
  • document_tags_match_all: boolean. Defaults to false. If set to true documents must have ALL specified tags (AND). If false, documents must have ANY of the specified tags (OR).
  • metadata_filter: optional metadata filter. Accepts either an object or a filter expression string. Only fields declared in collection metadata_indexes can be filtered. More details below.
  • join: optional join of other collections onto each search result. Details below.
  • raw_scores: boolean. Default is false. Whether to include individual vector scores in results.
  • wmtr_trigram_weight: float. Defaults to 1.0. WMTR trigram channel multiplier for this search query. Allows to boost (or downplay) the influence of trigrams in wmtr vectors. For the detailed discussion and examples see this blog post.
  • fusion_mode: 'rrf' (default) or 'linear'. Defines how to combine multi-vector search results. More details can be found here.

Vector Weights

vector_weights field allows you to control how much influence a particular vector on a particular field has in the ranking of the fused results (RRF or Linear). By default, when the field is not specified, Amgix searches with all the vectors and fields defined on the collection with equal weight of 1.0. Let's look at an example:

{
    "query": "query text",
    "vector_weights": [
        {"vector_name": "my_keywords", "field": "name", "weight": 0.5},
        {"vector_name": "my_keywords", "field": "content", "weight": 1.0},
        {"vector_name": "my_dense", "field": "content", "weight": 2.0}
    ]
}
In this example, we are searching with 3 vectors: my_keywords vector on name and content fields and my_dense vector on content. We chose to boost dense vector search on content by setting it's weight to 2.0, gave medium importance to keyword search on the same field, and down-weigh the keyword search on name by setting it to 0.5.

The exact weights you use depend entirely on your dataset and query requirements. You may have to experiment with different value combinations to find the ratios that work best for your use case.

Setting vector-field weight to zero excludes that vector from search (performance boost). The same happens if you omit one of the vector-field combinations while specifying weights for others. In other words, if vector_weights values are provided, the search is only performed on the vector-field combinations that were specified and have non-zero weights.

Why is this interesting? Because this allows you to not only control how the results are fused together, but also gives you control of what specific vector-field combinations are being executed on per-query basis. You may index your collection with multiple vectors but use only a subset of them for different queries. For example, you may use fast WMTR vectors only for typeahead/autocomplete searches and then fire full hybrid search to display final results. We used this trick in our findgovdata.org implementation. You can read the details of that deployment here.

Custom Vectors

custom_vectors is the way to search the collection with custom (pre-embedded) vectors if the collection was created with dense_custom or sparse_custom vector types:

{
    "query": "query text",
    "vector_weights": [
        {"vector_name": "my_keywords", "field": "name", "weight": 0.5},
        {"vector_name": "my_keywords", "field": "content", "weight": 1.0},
        {"vector_name": "my_custom_dense", "field": "content", "weight": 2.0},
        {"vector_name": "my_custom_sparse", "field": "content", "weight": 1.5}
    ],
    "custom_vectors": [
        {
            "vector_name": "my_custom_dense",
            "vector": [0.12, -0.34, 0.56, 0.78, ...]
        },
        {
            "vector_name": "my_custom_sparse",
            "vector": [[1, 0.8], [2, 0.6], [3, 0.4], ...]
        }
    ]
}

Metadata Filters

metadata_filter is where you define how you want to filter the data before the search. This field accepts two forms of filters: structural and string.

Structural Metadata Filters

Example:

"metadata_filter": {
    "not": {
        "and": [
            {"key": "year", "op": "gt", "value": 2022},
            {"key": "author", "op": "eq", "value": "Bob"}
        ]
    }
}

'not', 'and' and 'or' are supported. key is the key of your metadata field, op is the operator (gt, lt, eq, ne, gte, lte).

String Metadata Filters

Simpler to read and reason about than the structural form above is string logical expression for the filter. For example, the above example can be specified as such:

"metadata_filter": "NOT (year > 2022 AND author = \"Bob\")"

Joins

join query option allows you to pull related documents from other collections. This is a LEFT join in SQL terminology. Only the primary collection is searched and then documents from the joined collections are appended to the results of the search.

Simple Join

In the simplest form the join is just the name of the other collection:

"join": "my_other_collection"

In this form, the join is performed on document id fields. For each search result in the collection my_collection (for example) Amgix finds documents in my_other_collection with matching document IDs and appends them to the result in joined field.

You can join multiple collections by supplying a list of them:

"join": ["my_other_collection","my_other_collection2"]

Specifying Join Fields

The above example can also be written in an expanded form:

"join": "my_other_collection[$id = $$id]"

single $ refers to the main collection (collection we are searching) and double $$ is referring to the collection we are joining. So this example is equivalent to the simple join example: id of the document in the searched collection must be equal to the id of the document in joined collection.

But we can also match on metadata fields (they must be indexed when collections are created):

"join": "my_other_collection[$id = $$.meta.parent_id]"

or 

"join": "my_other_collection[$.meta.child_id = $$id]"

or

"join": "my_other_collection[$.meta.some_id = $$.meta.ref_id]"

Filtering Joined Collection

Joined collection can also be filtered:

"join": "my_other_collection[$.meta.some_id = $$.meta.ref_id](price > 100)"

This appends documents from my_other_collection only when they match the filter expression (price greater than 100, in this example).

This is the same syntax we use in metadata_filter specs that allows for complex logic expressions:

"join": "my_other_collection[$.meta.some_id = $$.meta.ref_id](NOT (year > 2022 AND author = \"Bob\") OR price > 1000)"