Skip to content

Document Upload and Deletion in Amgix

This guide explains how Amgix handles document upload (add/update), deletion, asynchronous processing and queue management.

Upserts (add/update)

Amgix API exposes 3 document upsert (add/update) endpoints:

POST /v1/collections/<collection-name>/documents

POST /v1/collections/<collection-name>/documents/sync

POST /v1/collections/<collection-name>/documents/bulk

The first two are for single document uploads and expect a single document-shaped JSON payload. The last one is for bulk upload of up to 100 documents in a batch:

{
    "documents": [{doc1}, {doc2}, ...]
}

Note

Because all the documents in a collection required to have a unique ID, add or update is decided based on whether the ID is already in collection or not.

/sync endpoint is the only endpoint that processes a document synchronously: Amgix indexes and embeds the document before returning the success (or failure). This allows for applications that require immediate search consistency guarantees to ensure that the document is successfully processed and available for search/retrieval as soon as the API call succeeds.

The other two endpoints are asynchronous: the documents are stored internally and queued for processing before the API responds. This is much faster, in most cases, since the calling application doesn't have to wait for the heavy indexing/embedding work to complete. This is also an eventual consistency contract - there is no way of knowing when the document will appear in the search system: it could be milliseconds, it could be hours, depending on many factors: current queue depth (how many documents are already in the queue), system resources (how quickly can system process documents), collection configuration (how many and what kind of vectors the system has to generate), system capacity (how busy is the system processing other requests), etc.

Ingestion in Amgix Now

All three endpoints in Amgix Now behave synchronously. Amgix Now doesn't have an internal queue (at least not the kind that Amgix has) and therefore cannot defer indexing/embedding for later processing.

For collections vectorized with lightweight lexical tokenizers and small models, this is usually not a problem. But heavier models and/or large documents sent to /bulk endpoint can take some time to embed, which could be a problem for some applications.

If this is your use case, we recommend switching to Amgix One and using its asynchronous capabilities. Alternatively, if you still want Amgix Now search performance, you can join Amgix Now instance to an Amgix cluster and direct ingestion traffic to asynchronous Amgix nodes, while serving search requests with Amgix Now. For ideas on how something like this can be setup, you can read our Amgix Building Blocks page or see how we implemented this in findgovdata.org.

Note

Internally, Amgix keeps bulk upload requests in a separate queue from single-document upload requests. This is done to prevent massive bulk upload jobs from blocking single-document processing for long periods of time. The two queues are processed in a round-robin fashion, interleaving single-doc indexing with bulk requests.

Think of an application that constantly does single writes (users update and add documents as a part of their workflow in the app all the time). While this is happening, some background process dumps 1M documents into the system. If the bulk and single-doc queues were not separated, the bulk upload would delay single-document updates for a lengthy period of time. Our architecture works around this problem by separating bulk and single-document queues. Bulk upload of 1M documents will slow things down (system has to index/embed large batches), but it will still process single-document updates in a reasonable timeframe.

Note

When the document is updated, Amgix automatically skips indexing/embedding (vector generation) part of the document processing if name, description and content fields of the document did not change from what is already in the system. This effectively patches the metadata of the document without heavier processing.

The only notable exception to this is when the collection is defined with store_content set to false. In this case, Amgix has not stored the content information with the document and has no way of knowing if it has changed, therefore it regenerates the vectors. There is a technical solution to this, of course, but it has not been implemented yet. Stay tuned.

Deletes

Amgix implements two delete endpoints:

DELETE /v1/collections/<collection-name>/documents/<document-id>?request_timestamp=<UTC-timestamp>

DELETE /v1/collections/<collection-name>/documents/<document-id>/sync?request_timestamp=<UTC-timestamp>

Similarly to document uploads, the first endpoint is async (it queues delete request for later processing). The second endpoint is synchronous - it returns only when the document is deleted from the system.

Please note, that both endpoints require request_timestamp query string parameter. This is used for automatic deduplication.

In Amgix Now both delete endpoints behave synchronously, for the reasons explained above.

Automatic Deduplication

Amgix provides automatic document upload and deletion timestamp deduplication. All documents uploaded to the system are required to have a UTC timestamp and all delete requests must provide a UTC request_timestamp. This assures the following benefits:

  • Older updates cannot overwrite newer documents.
  • Older document deletion requests won't delete newer documents.
  • Older updates/deletes do not waste system resources on processing stale documents.
  • Delete requests automatically remove older upsert documents from the queue, saving processing time on the documents that should not be in the system (were later deleted).
  • This guards against any race conditions where you may have multiple processes writing to the search engine at the same time.

You and your application don't have to worry about your search system correctness as long as you have a coherent document timestamp strategy when you send your documents. For example, if your application already stores last_modified or equivalent datetime on your records, simply supplying these values as document timestamps ensures that Amgix documents accurately reflect the state of your source documents.

This also allows you to, for example, bulk update the entire corpus of the documents (say in a nightly batch process), without having to worry (or needing to write complex custom logic) that your bulk updates will nuke some newer records (say updated by a user in the time window between your bulk process reads the records from your source DB and uploads them to Amgix).

Atomic Writes

Amgix can be deployed as a single container or as a distributed cluster. In the cluster scenarios multiple Amgix nodes maybe processing document updates and deletes at the same time. To ensure the system correctness, Amgix takes distributed locks on every document write (add/update/delete) operation. This prevents race conditions and combined with automatic document deduplication, guarantees that only the latest update ends up in the system. This also ensures that the documents are written atomically across the cluster: document write operation and queue update are treated as a single transaction.

Automatic Retries

For asynchronous operations, Amgix automatically retries on failures with progressive back-off delays. But it doesn't treat all failures equally. Vectorization failures (CPU-bound work) retried only a couple of times. I/O-bound work (DB read/writes or communication failures) retried for much longer.

Why?

Unlike a typical search engine, Amgix is not tightly coupled with the database backend (Qdrant, PostgreSQL, MariaDB) or the message broker (RabbitMQ). Those systems are treated as "remote" even if they are packaged in the same container (like can be the case with Amgix One or Amgix Now). Therefore, temporary network glitches and other operational hiccups in these systems are possible. Longer retries on I/O failures ensure that your document updates (or deletes) will be more likely to survive these temporary "outages" and will eventually make it into the storage backend.

Working With Asynchronous Queue

Amgix Now does not support asynchronous operations and therefore does not implement the endpoints to work with asynchronous queue

Individual Document Status

At any point, you may check the status of an individual document in the system by calling this endpoint:

GET /v1/collections/<collection-name>/documents/<document-id>/status

The endpoint responds with a list (array) of statuses for the document:

{
  "statuses": [
    {
      "status": "queued",
      "op_type": "upsert",
      "timestamp": "2026-06-21T12:05:00Z",
      "queue_id": "...",
      "try_count": 0,
      "info": null
    },
    {
      "status": "indexed",
      "timestamp": "2026-06-21T12:00:00Z",
      "op_type": null,
      "queue_id": null,
      "try_count": null,
      "info": null
    }
  ]
}
In the above example, the document is already indexed in the system, but has a pending upsert operation.

Possible statuses are:

  • indexed: a version of the document is already in the collection
  • queued: a version of the document is in the queue and is pending processing
  • requeued: a version of the document is in the queue, processing failed and it's requeued to attempt again.
  • failed: a queued version of the document exhausted all processing attempts and processing permanently failed. It will not be tried again.

info field will have a text description of the last error for requeued or failed queue entries.

The operation type (op_type) is either upsert or delete.

Collection Queue Stats

GET /v1/collections/<collection-name>/queue/info endpoint returns queue statistics for a given collection:

{
  "queued_upsert": 0,
  "queued_delete": 0,
  "requeued_upsert": 0,
  "requeued_delete": 0,
  "failed_upsert": 0,
  "failed_delete": 0,
  "total": 0
}

The same information can also be obtained by calling the collection stats endpoint:

GET /v1/collections/<collection-name>/stats

{
  "doc_count": 0,
  "queue": {
    "queued_upsert": 0,
    "queued_delete": 0,
    "requeued_upsert": 0,
    "requeued_delete": 0,
    "failed_upsert": 0,
    "failed_delete": 0,
    "total": 0
  }
}

In addition to queue statistic, this endpoint also returns the number of the documents already indexed in the collection in doc_count field.

Deleting Collection Queue

You can delete all the entries from the queue per collection by calling this endpoint:

DELETE /v1/collections/<collection-name>/queue

Conclusion

Amgix provides a robust set of features and options for document ingestion. You can chose synchronous or asynchronous workflow, or a mix of both, based on your application needs. We also offer a set of unique features (automatic timestamp deduplication, atomic writes, intelligent retries) that allows you to integrate with our search engine without having to write complex logic, or design and maintain a ton of glue code and your own asynchronous ingestion pipelines, just to keep your search system in sync and your search content up-to-date. We tried to make the system as simple, robust and intuitive as possible. You send us your documents, we take care of the complexity. You can read our walkthrough on integrating your application with Amgix for more information.