Cluster Alerts
The account owner receives these alerts by email. Each row gives the condition that triggers the alert, what it means for your cluster, and how to resolve it. For the metrics behind them, see Monitor Clusters. To read firing and resolved alerts in your own automation, run qcloud cluster alert list <CLUSTER_ID> in the Cloud CLI or call the ListClusterAlerts method of the Cloud API.
Capacity
| Alert | Triggers when | What it means | How to resolve |
|---|
| Memory Overutilized | Memory use is above 80% of the allocation for more than 5 minutes | If usage keeps growing past the allocation, the cluster fails from resource pressure and you see disruption | Scale vertically for more capacity per node, scale horizontally to spread the load, or reduce memory with quantization or the cold memory tier |
| Disk Space Overutilized | Disk use is above 80% of the allocation for more than 5 minutes | Same as memory: the cluster fails from resource pressure if usage keeps growing | Scale vertically for more disk per node, or scale horizontally to spread the data. See Capacity Planning |
| A Node Ran Out of Memory | A node tried to allocate more RAM than available and the operating system stopped the Qdrant process | A highly available cluster may avoid total downtime, but it is unstable. Without high availability, expect disruption until the cluster is scaled up | Scale vertically or horizontally, reduce memory with quantization, and make the cluster highly available |
| A Node Ran Out of Disk Space | Disk use on a node is above 99% | The node cannot store new vectors | Scale vertically for more disk, or scale horizontally and reshard if there are too few shards to use the new nodes. See Capacity Planning |
| One or More Nodes Are in Recovery Mode | A node failed to start, usually because it ran out of memory, and restarted in recovery mode | The node loads only collection metadata, so the only collection operation available is deleting a collection. The node does not serve search or writes | Scale the cluster vertically for more memory, or reduce memory with quantization. Delete a collection only on a proof-of-concept or toy cluster where the data does not matter |
| A Node is CPU Throttled | CPU throttling is above 25% for 30 minutes | CPU is saturated, so searches and writes respond much slower. The usual cause is a heavy write load while indexes are created or updated, or heavy search load with inefficient queries | Write in batches when search traffic is low, run fewer parallel writes, index your data, simplify heavy hybrid and multi-stage queries, tune the optimizers, or scale the cluster |
Deleting data is a last resort for these alerts. Use it only on a proof-of-concept or toy cluster where the data does not matter. On any other cluster, scale or reconfigure instead.
Availability and Health
| Alert | Triggers when | What it means | How to resolve |
|---|
| Cluster is Unhealthy | One or more nodes have been unhealthy for more than 5 minutes | There is a serious issue that needs attention | Check the other alerts first. Then look at recent client code or configuration changes, and at increases in search or write requests |
| Cluster Has Too Many Collections | The cluster has more than 500 collections | Many collections add significant resource overhead and, left alone, degrade resilience and can cause outages | Use one collection with payload-based partitioning in place of one collection per tenant. You can also split collections across clusters with the Migration CLI |
Versions and Keys
| Alert | Triggers when | What it means | How to resolve |
|---|
| Cluster Version is Not Covered by SLA | The cluster runs a version older than the latest and previous 3 minor versions | Support requests against the cluster are not covered by the SLA | Update the cluster |
| Database API Key is About to Expire | A Database API key expires within 7 days | Requests with an expired key fail, which can break queries and applications | Create a new Database API key and update your application |
Uneven Load
| Alert | Triggers when | What it means | How to resolve |
|---|
| Node CPU Usage is Not Distributed Equally | CPU usage differs across nodes | Some nodes carry more load than the rest, which slows search and can cause node failures. Uneven data distribution is the usual cause | Reshard and rebalance shards. See Shard Rebalancing |
| Node RAM or Disk Space Usage is Not Distributed Equally | RAM or disk usage differs across nodes | Same as CPU: hotspots slow search and can cause node failures | Reshard to split a collection across more nodes, and let rebalancing even out the shards |
Qdrant Cloud rebalances shards in the background by default. It balances the total shard count and size per node across all collections combined, so a single collection can look uneven while the cluster as a whole is balanced.
Ready to try Qdrant?