Too Many Buckets Exception

Problem

Elasticsearch throws too_many_buckets_exception during large aggregations

Root Cause

Aggregation exceeds the maximum allowed buckets setting, often due to high cardinality fields or insufficient shard filtering

How to Detect

curl -XGET 'localhost:9200/_cluster/health?pretty'
curl -XGET 'localhost:9200/_stats?pretty'
curl -XGET 'localhost:9200/_search?scroll=1m' -H 'Content-Type: application/json' -d '{"size":0,"aggs":{...}}'

Remediation Steps

1. Increase 'search.max_buckets' setting in elasticsearch.yml to a higher value, e.g., 200000
2. Refine aggregation queries to reduce the number of buckets, such as applying filters or using composite aggregations
3. Implement index patterns with lower cardinality fields or use doc values to optimize aggregations
4. Use composite aggregations for large datasets to paginate results efficiently
5. Reindex data with optimized mappings to reduce high-cardinality fields

Prevention

• Configure 'search.max_buckets' appropriately based on expected query complexity
• Design data models to minimize high-cardinality fields
• Limit aggregation scope with filters and query constraints
• Monitor aggregation performance and bucket counts regularly

Production Example

curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent": {"search.max_buckets": 200000}}'