Implicitly rollover data streams / aliases based on max_primary_shard_docs #94065
Conversation
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
|
Pinging @elastic/es-data-management (Team:Data Management) |
| if (targetIsTsdb && currentMaxPrimaryShardDocs > MAX_PRIMARY_SHARD_DOCS_FOR_TSDB) { | ||
| Map<String, Condition<?>> conditions = new HashMap<>(rolloverRequest.getConditions().getConditions()); | ||
| conditions.put(MaxPrimaryShardDocsCondition.NAME, new MaxPrimaryShardDocsCondition(MAX_PRIMARY_SHARD_DOCS_FOR_TSDB)); | ||
| rolloverRequest.setConditions(new RolloverConditions(conditions)); |
There was a problem hiding this comment.
Nit: we also have a builder that copies from an existing instance to simplify this code:
rolloverRequest.setConditions(
RolloverConditions.newBuilder(rolloverRequest.getConditions())
.addMaxPrimaryShardDocsCondition(MAX_PRIMARY_SHARD_DOCS_FOR_TSDB)
.build()
);
There was a problem hiding this comment.
I think that addMaxPrimaryShardDocsCondition(...) fails if max primary doc count has already been defined? This can happen if it has been defined, but with a value higher than MAX_PRIMARY_SHARD_DOCS_FOR_TSDB.
There was a problem hiding this comment.
Hm.... ah damn I didn't think of it, I will add a note to fix it, that was not my intention with this builder :p. Thanks @martijnvg
…t index has beyond 200M documents.
|
@gmarouli I updated the PR to always implicitly rollover data streams (and aliases) if a shard of most recent index has 200M or more documents. |
|
Hi @martijnvg, I've created a changelog YAML for you. |
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/WaitForRolloverReadyStep.java
Outdated
Show resolved
Hide resolved
I think it makes sense. Passing this number is almost guaranteed to cause problems right? |
Yes. I think this will cause issues a search time. Shard level searches getting slower without a real way of speeding that up. Certain bites cache entries getting uncomfortable large. |
This was introduced via elastic#94065 Relates to elastic#87246
This was introduced via elastic#94065 Relates to elastic#87246
This was introduced via elastic#94065 Relates to elastic#87246
In elastic#87246 we describe some reasons why it's a good idea to limit the doc count of a shard, and we started to do so in elastic#94065, so this commit adjusts the sizing guidance docs to match.
In elastic#87246 we describe some reasons why it's a good idea to limit the doc count of a shard, and we started to do so in elastic#94065, so this commit adjusts the sizing guidance docs to match.
In elastic#87246 we describe some reasons why it's a good idea to limit the doc count of a shard, and we started to do so in elastic#94065, so this commit adjusts the sizing guidance docs to match.
Implicitly rollover a data stream or alias if primary shard doc count of most recent backing index is on or beyond 200M (seems like a good catch all upper bound).
Closes #87246