[DLM] Introduce default rollover cluster setting & expose it via APIs

[gmarouli]

In this PR we are introducing the following:

• A cluster setting that configures the default rollover used by DLM
• Introduces a new query param, namely include_defaults for GET /data_streams, GET /_index_template and GET /_component_template, this will expose default configuration related to these endpoints.
• Exposes the default rollover configuration currently in place when the include_defaults
  parameter is true in the above endpoints
• Uses the rollover configuration from the cluster setting in DLM.

Part of: #93596

Docs preview:

• get data streams
• get index templates
• get component templates

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[andreidan]

Thanks for working on this Mary

This looks great, I left some minor suggestions

[gmarouli]

Thanks for the prompt review @andreidan . Along with implementing the feedback from your review I added a test for setting parser in DataLifecycle which I removed accidentally during the development of this feature.

[elasticsearchmachine]

Hi @gmarouli, I've created a changelog YAML for you.

[gmarouli]

Thanks for the prompt review @andreidan . Along with implementing the feedback from your review I added a test for setting parser in DataLifecycle which I removed accidentally during the development of this feature.

Ignore the comment above ^^.... The test was at a different class..... face_palm fixing it

[andreidan]

Thanks for iterating on this Mary

Left a few minor suggestions, but this is almost ready to go 🚀

[andreidan]

I'm in two minds about this validator.
The setting is meant to allow us to make changes, it'll not be something the users will configure.

I'm tempted to say we shouldn't add this validation, but only validate that the setting is not empty (which should be covered by the RolloverConditions.parseSetting ? )
What do you think?

Note: If we do keep it I think it shouldn't be restricted to min_docs. The are other min_* conditions that could be valid to avoid rolling over empty indices

[gmarouli]

The setting is meant to allow us to make changes, it'll not be something the users will configure.
That is true, but I think even we can make a mistake, so avoiding creating empty indices is a safe bet. I see this validation helping the us not blocking us.

If we do keep it I think it shouldn't be restricted to min_docs. The are other min_* conditions that could be valid to avoid rolling over empty indices
That is true, I did consider it but I had some concerns about the following:

• min_age doesn't protect us from empty indices, you can have an index that has been there for 7 days and it can still be empty.
• min_size & min_primary_shard_size concerns me because an index doesn't have an empty footprint so if you set it up wrongly you might still have an empty index (if I am not mistaken).
• min_primary_shard_docs I think is a good addition :).

I would suggest that we keep the validation but we add the min_primary_shard_docs as an option too. What do you think? I am thinking that there are no use cases that would not want at least one of the two.

[gmarouli]

Discussed offline. We will remove the checks for min_* conditions.

[andreidan]

I think the whole validator could be removed and leave the RolloverConditions#parser to make sure there's at least one condition, regardless what that is.

[gmarouli]

Opened #94447

[gmarouli]

@elasticmachine run elasticsearch-ci/part-1, I believe the failure is related (#94281) I have contacted the @HiDAl to verify the assumption and fix it.