Add aws.metrics_names_fingerprint field and mark it as a dimension

[kaiyan-sheng]

Proposed commit message

This PR adds aws.metrics_names_fingerprint which is a hash of the list of metric names exist in each document. For metrics coming in from Firehose, there can be cases two documents have the same timestamp, dimension, namespace, accountID, exportARN and region BUT from two different requests. With adding the aws.metrics_names_fingerprint as TSDB dimension, we are able to ingest these two documents into ES.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

backport #11281 to 2.25

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1d15650

cc @kaiyan-sheng

[elastic-sonarqube]

Quality Gate failed

Failed conditions
5.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[agithomas]

LGTM!

[constanca-m]

@kaiyan-sheng I would include in the description this part you added in the issue:

For metrics coming in from Firehose, there can be cases two documents have the same timestamp, dimension, namespace, accountID, exportARN and region BUT from two different requests.

Does it cause document loss because of TSDB? I would include it.

I think it would help to navigate better when looking at the changelog in case this appears in a SDH in the future.

[kaiyan-sheng]

@constanca-m Yes it does cause document loss because of TSDB. Let me add it in the aws firehose integration since that's where the actual fingerprint processor is added: https://github.com/elastic/integrations/blob/main/packages/awsfirehose/data_stream/metrics/elasticsearch/ingest_pipeline/default.yml#L29
changelog: https://github.com/elastic/integrations/blob/main/packages/awsfirehose/changelog.yml#L7

[elastic-vault-github-plugin-prod]

Package aws - 2.25.1 containing this change is available at https://epr.elastic.co/search?package=aws