[Behavioral Analytics] Add a final_pipeline to event data streams. #95198

afoucret · 2023-04-12T15:22:54Z

Description:

Add a final_pipeline to behavioral analytics events data streams
Process URLs in the event (page.url, page.referrer and search.results.items[*].page.url
Mapping update
- Update page.url to be an object
- Update page.referrer to be an object
- Update search.results.items[*].page.url to be an object

Other:

~~Blocked by [Ingest Processor] Add ignore_missing param to the uri_parts ingest processor. #95068~~

afoucret · 2023-04-12T15:25:32Z

...ources/org/elasticsearch/xpack/entsearch/analytics/behavioral_analytics-events-settings.json

@@ -9,7 +9,8 @@
        "refresh_interval": "1s",
        "number_of_shards": 1,
        "number_of_replicas": 0,
-        "auto_expand_replicas": "0-1"
+        "auto_expand_replicas": "0-1",
+        "final_pipeline": "behavioral_analytics-events-final_pipeline"


ℹ️ Updating the template to set the pipeline (as a final_pipeline)

afoucret · 2023-04-12T15:31:33Z

.../org/elasticsearch/xpack/entsearch/analytics/behavioral_analytics-events-final_pipeline.json

@@ -0,0 +1,51 @@
+{


ℹ️ The pipeline that will installed automatically when the cluster start by the AnalyticsIngestPipelineRegistry

afoucret · 2023-04-12T15:32:31Z

...-search/src/main/java/org/elasticsearch/xpack/application/utils/ingest/PipelineRegistry.java

@@ -0,0 +1,201 @@
+/*


ℹ️ This class may be moved to xpack core but I preferred to keep the code in our utils for now.

From my perspective, it makes sense to move it to x-pack/core from the beginning.

I would prefer to move it later because it will require other teams to validate which can add an additional delay to gets reviewed.

I am OK with moving it later considering we're very close to FF.

@kderusso Being very close to FF is indeed part of my rationale here.

...main/java/org/elasticsearch/xpack/application/analytics/AnalyticsIngestPipelineRegistry.java

.../org/elasticsearch/xpack/entsearch/analytics/behavioral_analytics-events-final_pipeline.json

tutelaris · 2023-04-17T11:17:19Z

...t-search/src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsConstants.java

+package org.elasticsearch.xpack.application.analytics;
+
+public class AnalyticsConstants {
+    // Data stream related constants.


tutelaris

Great job! 🚀

afoucret · 2023-04-17T11:45:18Z

...ources/org/elasticsearch/xpack/entsearch/analytics/behavioral_analytics-events-mappings.json

@@ -62,16 +62,84 @@
        "page": {
          "properties": {
            "url": {
-              "ignore_above": 1024,
-              "type": "keyword"
+              "properties": {


ℹ️ URLs are now object into the mapping (previously strings).

HiDAl · 2023-04-17T12:00:27Z

assigning to the right team

elasticsearchmachine · 2023-04-17T12:00:29Z

Hi @afoucret, I've created a changelog YAML for you.

kderusso

👍

kderusso · 2023-04-17T12:08:20Z

...-search/src/main/java/org/elasticsearch/xpack/application/utils/ingest/PipelineRegistry.java

@@ -0,0 +1,201 @@
+/*


I am OK with moving it later considering we're very close to FF.

carlosdelest

LGTM. Some lang level nitpicking.

I think we would need some review from the Platform teams on the approach for the PipelineRegistry - you probably did that already - to ensure we're not reinventing the wheel.

carlosdelest · 2023-04-17T15:12:08Z

...t-search/src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsConstants.java

+
+package org.elasticsearch.xpack.application.analytics;
+
+public class AnalyticsConstants {


can we add a private constructor to make the class non-instantiable?

carlosdelest · 2023-04-17T15:16:17Z

...t-search/src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsConstants.java

+
+package org.elasticsearch.xpack.application.analytics;
+
+public class AnalyticsConstants {


I'm not super fond of making a specific class for holding constants that are related to different concerns in the codebase.

Can't these constants be included into already existing classes?

I will probably refactor this in the near future to create an AnalytisEventDataStream class with more utils methods.
I propose to keep it for now and remove it later since these constants does not clearly belong to one of the existing class.

carlosdelest · 2023-04-17T17:59:39Z

...-search/src/main/java/org/elasticsearch/xpack/application/utils/ingest/PipelineRegistry.java

+                if (ingestPipelineExists(state, newPipeline.getId())) {
+                    IngestMetadata ingestMetadata = state.metadata().custom(IngestMetadata.TYPE);
+                    PipelineConfiguration existingPipeline = ingestMetadata.getPipelines().get(newPipeline.getId());
+                    boolean newPipelineHasVersion = Objects.isNull(newPipeline.getVersion()) == false;


why not a direct comparison with null? 🤔

It makes more sense right now because the code is

boolean newPipelineHasVersion = Objects.nonNull(newPipeline.getVersion())

It removes the need of == and I find it easier to read.

carlosdelest · 2023-04-17T18:02:23Z

...-search/src/main/java/org/elasticsearch/xpack/application/utils/ingest/PipelineRegistry.java

+     * @param pipelineId the pipeline that failed to be created.
+     * @param e The exception that caused the failure.
+     */
+    protected void onPutPipelineFailure(String pipelineId, Exception e) {


Is this method meant to be overriden by subclasses? We might use other visibility otherwise

I can see valid use case for overriding it.
The same is done in the template registry.

… events.

## Summary Fixes #156015 Fixes #156014 Fixes #155924 Fixes #155914 The Ingest Pipelines tests were broken because of a new pipeline introduced in elastic/elasticsearch#95198 but there was a fix added in elastic/elasticsearch#95621. The last PR fixes the error where it was not possible to delete the pipeline in the tests. This PR unskips the tests since they should now work as expected.

## Summary Fixes elastic#156015 Fixes elastic#156014 Fixes elastic#155924 Fixes elastic#155914 The Ingest Pipelines tests were broken because of a new pipeline introduced in elastic/elasticsearch#95198 but there was a fix added in elastic/elasticsearch#95621. The last PR fixes the error where it was not possible to delete the pipeline in the tests. This PR unskips the tests since they should now work as expected. (cherry picked from commit 32bc5c7) # Conflicts: # x-pack/test/accessibility/apps/ingest_node_pipelines.ts # x-pack/test/functional/apps/ingest_pipelines/ingest_pipelines.ts

…157257) # Backport This will backport the following commits from `main` to `8.8`: - [[Ingest Pipelines] Unskip functional and a11y tests (#157173)](#157173)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

elasticsearchmachine added external-contributor Pull request authored by a developer outside the Elasticsearch team v8.8.0 labels Apr 12, 2023

afoucret commented Apr 12, 2023

View reviewed changes

afoucret changed the title ~~Pipeline registry~~ [Behavioral Analytics] Add a final_pipeline to event data streams. Apr 12, 2023

afoucret commented Apr 12, 2023

View reviewed changes

...main/java/org/elasticsearch/xpack/application/analytics/AnalyticsIngestPipelineRegistry.java Outdated Show resolved Hide resolved

afoucret commented Apr 12, 2023

View reviewed changes

.../org/elasticsearch/xpack/entsearch/analytics/behavioral_analytics-events-final_pipeline.json Outdated Show resolved Hide resolved

afoucret force-pushed the pipeline-registry branch 2 times, most recently from 0945429 to fd92bce Compare April 13, 2023 13:15

tutelaris reviewed Apr 17, 2023

View reviewed changes

tutelaris approved these changes Apr 17, 2023

View reviewed changes

afoucret commented Apr 17, 2023

View reviewed changes

afoucret marked this pull request as ready for review April 17, 2023 11:46

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Apr 17, 2023

HiDAl added >feature cloud-deploy Publish cloud docker image for Cloud-First-Testing :EnterpriseSearch/Application Enterprise Search Team:Enterprise Search Meta label for Enterprise Search team and removed needs:triage Requires assignment of a team area label labels Apr 17, 2023

afoucret removed external-contributor Pull request authored by a developer outside the Elasticsearch team cloud-deploy Publish cloud docker image for Cloud-First-Testing labels Apr 17, 2023

kderusso reviewed Apr 17, 2023

View reviewed changes

carlosdelest approved these changes Apr 17, 2023

View reviewed changes

afoucret added 4 commits April 18, 2023 08:32

Adding a PipelineRegistry component to utils.

d0901d2

Use pipeline registry to create Analytics ingest events pipeline.

49a7dbb

Adding tests for the AnalyticsIngestPipelineRegistry

0e5d14d

Implementing URL fields into ingest pipeline for behavioral analytics…

cac34a7

… events.

afoucret and others added 4 commits April 18, 2023 08:32

Refactoring: grouping constants in a single place.

7badaca

Update mapping: URL fields are now objects.

35a47d2

Finalizing the events ingest pipeline.

ecc8d4a

Update docs/changelog/95198.yaml

a08a17f

afoucret force-pushed the pipeline-registry branch from a3ba148 to 6233ed5 Compare April 18, 2023 06:34

Fix feedback review.

4d39402

afoucret force-pushed the pipeline-registry branch from 6233ed5 to 4d39402 Compare April 18, 2023 06:36

Fix regression.

e8374bc

afoucret merged commit e03378f into elastic:main Apr 18, 2023

This was referenced Apr 30, 2023

[Logs+] Assign a default @timestamp if missing #95551

Closed

Adding ability to auto-install ingest pipelines #95782

Merged

yuliacech mentioned this pull request May 9, 2023

[Ingest Pipelines] Unskip functional and a11y tests elastic/kibana#157173

Merged


		package org.elasticsearch.xpack.application.analytics;

		public class AnalyticsConstants {

[Behavioral Analytics] Add a final_pipeline to event data streams. #95198

[Behavioral Analytics] Add a final_pipeline to event data streams. #95198

Uh oh!

Conversation

afoucret commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Other:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tutelaris left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HiDAl commented Apr 17, 2023

Uh oh!

elasticsearchmachine commented Apr 17, 2023

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

afoucret commented Apr 12, 2023 •

edited

Loading