[Enterprise Search][Behavioral Analytics] Events ingest API #95027

afoucret · 2023-04-04T21:16:01Z

Description:

In order to populate our analytics data streams we need an API to collect events.
This API will validate the event and emit them to bulk processor that will persist event into events data streams

A new REST action is added to the code to receive event.

curl -H "Content-Type: application/json" -XPOST -u "elastic-admin:elastic-password" "localhost:9200/_application/analytics/bar/event/pageview" -d '

{
  "session": {
    "id": "dghdjdk"
  },
  "user": {
    "id": "bdjhdkdjdl"
  },
  "page": {
    "url": "http://elastic.co/blog"
  }
}'

Checklist

afoucret · 2023-04-11T17:30:04Z

.../src/main/java/org/elasticsearch/xpack/application/analytics/ingest/BulkProcessorConfig.java

+    public static final Setting<TimeValue> FLUSH_DELAY_SETTING = Setting.timeSetting(
+        Strings.format("%s.%s", SETTING_ROOT_PATH, "flush_delay"),
+        TimeValue.timeValueSeconds(10),
+        TimeValue.timeValueSeconds(1),
+        TimeValue.timeValueSeconds(60),
+        Setting.Property.NodeScope
+    );
+
+    public static final Setting<Integer> MAX_NUMBER_OF_EVENTS_PER_BULK_SETTING = Setting.intSetting(
+        Strings.format("%s.%s", SETTING_ROOT_PATH, "max_events_per_bulk"),
+        1000,
+        1,
+        10000,
+        Setting.Property.NodeScope
+    );
+
+    public static final Setting<Integer> MAX_NUMBER_OF_RETRIES_SETTING = Setting.intSetting(
+        Strings.format("%s.%s", SETTING_ROOT_PATH, "max_number_of_retries"),
+        3,
+        0,
+        5,
+        Setting.Property.NodeScope
+    );


ℹ️ This settings will allow to make the bulk processor configurable

afoucret · 2023-04-11T17:31:25Z

...rc/main/java/org/elasticsearch/xpack/application/analytics/ingest/AnalyticsEventEmitter.java

+    @Override
+    protected void doClose() {
+        // Ensure the bulk processor is closed, so pending requests are flushed.
+        bulkProcessor.close();


ℹ️ Ensure the bulk processor events are drained when closing.

I think it should attempt to execute the remaining 'under construction' bulk request. But agree it would be great to double check this

afoucret · 2023-04-11T17:32:27Z

...ck/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/EnterpriseSearch.java

+    public List<Setting<?>> getSettings() {
+        return List.of(
+            BulkProcessorConfig.MAX_NUMBER_OF_EVENTS_PER_BULK_SETTING,
+            BulkProcessorConfig.FLUSH_DELAY_SETTING,
+            BulkProcessorConfig.MAX_NUMBER_OF_RETRIES_SETTING
+        );


ℹ️ Event ingest related settings.

afoucret · 2023-04-11T17:33:39Z

...src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsCollectionResolver.java

ℹ️ Changes to made to this class are syntaxic sugar.

afoucret · 2023-04-11T17:38:03Z

...main/java/org/elasticsearch/xpack/application/analytics/action/PostAnalyticsEventAction.java

+        ) {
+            this.eventCollectionName = eventCollectionName;
+            this.eventType = eventType;
+            this.debug = debug;


@kderusso I kept debug instead of suggestions like verbose.

Indeed it is intended to be used with the debug flag of the JS tag that will output event processed by the server into the dev console of the browser (feature that I plan to implement later).

kderusso

Nice work!

kderusso · 2023-04-11T20:36:28Z

...main/java/org/elasticsearch/xpack/application/analytics/action/PostAnalyticsEventAction.java

+            }
+
+            try {
+                if (eventType.toLowerCase(Locale.ROOT).equals(eventType) == false) throw new IllegalArgumentException();


Nitpick: Curly brackets

Fixed! Thank you.

carlosdelest

Looks really good to me. I just have a question on privileges for the POST action - could they be index privileges instead of manage cluster type privileges?

Very good job on tests! 💯

docs/reference/behavioral-analytics/apis/post-analytics-collection-event.asciidoc

...ain/java/org/elasticsearch/xpack/core/security/authz/privilege/ClusterPrivilegeResolver.java

...c/yamlRestTest/resources/rest-api-spec/test/entsearch/90_behavioral_analytics_event_post.yml

...src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsEventIngestService.java

carlosdelest · 2023-04-12T07:07:43Z

...h/src/main/java/org/elasticsearch/xpack/application/analytics/AnalyticsTemplateRegistry.java


    // This number must be incremented when we make changes to built-in templates.
-    public static final int REGISTRY_VERSION = 1;
+    protected static final int REGISTRY_VERSION = 1;


Why protected? Are we expecting subclasses for AnalyticsTemplateRegistry?

I have another PR with a class in the same package that use this variable.
I will probably push those into a file that hold all these constant.

I have another PR with a class in the same package that use this variable.

Maybe use default package visibility then? Sorry for nagging, I see protected as a strong signal that it is used in a subclass. Let's use default package visibility if that's the use case 👍

...main/java/org/elasticsearch/xpack/application/analytics/action/PostAnalyticsEventAction.java

...org/elasticsearch/xpack/application/analytics/event/parser/event/AnalyticsEventPageView.java

...src/main/java/org/elasticsearch/xpack/application/analytics/ingest/BulkProcessorFactory.java

pgomulka

LGTM,
I left questions, but feel free to merge anyway once CI is green

pgomulka · 2023-04-12T09:02:40Z

...src/main/java/org/elasticsearch/xpack/application/analytics/ingest/BulkProcessorFactory.java

+    }
+
+    public BulkProcessor2 create(Client client) {
+        return BulkProcessor2.builder(client::bulk, new BulkProcessorListener(), client.threadPool())


this looks good, but do we want to have control over indexing being enabled or not?
let's say a user is sending a lot of requests that results in analyticsEventEmitter to keep on creating index requests What if cluster is struggling at the moment? Do we want to have control to turn that off( that would mean that data is lost ..)

I will probably implement it into the AnalyticsEventEmitter.
If you don't mind I will do it in a follow-up PR to be able to merge the PR and unblock other work streams that rely on the feature.

yes, follow up is good. But I will this up to ent search team to decide if this is even needed at all. I thought that it might be good as an emergency switch off button

We have that for deprecation logging, but we started with that feature turned off so we needed this to enable this.

This is a good call and I will add these feature. It will probably be part of the upcoming PR to add metering to the event ingest API.

...est/java/org/elasticsearch/xpack/application/analytics/ingest/BulkProcessorFactoryTests.java

pgomulka · 2023-04-12T10:11:26Z

...rc/main/java/org/elasticsearch/xpack/application/analytics/ingest/AnalyticsEventEmitter.java

+    @Override
+    protected void doClose() {
+        // Ensure the bulk processor is closed, so pending requests are flushed.
+        bulkProcessor.close();


I think it should attempt to execute the remaining 'under construction' bulk request. But agree it would be great to double check this

afoucret · 2023-04-12T12:25:50Z

@elasticsearchmachine run elasticsearch-ci/part-2

…entField

The cluster privileges are listed in alphabetic order on get-privileges response. This PR ensures that is matched in doc. Relates: elastic#95027 Resolves: elastic#95210

The cluster privileges are listed in alphabetic order on get-privileges response. This PR ensures that is matched in doc. Relates: #95027 Resolves: #95210

afoucret and others added 30 commits April 4, 2023 22:58

A placeholder REST controller to push analytics events.

3fbfa85

Introduce EventEmitterService

330a6da

Introduce SearchAnalyticsEvent

d4bc7cc

Introduce InteractionAnalyticsEvent

397c5a8

Introduce PageViewAnalyticsEvent

35433d8

WIP

cebcc5b

Event emitter refactoring.

c80c6fe

Adding interaction events

3336873

Updating data stream mapping.

b210a61

Refactoring to ease testing.

f739385

Fix AnalyticsEvent serializing.

c5104a1

Revert useless change in the module dependencies.

ff9e305

Code refactoring.

7fc3c92

Adding tests.

c181e40

Fixing forbidden API call.

4994f8a

Added debug flag on the API.

808f4fc

Fix failing test.

c236abc

Refactored with one class per event type.

333bc28

Refactored with one class per event type.

4bb2688

Revert debug code.

b110086

Replacing interaction event type with search_click

e88aee1

A bunch of unit tests.

e520033

Wrap resource init into a try

768e2ed

update mapping

75a6fbf

remove top level interaction

8c6c9b7

add document top level

9d6a8ba

add document top level

cff765a

remove event.dataset

d49ab3b

Add licence gate to the post event action.

3cce2bc

Refactoring

68bfd65

afoucret added 4 commits April 11, 2023 14:49

Ensure the bulk processor is closed with the event emitter.

83ac15f

Add the ability to configure the bulk processor.

b4e7751

Add more tests.

98c6807

Add more tests.

2a3d993

afoucret requested a review from jimczi April 11, 2023 17:11

Lint fixes.

ffa676e

afoucret commented Apr 11, 2023

View reviewed changes

afoucret requested a review from pgomulka April 11, 2023 18:44

kderusso approved these changes Apr 11, 2023

View reviewed changes

Introduce behavioral analytics post even yml tests (#4)

d95c073

carlosdelest reviewed Apr 12, 2023

View reviewed changes

afoucret added 4 commits April 12, 2023 11:26

Fixing and improving YAML REST tests.

9ab6296

Apply suggestions from reviews.

f13e963

Fix HTTP status code of the post event.

bbb47e7

Fix REST API spec.

cfdbd5f

pgomulka approved these changes Apr 12, 2023

View reviewed changes

afoucret added 4 commits April 12, 2023 12:42

Fix max bulk size test.

bdfa90c

Fix failing YAML REST test.

3de5ba3

Use assertBusy in testFlushDelay

2baddca

linting

73c047a

Renaming AnalyticsEventSearchResultField into SearchResultAnalyticsEv…

36396a2

…entField

afoucret merged commit 675163b into elastic:main Apr 12, 2023

ywangd mentioned this pull request Apr 13, 2023

[Test] Fix cluster privileges ordering in doc #95213

Merged

ywangd added a commit that referenced this pull request Apr 25, 2023

[Test] Fix cluster privileges ordering in doc (#95213)

3afed5f

The cluster privileges are listed in alphabetic order on get-privileges response. This PR ensures that is matched in doc. Relates: #95027 Resolves: #95210

[Enterprise Search][Behavioral Analytics] Events ingest API #95027

[Enterprise Search][Behavioral Analytics] Events ingest API #95027

Uh oh!

Conversation

afoucret commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afoucret Apr 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pgomulka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afoucret commented Apr 12, 2023

Uh oh!

Uh oh!

afoucret commented Apr 4, 2023 •

edited

Loading

afoucret Apr 11, 2023 •

edited

Loading