Closing the Loop in Extended Reality with Kafka Streams and Machine Learning (Rob Drysdale and Sarah Healy, Accenture) Kafka Summit London 2019

Closing the Loop in Extended
Reality with Kafka Streams and
Machine Learning
Rob Drysdale | Senior Principal | robert.drysdale@accenture.com
Sarah Healy | Software Engineer | s.healy@accenture.com

The Dock is Accenture’s flagship
R&D and Global Innovation
Centre
300+ multi-disciplinary team
42% female
35+ nationalities
142 patents filed
Based in Silicon Docks, Dublin.
A hub for many global tech companies

• Extended reality in mainstream
• XR Insights : Usecase
• Building Generic Streaming Platform: Architectureand Schema
• Deep dive into our implementation
• Machine Learning 101 : Finding Patterns
• Learnings & Takeaways
• Demo
Agenda

XR Industry
• Exponential Growth
• Industry Adoption:
• Healthcare
• Aviation
• Education & Training
• Retail & Advertising
• Training: VR
• Safer Healthcare
• Cheaper
• Monitorable
• On-the-job: AR
• Future Worker
• Hands-free Manuals
• Remote Guidance
• Automated Guidance
• Prediction/Prevention

A Streaming platform with capability to
plug & play VR/AR devices and ML models.
XR Insights
Data Stream Capture:
• State Stream
• Event Stream
Data Analyses:
• Data-Driven User Assessment:
o Improving Accuracy of Assessment
o Removing Subjectivity of Assessment
• Insights on Performance and behaviour
• Personalise Training
• Improve Work/Training Process
• Predict/Guide User behaviour

TrainingData
WebSocket
Server
(STOMP)
/topic/feedback
XR Insights
/app/state
/app/event
Kafka
Producer
Kafka
Consumer
Kafka Stream Data Transformations
Kafka Connect
Master DB
BuildingML Model
Model & Features
S3 Bucket
Production ML Model
Personalization
Real Time Reporting
XR Devices

• Speed of Data Transmation
• Consumer Groups
• Easy Embedding of Kafka clients in existing servicedeployments
• Avro Schema support
• Clients for Java & Python
• Eventdriven & supportto query on real-time data
Why Kafka Streams?

Kafka
Producer
Kafka
Consumer
Schema Registry
XR Devices
Schema Registry: Single point of Success

• Kubernetes 1.13 & Docker (Statefulsets)
• Spring boot with WebSocket + STOMP
• Unity & C#
• Kafka 2.1.0, Kafka Streams DSL
• Schema Registry, Avro, ConnectCluster
• Couchbase, Grafana, Elastic search
Technologies

{
"type": "record",
"namespace": "com.accenture",
"name": "Event",
"version": "1",
"fields": [
{ "name": "eventId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type":"string", "doc": "session Id per record" },
{ "name": "timestamp", "type": "float", "doc": "Timestamp of Event" },
{ "name": "type", "type": "string", "doc" : "Type of Event" },
{ "name": "name", "type": "string", "doc": "Name of Event" },
{ "name": "desc", "type": {"type": "map", "values": "string"} , "doc":
"description of event"}
]
}
{
"topic":"event-topic",
"key":"d433b1db-4f89-4128-896f-95ac204b8681",
"value":{
"eventId":"d433b1db-4f89-4128-896f-95ac204b8681",
"sessionId":"13937ec1-82e1-4071-a6e7-31d31674de25",
"timestamp":148.45999,
"type":"Error",
"name":"Drop",
"desc":{
"collided":"Floor",
"Task id":"0",
"point of contact":"(1.1 1.0 -1.1)",
"lhand":"NA",
"collider":"spray_bottle",
"rhand":"NA"
}
},
"partition":2,
"offset":10
}
Event Schema: Example

{
"type": "record",
"name": "State",
"version": "1",
"fields": [
{ "name": "stateId", "type":["null", "string"], doc:”NA” },
{ "name": "sessionId", "type":["null", "string"], "doc": ”NA" },
{ "name": "timestamp", "type": "double", "doc": ”NA" },
{ "name": "task", "type": "long", "doc": ”NA" },
{ "name": "subtask", "type":["null", "string"], "doc": ”NA" },
{ "name": "lx", "type": "double", "doc": "NA" },
{ "name": "ly", "type": "double", "doc": "NA" },
{ "name": "lz", "type": "double", "doc": "NA" },
{ "name": "rx", "type": "double", "doc": "NA" },
{ "name": "ry", "type": "double", "doc": "NA" },
{ "name": "rz", "type": "double", "doc": "NA" },
{ "name": "hx", "type": "double", "doc": "NA" },
{ "name": "hy", "type": "double", "doc": "NA" },
{ "name": "hz", "type": "double", "doc": "NA" },
{ "name": "rItemHeld", "type":["null", "string"], "doc": "NA"}
{ "name": "lItemHeld", "type":["null", "string"], "doc": "NA"},
{ "name": "ltrigger", "type":["null", "boolean"], "doc": "NA"},
{ "name": "rtrigger", "type":["null", "boolean"], "doc": "NA"} ]
}
{
"topic":"state-topic",
"key":"13937ec1-82e1-4071-a6e7-31d31674de25",
"value":{
"stateId":{
"string":"cdfc5d94-3f1e-404a-ba27-632b3616dbfd"
},
"sessionId":{
"string":"13937ec1-82e1-4071-a6e7-31d31674de25"
},
"timestamp":126.98802185058594,
"task":0,
"subtask":{
"string":"0"
},
"lx":0.6099972724914551,
"ly":2.122999906539917,
"lz":-0.22100147604942322,
"rx":0.9099972248077393,
"ry":2.122999906539917,
"rz":-0.22099855542182922,
"hx":0.8208140134811401,
"hy":2.4463324546813965,
"hz":0.6848392486572266,
"rItemHeld":null
"lItemHeld":null,
"partition":2,
"offset":15
}
}
State Schema: Example

1. Maven Plugin to generate Java Classes for Avro Schema: org.apache.avro.avro-maven-plugin
2. Parse STOMP messages as Generic Avro Record.
Inspired by stackoverflow answer from @lucapette here
eventSchemaPath = IOUtils.toString(classLoader.getResourceAsStream("event.avsc"));
Schema eventSchemaParser =parser.parse(eventSchemaPath);
@MessageMapping("/event")
public void pushEvents(Eventevent){
String key=event.getSessionId();
GenericRecord eventAvroRecord=mapObjectToRecord(event, eventSchemaParser);
ProducerRecord<Object,Object>record =new ProducerRecord<>(config.getEventTopicName(), event.getSessionId(),
eventAvroRecord);
}
public GenericData.Record mapObjectToRecord(Objectobject,Schema schema) {
final GenericData.Record record =new GenericData.Record(schema);
schema.getFields().forEach(r->record.put(r.name(),PropertyAccessorFactory.forDirectFieldAccess(object).getPropertyValue(r.name())));
return record;
}
KAFKA Producer: Generic Avro Record

Using GlobalKTable to maintain user's active status
String[] state = {"GameStart","GameEnd"};
valueGenericAvroSerde.configure(serdeConfig, false); // `false` for record values
StreamsBuilder builder = new StreamsBuilder();
builder.stream("event-topic", Consumed.with(keySerde,valueGenericAvroSerde))
.map((key,value) -> KeyValue.pair(value.get("sessionId").toString(),value.get("name").toString()))
.filter((key,value) -> Arrays.stream(state).anyMatch(value::equalsIgnoreCase))
.to("user-status");
GlobalKTable<String, String> userStatusTable=builder.globalTable("user-status",Consumed.with(Serdes.String(),
Serdes.String()),Materialized.as(”active-user-status"));
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
streams.cleanUp();
streams.start();
Active User Status: GlobalKTable

ML 101: Finding Patterns
Mining Association Rules Based on
Apriori Algorithm
Problems
• One hot Encoding Numeric Data
• Updating the model needs to update the application.
• Computational Complexity Problem.
Prediction Pattern:
• Understanding Past User Behavior
• Personalization of current User

Events:
• No. of Frames
• No. of Tasks
• Grip Right
• Grip Left
• Trigger Right
• Trigger Left
• Both Right
• Both Left
• Only Uses One Hand
• No. of Drops
• No. of Incorrect Objects Picked Up
• No. of times looking at trainer
Derived Events:
• No. of times not looking at object of interest
• Hands moving towards/away from object of interest:
• This is not an once-off event, but runs over time through the task.
• Hands jittering
• Picking up the wrong item
• Touching the wrong item
• Dropping an object
• Hitting an object (including surfaces or hood)
• Facing away from Objects of interest
• Task Start/End

For creating association rules, algorithm needs a
boolean matrix as a parameter. Because of this user /
event count data should be converted a boolean
matrix.
Data Encoding Challenge
PossibleApproaches
• Thresholding
• Ranging
• Bucketing
• Smart Bucketing
Colour Red Yellow Green
Red
Red
Yellow
Yellow
Green
1 0 0
1 0 0
0 1 0
0 0 1

Predicting Responses:
eventStream.join(activeUserGlobalKTable,
(session, event) -> event.getSessionId(),
(event, userStatus) -> model.predict(event, userStatus))
.filter((key , response) -> response.getMessageText()!=null)
.to("response-topic");
{
"type": "record",
"name": "Response",
"version": "1",
"fields": [
{ "name": "responseId", "type": "string", "doc": "Unique Id per record" },
{ "name": "sessionId", "type": "string", "doc": "session Id per record" },
{ "name": "action", "type": "string", "doc": "NA" },
{ "name": "severity", "type": "int", "doc" : "NA" },
{ "name": "messageText", "type": "string", "doc": "NA" },
{ "name": "messageType", "type": "string", "doc": "predefined message types"},
{ "name": "messageObject", "type": "string" , "doc": "description of event"}
]
}
Response Schema

Low Latency Predictions: Kafka Producers in Python
oarp = OnlineAssociationRulePrediction()
try:
oarp.initialize_model(application_id, Config.S3_ADDRESS,Config.S3_KEY,Config.S3_SECRET)
except:
print("no model")
execute = False
while (execute):
event_msg = event_consumer.poll(600000)
state_msg = state_consumer.poll(600000)
eventRecord = decode_message(event_msg, event_schema)
stateRecord = decode_message(state_msg, state_schema)
append_windowed_df(state_msg)
analytics_response= oarp.predict(state_data=state_df, event_data=event_df)
# Return responseto Unity
key = {"name": eventRecord["sessionId"]}
for responseinanalytics_response:
value = {"responseId": str(uuid.uuid4()), "sessionId": eventRecord["sessionId"], "action": response["action"],"severity":
response["severity"], "messageText": response["messageText"], "messageObject":
response["messageObject"], "messageType": response["messageType"] }
avro_producer.produce(topic=Config.RESPONSE_TOPIC, value=value, key=key)

public void run(String... strings)
{
final Properties streamsConfiguration =newProperties();
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,Serdes.String().getClass());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
KStream<String, GenericRecord>responseStream =builder.stream(“response-topic”,
Consumed.with(keySerde,valueGenericAvroSerde));
responseStream.foreach((key,value)->publishToVR(key,value.toString()));
final KafkaStreams streams =newKafkaStreams(builder.build(), streamsConfiguration);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
public void publishToVR(String sessionId,String message)
{
brokerMessagingTemplate.convertAndSendToUser(sessionId,"/topics/feedback",JSON.toJSON(message));
}
Stream Response: Generic Avro Record

Challenges and Learnings
• Handling Multi-tenancy in streaming
• Pushing new models in streaming app.
• Handling offline devices could be challenging.
• Featuredetection

Key Takeaways
• Extended Reality needs streaming.
• Transfer learning between environments.
• Using Avro map with Schema Registry is powerful.
• Everything is an event except state.
• Node based deployments in Kubernetes for robustcluster.
• VR data is biometric, so ask for Consent.
• STOMP compatibility with Kafka
(Simple TextOrientated Messaging Protocol)

Thank You
accenture.com/thedock
Navdeep Sharma | Data & AI Engineer
navdeep.a.sharma@accenture.com
/showcase/accenture-the-dock @AccentureDock

Closing the Loop in Extended Reality with Kafka Streams and Machine Learning (Rob Drysdale and Sarah Healy, Accenture) Kafka Summit London 2019

Recommended

More Related Content

What's hot (20)

Similar to Closing the Loop in Extended Reality with Kafka Streams and Machine Learning (Rob Drysdale and Sarah Healy, Accenture) Kafka Summit London 2019 (20)

More from confluent (20)

Recently uploaded (20)

Closing the Loop in Extended Reality with Kafka Streams and Machine Learning (Rob Drysdale and Sarah Healy, Accenture) Kafka Summit London 2019