MicroStrategy & Real Time Analytics

Tony Maresco
Stephen Lippens
HF Chadeisson

1 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Safe Harbor Notice

This presentation describes features that are under development by MicroStrategy. The objective of this presentation is to provide insight into
MicroStrategy’s technology direction. The functionalities described herein may or may not be released as shown.

This presentation contains statements that may constitute “forward-looking statements” for purposes of the safe harbor provisions under the
Private Securities Litigation Reform Act of 1995, including estimates of future technology releases. Forward-looking statements inherently
involve risks and uncertainties that could cause actual results of MicroStrategy Incorporated and its subsidiaries (collectively, the “Company”)
to differ materially from the forward-looking statements.

Factors that could contribute to such differences include: the Company’s ability to develop, market and deliver on a timely and cost-effective
basis new or enhanced offerings that respond to technological change or new customer requirements; delays in the Company’s ability to
develop or ship new products; the extent and timing of market acceptance of MicroStrategy’s new offerings; continued acceptance of the
Company’s other products in the marketplace; competitive factors; general economic conditions; and other risks detailed in the Company’s
registration statements and periodic reports filed with the Securities and Exchange Commission. By making these forward-looking statements,
the Company undertakes no obligation to update these statements for revisions or changes after the date of this presentation.

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Real Time Analytics - Team Involved
Cross departmental initiative to support market demands with a new model

MicroStrategy Technology Solutions Architects

Sanctions the architecture. Adjust the roadmap based on Implement and publish the framework. Internal trainings and
Solutions Architects findings knowledge sharing. Customer facing related activities

Sandipto Banerjee Tony Maresco

SVP & CPO Principal Solutions Architect
Enterprise Assets Big Data

Ben Reyes Stephen Lippens

VP Product Management Principal Solutions Architect
Gateways SDK

Scott Rigney HF Chadeisson

Principal Product Manager Principal Solutions Architect
APIs & Machine Learning

Special thanks to Alex, Kyung-Whu, Robert, Pradyut, Matt, Michael and Zander for contributing
to our brainstorming sessions

3 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 What is Real Time Analytics ?
First Google result – Techtarget says

Real-time analytics is the use of, or the capacity to use, data and related resources as soon as the data
enters the system. The adjective real-time refers to a level of computer responsiveness that a user senses as
immediate or nearly immediate. Real-time analytics is also known as dynamic analysis, real-time analysis,
real-time data integration and real-time intelligence.
Technologies that support real-time analytics include:
Processing in memory (PIM) -- a chip architecture in which the processor is integrated into a memory chip to
reduce latency.
In-database analytics -- a technology that allows data processing to be conducted within the database by
building analytic logic into the database itself.
Data warehouse appliances -- combination hardware and software products designed specifically for analytical
processing. An appliance allows the purchaser to deploy a high-performance data warehouse right out of the
box.
In-memory analytics -- an approach to querying data when it resides in random access memory (RAM), as
opposed to querying data that is stored on physical disks.
Massively parallel programming (MPP) -- the coordinated processing of a program by multiple processors that
work on different parts of the program, with each processor using its own operating system and memory.
5 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
What is Real Time Analytics ?
Second Google result - Techopedia says

Real-time analytics is a term

used to refer to analytics that are
able to be accessed as they
come into a system. In general,
the term analytics is used to
define data patterns that provide
meaning to a business or other
entity, where analysts collect
valuable information by sorting
through and analyzing that data.

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

What is Real Time Analytics ?
What does Gartner say ?

Real-time analytics is the discipline that applies logic and mathematics to data to
provide insights for making better decisions quickly. For some use cases, real time
simply means the analytics is completed within a few seconds or minutes after the
arrival of new data. On-demand real-time analytics waits for users or systems to
request a query and then delivers the analytic results. Continuous real-time
analytics is more proactive and alerts users or triggers responses as events
happen.

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

What is Real Time Analytics ?
What does Gartner say ?

Real-time analytics is the discipline that applies logic and mathematics to data to
provide insights for making better decisions quickly. For some use cases, real time
simply means the analytics is completed within a few seconds or minutes after
the arrival of new data. On-demand real-time analytics waits for users or
systems to request a query and then delivers the analytic results. Continuous
real-time analytics is more proactive and alerts users or triggers responses as
events happen.

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Observations – is this real time ?
1. I get speed of thought answers on my database in real time
On Demand

9 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Observations – is this real time ?
2. I’m getting new data every time I update my Dossier
Near Real Time

10 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Observations – is this real time ?
3. I get visualizations self updating as the data flows
Real Time

11 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

12 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Observations - Alerting / Notifications
4. Probably more important than Real Time visualizations
Real Time

13 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Conclusion Real Time Analytics capabilities
Getting the Right Data at the Right Time

Speed of thoughts Near Real Time Dossier Real Time Visualizations Real Time Notifications
Already available Stream support - Q3 2018 Q4 2018 Future
I get speed of thought answers Get near real time information in Shared spaces Instant remediation - problems
on my database in real time. MicroStrategy cube being solved in real time
TV Walls
The system answers to ad-hoc Dossier available on demand Trigger process in other system
requests
Call centers • Webhook
Use of client based Autorefresh • Human notification : Slack /
Mail / SMS
Supervisor multi-screen
Stream Processing • Run a script

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Near Real Time Dossier
Phase 1a – Kafka Source – Q3 2018
Definitions: Streams and Tables Getting Data in and out of a Streaming source

Getting Data in Kafka : Producer / Source

• A Kafka Producer is a third party system that
publishes new data to Kafka
• A Kafka Source is configured within Kafka. It fetch
new data on a regular basis

Getting Data out of Kafka : Consumer / Sink

• A Kafka Consumer is a third party system that
subscribes to Kafka
• A Kafka Sink is configured in Kafka. It sends newly
received data as it flows

External External
Producer Consumer

Kafka Kafka
Source: https://docs.confluent.io/current/streams/concepts.html Source Sink

15 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Near Real Time Dossier
Accessing the data until Q3

Auto
Refresh

MicroStrategy
Platform

ODBC
Stream Sources Stream Brokers Stream Processors Database

Data Lake
16 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Framework for Near Real Time Dossier
Accessing the data that resides in Streaming sources

Got 3M Rows streamed

every minute ? Auto
Refresh
MicroStrategy
Platform
MicroStrategy APIs

Sink Sink ODBC

Stream Sources Stream Brokers Stream Processors Database

Data Lake
17 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Framework for Near Real Time Dossier
Accessing the data that resides in Streaming sources

KSQL – Runs Real Time

stream aggregates Auto
Refresh
MicroStrategy
Platform
MicroStrategy APIs

Sink Sink ODBC

Stream Sources Stream Brokers Stream Processors Database

Data Lake
18 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Using KSQL for Stream Processing

https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka

19 Copyright © 2018 MicroStrategy Incorporated. All Rights Reserved.

 Example KSQL for Analytics with Stream and Table
CREATE STREAM twitter_raw (CreatedAt BIGINT, Id BIGINT, Text VARCHAR) WITH
(KAFKA_TOPIC='twitter_json_01', VALUE_FORMAT='JSON’);

CREATE STREAM twitter AS \

SELECT TIMESTAMPTOSTRING(CreatedAt, 'yyyy-MM-dd HH:mm:ss.SSS') AS CreatedAt,\
EXTRACTJSONFIELD(user,'$.Name') AS user_Name,\
EXTRACTJSONFIELD(user,'$.ScreenName') AS user_ScreenName,\
EXTRACTJSONFIELD(user,'$.Location') AS user_Location,\
EXTRACTJSONFIELD(user,'$.Description') AS user_Description,\
Text,hashtagentities,lang \FROM twitter_raw ;

CREATE TABLE user_tweet_count AS \

SELECT user_screenname, count(*) AS tweet_count
\
FROM twitter WINDOW TUMBLING (SIZE 1 HOUR) \
GROUP BY user_screenname ;

https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka

20 Copyright © 2018 MicroStrategy Incorporated. All Rights Reserved.

 Solution Combining Push API – KSQL - and Push to Viz
Connect with objects – scalable with near real-time performance

MicroStrategy Analytics***

Customer Stream
processing *
Historical
Cubes for RT
Long-term Storage of Blending,
Structured Detail ** Analysis, and
Alerts

Node.js/
Socket.io Web Sockets
Push API

Aggregated/
Sources Analytics

Alerts/Transactions
• High user scalability
• Moderately high rate and volume scalability * Note that all the “Customer Stream processing” components are customer-provided,
built with streaming components such as Spark, Storm and Kafka.
• Uses objects ** A typical architecture stores data from the stream in HDFS, ElasticSearch, HBase,
etc. for Historical Reporting and Machine Learning
• Supports queries for aggregations and metrics *** The full MicroStrategy Analytics solutions requires some custom development with

• KSQL adds Stream Analytics the SDK

21 Copyright © 2018 MicroStrategy Incorporated. All Rights Reserved.

 Framework for Near Real Time Dossier se-realtime
Phase 1 - Summary

What's new – Phase 1a What's next – Phase 1b

Concept of Sink – connector is built on • Sinks for other sources : Spark
Streaming, Storm, Flume, Flink
Streaming source side
• In the meanwhile, try to get these
Concept of Real Time aggregates – data to Kafka
KSQL
• Productionize the Kafka Sink
Kafka Sink available on Github :
https://github.com/hchadeisson/mstr-
kafka-sink

22 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Real Time Analytics capabilities
Getting the Right Data at the Right Time

Speed of thoughts Near Real Time Dossier Real Time Visualizations Real Time Notifications

I get speed of thought answers Get near real time information in Shared spaces Instant remediation - problems
on my database in real time. MicroStrategy cube being solved in real time
TV Walls
The system answers to ad-hoc Dossier available on demand Trigger process in other system
requests
Call centers • Webhook
Use of client based Autorefresh • Human notification : Slack /
Mail / SMS
Supervisor multi-screen
• Run a script

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Real Time Visualizations
Phase 2 – Real Time Engine - Q4 2018

Problems we are trying to solve

• How to update viz in real time with information that goes through the Sink ?

• How to distribute the right information to the right client / user ?

• HTTP connections limitation ?

• Need for Node.js and Socket.io

24 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Real Time Visualizations
Building a Real Time Framework

Real Time
Custom
Visualizations Historical Auto
Visualizations Refresh
Real Time APIUpdate MicroStrategy
Engine Listener Platform
MicroStrategy APIs
(internal Kafka) ODBC
Sink Sink
Stream Sources Stream Brokers Stream Processors Database

Data Lake
26 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
27 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Real-time Engine Responsibilities

Kafka Notification Receiver Real-time engine Data Broker

MSTR Cube
Cube Socket Connected visualizations
Definitions/Data rooms
Broadcast data

Push data Success

to cube response
Execute
Cube
Create or New visualization request
join room
Data Stream Success
notification

Drop zones
Attributes Metrics
Request room to
kafka satisfy data request

28 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

29 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Real-time Engine Benefits
• Visualization updates the moment new data is available (Push over Poll)

W/O real-time engine

MSTR - Only as accurate as refresh window (doc refresh)
Dataset/Cube
- Constant refreshing of entire Dossier interface
Data stream
X min. - Refreshes regardless if data is actually new (polling)
Document refresh
Visualization
Dossier
W/ real-time engine
Kafka RT Engine Run Broadcast
Data stream Is a vis cube
notification
requesting Visualization
data? Do
nothing

+ Can support 2-5 second updates

+ Visualizations update without interfering with Dossier interface
+ Visualizations update the moment new data is provided (push)
30 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Real-time Engine Benefits
• Supports massive concurrency through shared socket broadcasts

W/O real-time engine

- #Jobs = [clients] x [#datasets] x [refreshRate]
X min. - Server load scales linearly by number of clients
Document refresh
MSTR Web Clients
dataset/cube

W/ real-time engine
Rooms
• (A) Callcenter Execution with filter1
Kafka RT Engine
• OnCall
Data stream notification • (A) Employee Execution with filter2
• OnCall
• (A) Employee Execution with filter3
• Call Duration
X 100

+ #Jobs= [# active rooms] x [kafka notification frequency]

+ Note: # connected clients is NOT in formula
31 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
32 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.
 Framework for Real Time Visualizations
Phase 2 – Real Time Engine - Q4 2018

Features

• Same Drag & Drop User Experience as with standard datasets

• Multi source Dossier : showing historical data and real time data

• Filtering historical data with real time data points

• Real Time data uses Intelligent Cubes for Platform Analytics, Blending, Caching

• High user scalability, supports less than 3s rate

33 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Real Time Visualizations
Phase 2 – Real Time Engine - Q4 2018

Community & Collaborative approach

• Reference Architecture sanctioned by MicroStrategy Technology

• Reusable code-based framework, published and on Github

• Anyone can contribute, inside or outside of MicroStrategy

• Led by Solutions Architects, advised by Technology

34 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Framework for Real Time Visualizations
Phase 2 - Summary

What's new – Phase 2 What's next

Node.js real time engine More Visualizations
• Heatmap
Real Time KPI Widget • Mapping
• 1 metric
Push Down aggregates
Real Time Line Chart
• Time attribute + 1 or more metrics
• Time attribute + qualitative attribute + 1
metric
• Qualitative attribute + 1 or more metrics

35 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Real Time Analytics capabilities
Getting the Right Data at the Right Time

Speed of thoughts Near Real Time Dossier Real Time Visualizations Real Time Notifications

I get speed of thought answers Get near real time information in Shared spaces Instant remediation - problems
on my database in real time. MicroStrategy cube being solved in real time
TV Walls
The system answers to ad-hoc Dossier available on demand Trigger process in other system
requests
Call centers • Webhook
Use of client based Autorefresh • Human notification : Slack /
Mail / SMS
Supervisor multi-screen
• Run a script

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Where does Machine Learning fit
Example : pipeline.ai

37 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Where does Machine Learning fit
Example : pipeline.ai

38 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Materials

• Kafka Sink
• 2 min video: https://www.youtube.com/watch?v=EMz_GCurv4o
• Community Article: https://community.microstrategy.com/s/article/Kafka-connector-Sink-to-MicroStrategy-Cubes
• Code: https://github.com/hchadeisson/mstr-kafka-sink

• These slides : will be shared on MicroStrategy World 2019 website

• White paper : (coming soon)

• Blog article : (coming a little sooner)

• Useful reads :
• Streams concepts: https://docs.confluent.io/current/streams/concepts.html
• Top Streaming Technologies for Analytics (pros/cons): https://resources.zaloni.com/blog/top-streaming-technologies-for-data-lakes-and-real-time-data

39 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Conclusion Real Time Analytics capabilities
Getting the Right Data at the Right Time

Speed of thoughts Near Real Time Dossier Real Time Visualizations Real Time Notifications
Already available Stream support - Q3 2018 Q4 2018 Future
I get speed of thought answers Get near real time information in Shared spaces Instant remediation - problems
on my database in real time. MicroStrategy cube being solved in real time
TV Walls
The system answers to ad-hoc Dossier available on demand Trigger process in other system
requests
Call centers • Webhook
Use of client based Autorefresh • Human notification : Slack /
Mail / SMS
Supervisor multi-screen
• Run a script

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

 Conclusion

Call to Action Project Next Steps

1. Update your stories on Real Time Analytics 1. Publish Real Time Viz Framework on
Github
2. Bring us more use cases
2. Research on Machine Learning prototypes
3. JavaScript developers? Help us build more
visualizations!

4. Java developers? Help us build more Sinks !

41 Copyright © 2018 MicroStrategy Inc. All Rights Reserved.

Thanks

Copyright © 2018 MicroStrategy Inc. All Rights Reserved.