Big Data and Data Visualization

Dr. Shantanu Ganguly

Fellow, Knowledge Resource Center
The Energy and Resources Institute (TERI)
Email: [email protected]
 Lesson Topics
• Big Data

• Big Data and Analytics

• Powerful data visualizations

• Understand visualizations

• Understand design considerations that lead to powerful data visualizations

• Understand effective techniques to create data visualizations

• Understand best practice tips for presenting data visualizations

The Information Continuum

Cartoon by David Somerville, based on a two pane version by Hugh McLeod

Types of Data
Quantitative Data Qualitative Data
• Measurable • Descriptive
• Collected through observation,
• Collected through field work, focus groups,
measuring things that have interviews, recording or filming
a fixed reality conversations
• Close ended • Open ended
Big Data
Data that is too large or too
complex to be managed
using traditional data
processing, analysis, and
storage techniques.
What is Big Data?
• Big Data is a collection of large datasets that cannot be adequately
processed using traditional processing techniques. Big data is not only data
it has become a complete subject, which involves various tools, techniques
and frameworks.
• Big data term describes the volume amount of data both structured and
unstructured manner that adapted in day-to-day business environment. It’s
important that what organizations utilize with these with the data that
matters.

• Big data helps to analyze the in-depth concepts for the better decisions and
strategic taken for the development of the organization.
The Evolution of Big Data
Volume: scale of data
 Volume: scale of data

• 90% of today’s data has been created in just the last 2 years
• Every day we create 2.5 quintillion bytes of data or enough to fill 10 million Blu-ray discs
• 40 zettabytes (4o trillion gigabytes) of data will be created by 2020, an increase of 300
times from 2005, and the equivalent of 5,200 gigabytes of data for every man, woman and
child on Earth
• Most companies in the US have over 100 terabytes (100,000 gigabytes) of data stored
Categories of Big Data - I
Categories of Big Data - II
What is the importance of Big Data?
Who are the ones who use the Big Data
Technology?
Brief explanation of how
exactly businesses are utilizing Big Data
Big Data Technologies
 “Big Data” on PubMed
1400
1196
1200
Instances of “Big Data”

1000

800 723

600
463
400
201
200
2 1 9 3 2 7 41
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Year
Big Data and Librarians

• What role will librarians play in the

Big Data revolution?
• Do you see yourself playing a part?
• How will you prepare yourself?
• What resources will you use?

Patricia Brennan, RN, PhD, NNLM Director

Operational Big Data vs. Analytical Big Data
Barriers
Big Data and Analytics
Where does Big Data come from?
Email Transactions
Enterprise Partner, Employee
“Dark Data” Customer, Supplier

Contracts Monitoring

Public Commercial
Sensor
Credit
Weather
Industry
Population Social Media Sentiment
Economic
Network
Types of Data
Which Big Data characteristic is the
biggest issue for your organization?

Velocity of
data
16%
Variety of
data
Volume of 48%
data
35%

Source: Getting Value from Big Data, Gartner Webinar, May 2012
Biggest opportunity for Big Data in your
organization?
• 85% of Fortune 500
organizations will be unable
to exploit big data for
competitive advantage.

• Business analytics needs

will drive 70% of
investments in the
expansion and
modernization of
information infrastructure.
Source: Getting Value from Big Data, Gartner Webinar, May 2012
Business Analytics
 Business Analytics/ Business Intelligence

• Business Analytics/Business intelligence (BI) is a broad category of

applications, technologies, and processes for:
• gathering,
• storing,
• accessing, and
• analyzing data

• to help business users make better decisions.

 Things Are Getting More Complex

• Many companies are performing new kinds of analytics (**sentiment analysis, etc.), to
better and more quickly understand and respond to what customers are saying about them
and their products.
• The cloud, and appliances are being used as data stores
• Advanced analytics are growing in popularity and importance

**Sentiment analysis (also known as opinion mining) refers to the use of natural language processing, text
analysis and computational linguistics to identify and extract subjective information in source materials.
Analytics Models How can we make it
happen?

What will Prescriptive

happen? Analytics

Why did it Predictive

VALUE

happen? Analytics

Diagnostic
What happened? Analytics

Descriptive
Analytics

DIFFICULTY
 Descriptive Analytics
• Descriptive analytics, such as reporting/ OLAP, dashboards, and data visualization, have
been widely used for some time.
• They are the core of traditional BI.

What has occurred?

Descriptive analytics, such as data visualization, is important in
helping users interpret the output from predictive and predictive
analytics.
 Predictive Analytics

• Algorithms for predictive analytics, such as regression analysis, machine learning, and
neural networks, have also been around for some time.

What will occur?

• Marketing is the target for many predictive analytics applications.

• Descriptive analytics, such as data visualization, is important in helping users
interpret the output from predictive and prescriptive analytics.
 Prescriptive Analytics
• Prescriptive analytics are often referred to as advanced analytics.
• Often for the allocation of scarce resources
• Optimization

What should occur?

Prescriptive analytics can benefit healthcare strategic planning by using analytics to

leverage operational and usage data combined with data of external factors such as
economic data, population demographic trends and population health trends, to more
accurately plan for future capital investments such as new facilities and equipment
utilization as well as understand the trade-offs between adding additional beds and
expanding an existing facility versus building a new one.
A Brief History of Data Visualization
What makes a good chart?

Napoleon’s 1812 March by

Charles Joseph Minard
Reprinted in Tufte (2009), p. 41
Perhaps the most famous data presentation…
Florence Nightingale's 'Coxcombs‘1858

• Pioneer hospital sanitation

• Meticulously gathered data
• Pioneer in applied statistics and
visualization
• Nurse
Willard C. Brinton, 1914
First business book about visualization

• Rules for presenting data

• American consulting engineer
 Mary Eleanor Spear 1952, 1969

• Common-sense advice
• Invented box plot
• Worked for various US
government agencies
Jacques Bertin1967
• Principle of expressiveness:
• Say everything you want to say — no
more, no less
• Don’t mislead
• Principle of effectiveness:
• Use the best method available for
showing your data
• Cartographer
Jacques Bertin
Seven Visual Variables
• Position
• Size
• Shape
• Color
• Brightness
• Orientation
• Texture
Edward Tufte
1983

• Disciplined design principles

• Minimalist approach
• Professor emeritus at Yale University
 Jock Mackinlay
1986

• Automatically encode data with software

• Enable people to focus on ideas, concepts
• Added eighth variable to Bertin’s list: motion
• VP of Research and Design, Tableau Software
When a Chart
hits our Eyes

1. Visuals aren’t read in a predictable, linear way

• Create charts spatially, from the visual outward
2. We see first what stands out
• Whatever stands out should support idea
3. We see only a few visuals at once
• Plot as few visual elements as possible
4. We seek meaning and make connection
• Relate visual elements in a meaningful way
5. We rely on conventions and metaphors
• Embrace deeply ingrained
49 conventions
Example: USA Energy Resources
What is data visualization and why is
it important?
Is data visualization a part of data
science?
What are the best data visualization
software of 2019?
What is data discovery and
visualization?
What are data visualization tools?
Is Excel a data visualization tool?
How do you create good data visualization?
What kind of visual communication
do you want to create?

1. Is my information conceptual or data-

driven?
• Conceptual information is qualitative
• Data-driven information is quantitative
2. Are my visuals meant to be declarative or
exploratory?
• A declarative purpose is to make a statement
• An exploratory purpose is to look for new ideas
Four Types of Data Visualizations
Declarative
Idea
illustration
Conceptual Data-Driven

Idea
generation
Exploratory
 Four Types of Data Visualizations
Declarative
Idea Everyday
illustration dataviz
Conceptual Data-Driven

Idea Visual
generation discovery
Exploratory
 Data Visualization

provide clear understanding of patterns in data

detect hidden structures in data

condense information
 What makes a good chart?

Wikipedia: Patriotic War of 1812

Video: Napoleonic Wars in 8 Minutes Another video http://en.wikipedia.org/wiki/File:Patriotic_War_of_1812_ENG_map1.svg
What can you learn from this map?

http://www.popvssoda.com/countystats/total-county.html
Some basic principles (adapted from Tufte 2009)

1 • The chart should tell a story

• The chart should have graphical

2 integrity

• The chart should minimize graphical

3 complexity

Tufte’s fundamental principle:

Above all else show the data
Principle 1: The chart should tell a story

Graphics should be clear on their own

The depictions should enable meaningful comparison

The chart should yield insight beyond the text

“If the statistics are boring, then you’ve got the wrong
numbers.” (Tufte 2009)
 Principle 2: The chart should have graphical
integrity
• Basically, it shouldn’t “lie” (mislead the reader)
• Tufte’s “Lie Factor”:
𝑠𝑖𝑧𝑒 𝑜𝑓 𝑒𝑓𝑓𝑒𝑐𝑡 𝑠ℎ𝑜𝑤𝑛 𝑖𝑛 𝑔𝑟𝑎𝑝ℎ𝑖𝑐
• 𝐿𝑖𝑒 𝐹𝑎𝑐𝑡𝑜𝑟 =
𝑠𝑖𝑧𝑒 𝑜𝑓 𝑒𝑓𝑓𝑒𝑐𝑡 𝑖𝑛 𝑑𝑎𝑡𝑎

Should be ~ 1

> 1 = exaggerated < 1 = understated

effect effect
 Examples of the “lie factor”

5.3/0.6 8.83
𝐿𝐹 = = = 5.77
27.5/18 1.53

Reprinted from
Tufte (2009), p. 4280% (𝑐ℎ𝑎𝑛𝑔𝑒 𝑖𝑛 𝑣𝑜𝑙𝑢𝑚𝑒)
57 & p. 62 𝐿𝐹 = = 9.4
454% (𝑐ℎ𝑎𝑛𝑔𝑒 𝑖𝑛 𝑝𝑟𝑖𝑐𝑒)
Principle 3: The chart should minimize
graphical complexity

Generally, the simpler the better…

Key concepts
Sometimes
a table is Data-ink Chart junk
better
When a table is better than a chart
For a few data points, a table can do just as well…

Salesperson Total Sales

Total Sales by Salesperson Peacock $225,763.68
$250,000.00
Leverling $201,196.27
$200,000.00
Davolio $182,500.09
$150,000.00
Fuller $162,503.78
$100,000.00
Callahan $123,032.67
$50,000.00 King $116,962.99
$0.00 Dodsworth $75,048.04
Suyama $72,527.63
Buchanan $68,792.25

The table carries more information in less space

and is more precise.
The Ultimate Table: The Box Score

• Large amount of
information in a very
small space

• So why does this work?

• Depends on the reader’s
knowledge of the data
Data Ink
• The amount of “ink” devoted to data in a chart
• Tufte’s Data-Ink ratio:
𝑑𝑎𝑡𝑎−𝑖𝑛𝑘
• 𝐷𝑎𝑡𝑎 − 𝑖𝑛𝑘 𝑟𝑎𝑡𝑖𝑜 =
𝑡𝑜𝑡𝑎𝑙 𝑖𝑛𝑘 𝑢𝑠𝑒𝑑 𝑖𝑛 𝑔𝑟𝑎𝑝ℎ𝑖𝑐

Should be ~ 1
< 1 = more non-data = 1 implies all ink
related ink in graphic devoted to data
Tufte’s principle:
Erase ink whenever possible
Being conscious of data ink
Lower data-ink ratio
Hypothetical City Crime
(worse) 425

375

Thefts per 100000 citizens

325

275

225

175
Hypothetical City Crime 125
425 75
375 25
Thefts per 100000 citizens

325 2003 2004 2005 2006 2007 2008 2009 2010

275

225

175 Hypothetical City Crime

125 400
370 370
75 350
320 330
25
2003 2004 2005 2006 2007 2008 2009 2010 270

200

Higher data-ink ratio

2003 2004 2005 2006 2007 2008 2009 2010
(better)
What makes a good chart?
Sum of Extended Price

2011 Total Sales

160000

Sometimes it’s
140000
120000
100000
80000
60000
40000
really a matter of
20000
0 preference.
Order Date

These both
Sum of Extended Price

2011 Total Sales minimize data ink.

160000
140000
120000
100000
80000 Why isn’t a table
60000
40000
20000
better here?
0

Order Date
3-D Charts

Evaluate this from a data-ink perspective.

How does it affect the clarity of the chart?
One of the golden rules of data
visualization is…..
Never use 3D!
• 3D skews numbers,
Data Integrity/
making them difficult
Lie Factor
to interpret or compare
• Adding 3D to graphs
introduces
Graphical
unnecessary chart
Complexity
elements like side and
floor panels

Source: Knaflic (2015). Storytelling with Data: A Data Visualization Guide for Business Professionals. Chapter 2.
Chartjunk: Data Ink “gone wild”

Unnecessary visual clutter that doesn’t

provide additional insight

Distraction from the story the chart is

supposed to convey

When the data-ink ratio is low, chartjunk is

likely to be high
 Example: Moiré effects (Tufte 2009)
Total Sales by Salesperson Creates illusion of
$250,000.00 movement
$200,000.00

$150,000.00 Stands out, in a bad

$100,000.00 way
$50,000.00

$0.00
Hypothetical City Crime
425

375

Thefts per 100000 citizens

325

275

225

175

125

75

25
2003 2004 2005 2006 2007 2008 2009 2010
Example: The Grid
Hypothetical City Crime
425

375
Why are these
Thefts per 100000 citizens

325

275 examples of
225

175 chartjunk?
125

75

25
2003 2004 2005 2006 2007 2008 2009 2010 Hypothetical City Crime
425

375

Thefts per 100000 citizens

325

275
What could you 225

do to remedy it? 175

125

75

25
2003 2004 2005 2006 2007 2008 2009 2010
Data Ink Working For Us

Evaluate this
chart in terms of
Data Ink.

Imagine this as
a bar chart. As a
table!!
Review: Data principles (adapted from
Tufte 2009)
1 • The chart should tell a story

• The chart should have graphical

2 integrity

• The chart should minimize graphical

3 complexity

Tufte’s fundamental principle:

Above all else show the data
Infographics
• Information graphics

• Visualization of information, data

or knowledge intended to present
information quickly and clearly

• We will have an ICA to create

infographics using Piktochart.

http://the-digital-reader.com/2015/04/13/infographic-ebooks-on-track-to-double-dutch-ebook-market-in-2014/
 Summary
• Use data visualization principles to assess a visualization
• Tell a story
• Graphical integrity (lie factor)
• Minimize graphical complexity (data ink, chartjunk)
• Explain how a visualization can be improved based on those principles
• Types of visualization
 Resources…
• DataMed https://datamed.org/
• Institute for Health Metrics and Evaluation’s Global Health Data Exchange
http://ghdx.healthdata.org/
• NNLM RD3: Resources for Data-Driven Discovery https://nnlm.gov/data/
• NNLM’s YouTube Channel https://www.youtube.com/channel/UCmZqoegBFKJQF69V8d-
05Bw
• OHSU’s Big Data to Knowledge https://dmice.ohsu.edu/bd2k/topics.html
• Registry of Research Data Repositories (re3data.org) http://www.re3data.org/
References
• Borgman, Christine L. Big data, little data, no data: Scholarship in the
networked world. MIT Press, 2015.
• Federer, Lisa. Beyond the SEA: Data Science 101: An introduction for
librarians https://www.youtube.com/watch?v=i78ciP1eGxo&t=3s
• Mayer-Schönberger, Viktor, and Kenneth Cukier. Big data: A
revolution that will transform how we live, work and think. Houghton
Mifflin Harcourt, 2013.
 Bibliography…
• A Good Example of Misleading Visualization
• http://spatial.ly/2009/09/a-good-example-of-misleading-visualization/
• A quick guide for better data visualizations
• https://www.tableau.com/good-to-great
• The analysis of visual variables for use in the cartographic design of point symbols for mobile Augmented Reality applications
• Łukasz Halik, Adam Mickiewicz University Poznan
• http://www.iag-aig.org/attach/30dee1f85f7bd479367f1f933d48b701/V61N1_2FT.pdf
• The Benefits and Future of Data Visualization
• StatSilk founder Frank van Cappelle
• https://www.statsilk.com/blog/benefits-and-future-data-visualization
• Charting Statistics
• Mary Eleanor Spear
• https://archive.org/details/ChartingStatistics
• Color Brewer
• http://colorbrewer2.org/#type=sequential&scheme=BuGn&n=3
Bibliography…
• Data: The language of modern business leaders
• Steve Proctor, March 17, 2017
• http://www.itbusiness.ca/sponsored/data-the-language-of-modern-business-leaders

• Data Visualization: The Best Infographic Tools For 2017

• Bernard Marr, October 10, 2017
• https://www.huffingtonpost.com/entry/data-visualization-the-best-infographic-tools-for_us_59ca128fe4b0f2df5e83b134

• Data Visualization: The future of data visualization

• Will Towler, January/February 2015
• http://analytics-magazine.org/data-visualization-the-future-of-data-visualization

• Data Visualization 101: How to Choose the Right Chart or Graph for Your Data
• Jami Oetting
• https://blog.hubspot.com/marketing/types-of-graphs-for-data-visualization

• Data-Driven Design: Dare to Wield the Sword of Data – Part I

• Brent Dykes, December 4, 2012
• http://www.analyticshero.com/2012/12/04/data-driven-design-dare-to-wield-the-sword-of-data-part-i/

• Datavis.ca
• http://www.datavis.ca/index.php
 Bibliography
• Diverging color schemes: Showing good data isn't enough; you need to show it well
• Alberto Cairo, June 26, 2016
• http://www.thefunctionalart.com/2016/06/diverging-color-schemes-showing-good_26.html
• 8 Horrible Data Visualizations That Make No Sense
• Eric Limer, September 02, 2013
• http://gizmodo.com/8-horrible-data-visualizations-that-make-no-sense-1228022038
• 55 Striking Data Visualization and Infographic Poster Designs
• Igor Ovsyannykov, May 16, 2011
• http://inspirationfeed.com/inspiration/infographics/55-striking-data-visualization-and-infographic-poster-designs/
• 4 Tips for Promoting Predictive Analytics in Your Organization
• Fern Halper, September 26, 2017
• https://tdwi.org/articles/2017/09/26/ADV-ALL-4-Tips-for-Promoting-Predictive-Analytics.aspx
THANK YOU