2012 summarytables

2 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
ORACLE
PRODUCT
LOGO
MySQL: Summary Tables
Keith Larson
keith.larson@oracle.com
MySQL Community Manager
sqlhjalp.blogspot.com
sqlhjalp.com/pdf/2012_summarytables.pdf

3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product
direction. It is intended for information purposes only,
and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at
the sole discretion of Oracle.

localhost ~]$ whoami
Keith Larson
keith.larson@oracle.com
MySQL Community Manager
sqlhjalp.blogspot.com
Started with MySQL 3.23 during the dot.com days.
Primary real world work was with a MySQL InnoDB delayed replicated environment
that easily held over 4 billion rows of user data. Did a lot of data mining of that data
over the years.
Numerous other sites developed on LAMP stack over the last 13 years.
Also a movie buff :)
Who are you?
DBAs?
Developers?
sqlhjalp.blogspot.com/2012/02/summary-tables-with-mysql.html

Program Agenda
MySQL Summary Tables are not a new concept. They are
exactly what they sound like, basically a summary of existing
data. We will review how to take advantage of summary table
information and ETL the processes to provide quick results for
your application.
• A Situation
• Some Solutions
• Resources to learn more

“I know why you're here, ... I know what you've been
doing... why you hardly sleep, why you live alone, and why
night after night, you sit by your computer. You're looking
for him. I know because I was once looking for the same
thing. And when he found me, he told me I wasn't really
looking for him. I was looking for an answer. It's the
question that drives us.... “
-- Trinity , The Matrix
“Perhaps we are asking the wrong questions.”
-- Agent Brown , The Matrix
http://keikonium.deviantart.com/art/The-Matrix-Background-32136837
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.

Temet Nosce - thine own-self you must know.
How well do you know your data....
What if your data was as big as the matrix?
http://thematrix101.com/media/still/photo-mat_temetnosce.jpg
●
How to sort data efficiently
●
How to data mine all of this data
●
Reporting of data trends takes a long time
●
How would you gather data for rendering
A Situation

http://www.grinningplanet.com/3001/mp3news_other/matrix-pods_222x127.jpg
Table of all the structures
CREATE TABLE `structure` (
`structure_id` int(5) unsigned NOT NULL auto_increment,
`latitude` Float (10,6),
`longitude` Float (10,6),
`date_online` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`levels` int(5) unsigned DEFAULT '0',
`max_pods` int(5) unsigned DEFAULT '0',
`ip` INT UNSIGNED NOT NULL,
`communication_protocol`
enum('OFF','IP.PIPE','IP.UDP','IP6.PIPE','IP6.UDP','IP.SPIPE','IP6.SPIPE','SNA.PIPE'
) default NULL,
PRIMARY KEY (`structure_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB ;
The Situation

The Situation
http://www.tony5m17h.net/MatrixNet.gif
Table of all the pods
CREATE TABLE `pod` (
`pod_id` int(5) unsigned NOT NULL auto_increment,
`structure_id` int(5) unsigned DEFAULT '0',
ònline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
òffline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`level` int(5) unsigned DEFAULT '0',
`position` int(5) unsigned DEFAULT '0',
`communication_protocol`
enum('OFF','IP.PIPE','IP.UDP','IP6.PIPE','IP6.UDP','IP.SPIPE','IP6.SPIPE','SNA.PIPE'
) default NULL,
UNIQUE KEY `pods` (`pod_id`,`structure_id`,ònline_date`),
KEY ònline_date` (ònline_date`) ,
KEY òffline_date` (òffline_date`)
) ENGINE=InnoDB ;

CREATE TABLE `human` (
`human_id` int(5) unsigned NOT NULL auto_increment,
`human_key` varchar(32) default NULL,
`pod_id` int(5) unsigned DEFAULT '0',
ònline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
òffline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`human_id`),
UNIQUE KEY `human` (`human_id`,`pod_id`,ònline_date`),
KEY ònline_date` (ònline_date`) ,
KEY òffline_date` (òffline_date`)
) ENGINE=InnoDB ;
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Situation
Table of all the humans

CREATE TABLE `data` (
`data_id` int(5) unsigned NOT NULL auto_increment,
`question` varchar(100) default NULL,
`create_date` datetime default NULL,
`date_updated` datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (`data_id`),
KEY `question` (`question`(5))
) ENGINE=InnoDB ;
mysql> select data_id , question from data limit 7;
+---------+-------------+
| data_id | question |
+---------+-------------+
| 1 | First Name |
| 2 | Last Name |
| 3 | Gender |
| 4 | Address |
| 5 | City |
| 6 | State |
| 7 | Zip |
+---------+------------+
The Situation

CREATE TABLE `human_data` (
`human_data_id` bigint(12) unsigned NOT NULL auto_increment,
`human_id` int(9) unsigned default NULL,
`data_id` int(5) unsigned default NULL,
`value` varchar(100) default NULL,
PRIMARY KEY (`human_data_id`),
UNIQUE KEY `hd_id` (`human_id`,`data_id`),
KEY `dv` (`data_id`,`value`(5))
) ENGINE=InnoDB;
The Situation
+---------------+----------+---------+------------------------------+
| human_data_id | human_id | data_id | value |
+---------------+----------+---------+------------------------------+
| 1 | 1 | 1 | Thomas |
| 2 | 1 | 2 | Anderson |
| 3 | 1 | 3 | M |
| 13 | 1 | 4 | 1060 West Addison Street |
| 4 | 1 | 5 | Chicago |
| 5 | 1 | 6 | Illinois |
| 6 | 1 | 7 | 60613 |
| 7 | 1 | 16 | Lower Downtown, Capitol City |
| 8 | 1 | 17 | 03 |
| 9 | 1 | 18 | 11 |
| 10 | 1 | 19 | 1962 |
| 11 | 1 | 20 | Central West Junior High |
| 12 | 1 | 21 | Owen Paterson High School |
+---------------+----------+---------+------------------------------+

CREATE TABLE `job` (
`job_id` int(5) unsigned NOT NULL DEFAULT '0',
`title` varchar(15) DEFAULT NULL,
`salary` decimal(5,3) default '0.000',
PRIMARY KEY (`job_id`)
) ENGINE=InnoDB ;
CREATE TABLE `color` (
`color_id` int(5) unsigned NOT NULL DEFAULT '0',
`color` enum('Brown','Blue','Hazel','Green','Red','Yellow','Blonde','Gold') default NULL,
PRIMARY KEY (`color_id`)
) ENGINE=InnoDB ;
The Situation

CREATE TABLE èyes` (
èyes_id` int(5) unsigned NOT NULL auto_increment,
`color_id` int(5) unsigned DEFAULT '0',
`species_id` int(5) unsigned DEFAULT '0',
`style_id` int(5) unsigned DEFAULT '0',
`shape_id` int(5) unsigned DEFAULT '0',
PRIMARY KEY (èyes_id`)
) ENGINE=InnoDB ;
CREATE TABLE `hair` (
`hair_id` int(5) unsigned NOT NULL auto_increment,
`style_id` int(5) unsigned DEFAULT '0',
`color_id` int(5) unsigned DEFAULT '0',
`shape_id` int(5) unsigned DEFAULT '0',
PRIMARY KEY (`hair_id`)
) ENGINE=InnoDB ;
The Situation

SELECT h.human_key , hd1.value as 'First Name' , hd2.value as 'Last Name' , hd3.value as
'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' , hd7.value as
'ZipCode' , hd16.value as 'Place of Birth' , CONCAT(hd17.value, '-',hd18.value,'-', hd19.value )
AS DOB, hd20.value as 'Junior High' , hd21.value as 'High School', INET_NTOA(h.ip) as ip
FROM human h
INNER JOIN human_data hd1 ON h.human_id = hd1.human_id AND hd1.data_id = 1
LEFT JOIN human_data hd16 ON h.human_id = hd16.human_id AND hd16.data_id = 16
WHERE h.human_id = 1G
Lots of Data Across lots of Tables
The Situation

*************************** 1. row ***************************
human_key: 63b28bb2be0bf40085646decce81ae9f
First Name: Thomas
Last Name: Anderson
Gender: M
Address: 1060 West Addison Street
City: Chicago
State: Illinois
ZipCode: 60613
Place of Birth: Lower Downtown, Capitol City
DOB: 03-11-1962
Junior High: Central West Junior High
High School: Owen Paterson High School
ip: 192.168.5.221
1 row in set (0.00 sec)
The Situation

SELECT h.human_key , hd1.value as 'First Name' , hd2.value as 'Last Name' ,
hd3.value as 'Gender'
FROM human h
WHERE hd2.value like 'Mc%';
The Situation

Define snowflake
A snowflake schema is a logical arrangement of tables
in a multidimensional database such that the entity
relationship diagram resembles a snowflake in shape. ...
-- en.wikipedia.org/wiki/Snowflake_schema
http://thumbs.dreamstime.com/thumblarge_316/12227643929M1f6o.jpg
http://www.informix.com.ua/articles/rolap/rolap.htm

Solutions
Data Warehouse**
Noun:
A large store of data accumulated from a wide range of
sources within a company and used to guide
management decisions.
-- Google.com
** Same place, same time on Saturday for more on Data Warehouses.

Solutions
Bill Inmon's paradigm: Data warehouse is one part of the
overall business intelligence system. An enterprise has
one data warehouse, and data marts source their
information from the data warehouse. In the data
warehouse, information is stored in 3rd normal form.
Ralph Kimball's paradigm: Data warehouse is the
conglomerate of all data marts within the enterprise.
Information is always stored in the dimensional model.
http://www.1keydata.com/datawarehousing/inmon-kimball.html

Define Summary Table
A collection of one or more data elements that are
classified into some logical structure, either as
dimensions or data points
-- webster's dictionary
Solutions

CREATE TABLE `the_matrix` (
`the_matrix_id` bigint(12) unsigned NOT NULL AUTO_INCREMENT,
`human_id` int(9) unsigned DEFAULT NULL,
`firstname` varchar(100) DEFAULT NULL,
`lastname` varchar(100) DEFAULT NULL,
àlias` varchar(50) DEFAULT NULL,
`gender` varchar(2) DEFAULT NULL,
àddress` varchar(100) DEFAULT NULL,
`city` varchar(100) DEFAULT NULL,
`state` varchar(100) DEFAULT NULL,
`zipcode` varchar(100) DEFAULT NULL,
`dob_month` varchar(5) DEFAULT NULL,
`dob_day` varchar(5) DEFAULT NULL,
`dob_year` varchar(5) DEFAULT NULL,
`hair_color` varchar(50) DEFAULT NULL,
`hair_style` varchar(50) DEFAULT NULL,
èye_color` varchar(50) DEFAULT NULL,
`date_updated` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`the_matrix_id`),
UNIQUE KEY `human_id` (`human_id`),
KEY `date_updated` (`date_updated`)
) ENGINE=InnoDB
Solutions: Summary Table

DELIMITER //
CREATE PROCEDURE build_the_matrix()
BEGIN
SELECT MAX(date_updated) INTO @matrix_date FROM the_matrix ;
REPLACE INTO the_matrix SELECT NULL, h.human_id, hd1.value as 'First Name' , hd2.value as 'Last Name',
hd13.value as alias , hd3.value as 'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' ,
hd7.value as 'ZipCode' , hd17.value as dob_month, hd18.value as dob_day, hd19.value as dob_year, hd23.value as
hair_color, hd24.value as hair_style, hd25.value as eye_color, hdd.date_updated
FROM human h
INNER JOIN human_data hdd ON h.human_id = hdd.human_id
WHERE hdd.date_updated >= @matrix_date ;
END //
DELIMITER ;
Solutions: ExtractLoad EL

CREATE EVENT build_the_matrix_10minruns
ON SCHEDULE
EVERY 10 MINUTE
COMMENT ' it is the question that drives us '
DO
CALL build_the_matrix();
show variables like '%event_s%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| event_scheduler | ON |
+-----------------+-------+
mysql> set GLOBAL event_scheduler = ON || OFF ;
mysql> show processlistG
.....
User: event_scheduler
Host: localhost
db: NULL
Command: Daemon
Time: 26
State: Waiting for next activation
Info: NULL
Solutions: ExtractLoad EL

Solutions: Data Mining
mysql> SELECT SQL_no_cache COUNT(h.human_id) , g.value , s.value , c.value
FROM human h
INNER JOIN human_data g ON g.human_id = h.human_id AND g.data_id = 3
LEFT JOIN human_data s ON s.human_id = h.human_id AND s.data_id = 6
LEFT JOIN human_data c ON c.human_id = h.human_id AND c.data_id = 25
GROUP BY g.value , s.value , c.value;
+-------------------+-------+----------+-------+
| COUNT(h.human_id) | value | value | value |
+-------------------+-------+----------+-------+
| 119932 | F | NULL | NULL |
...
| 14 | M | New York | Blue |
| 13 | M | New York | Brown |
| 7 | M | New York | Green |
| 5 | M | New York | Hazel |
| 7 | M | Oregon | Blue |
| 8 | M | Oregon | Brown |
| 9 | M | Oregon | Green |
| 4 | M | Oregon | Hazel |
+-------------------+-------+------------+-------+

Data Mine the matrix
mysql> SELECT SQL_no_cache COUNT(human_id), gender , state , hair_color
FROM the_matrix
GROUP BY gender , state , hair_color
ORDER BY gender , state , hair_color ;
| COUNT(human_id) | gender | state | hair_color |
+-----------------+--------+------------+------------+
| 0 | NULL | NULL | NULL |
| 109196 | F | NULL | NULL |
| 163435 | M | NULL | NULL |
.....
| 1 | M | Illinois | BLACK |
| 25 | M | New York | BLACK |
| 14 | M | New York | BLOND |
| 12 | M | Oregon | BLACK |
| 16 | M | Oregon | BLOND |
+-----------------+--------+------------+------------+

mysql> SELECT SQL_no_cache COUNT(h.human_id) , g.value , s.value , c.value
FROM human h
INNER JOIN human_data g ON g.human_id = h.human_id
LEFT JOIN human_data s ON s.human_id = h.human_id
LEFT JOIN human_data c ON c.human_id = h.human_id
WHERE g.data_id = 3 AND s.data_id = 6 AND c.data_id = 25
+-------------------+-------+----------+-------+
| COUNT(h.human_id) | value | value | value |
+-------------------+-------+----------+-------+
...
+-------------------+-------+------------+-------+

CREATE VIEW matrix.matrix_view AS
SELECT SQL_NO_CACHE COUNT(h.human_id) , g.value as gender, s.value as state , c.value as hair_color
FROM human h
LEFT JOIN human_data g ON g.human_id = h.human_id
LEFT JOIN human_data s ON s.human_id = h.human_id
LEFT JOIN human_data c ON c.human_id = h.human_id
WHERE g.data_id = 3 AND s.data_id = 6 AND c.data_id = 25
+-------------------+--------+------------+------------+
| COUNT(h.human_id) | gender | state | hair_color |
+-------------------+--------+------------+------------+
.....
+-------------------+--------+------------+------------+
Solutions: A View

DELIMITER //
CREATE PROCEDURE build_the_matrix()
BEGIN
SELECT MAX(date_updated) INTO @matrix_date FROM the_matrix ;
REPLACE INTO the_matrix SELECT NULL, h.human_id, hd1.value as 'First Name' , hd2.value as 'Last Name', hd13.value as alias ,
hd3.value as 'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' , hd7.value as 'ZipCode' , hd17.value as dob_month,
hd18.value as dob_day, hd19.value as dob_year, hd23.value as hair_color, hd24.value as hair_style, hd25.value as eye_color,
hdd.date_updated
FROM human h
INNER JOIN human_data hdd ON h.human_id = hdd.human_id
WHERE hdd.date_updated >= @matrix_date ;
END //
DELIMITER ;
Solutions: Populating Tables

CREATE EVENT build_the_matrix
ON SCHEDULE
EVERY 2 MINUTE
COMMENT 'It is the question that drives us'
DO
CALL build_the_matrix();
mysql> show events;
Db: matrix
Name: build_the_matrix
Definer: root@localhost
Time zone: SYSTEM
Type: RECURRING
Execute at: NULL
Interval value: 2
Interval field: MINUTE
Starts: 2012-05-23 23:09:24
Ends: NULL
Status: ENABLED
Originator: 1
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: latin1_swedish_ci
mysql> select * from information_schema.eventsG
Solutions: Events

Triggers
delimiter |
CREATE TRIGGER some_trigger BEFORE INSERT ON Table_A
FOR EACH ROW BEGIN
INSERT INTO TEST_B SET a2 = NEW.a1;
DELETE FROM TEST_C WHERE a3 = NEW.a1;
UPDATE TEST_D SET b4 = b4 + 1 WHERE a4 = NEW.a1;
END; |
delimiter ;
mysql> show triggers;
Solutions: Populating Tables

Transform---
– This is your chance to clean data
– Enable the special needs of others
– Rectify Problems
– Know your data....
• Events
• Procedures
• Triggers
• Scripts
Solutions: ExtractTransformLoad ETL

mysql> SELECT SQL_no_cache
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender
FROM human h
INNER JOIN the_matrix m ON m.human_id = h.human_id
INNER JOIN pod p ON p.pod_id = h.pod_id
INNER JOIN structure s ON s.structure_id = p.structure_id
WHERE s.date_online BETWEEN 19850101000000 AND 19890101000000
GROUP BY m.gender;
+--------+--------+
| Amount | gender |
+--------+--------+
| 23783 | F |
| 35472 | M |
+--------+--------+
Solutions: Reporting

First start with our lowest common denominator beyond your root table.
CREATE TABLE `structure` (
`structure_id` int(5) unsigned NOT NULL DEFAULT '0',
`latitude` float(10,6) DEFAULT NULL,
`longitude` float(10,6) DEFAULT NULL,
`date_online` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`levels` int(5) unsigned DEFAULT '0',
`max_pods` int(5) unsigned DEFAULT '0',
`communication_protocol` enum('OFF','EMAIL','FTP','BATCH POST','BATCH GET','REAL
TIME GET','TRACKTAGS','FTP-SSL','REAL TIME POST','CUSTOM','POST') DEFAULT
NULL,
PRIMARY KEY (`structure_id`),
) ENGINE=InnoDB DEFAULT

CREATE TABLE `report_structure_24hour` (
`report_structure_24hour_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`totals` int(5) unsigned DEFAULT '0',
`date_online` date NOT NULL DEFAULT '0000-00-00',
PRIMARY KEY (`report_structure_24hour_id`),
) ENGINE=MEMORY;
REPLACE INTO report_structure_24hour
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender ,
DATE_FORMAT(s.date_online, '%Y-%m-%d %h ') as date
FROM human h
WHERE s.date_online >= NOW() - interval 24 hour
GROUP BY m.gender, date;

CREATE TABLE `report_structure_perday` (
`report_structure_perday_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`report_structure_perday_id`),
) ENGINE=InnoDB;
REPLACE INTO report_structure_perday
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender ,
DATE_FORMAT(s.date_online, '%Y-%m-%d') as date
FROM human h
WHERE s.date_online >= NOW() - interval 1 week

CREATE TABLE `report_structure_permonth` (
`report_structure_permonth_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`report_structure_permonth_id`),
) ENGINE=InnoDB;
REPLACE INTO report_structure_permonth
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y-%m-00') as date
FROM report_structure_perday
WHERE date_online >= NOW() - interval 2 month
GROUP BY gender, date;

CREATE TABLE `report_structure_per_year` (
`report_structure_year_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`date_online` int(5) unsigned DEFAULT '0000',
PRIMARY KEY (`report_structure_year_id`),
) ENGINE=InnoDB;
REPLACE INTO report_structure_per_year
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y') as date
FROM report_structure_permonth
WHERE date_online >= NOW() - interval 2 YEAR

Populate tables with :
DELIMITER //
CREATE PROCEDURE reporting_perday()
BEGIN
REPLACE INTO report_structure_perday
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender , DATE_FORMAT(s.date_online, '%Y-%m-%d')
as date
FROM human h
WHERE s.date_online >= NOW() - interval 1 week
END //
DELIMITER ;
CREATE EVENT reporting_perday
ON SCHEDULE
EVERY 24 HOUR
DO
CALL reporting_perday();

DELIMITER //
CREATE PROCEDURE reporting_permonth()
BEGIN
REPLACE INTO report_structure_permonth
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y-%m-00') as date
FROM report_structure_perday
WHERE date_online >= NOW() - interval 2 month
END //
DELIMITER ;
CREATE EVENT reporting_permonth
ON SCHEDULE EVERY 2 WEEK
DO
CALL reporting_permonth();

DELIMITER //
CREATE PROCEDURE reporting_peryear()
BEGIN
REPLACE INTO report_structure_per_year
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y') as date
FROM report_structure_permonth
WHERE date_online >= NOW() - interval 2 YEAR
END //
DELIMITER ;
CREATE EVENT reporting_peryear
ON SCHEDULE
EVERY 2 MONTH
DO
CALL reporting_peryear();

It all comes down to who do
you want to be?
The One or just someone ?
The Solutions are up to you.
http://images1.wikia.nocookie.net/__cb20070215050459/uncyclopedia/images/7/75/Neo.JPG
Solutions - Solution

Resources
http://code.google.com/p/flexviews/

2012 summarytables

Recommended

More Related Content

What's hot (20)

Similar to 2012 summarytables (20)

Recently uploaded (20)

2012 summarytables