Understanding & Best Practices

with the QVD File Format

We will begin at the top of the hour

Rob Patterson
Senior Manager, User Enablement
QlikTech

Rob Wunderlich
Senior QlikView Consultant
Factory Information Management Solutions

Factory
Information Management Solutions
UNDERSTANDING & BEST PRACTICES
WITH THE QVD FILE FORMAT

WHAT?

WHY?

HOW?

ARCHITECTURES

2
WHAT?
 Basic Single Tier Model
Datasource

HrDashboard.qvw
Employee

Salary

Emergency
Vacation.qvw

Vacation

PayrollDetail

A *.qvw contains the ETL script, loaded data & screen objects.

4
 Two-Tier Concept

Loader QVWs
Operational QVD
QVD
Data QVD
QVD
QVD

Operational
Data

Operational
Data

User facing qvws

5
 QVDs

QlikView Data file.

One Qlikview table per *.qvd file.

Loads very fast.

Provides reusable extract once single version of ETLed

data.

May be used to provide history beyond current

datamart.
 Creating QVDs

Employee Loader.qvw PayrollHist.qvd

SQL Select
Salary
STORE Vacation.qvd
EmployeeMaster INTO
Emergency EmployeeMaster.qvd;

Vacation One qvw may create

EmployeeMaster.qvd
one or more QVDs.
Typically organized Employee, Salary,
PayrollDetail around subject matter, Position, Emergency
performance and
script requirements.

7
 Consuming QVDs

HrDashboard.qvw

EmployeeMaster.qvd

Vacation.qvd

Vacation.qvw

8
WHY?
 QVD Benefits

Loads fast

Reduced DBMS load & improved scalability

Ease of maintenance

Separation of Roles

De-normalization

Incremental Load

Data consistency, SVT

QVD Data Dictionary
 Loads Fast!

Typically 10 to 100 times faster than loading from DBMS.

DBMS 20 minutes

QVD 30 seconds!

Throughput is based on a number of factors including

number of fields, network configuration and hard disk
speed.
Reduced DBMS Load & Improved Scalability

Every QVW repeats the SQL Select

Reduced DBMS Load & Improved Scalability

Orone SQL Select, then load from QVD

 Ease of Maintenance
Redundant Scripting

A new business rule.

Change all occurrences of CostCenter 3502 to3501.

Revenue.qvw
// Remap CostCenter 3502
if(CostCenter='3502', '3501', CostCenter) as CostCenter

Sales.qvw
// Remap CostCenter 3502
if(CostCenter='3502', '3501', CostCenter) as CostCenter

Commission.qvw
// Remap CostCenter 3502
if(CostCenter='3502', '3501', CostCenter) as CostCenter
 Ease of Maintenance
Eliminate Redundant Script

LoaderCostCenter.qvw
// Remap CostCenter 3502
if(CostCenter='3502', '3501', CostCenter) as CostCenter

STORE CostCenterTable INTO CostCenter.qvd;

Revenue.qvw CostCenter.qvd
LOAD * FROM CostCenter.qvd (qvd);

Sales.qvw
LOAD * FROM CostCenter.qvd (qvd);

Commission.qvw
LOAD * FROM CostCenter.qvd (qvd);
 Separation of Roles

Employee

Salary
Employee.qvd

Emergenc Vacation.qvd
Data Experts

Vacation

Payroll
Business
Experts
 Denormalization

LEFT JOIN (Employee)

Employee: SQL SELECT DepartmentID,
EmployeeName DepartmentName
Position
DepartmentID * FROM dbo.Department ;

Department: 1
DepartmentID
DepartmentName

STORE Employee INTO Employee.qvd;

Sales.qvw
Employee.qvd

Commission.qvw
 Data consistency, SVT

6:00am---------------6:30am--------------8:00am

Sales.qvw
sum(Orders)=$1,000,000

Orders
Database Commission.qvw
sum(Orders)=$1,023,482
 QVD Data consistency, SVT

6:00am---------------6:30am--------------8:00am

Loader_Orders.qvw
sum(Orders)=$1,000,000

Orders
Orders.qvd
Database

Sales.qvw
sum(Orders)=$1,000,000

Commission.qvw
sum(Orders)=$ 1,000,000
 Incremental Load

Fetch only new or changed rows from DBMS

New rows are merged with existing rows in QVD

Greatly reduces load on DBMS

Full Table: 10M rows, 30 minutes

Changed data, 50k rows, 1 minute

Reduced load time may enable intra-day reloads

 History & Snapshots

QVDs may be used to store data longer than DBMS

Some applications may require periodic snapshots
for auditing or delta reporting:
 Customers_2010_06.qvd
 Customers_2010_07.qvd
HOW?
 Creating QVDs

QVDs may be created in three ways:

 Explicitly using STORE script statement

 Implicitly using BUFFER script statement

 Via UI chart Export

 Creating QVD - STORE

STORE may be done in script of

user-facing qvw
HrDashboard.qvw

More typically done in special purpose Loader qvw

QVD
Employee

Department

Loader_Employee.qvw
 STORE Syntax

STORE table INTO filename.qvd (qvd|txt);

Default is (qvd) typically not specified

(txt) creates a CSV file
may include path

filename
data\employee.qvd;

filename respects Directory statement. e.g.
Directory data;
STORE Employee INTO employee.qvd;
Writes to data\employee.qvd
 Creating QVD - BUFFER

BUFFER prefix on LOAD signifies that a QVD should

be automatically created on first execution, and read
from on subsequent executions.
BUFFER LOAD * FROM Access*.log;

QVD is created in a special location. The QVD is
automatically dropped and recreated if the LOAD
statement changes.

Very useful with text files (like logs) where
incremental load is automatic.

Less useful with DBMS tables, but often used to
speed development.
 Creating QVD UI Export

Charts (not TableBoxes) may be exported to QVD

using context menu Export
Creating QVD UI Export
 Loading QVDs

QVDs may be loaded

 Explicity LOAD * FROM data.qvd (qvd);

 Implictly BUFFER LOAD

 Loading QVDs Explicit

Script wizard support provided by Table Files button

 Loading QVDs Explicit

LOAD fieldlist FROM filename.qvd (qvd);

(qvd) parameter is critical. If
ommitted, script error or unexpected
(xml) data.

filename respects Directory statement. e.g.
Directory data;
LOAD * FROM Employee.qvd (qvd);
Reads from data\employee.qvd
 Loading QVD - BUFFER

BUFFER LOAD * FROM Access*.log;

Will load from QVD if it exists, otherwise will read
FROM source.

If source is text files, will read only new data.
 Loading QVD - BUFFER

May be used to reload DBMS tables in the same qvw on

different frequencies.

This qvw is reloaded hourly.
Orders:
// Orders are loaded on every reload
SQL SELECT * FROM Orders;
Customers:
// Customers are loaded only once per day
BUFFER (stale after 1 days) SELECT * FROM
Customers;
 Optimized Load

Optimized load (super fast) will occur if no

transformations or WHERE predicates.
Optimized load indicated by message in
script progress window.
 Optimized Load Limitations

Optimized Load Limits

1) F1 as newfield - yes, rename allowed
2) F1 * 2 as newfield no, transformation

3) 1 as newfield no, cannot add new field

4) where exists(F1) yes, single field exists()

5) where exists(F1, F3) no, multi-field exists()

6) F1 as newfield where exists(F1) - no

 QVD as Map

QVDs may be used in MAPPING LOAD except

Optimized load cannot be used. No error, but no
mapping results.

Solution is to force a non-optimized load:
EmpMap:
MAPPING LOAD EmpID, EmpName
FROM Emp.qvd (qvd)
WHERE 1=1
;
 QVD Functions

Several useful script functions obtain metadata from

a QVD.

Documented in File Functions section of Ref
Guide. Examples:
 QvdNoOfRecords( filename )
 QvdNoOfFields( filename )

Qvd Functions do not respect Directory statement.
Specify full path relative to qvw.

See Ref Guide for complete list
 QVD Dictionary

QVD Functions may be used with a filelist loop to

create a dictionary of QVDs for reference
 QVD Dictionary

QVD Field Analysis

 QVX

New with V10 -- QlikView data eXchange

Open variant of QVD

Not quite as fast as QVD

May be used to create high performance input files
from sources where ODBC drivers are not available.

Read by standard LOAD script statement.

MetaFunctions --QvdNoOfFields( filename )
ARCHITECTURES
 Basic Single Tier Model
Datasource

HrDashboard.qvw
Employee

Salary

Emergency
Vacation.qvw

Vacation

PayrollDetail

A *.qvw contains the ETL script, loaded data & screen objects.

42
 Two Tier Model
Extract & Transform
Employee EmployeeMaster.qvd

Salary
Vacation.qvd

Emergency

Vacation

PayrollDetail

User facing qvws

43
 Three Tier Model
Extract Transform

Employee QVD
EmployeeMaster.qvd

Salary QVD
Vacation.qvd

Emergency QVD

Vacation QVD

PayrollDetail QVD
User facing qvws

44
 Bonus -- QlikMarts

QVWs may also be consumed as datasources.

Script: BINARY Vacation.qvw;

Vacation.qvw
EmployeeMaster.qvd

Repurpose the data

model of a qvw for
Vacation.qvd different UI audiences.

VacationByDept.qvw VacationForecast.qvw VacationSelfServe.qvw

 Questions and Next Steps

Learn More about our QVD Data Model Pack at

www.factory-ims.com
 Questions?

Please use the Q&A feature in WebEx

 Contact Us

Rob Patterson Rob Wunderlich

Senior Manager, User Enablement Senior QlikView Consultant

[email protected] [email protected]

community.qlikview.com

www.factory-ims.com