1

Module 4 In this module we will:

• Walkthrough of a BigQuery Job
• Calculate BigQuery Pricing: Storage, Querying,
Google BigQuery
and Streaming Costs
Pricing • Optimize Queries for Cost

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 2

The Unit of Work in BigQuery is Called a Job

Each Job:

● Given a Unique ID by Web UI

● Can run concurrently

● Perform Tasks

● History Stored for 6 Months

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 3

The Four Types of BigQuery Jobs

Job Types:

● Query

● Load Data into a Table

● Extract to GCS

● Copy Existing Table

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 4

You Only Incur Query Job Processing Costs

Job Types:

● Query - charged by bytes processed

● Load Data - free

● Extract - free

● Copy - free

Note that storing data in BigQuery is a separate cost

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 5

Module 4 In this module we will:

• Walkthrough of a BigQuery Job
• Calculate BigQuery Pricing: Storage,
Google BigQuery
Querying, and Streaming Costs
Pricing • Optimize Queries for Cost

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 6

Three categories of BigQuery pricing

Storage Processing Free

● Amount of data in ● On-demand OR ● Loading
table Flat-rate plans ● Exporting
● Ingest rate of ● On-demand based on ● Queries on metadata
streaming data amount of data ● Cached queries
● Automatic discount processed ● Queries with errors
for old data ● 1 TB/month free
● Have to opt-in to run
high-compute queries

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 7

On-demand Query and Storage Pricing (as of August 2017)

Query - charged by bytes processed

● $5 per TB (first TB each month
is free)

Storage - charged by GB stored per

month
● $0.02 per GB, per month

Streaming Inserts
● $0.05 per GB
Latest Pricing
https://cloud.google.com/bigquery/pricing
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 8

Storage Pricing is Prorated

Total Size of Tables Stored Cost

100 MB for half a month You pay $0.001

500 GB for half a month You pay $5

1 TB for a full month You pay $20

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Demo: How much is 5TB of monthly storage and
5TB querying?

Access the price calculator:

https://cloud.google.com/products/calculator/

Latest Bigquery Pricing

https://cloud.google.com/bigquery/pricing
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Pricing Model - Reserved Slots

Slots are the amount of total query throughput

• Guarantee resources, regardless of demand on the overall
multi-tenant pool
• Minimizes variability in query performance
• More concurrent queries without slowing down
• Larger queries may run faster

The more you use BigQuery, the more slots you get
automatically

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Use the validator in the web UI to help calculate cost

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Quotas are used to protect all BigQuery tenants

● 50 Concurrent Queries
You can set custom quotas
by project and even by user
● Query timeout: 6 hours for your organization

● 1,000 updates to a Table per day

● 1,000 Tables Referenced by a Single Query

● Max result size: 128MB Compressed*

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Pricing Pitfalls: Only query what you actually need

● SELECT column WHERE

column = 123 LIMIT 10 will
still need to process all rows
to satisfy the filter condition

● SELECT * with very long string

length columns means more
bytes processed
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 14

Module 4 In this module we will:

• Walkthrough of a BigQuery Job
• Calculate BigQuery Pricing: Storage, Querying,
Google BigQuery
and Streaming Costs
Pricing • Optimize Queries for Cost

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Apply cost optimizing principles when writing your queries

● Only include the columns and rows you need (filter early)

● Use cached results when possible

○ (i.e. Permanent Tables instead of Views)
○ Views are saved Queries -- covered later

● Limit the use of User-Defined Functions

○ UDFs covered later

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 16

Summary: GCP offers

Summary: Calculate youoptimize
costs and the ability to:
your queries

BigQuery jobs BigQuery charges 1TB / month of Consume only

include query, for what data you free data the rows and
load, extract, consume in your processing. No columns of data
copy queries (bytes charges for you need
processed) queries using
cache.

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 17

Lab 3
Calculate Google BigQuery
Pricing

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
 Calculate BigQuery Pricing

In this lab, you will explore Google

BigQuery pricing and how to estimate
query and storage costs. Additionally,
you will see how modifying a query can
affect the cost.

BigQuery is a pay-for-what-you-use tool.

You don’t pay for infrastructure costs.

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.