April 21 23.

Neuss, Germany
w w w. s q l p a s s . o r g / e u 2 0 1 0

Inside SQL Server Wait Types SQL 2005 and

SQL 2008
Bob Ward, Principal Escalation Engineer, Microsoft Corporation

Welcome to My World

This is a 500 level talk

I assume SQL knowledge

DMVs,
Code, Debugger, and APIs

We will move fast and furious

This means your brain may hurt

Stay for Stuff you

can use

questions as long as you want

Inside SQL Server Wait Types

What is a Wait Type?

Not the best of docs

We created this to help us find bottlenecks In a galaxy, far, far, away we had locks, I/O and network But as time has moved on we went a bit overboard The name of the type is up to the developer
485 in SQL Server 2008 Resource Synchronization Forced External
I/O, Network, Thread, Memory Locks, Latches, and bunch of others Yield or Sleep

Preemption
Background tasks

Queue

Inside SQL Server Wait Types

How does it work?

Common for a SELECT

We know we need to wait

Request LCK_M_IS (Shared Intent) lock

A conflict exists

Setup a SOS_WaitInfo with LCK_M_IS

Understands SQLOS scheduling

Wait() results in SignalObjectAndWait()

Use SOS_EventAuto class to wait

Call LockOwner::Sleep

Ultimately it always comes down to WaitForSingleObject() or SignalObjectAndWait()

Inside SQL Server Wait Types

SOS_EventAuto is a wrapper for Windows Kernel Event object

Where do you see Wait Types?

sys.dm_os_wait_stats sys.dm_exec_requests sys.dm_os_waiting_tasks sys.sysprocesses Extended Events Management Data Warehouse Activity Monitor Performance Monitor Counters
Wait Statistics Counter
Legacy Tracing in 2008 In the tools Historical stats Live state

Inside SQL Server Wait Types

last_wait_type explained
Last wait type used to be stored with session Now stored with the worker Only valid while worker bound to task

Last_wait_type = wait_type when task is waiting

true last wait type when task is still running but not waiting
Good example is SOS_SCHEDULER_YIELD

Pre-emptive waits can confuse this but we will explain later

Inside SQL Server Wait Types

Dive into Wait Types

Inside SQL Server Wait Types

Common Wait Types

Locks sync Hint: Your app

Hint: System table or allocation

BUF latch - sync

LCK_XX

PAGELATCH and PAGEIOLATCH

Hint: I/O delay

Resource

Make up ~50 of the wait types

ASYNC_NETWORK_IO
Hint: Network or your app

Inside SQL Server Wait Types

Some Waits may not be bottlenecks MISCELLANEOUS

Should be called not waiting

Background Task Waits

LAZYWRITER_SLEEP SQLTRACE_BUFFER_FLUSH LOGMGR_QUEUE CHECKPOINT_QUEUE REQUEST_FOR_DEADLOCK_SEARCH
BOL calls these

Queue Waits

CLR_AUTO_EVENT
Inside SQL Server Wait Types

Normal for SQL CLR

Busting the Myth of CXPACKET

Craig Freedman Paper is a must read Sync

Used to synchronize parallel query workers

Just means you have a parallel query Do you expect parallel queries? Do you have high wait times? wait_resource shows coordination

High wait times mean long running parallel queries

Look at the Tasks

What Should I Do?

You may not need to do anything Find queries and tune them Use the MAXDOP query hint Modify max degree of parallelism

Which one is not CXPACKET? Some other wait may be the issue

Dont panic. Some think MAXDOP=1 should be on OLTP systems

Inside SQL Server Wait Types

The Non Buffer Latch

Sync

Latch can be generic Appears as LATCH_XX How many are there?

Inside SQL Server Wait Types

Not just for BUFs Thread sync of memory structure

As opposed to PAGELATCH or PAGEIOLATCH

Same modes as BUFs (KP, SH, UP, EX, DT) Latch class

sys.dm_os_latch_stats sys.dm_exec_requests.wait_resource

FGCB_ADD_REMOVE Latch
SQL Server Engine
INSERT Need space INSERT Need space Need space Need space
LATCH_SH: FGCB_ADD_REMOVE

Sync

I need to grow LATCH_EX: FGCB_ADD_REMOVE

FGCB

Autogrow

mydb.mdf

INSERT

INSERT

Moral of the story: Use instant file initialization butit doesnt work for the tlog

Inside SQL Server Wait Types

SOS_SCHEDULER_YIELD
I/O, Lock, Latch Forced

A task that does not naturally wait must yield When quantum of 4ms is exceeded What if we dont do this right?
************************ * * BEGIN STACK DUMP: * 10/17/09 15:51:52 spid 0 * * Non-yielding Scheduler * ************************

Examples
No I/O needed for pages T-SQL variables only or just expressions Query compile Small hashes and sorts

Indicators
High count/low wait this is actually good High count/High wait CPU queries competing Low count/High wait someone not yielding or preemptive threads hogging CPU

Inside SQL Server Wait Types

THREADPOOL
Applies to any task TDS Login
Receive TDS packet Engine creates SQLOS Task

Resource

Find available worker on scheduler

If none, set THREADPOOL wait type

Next available worker runs task

Login Timeout

These are pure victims

Only seen in stats and tasks

You may need DAC to see it live

Look for other waits

Request = task + worker

PENDING tasks and work_queue_count in schedulers > 0

Often a long blocking chain DO NOT assume you need more worker threads

Inside SQL Server Wait Types

What about I/O Waits?

COMMIT TRAN

Sync Flush Log Buffer

WRITELOG LOGBUFFER

Log Writer Log Buffer

Mylog.ldf

INSERT

Copy model SQLTrace File Sort I/O

Request Log Buffer Resource

IO_COMPLETION File Resource

Log Cache
All buffers in use

Page I/O
DISKIO_SUSPEND

Mylog.ldf and .mdf

Create database files Backup reading files

ASYNC_IO_COMPLETION

Engine Workers

VDI App BACKUP WITH SNAPSHOT Backup Sync media

Resource

Inside SQL Server Wait Types

Queries, Memory, and RESOURCE semaphores Resource

Hashes and sorts
Limited memory or too many concurrent users MEMORYCLERK_SQLQUERYEXEC and MEMORYCLERK_SQLQERESERVATIONS clerks dm_exec_query_resource_semaphores dm_exec_query_memory_grants RESOURCE_SEMAPHORE_SMALL_QUERY waits

RESOURCE_SEMAPHORE (Query Memory)

compiles

sys.dm_os_memory_brokers DBCC MEMORYSTATUS

Why are you compiling so much? Factor of limited memory or memory hungry compiles Throttled on a system of levels (gateways) with thresholds High Query Memory lowers thresholds Not often seen on 64bit systems

RESOURCE_SEMAPHORE_ QUERY_COMPILE

Inside SQL Server Wait Types

Pre-Emptive Waits
Workers go pre-emptive when calling external APIs that may take some time

May wrap more code than just the API

External

Windows API Xproc Status = RUNNING Wait_type = NULL

What does this look like preSQL 2008?

************************ * * BEGIN STACK DUMP: * 10/17/09 15:51:52 spid 0 * * Non-yielding Scheduler * ************************

What does this look like in SQL 2008?

Status = RUNNING Wait_type = PREEMPTIVE_XXXX

Can see along with normal wait type

We can go pre-emptive while another wait type is set (Ex. ASYNC_NETWORK_IO)

Inside SQL Server Wait Types

What are some I might see?

Type ~190 of these Description Scenario

PREEMPTIVE_OS_GETPROCADDRESS

Wraps calls to GetProcAddress() and xproc function Wraps calls to WriteFileGather() to zero out a section of a file Wrapped calls to LookupAccountSid()

Measure of xproc execution time

PREEMPTIVE_OS_WRITEFILEGATHER

Long autogrow for tlog file or database files (if not using instant file init) Mostly used during login authentication. Long waits could indicate DC issues. Helps fill in gaps where OLEDB wait not set.

PREEMPTIVE_OS_LOOKUPACCOUNTSID

PREEMPTIVE_OLEDBOPS

Wrapped around various code fragments that will call OLE-DB methods for linked server queries.. Wrapped around WaitForSingleObject() calls in network I/O and waiting for recovery to complete

PREEMPTIVE_OS_WAITFORSINGLEOBJECT

Can sometimes be seen in conjunction with ASYNC_NETWORK_IO

Inside SQL Server Wait Types

Extended Events and Waits

wait_info wait_info_external

wait_type

Normal waits

Preemptive waits

opcode
Timings

Begin and End

Duration (Total and Max are per scheduler)

Get query, session, or stack dump On by Default System_Health Session Has These SQLCAT Waits Stats Per Session Project
Inside SQL Server Wait Types

Where is THE LIST?

In a header file in the source code and in sys.dm_xe_map_values for SQL Server 2008 The BOL list

KB article on waittypes is only for SQL 2000 and prior

The plan

The Wait Type Repository Blog

Post new findings on this blog post Comment on the blog or send email to [email protected] Use the blog to update the BOL Blog may contain scenarios and more details

Inside SQL Server Wait Types

Resources
Our CSS Escalation Blog The Wait Type Repository Blog Post BOL reference on sys.dm_os_wait_stats SQLCAT Waits Stats Per Session CodePlex Craig Freedman blog posts on Parallelism CLR Wait Types blog post SQL Server 2005 Waits and Queues Whitepaper The System_Health XEvent Session Blog

Inside SQL Server Wait Types

Questions

Appendix

Inside SQL Server Wait Types

The General Algorithm

Developer writes code that runs Code is signaled to wake-up Code clears wait type and time.

Developer knows they might execute code that waits

Anyone querying the DMVS sees the wait type and accumulated wait time

last wait type in worker not cleared until task is complete

Developer sets a wait type

Developer calls SQLOS routines to wait

Inside SQL Server Wait Types

What does MDW tell you about I/O waits?

sync reads, sorts, SQLTrace I/O, load CLR assembly

Buffer Pool I/O for pages

Backups, Recovery, DBM

WRITELOG wait time = Log Flush Wait (perfmon)

LOGBUFFER is just waiting on folks waiting on WRITELOG 2

Inside SQL Server Wait Types

The Mapping for sysprocesses has changed

sys.sysprocesses.waittype is a binary value

KB 822101 wrong for 2005 and 2008

Binary to string mapping changed in SQL 2005

sys.dm_xe_map_values is to be used for the mapping

Inside SQL Server Wait Types

What about these?

Forced
SLEEP_TASK
Fixed time Hard to figure out scenario DBMIRROR_DBM_EVENT

Resource

Log shipping delayed to secondary

OLEDB Wrapped around linked server OLE-DB API calls Wait time will fluctuate since set and cleared for each call wait_resource is remote server and remote SPID PREEMPTIVE_XX type can now also show up

External

Sync

CMEMTHREAD

Thread synchronization for memory allocation High wait times = A likely bug

Hot stored proc in SQL Server 2005

2

Inside SQL Server Wait Types

The are other waits

Why cant I truncate the log
A poorly written DLL PRECONNECT status

backoffs in sys.dm_os_spinlock_stats

You decide to throttle

Inside SQL Server Wait Types

Thank you!

Dont forget to fill out your session evaluation forms