How Microsoft IT Implements New Storage

Designs for Exchange Server 2007

Published:
May 2008
Solution Overview
Business Challenge Solution Results/Benefits

The future Mailbox server Microsoft IT eliminated • Lower costs

design required a cost- shared storage as a
• Improved reliability
efficient solution to critical single point of
support greater mailbox failure by maintaining • Simplified maintenance
sizes. Increased mailbox separate up-to-date and troubleshooting
sizes made it necessary copies of mailbox data at
• Higher performance and
to provide 10 times more all times. This provided
lower I/O requirements
data capacity on mailbox an opportunity to replace
servers and implement SAN technology with • Larger mailbox size
redundancy at the DAS equipment and capacities andmore
storage level to support employee mailboxes per server
eliminate the need for productivity with
• End-to-end messaging
restores from backup as substantially larger
the primary recovery mailboxes that service ownership from
mechanism after a accommodated higher server and storage
storage failure. redundancy levels. design to operations
procedures
Products and Technology

● Windows Server 2003

● Microsoft Exchange Server
2007
● Microsoft System Center Data
Protection Manager 2007
● Storage Area Networks (SANs)
● Direct-attached storage (DAS)
● Cluster continuous replication
(CCR)
Agenda
● Situation
● Solution Requirements
● Advantages of DAS-Based Storage Designs

● Microsoft IT Storage Design

● Best Practices
● Conclusion
● For More Information
SITUATION
Though the development team was excited about CCR and its potential
to support larger mailboxes at lower storage costs, provide shorter
failover times, reduce the need for restores from backup, and
decrease storage complexity and eliminate maintenance overhead,
Microsoft IT initially hesitated to embrace CCR on DAS due to the
following concerns:
● Need to protect existing IT investments   
● Desire to capitalize on existing expert knowledge   
● Perception that DAS was not an enterprise storage technology and that
CCR was not enterprise ready   
● Concerns that DAS would create storage silos and hidden operational
costs   
● Fear that replication latencies would introduce the potential for data
loss   
Business Requirements

● Microsoft Messaging Statistics and

Projections
● Increased Mailbox Capacities
● Service Level Agreements    
Business Requirements
Microsoft Messaging Statistics and Projections
Category 2002/2003 2003/2004 2004/2005 2005/2006 2006/2007 2007/2008
Total mailboxes 71,000 80,000 95,000 110,000 130,000 147,000

Microsoft Exchange
ActiveSync® users Not applicable 6,000 13,000 21,000 31,000 48,000
per month

Outlook® Anywhere Not applicable 20,000 25,000 60,000 60,000 100,000

users per month

Internet message
submissions per day 6,000,000 9,000,000 11,300,000 13,000,000 13,500,000 30,000,000
(unfiltered)

Blocked message 2,500,000 7,500,000 10,000,000 10,500,000 11,000,000 28,000,000

submissions per day

Maximum message 2 MB 5 MB 10 MB 10 MB 10 MB 10 MB

size

E-mail volume per

user per calendar 10 MB 15 MB 15 MB 20 MB 20 MB 26 MB
day

Number of clustered
Mailbox servers 113* 38 34 30 62 34

Typical mailbox 500 MB or 500 MB or

quota 100 MB 200 MB 200 MB 200 MB 2 GB 2 GB

Total mailbox data 7 terabytes 17 terabytes 19 terabytes 22 terabytes 60 terabytes Up to 300

terabytes
Business Requirements
Increased Mailbox Capacities

Demographic messaging profile at Microsoft

User Type Messages Received Range Percent

Light Messages received <= 20 43.7

Medium 20 < Messages received <= 40 9.6

Light users
Heavy 40 < Messages received <= 80 15.4

Very heavy 80 < Messages received <= 120 10.0

Heaviest 120 < Messages received 21.2

Medium
users Heaviest User Type Messages Sent Range Percent
users
Light Messages sent <= 5 37.3
Very
Medium 5 < Messages sent <= 10 15.4
Heavy heavy
users users Heavy 11 < Messages sent <= 20 19.6

Very Heavy 21 < Messages sent <= 30 11.4

Heaviest 30 < Messages sent 16.3

Business Requirements
Service Level Agreements
Organization-Wide SLAs with Impact on Mailbox Server Designs
Service level definition Resolution target Comments

This SLA gives an end-to-end view of messaging as a managed

service and includes Mailbox server availability as well as the
End-to-end availability of availability of Client Access servers, Active Directory, and the
messaging services 99.99 percent or greater network infrastructure. On Mailbox servers, Microsoft IT measures
the availability of system services based on stop and start events
and the availability of messaging databases based on events
generated by the Exchange Information Store service.

This SLA defines client availability as the percentage of successful

End-to-end client availability 99.5 percent or greater remote procedure call (RPC) activity in relationship to failed RPC
activity between Microsoft Office Outlook clients or Client Access
servers and Mailbox servers.

This SLA requires RPC client/server operations between Outlook

End-to-end client performance 95 percent or greater clients or Client Access servers and Mailbox servers to finish in
less than two seconds.

Business will continue with One hour or less This SLA requires individual database restores through reseeding
messaging service or from backup to finish in less than one hour.

Retention of mailbox database This SLA enables Microsoft IT to discard Exchange Server
backups 14 days or more backups after 14 days. Microsoft IT does not use database
backups for archiving purposes.

Retention of deleted items 14 days or more This SLA influences the calculation of server capacity needs, as
explained later in this white paper.
ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
● Increased Exchange data resilience
● Ongoing backup operations during regular
business hours
● Reduced reliance on traditional backups to
restore data  
● Simplicity in the storage design and low
maintenance overhead      
ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
Increased Exchange Data Resilience through Redundancies
Failover cluster

4 Active nodes
1 Primary passive node
2 Secondary passive nodes
4,000 Mailboxes
per Mailbox server
(active node)

SAN SAN
switched switched
fabric A fabric B
Microsoft IT server SAN subsystem
cluster configuration
for Exchange Fibre
Channel
Server 2003 Mailbox ` switches

servers Control Console

SAN storage array Exchange

information store
databases of four
Mailbox servers

Fibre Channel disks for

data and hardware VSS
backups (RAID 10)
ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
Increased Exchange Data Resilience through Redundancies-
continued
CCR-based Mailbox server configuration

Transport dumpster Hub Transport server File share witness

Public network

Private network
RAID RAID
CCR

Transaction log
shipping

RAID 10 across enclosures RAID 10 across enclosures

Active node Passive node
Mailbox server
ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
Ongoing Backup Operations During Regular Business Hours
Mailbox server

Data Protection
Manager 2007
Data Protection Active node Passive node

Manager 2007– DPM 2007 agent DPM 2007 agent

based backup
on passive Mailbox server
nodes

Active node Passive node

DPM 2007 agent DPM 2007 agent

ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
Reduced Need for Restores from Backup
The typical CCR-based recovery scenario includes four phases:
● Normal operation   
Mailbox server is available and new transactions result in new
transaction log files on the active node.
● Lossy failover and recovery   
Active node fails right after a Hub Transport server delivered
messages and before CCR had a chance to replicate the current
transactions.
● Node repair   
Mailbox server is up to date and running on the remaining cluster
node while Microsoft IT performs the necessary repair activities on the
failed node.
● Normal operation   
Mailbox server resumes normal operations after the failed node is
repaired, and any affected messaging databases are reseeded.
ADVANTAGES OF DAS-BASED STORAGE
DESIGNS
Predictable Mailbox Server Performance
Different usage patterns on shared storage

Other application Shared physical media Exchange Server

Sequential I/O in large chunks

of data (megabytes)

Random database I/O in small

chunks of 8 kilobytes
MICROSOFT IT STORAGE DESIGN

● Design Emphasis
● Mailbox Server Performance
● Mailbox Store Capacity
● Standardized Storage Layout
● Microsoft IT Scalability Experiment
MICROSOFT IT STORAGE DESIGN
Design Emphasis

● Reliability  
● Availability   
● Performance   
● Capacity   
● Costs   
● Simplicity  
● Recoverability   
● Scalability   
 MICROSOFT IT STORAGE DESIGN
Mailbox Server Performance
Outlook client

Database buffer
cache and
database I/O
Client requests and server responses
Mailbox server

Database buffer cache

Database I/O

Database files on disk

MICROSOFT IT STORAGE DESIGN
Mailbox Store Capacity

Mailbox server capacity needs per cluster node: 13,210.10 GB

Transaction logs (~10% of database drive capacity):

16 disks / 2 (for RAID 10)
* 137.2 GB/disk
= 1,097.60 GB
Storage capacity
Unexpected growth (10% of mailbox data):
needs for 4,000 8,000 GB
mailboxes with 2-GB * 10 / 100
= 800 GB
quotas at Microsoft IT
Content indexes (5% of mailbox data):
8,000 GB
* 5 / 100
= 400 GB

Database overhead (20% of database size):

8,000 GB + 1,093.75 GB
* 20 / 100
= 1,818.75 GB

Deleted items retention (20 MB per user per day):

20 MB / day * 14 days
* 4,000 Mailboxes
= 1,093.75 GB

Mailbox data:
4,000 Mailboxes
* 2 GB Quotas
= 8,000 GB
MICROSOFT IT STORAGE DESIGN
Standardized Storage Layout
Mailbox server
(4,000 users
2 GB quotas)

Active node
Passive node

2 x Quad-Core
Intel Xeon X5355 16 GB RAM 2 x Quad-Core
Intel Xeon X5355 16 GB RAM

Microsoft IT

RAID
RAID

RAID

RAID
Mailbox server
design for E: F: G: Q: E: F: G: Q:

4,000 users
with 2-GB
quotas H: I: J: R: H: I: J: R:

K: L: M: S: K: L: M: S:

N: O: P: T: N: O: P: T:
MICROSOFT IT STORAGE DESIGN
Microsoft IT Scalability Experiment
Mailbox server
(6,000 users
500 MB quotas)

Active node
Passive node

2 x Quad-Core
Intel Xeon X5355 16 GB RAM 2 x Quad-Core
Intel Xeon X5355 16 GB RAM

Microsoft IT
Mailbox server
design for 6,000
RAID

RAID
users with 500-MB
quotas
E: F: G: K: E: F: G: K:

H: I: J: L: H: I: J: L:
Best Practices

● Eliminate storage as a critical single point of failure  

● Use enterprise storage hardware
● Focus on simplicity
● Standardize the storage layout 
● Design the storage subsystem for both capacity and
performance needs 
● Perform backups on the passive node
● Keep the size of individual messaging databases
small 
● Continuously monitor Mailbox servers
Summary

● CCR enables enterprise IT organizations to achieve

significant improvements in the Mailbox server design.
● It provides new design options to balance processor,
memory, and storage resources in order to achieve an
optimal system performance while remaining cost efficient.
● It substantially simplifies the storage design
● CCR provides Microsoft IT with the ability to eliminate the
Exchange databases as a critical single point of failure
● CCR reduces the need for restores from backup.
● CCR on DAS eliminates maintenance overhead.
● CCR on DAS is designed once and never touched again.
For More Information

● Additional content on Microsoft IT

deployments and best practices can be
found on http://www.microsoft.com
● Microsoft IT Showcase Webcasts
http://www.microsoft.com/howmicrosoftdoesitw
ebcasts
● Microsoft TechNet
http://www.microsoft.com/technet/itshowcase
This document is provided for informational purposes only.
MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

© 2006 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS
SUMMARY. Microsoft, Microsoft Press, Visual Studio, Visual SourceSafe, Windows and Windows NT are either registered trademarks
or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products
mentioned herein may be the trademarks of their respective owners.