<Insert Picture Here>

Levent Yavuz, ACS Sen.Principal Engineer

Support Engineer, 18+ years
GCS, ASM BDE 10+ years
GCS, Exadata BDE 4+ years
An internal look into ASM
metadata, processes and tools
 AGENDA

• ASM Architecture

• ASM Metadata

• Tools: kfed, amdu

• Creating a fake ASM instance

 What is ASM?

• A Database File System which provides cluster file system and

volume manager capabilities that are integrated into the Oracle
database kernel (since 10g version) at no additional cost

APPL1 APPL2 APPL3 APPL1 APPL2 APPL3

DB1 DB2 DB1 DB2

File System
ASM
LVM
OPERATING SYSTEM OPERATING SYSTEM
 Automatic Storage Management

• ASM is a file system and volume manager built into the database kernel
that allows the practical management of disk drives with high
availability.

• It is linked into “/bin/oracle” so that its code may be executed by all

database processes

• First portion of ASM code allows for the startup of a special instance
called “ASM Instance”

• Second portion of ASM code run in the database instance, in the I/O
path.
Segment <- Extent <- File

Files (Automatic Storage Management)

Extents (Dictionary Managed, Locally (Bitmap) Managed)

Tables (Freelists, ASSM)

 What is a ASM instance?
• How ORACLE knows what is what?
init+ASM.ora: INSTANCE_TYPE=ASM
• Oracle instance to manage the metadata for disk groups
• All metadata modifications are done by an ASM instance
to isolate failures
• Database instances connect to ASM instance to create,
delete, resize, open, or close files
• Database instances read/write directly to disk
• One ASM instance per node in a cluster
• ASM instance failure kills attached database instances
 ASM specific BG Processes
ASMB provide info to and from CSS, heartbeat mechanism

RBAL performs rebalancing of disk resources controlled by ASM

ARBx do the actual rebalancing of ASM disks resources, managed by RBAL

GMON maintains disk membership in ASM disk groups.

MARK marks AUs as stale following a missed write to an offline disk. This
essentially tracks which extents require resync for offline disks.

DBWn, LGWR, PMON, SMON Additional processes with different tasks

 ASM Metadata

• Describes the disk group and files

• Resides inside the disk group so it is self describing
• Each metadata block is 4K.
• Cached in ASM instance SGA
 ASM Metadata

There are three types of ASM metadata

• On Disk structures
• physically addressed metadata
• virtually addressed metadata

• In Memory structures
 Disk Metadata Structures
• Two types of On disk Metadata structures
• Physically addressed metadata –only relevant to a single disk and
thus is addressed directly by physical block number relative to the disk
number.

• Virtually addressed metadata – is metadata that is relative to the file

number and requires address translation.
 A Datafile with 3MB size => 4 AU‘s (4MB)

disk0 disk1 disk2

AU 0

... E X T 2
AU 86

AU 84 E X T 0 AU 85 E X T 1
AU 87 E X T 3

Physically Addressed
Virtually Addressed
Files
 Block 0 1 2 3 ............ 255

AU0 DH FST AT 1MB

AU1 PST PST

LH FD FD
AU2
AU3 DD DD DD

AU4 ACD ACD ACD

COD
...
50MB
AU50 D A T A F I L E S
...
 On Disk Metadata Structures
• All ASM metadata is broken down into blocks

• Each block is 4096 bytes, with 32 bytes as the block header

0 1 2
AU ....
1MB kfbh.endian: 1 ; 0x000: 0x01 255
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.datfmt: 2 ; 0x003: 0x02
kfbh.block.blk: 512 ; 0x004: blk=512
kfbh.block.obj: 2147483698 ; 0x008: disk=50
kfbh.check: 2183462039 ; 0x00c: 0x8224fc97
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
....
....
 Disk Metadata Structures
• Currently 17 block types:
– Disk Header
– Partnership and Status Table
– Allocation Table
– Free Space Table
– File Directory
– Disk Directory
– Active Change Directory (ACD)
– Continuing Operation Directory (COD)
– Template Directory
– Alias Directory
– Indirect Block

• Not all blocks are present on every disk

 Physically addressed metadata
(AU0 + AU1)

• ASM disk header AU0B0 (block type 1)

• Partner Status Table AU1B0 (block type 17/13, 13 is pst_none)
• Space Allocation Table AU0B2 (block type 3)
• Free Space Table AU0B1 (block type 2)
 Disk Header
0 1 2
AU ....
255
• Written to block 0 of every ASM disk
• Recognized by discovery
• Contains attributes of the disk group
• Disk group name and creation timestamp
• Allocation Unit size and Metadata block size
Redundancy type (external, normal, or high)
• Timestamp of creation and last mount
• Describes the disk
• ASM disk name and number
• Failure group name
• Size of disk in allocation units
• Root extent pointer (f1b1 location) if present
• A copy is stored at aun = 1 and blkn = ausize/4096 – 2
BLK254, BLK510, BLK1022 for AUSIZE 1M, 2M, 4M

Oracle Confidential
 Disk header (AU0,Block 0)
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: ASMDISK1_0000 ; 0x028: length=13
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 2000 ; 0x0c4: 0x000007d0
kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
 Binary dump of ASM Disk header
BYTE 0 1 2 3 4 5 6 7 8 9 A B C D E F
00000000 01 82 01 01 00 00 00 00 00 00 00 80 8A 8B D0 AE 01: Little/Big Endian
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 82: Metadata blocksize
00000020 4F 52 43 4C 44 49 53 4B 00 00 00 00 00 00 00 00 Driver being used
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000040 00 00 10 0A 00 00 01 03 41 53 4D 30 31 5F 30 30 Disk Name (32 bytes)
00000050 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060 00 00 00 00 00 00 00 00 41 53 4D 30 31 00 00 00 Diskgroup name (32 bytes)
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000080 00 00 00 00 00 00 00 00 41 53 4D 30 31 5F 30 30 Failgroup name
00000090 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000C0 00 00 00 00 00 00 00 00 8E 10 F5 01 00 70 BA 5C Creation timestamp
000000D0 8E 10 F5 01 00 AC 36 5D 00 02 00 10 00 00 10 00 AU size
000000E0 80 BC 01 00 73 23 00 00 02 00 00 00 01 00 00 00 Disk size(# of AU units)
000000F0 02 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Note: 284757.1 - ASM Disk Header Block Dump Analysis (kfed)

Oracle Confidential
 Allocation and Space Tables
0 1 2
AU0 FST AT .... AT
255

• Allocation tables contains an entry for each disk allocation unit.

• Allocation Table and File Extent Map must always be in-sync

• Free Space Table list blocks that in allocation table that contain free
allocation unit
 Partner Status Table (PST)
0 1 2
AU1 PST PST ... .... HBEAT
255

• Maintains membership of disks in disk group and status of the

disk (online/offline)
• Contains vital information for crash recovery
• Replicated on n number of disks for protection
• When a disk group is mounted the PST is read right after
discovery
– similar to a file system super block
 PST Heartbeat
0 1 2
AU1 HBEAT
255
• ASM heartbeats PST every 3 seconds
• Heartbeat writes are atomic writes
• Heartbeat prevents two instances in different clusters from
mounting the same disk group
– Second instance will hit ORA-15003
• Heartbeat verified at disk group mount
 Partner Status Table (PST)

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 17 ; 0x002: KFBTYP_PST_META
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 256 ; 0x004: T=0 NUMB=0x100
kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check: 3542934828 ; 0x00c: 0xd32ce52c
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
...
kfdpHdrB.time.hi: 32868652 ; 0x000: HOUR=0xc DAYS=0x9 MNTH=0x2 YEAR=0x7d6
kfdpHdrB.time.lo: 1364950016 ; 0x004: USEC=0x0 MSEC=0x2df SECS=0x15 MINS=0x14
...
ub1[2]: 0 ; 0x026: 0x00
ub1[3]: 0 ; 0x027: 0x00
 Virtually addressed metadata
(AU2 .. AU255)
• File directory
• ASM disk directory
• Active Change Directory (ACD)
• Continous Operations Directory (COD)
• Template Directory
• Alias Directory
• Indirect Block
 File directory

• Contains 1 entry per ASM file - this includes ASM metadata

files and database files.
• The File Directory is a self describing file
• The entry in the File directory points to the file data extents.
• Each entry contains information about the file. This is
reflected in v$ASM_FILE
• Is triple mirrored with ASM normal & high redundancy
• Files 1 – 255 are reserved for internal use
• Files 256 and up are for database files
• The File directory is File #1
 ASM Disk directory

• Contains 1 entry per ASM disk in the diskgroup

• This same information is located in the disk header, except
the Disk Directory includes disk status
• The Disk directory is file # 2
 Active Change Directory
• It’s the redo log of ASM. Records atomic changes to the
metadata files

• Examples of ACD records –

– Add or drop – templates, files or aliases
– Allocate/de-allocate extents

• ACD is file # 3
 Continous Operations Directory
• Used for ASM instance recovery
• Logs long/large changes to the ASM structure that cannot be done
atomically.
• Changes can be initiated by database instance or ASM instance
• Two types of operations – background & rollback
• background operations are invoked by the ASM instance to perform
diskgroup maintenance; such as rebalance
• Rollback operations are instigated by the database instance; examples
include adding or deleting files, templates, or aliases.
• In rollback operations the RBAL (db instance) actually performs the
operations.
• COD is file # 4
 Template and Alias Directory

• Template directory describes the attributes of the system and

user defined templates

• Alias Directory describes system and user defined aliases

 ASM Tools

• kfed
• dbverify
• examine corrupt diskheader (kfed)
• OS blockdump (dd)
• amdu
 kfed – kernel files editor
COMMAND:
kfed read /dev/rdsk/c0t0d0s7
this will dump out ASM disk header block

note:
you have to link kfed tool in 10g before starting (it‘s not available
with normal installation)
Note 284646.1 - Creating and using the kfed utility to view ASM
disk header
Note 284757.1 - ASM Disk Header Block Dump Analysis
 kfed, kfod – quick reference
• operations
read, write, merge
Command:
kfed read /dev/rdsk/c0t0d0s7
this will dump out ASM disk header block

• command line parameters

aunum
blknum
ausize
dd
• ASM instance has to be down dd can be used against
ASM file directly
$ dd if=/dev/rdsk/c0t0d0s7 of=asm.dat
count=100
• dd can be used to extract the block from a ASM file copy
of the datafile (RMAN)
$ dd if=df_7 of=f7b65.dd count=1 skip=65
bs=8192
• put the patched block back into datafile
$ dd if=f7b65.dd of=df_7 bs=8192 seek=65
count=1 conv=notrunc
 How to identify corrupt header
• corrupt disk header has kfbh structures only. kfdhdb structures are missing completely
kfbh.type = 1 -> KFBTYP_DISKHEAD -> ok.
kfbh.type <> 1 -> corruption or unformated
e.g. 77 ; 0x002: *** Unknown Enum ***

compare other flags in kfbh structure.

e.g. kfbh.endian = 0x00 and kfbh.hard = 0x82
If not, this is could be corruption hint
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message
 AMDU - UDMA
AMDU is an utility to extract all the available metadata from one or more ASM disks and/or generate
formatted printouts of individual blocks.The utility does not require that any diskgroup to be mounted.

Oracle Support may ask you to provide AMDU output, especially when facing internal errors related to
the ASM metadata

NOTE:
AMDU executable comes with installation from 11.1 onwards. Although this tool
was released with 11g, it can be used with ASM 10g as well.
UDMA (Internal) is the opposite utility
The usual syntax used for dumping metadata related to a diskgroup is: of AMDU. It can be used to recreate a
diskgroup's disk(s) using AMDU's
amdu -diskstring '<your_path_to_ASM_disks' -dump '<diskgroup>'
previously extracted metadata. Since the
recreated disks will be sparse files
containing only ASM metadata, UDMA can
be used to replicate a customer's
diskgroup in Oracle without having any
$ amdu -diskstring 'o/*/DATA*' -dump 'DATA' -noimage compromisable data, and thereby help to
diagnose issues.
 CreateASMInstanceEveryWhere
1. Create pfile init+MYA.ora

2. Create ASM disks and set file permissions

3. Start ASM init+MYA.ora

*._ASM_ALLOW_ONLY_RAW_DISKS=FALSE
*.instance_number=99
4. Create ASM Diskgroup *.instance_type='asm'
*.large_pool_size=12M
*.db_unique_name=MYA
*.remote_login_passwordfile='EXCLUSIVE'
*.asm_diskstring=' =/export/home/oracle/lyavuz/disks/disk*'
*.processes=25
*.sga_max_size=300M

Oracle Confidential
 CreateASMInstanceEveryWhere
1. Create pfile init+MYA.ora

2. Create ASM disks and set file permissions

3. Start ASM

4. Create ASM Diskgroup

# dd if=/dev/zero of=/export/home/oracle/lyavuz/disks/disk1 bs=1M count=200

# dd if=/dev/zero of=/export/home/oracle/lyavuz/disks/disk2 bs=1M count=200
# dd if=/dev/zero of=/export/home/oracle/lyavuz/disks/disk3 bs=1M count=200
# dd if=/dev/zero of=/export/home/oracle/lyavuz/disks/disk4 bs=1M count=200

# chown oracle:oinstall /export/home/oracle/lyavuz/disks/disk*

Oracle Confidential
 CreateASMInstanceEveryWhere
1. Create pfile init+MYA.ora

2. Create ASM disks and set file permissions

$ export ORACLE_SID=+MYA
$ sqlplus / as sysasm
3. Start ASM
SQL> startup pfile=init+MYA.ora
ASM instance started
4. Create ASM Diskgroup
Total System Global Area 313159680 bytes
Fixed Size 2227944 bytes
Variable Size 285765912 bytes
ASM Cache 25165824 bytes
ORA-15110: no diskgroups mounted

ps -ef|grep MYA
oracle 5012 17024 0 14:23 pts/8 00:00:00 grep MYA
oracle 7213 1 0 13:48 ? 00:00:00 asm_pmon_+MYA
oracle 7268 1 0 13:48 ? 00:00:00 asm_psp0_+MYA
oracle 7914 1 0 13:48 ? 00:00:00 asm_vktm_+MYA
oracle 7972 1 0 13:48 ? 00:00:00 asm_gen0_+MYA
oracle 8017 1 0 13:48 ? 00:00:00 asm_diag_+MYA
oracle 8068 1 0 13:48 ? 00:00:00 asm_dia0_+MYA

Oracle Confidential
 CreateASMInstanceEveryWhere
1. Create pfile init+MYA.ora

2. Create ASM disks and set file permissions

3. Start ASM SQL> select path from v$asm_disk where path like '%lyavuz%';

PATH
4. Create ASM Diskgroup --------------------------------------------------------------------------------
/export/home/oracle/lyavuz/disks/disk2
/export/home/oracle/lyavuz/disks/disk3
/export/home/oracle/lyavuz/disks/disk1

SQL> create diskgroup data1 external redundancy disk

'/export/home/oracle/lyavuz/disks/disk[1-3];

Diskgroup created.

SQL>

Oracle Confidential
 CreateASMInstanceEveryWhere
1. Create pfile init+MYA.ora
SQL> select name,state,type,total_mb,free_mb from v$asm_diskgroup;
2. Create ASM disks and set file permissions
NAME STATE TYPE TOTAL_MB FREE_MB
------------------------------ ----------- ------ ---------- ----------
3. Start ASM DATA1 MOUNTED EXTERN 600 546

4. Create ASM Diskgroup

PATH
--------------------------------------------------------------------------------
NAME TOTAL_MB FREE_MB OS_MB HEADER_STATU
------------------------------ ---------- ---------- ---------- ------------
/export/home/oracle/lyavuz/disks/disk4
0 0 200 CANDIDATE

/export/home/oracle/lyavuz/disks/disk3
DATA1_0000 200 182 200 MEMBER

/export/home/oracle/lyavuz/disks/disk1
DATA1_0001 200 182 200 MEMBER

/export/home/oracle/lyavuz/disks/disk2
DATA1_0002 200 182 200 MEMBER

Oracle Confidential