Ad disasters & how to prevent them

AD Disasters …and How to Prevent Them! Greg Shields, MVP, vExpert Head Geek, Concentrated Technology www.ConcentratedTech.com

This slide deck was used in one of our many conference presentations. We hope you enjoy it, and invite you to use it within your own organization however you like. For more information on our company, including information on private classes and upcoming conference appearances, please visit our Web site, www.ConcentratedTech.com . For links to newly-posted decks, follow us on Twitter: @concentrateddon or @concentratdgreg This work is copyright ©Concentrated Technology, LLC

Agenda Topics Part I: Hardware & Software Failures Part II: Human Errors Part III: Complete Disasters

Three Types of Disasters Hardware & Software Human Error Complete Disasters Microsoft Screwed Up You Screwed Up Somebody REALLY Screwed Up

Three Types of Disasters Hardware & Software Human Error Complete Disasters Increasing Problem Complexity Increasing Troubleshooting Complexity

Part I Hardware & Software Failures

Morphed SYSVOL Folders Problem: When SYSVOL replication finds a name conflict, one of the folders in conflict is renamed to foldername_ntfrs_???????? (hex) This can break the links to that folder. This can occur when users attempt to manually replicate folders, two users add folders of the same name at the same time, or during an improper restore of the SYSVOL.

Morphed SYSVOL Folders Solution: Three-steps: Rename all morphed folders to new names and allow replication of the new names to fully complete. This ensures a common name for the folder is available on all DC ’s and that the new names and GUID’s match. Once replication has completed, look in the folders and determine which is correct and which does not belong. Rename the correct folder back to its original name and again allow replication to complete. Delete the unnecessary folder. This is OK as FRS tracks files by their GUID.

Broken GPT/GPC Linkages Problem: Group Policy Objects are made up of two parts, the Group Policy Template and the Group Policy Container. GPC ’s are stored in Active Directory and replicate through AD replications. GPT ’s are stored in the SYSVOL and replicate through FRS. Broken GPT/GPC linkages can cause GPO ’s to malfunction and should be fixed. Same with mismatched version numbers.

Broken GPT/GPC Linkages Solution: Use GPOTOOL.EXE from the Resource Kit Tools to identify GPT ’s/GPC’s that are not synchronized. GPOTOOL.EXE with no switches validates and reports on the GPT/GPC linkage. If someone has accidentally changed permissions on the GPT, you can also use the /checkacl switch. GPMC will notify when permissions are not consistently set and request to reset those permission. GPMC will reset the permission on the GPT to match the permission on the GPC.

DNS Aging & Scavenging Not Enabled Problem: If DNS Aging & Scavenging is not enabled on a domain, stale DNS records caused by DHCP lease changes can pile up. Group Policy application as well as correct name resolution requires a one-to-one mapping between FQDN and IP. Stale DNS records mean multiple IP ’s per host and/or multiple host records per IP. Systems management tools like SMS can fail.

Aging & Scavenging When DNS was static, keeping active and inactive records straight was a nightmare. Now that DNS is dynamic, inactive recordkeeping is improved, when configured correctly. Aging All dynamically updated resource records have a time stamp That time stamp is reset whenever a record is created, modified, or refreshed. Windows hosts refresh their record… At startup At DHCP lease renewal Every 24 hours

Aging & Scavenging Windows DNS servers that accept dynamic updates need to have Scavenging enabled or records will quickly grow stale. This is especially problematic if DHCP is active and has a short lease time. Be aware the DNS scavenging on AD-integrated zones can have an impact on AD replication. Refresh Interval If a client does not refresh its record by the end of this period, the scavenging process will remove the record. No-Refresh Interval A period of time before the refresh interval where client refreshes are ignored by the server. This is done to reduce DNS replication requirements. Scavenging increases AD replication

Aging & Scavenging 7 Days Record Created 7 Days 7 Days 7 Days 7 Days 7 Days 7 Days 7 Days No Refresh Interval Refresh Interval Refresh Accepted Refresh Accepted Record Deleted Time

Aging & Scavenging Global Setting Per-Zone Setting Let ’s discuss strategies for Aging & Scavenging in Small, Large, & Enterprise Networks…

DNS Aging & Scavenging Not Enabled Solution: Enable DNS Aging & Scavenging on all zones populated by DHCP. DNS Aging & Scavenging enabled in two locations. Global Setting Per-Zone Setting

DNS Aging & Scavenging Not Enabled Solution: Use DNSCMD.EXE command-line tool to automatically age and scavenge all records after enabling Aging & Scavenging DNSCMD.EXE ageallrecords DNSCMD.EXE startscavenging

Disable Unused Network Cards Problem: Unused network cards can auto-populate DNS with incorrect entries. With regular servers this doesn ’t often cause a big problem, but with DC’s, auto-registration populates SRV records as well. This can cause bad resolution to DC services. Solution: Disable any unused network cards. Disabling unused network cards prevents them from registering their incorrect values into DNS.

Tombstones & Zombies Problem: When an AD object is deleted, it goes into a special container called “Deleted Items”. It’s movement there is replicated. The object is not removed until the tombstone lifetime is exceeded. Windows 2000 tombstone lifetime is 60 days Windows 2003 tombstone lifetime is 180 days Upgraded Windows 2003 tombstone lifetime is still 60 days. When a DC comes back on-line after being down for longer than the tombstone lifetime or a restore from a tape older than the tombstone lifetime, zombies are created.

Tombstones & Zombies Solution: Never bring on-line a DC that ’s been down for greater than 60/180 days. Never use tapes to restore objects older than 60/180 days. If you do, you ’re in a world of hurt. … but what if you forget…?

Lingering Objects Problem: So, you ’ve gone ahead and accidentally reanimated a tombstoned object? What now? Reanimation of these lingering objects can break replication in some cases.

Lingering Objects Solution: Use REPADMIN.EXE /REPLSUM from the Support Tools to verify if lingering objects are resident in Active Directory. REPADMIN.EXE to remove them. These tools only work on W2003. Step 1: Find the GUID of a DC: repadmin.exe /showrepl Step 2: Check for lingering objects: repadmin.exe /removelingeringobjects * <DC GUID> dc={mydomain},dc={com} /advisory_mode Step 3: Remove any lingering objects found: Remove the /advisory_mode switch from Step 2.

Lingering Objects Solution: Use REPADMIN.EXE /REPLSUM from the Support Tools to verify if lingering objects are resident in Active Directory. REPADMIN.EXE to remove them. These tools only work on W2003. Step 4: Enable strict replication consistency. Strict replication consistency is only enabled by default on 2003 DCs (not upgraded) that were promoted into a Forest that was built as 2003 (not upgraded from 2000). All other DCs will only have this setting enabled manually. Enable strict replication consistency on all DC ’s by setting the DWORD value for Strict Replication Consistency to 1 at the key HKLM\System\CurrentControlSet\Services\NTDS\Parameters.

Improper Time Synchronization Problem: Time synchronization is critical for Kerberos authentication and many applications. Time skew greater than 5 minutes can prevent logins and cause log files to barf. Users with administrator rights can reconfigure time sync to another time server. Very slight differences in time between stratum 1, 2, and 3 servers, usually caused by Internet conditions. Using different time servers in a network can cause problems for time-sensitive network applications.

Improper Time Synchronization Solution: Configure all machines in the domain to synchronize against the same time server. Choose to use NT5DS or NTP mode, but choose one for all systems. NT5DS is accurate to ~20 seconds. NTP can be accurate to <1 second. Some applications require greater time resolution, so consider a 3 rd party time sync tool with an on-site stratum 3 time device. “ Domain Time” from Symmetriccom

Bad DNS SRV Records Problem: Improperly decommissioning DC ’s can lead to their SRV records not being expunged from the DNS database. Also, missing DNS SRV records can prevent AD from functioning properly. This can cause error messages in the Event Log, replication problems, etc. due to the missing server. This happens most often when AD DNS is not hosted on Windows and dynamic updates are not enabled.

Bad DNS SRV Records Solution: Ensure DNS SRV records are consistent. Use ipconfig /registerdns to force DC to re-register DNS SRV records along with it ’s A records. Be careful of multiple interfaces on DC ’s. Disable any unused interfaces. Unused interfaces can register themselves in DNS. Bridged interfaces can cause routing problems. Delete stale DNS SRV records from DNS database (you ’ll know which are stale).

Orphaned Domains & DC ’s Problem: Old domains and Domain Controllers are still resident in Active Directory. These extra domains are unnecessary, can cause Event Log errors and odd problems during contact attempts. Orphaned DC ’s can prevent a domain from being decommissioned. Orphaned DC ’s in child domains can prevent a parent domain from being decommissioned.

Orphaned Domains & DC ’s Solution: Remove the offending Domains and/or DC ’s from your infrastructure. This is a multi-step process. NTDSUTIL.EXE to remove from Active Directory ADSIEDIT.MSC to remove from LDAP DNSMGMT.MSC to remove from DNS

Orphaned Domains & DC ’s Solution: Remove the offending Domains and/or DC ’s from your infrastructure. This is a multi-step process. NTDSUTIL.EXE to remove from Active Directory NTDSUTIL METADATA CLEANUP CONNECTIONS CONNECT TO SERVER {SERVER NAME} QUIT SELECT OPERATION TARGET SELECT SERVER {SERVER NAME} REMOVE SELECTED SERVER | QUIT

Orphaned Domains & DC ’s Solution: Remove the offending Domains and/or DC ’s from your infrastructure. This is a multi-step process. ADSIEDIT.MSC to remove from LDAP. This step is required if the domain is not at W2003 SP1. Navigate to DC={MYDOMAIN},DC={COM},OU=DOMAIN CONTROLLERS Delete the offending Domain Controller. DNSMGMT.MSC to remove from DNS Delete any FQDN ’s and/or associated GUID’s related to that DC.

Stale AD Site Links Problem: AD Site Links are usually created and managed by the KCC. However, some administrators want to get their hands in on replication. Once Site Links are manually created, the KCC no longer manages them, which can cause them to grow stale as the network changes.

Stale AD Site Links Solution: (Except in the very largest of networks) Remove any manually created Links and allow the KCC to manage links. In Windows 2003 SP1, the link-managing capabilities of the KCC are improved by multiple orders of magnitude. Older versions in larger networks had timing problems with KCC optimization passes. Also, improperly decommissioned DC ’s may need to be removed from AD S&S.

No DNS Reverse Zones Problem: DNS reverse zones must be enabled for proper functionality of Active Directory. Needed so clients can identify the site they reside in. Needed so clients can find the closest DNS server. Needed for correct processing of some attributes of Group Policy.

No DNS Reverse Zones Solution: Enable DNS reverse zones for each zone active in your network infrastructure. Ensure that all zones have similar configuration and dynamic updates enabled. Don ’t forget Aging & Scavenging.

DSRM Passwords Unknown Problem: Directory Services Restore Mode passwords are set individually on each Domain Controller as that controller is DCPROMO ’ed. This is arguably the most forgotten password in a Windows network because it is only used again during a restore operation. Not having this in a crisis can inhibit restoration activities. Who here knows their DSRM password?

DSRM Passwords Unknown Solution: Run NTDSUTIL.EXE to reset DSRM passwords before a failure occurs. NTDSUTIL.EXE SET DSRM PASSWORD RESET PASSWORD ON SERVER {Server Name} {Enter New Password} {Re-Enter New Password} QUIT / QUIT (Consider “bagging” the password…)

DSRM Passwords Unknown Solution: Windows Server 2008 + KB961320 enables DSRM password synchronization to a domain account. Create a standard domain user. This user does not need to be a member of any special groups or the Domain Admins group. NTDSUTIL SET DSRM PASSWORD SYNC FROM DOMAIN ACCOUNT <userName> This process can also be scheduled via a GPP scheduled task “ SET DSRM PASSWORD” “SYNC FROM DOMAIN ACCOUNT <userName>” Q Q

Why 2008 R2 is a good idea for AD AD Module for PowerShell and PowerShell cmdlets Every AD task is now automate-able via PowerShell AD Administrative Center Improved, task-based GUI for ADUC AD Recycle Bin Tough to use, but better than Authoritative Restore AD Best Practices Analyzer Are you the weakest link in your AD infrastructure? Offline Domain Join Handy for W7 upgrades and VDI

Why 2008 R2 is a good idea for AD Managed Service Accounts Eliminate service account nightmares AD Web Services & AD Management Gateway Simplified PowerShell and 3 rd party management integration Authentication Mechanism Assurance Deliver a different set of resources when users login via smart cards. AD OpsMgr Management Pack If you haven ’t incorporated OpsMgr yet, see me after class…

A Review of Useful AD Logs NTDS Diagnostics Logging By default, AD only records critical and error events to the Directory Service log. OK during normal operations, but during problem troubleshooting additional logging is necessary. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\NTDS\Diagnostics Set any of the 24 subkey ’s DWORD value to a number between 0 and 5

A Review of Useful AD Logs Extended DCPROMO Logging During a W2003 DCPROMO, two log files are created in %systemroot%\debug: dcpromo.log and dcpromoui.log. The log level on dcpromoui.log can be increased to help when troubleshooting promotions/demotions. HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Current Version\AdminDebug\dcpromoui Set the DWORD value for LogFlags to FF0003 (hex).

A Review of Useful AD Logs NETLOGON Logging Hunt down problems with client log-ins, repeatedly locked-out accounts and log-in activity across forest trusts by increasing the log level on NETLOGON. HKEY_LOCAL_MACHINE\SYSTEM\Current ControlSet\Services\NetlogonParameters Set the DWORD value for DBFlag to 2080FFFF (hex), then restart the NETLOGON service. NETLOGON.log is found in %systemroot%\debug.

A Review of Useful AD Logs Kerberos Logging Increasing the Kerberos logging level can track down problems with disabled or expired accounts, missing usernames, and clock synchronization. HKEY_LOCAL_MACHINE\System\CurrentControlSet\ Control\Lsa\Kerberos\Parameters Set the DWORD value for LogLevel to 1 and look for events in the System Event Log.

A Review of Useful AD Logs USERENV Debug Logging This logging helps identify problems with loading/unloading of user profiles, login/logout delays, and Group Policy application. HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon Set the DWORD value for UserEnvDebugLevel to 0x00030002 and watch %systemroot%\debug\usermode\userenv.log Match wall clock time.

A Review of Useful AD Logs GPO Client Logging To troubleshoot the enumeration and application of GPO ’s, increase the log level for GPO application at the client. HKEY_LOCAL_MACHINESoftware\Microsoft\ Windows\CurrentVersion\Diagnostics Set the DWORD value for RunDiagnosticLoggingGroupPolicy to 1, reboot the system and watch the Application Event Log.

A Review of Useful AD Logs Group Policy Logging changes with Vista/08 With Vista/08, Group Policy elements are moved to their own process. Out of WinLogon Enabling Group Policy logging is now done by setting the DWORD value for GpSvcDebugLevel to 10002 for HKLM\Software\Microsoft\Windows NT\CurrentVersion\ Diagnostics The userenv debug log is now moved to %systemroot%\debug\usermode\gpsvclog.log The format of this log is easier to use, aligned with the movement of GP to a service.

GPO ’s Not Easily Restorable Problem: Group Policy Objects can be restored if they ’re accidentally deleted, but this process involves a complicated authoritative restore, et cetra. This authoritative restore can be a source of downtime to complete the restore and takes a while to complete.

GPO ’s Not Easily Restorable Solution: Create a Scheduled Task that backs up all GPO ’s in the Domain to a text file using GPMC. This task will use both scripts to ensure that GPO settings and any logon/logoff/startup/shutdown scripts are also saved. Ensure that this text file is part of the nightly backup scheme. cscript.exe %PROGRAMFILES%\gpmc\scripts\ BackupAllGPOs.wsf %SYSTSEMDRIVE%\backup\ GPOData /domain:<DomainFQDN> cscript.exe %PROGRAMFILES%\gpmc\scripts\ GetReportsForAllGPOs.wsf %SYSTSEMDRIVE%\ backup\GPOReports (Can also backup and restore in R2 with PowerShell)

Incorrect FSMO Placement Problem: Incorrect FSMO role placement in domains where all DC ’s are not GC’s can cause a loss of data. The Infrastructure Master role cannot reside on a Domain Controller that also runs the Global Catalog role Except in the situation where the forest contains a single domain and all Domain Controllers are Global Catalogs. To check FSMO role placement, NETDOM QUERY FSMO

Incorrect FSMO Placement Solution: Either enable the Global Catalog role on all Domain Controllers or move the Infrastructure Master role to a DC that is not a GC. NTDSUTIL.EXE ROLES CONNECTIONS CONNECT TO SERVER {Server Name} QUIT TRANSFER {Role} QUIT | QUIT

Fat Finger (Not that I ’m calling your finger fat) Problem: You ’ve done it again and accidentally deleted a series of objects or an entire OU from AD. The default configuration of Active Directory allows everyone with administrative access to delete any object in AD. Though, we all make mistakes… In W2008, OU properties include a box box “Protect this Object/Container from Accidental Deletion”. But we ’re not at W2008 yet!

Fat Finger (Not that I ’m calling your finger fat) Solution: That checkbox, revealed in W2008 is actually just a skin for a supported feature of W2003. By checking this box: “ Deny Delete” and “Deny Delete Subtree” permissions for the Everyone group are set on the object. You can set these permissions manually for any AD object or container inside the ADUC: Enable ADUC Advanced Features Navigate to the Object, select Properties, and view the Security tab Apply the Advanced privileges “Deny Delete” and “Deny Delete Subtree” to the object

Unnecessary Apps Installed Problem: Every additional application installed to a server is an expansion of that server ’s attack surface. WinZip versions prior to v10. Java JRE prior to Version 5. Real Player Office Acrobat Solution: Never install applications to your Domain Controllers. Ensure that any apps installed are always patched. There ’s more to patching than just Microsoft patching.

Letting DCPROMO Do DNS Problem: Generally a bad idea to let DCPROMO handle configuring DNS for Active Directory. Tends to do a poor job of it, if at all. Better in W2008. Solution: Ensure DNS properly configured before starting a DCPROMO process. Three tests: nslookup dchostname nslookup dchostname.dcdomainame.com nslookup 10.1.3.4 If success, no errors, and no time delays, then OK.

VM-level Backups for AD DR Problem: With virtualized Domain Controllers using VM-level backups to backup DC ’s can corrupt AD. USN number mismatch between restored DC and existing DC ’s. Which USN ’s have correct high water mark? Solution: Always use authoritative/non-authoritative restore for AD DR. Never VM-level backups. In fact, never use VM-level backups for any transactional database for the same reason. Want more justification? http://support.microsoft.com/kb/888794

Snapping an Offline DC VM Problem: Need to create an offline DC VM for testing purposes, but am concerned about lingering objects. Solution: Use this process… Create a new site in AD Add a member server VM to the domain in the new site. DCPROMO. Wait for replication to complete, then shut down the DC. Copy/Paste the virtual machine, then restart the DC. Demote this DC back to a member server and remove it from the production network. Start the DC, reconfigure network, and seize all FSMO roles. Use the new DC to complete testing.

What you Need to Back Up Problem: What exactly needs to be backed up to ensure a successful DC restore. A successful authoritative restore of the AD database. Solution: Never try to restore the AD database from one DC to another DC. So, all files that make up that DC must be backed up: C:\ System State

Lack of Defined DR Policy & Procedures Problem: Most companies do not have a defined DR Policy and DR Restoration Procedures. This is usually the case because the project can get over-scoped. Consider just the steps necessary to start a recovery. Solution: Build a simple DR plan and recovery steps. Does not need to be complicated. Just the basic steps necessary to start recovery. When you ’re under the spotlight, you don’t want to be searching for recovery steps on TechNet…

3 DR Scenarios Scenario 1: A subset of objects within Active Directory or the SYSVOL is accidentally or maliciously removed from the database. Scenario 2: An Active Directory domain controller is functionally and irrecoverably down and must be rebuilt to return to operations. Scenario 3: The entire Active Directory forest and domain is functionally and irrecoverably down and must be rebuilt to return to operations.

Scenario 1: Deleted Objects Locate a DC that is also a GC. Disconnect this server from the network. Reboot that server into Directory Services Restore Mode using the DSRM password. Restore the AD database to the DC from tape or file backup (non-authoritative). Perform an authoritative restore of the deleted object: NTDSUTIL AUTHORITATIVE RESTORE RESTORE SUBTREE {Object to Restore} <Object to Restore> is the DN of the object to restore. For example, to restore the Accounts OU, the DN would be “OU=Accounts,DC={MyDomain},DC={com}” QUIT / QUIT

Scenario 1: Deleted Objects Reconnect the DC and reboot the DC into normal operations. Ensure the restored object has replicated to all DC ’s in the domain. As the DC reboots from DSRM mode, it will generate .LDF files that include back-link information for the restored objects. As an example, back-links are groups the object is a member of. These files are of the format ar_{date}-{time}_links_{Domain Name}.ldf. Restore the back-links for each file found: ldifde –i –k –f ar_{date}-{time}_links_{Domain Name}.ldf

Scenario 2: A DC Goes Down Validate the DC is completely failed and a restoration is not feasible. If the DC is functional, but the AD database is corrupt, attempt a forced demotion: DCPROMO /FORCEREMOVAL Remove the failed server ’s server objects from a functioning DC: NTDSUTIL METADATA CLEANUP REMOVE SELECTED SERVER {DN of Server} QUIT | QUIT Within the active DNS for the domain, manually remove any references to the failed DC and its SID in either A or SRV records.

Scenario 2: A DC Goes Down Build a replacement server at the same Service Pack and patch level. DCPROMO the member server Validate a complete promotion and verify the AD database has resynchronized to the domain.

Scenario 3: Corrupted Forest As of the last time I checked, Microsoft PSS has never been called to perform a complete forest restoration. Validate that the complete Active Directory is completely and irreparably failed and a restoration is not feasible. Call Microsoft PSS at 800-936-2200 and declare a Priority 1 “Crit-Sit”. http://www.microsoft.com/downloads/details.aspx? displaylang=en&FamilyID=3EDA5A79-C99B-4DF9-823C-933FEBA08CFE

Ad disasters & how to prevent them

Recommended

More Related Content

Viewers also liked (20)

Similar to Ad disasters & how to prevent them (20)

More from Concentrated Technology (19)

Recently uploaded (20)

Ad disasters & how to prevent them

Editor's Notes