Domino Server Health - Monitoring and Managing

Gabriella Davis - IBM Lifetime Champion
Technical Director - The Turtle Partnership
Brussels May 14th 2019
DOMINO SERVER HEALTH -
MONITORING AND MANAGING

• Admin of all things and especially quite
complicated things where the fun is
• Working with the design, deployment and security
of IBM technologies within global infrastructures
• working with the real world security and privacy
aspects of expanding data ecosystems
• Stubborn and relentless problem solver
• http://turtleblog.info
• https://www.turtlepartnership.com
• IBM Lifetime Champion

DOMINO SERVER LOGS
• Analysing
• Miscellaneous Events
• Mail Routing Events
• Security Events

NOTES CLIENT LOGS
• It’s not a race to read messages on your status bar when you can look in your local log.nsf for
those messages in more detail
• Security events show you issues with accessing the ID Vault for instance
• Miscellaneous events will show you network failures amongst other things
• Client_Clock=1 in local notes.ini for RPC activity from your Notes client
• Console_Log_Enabled=1

DOMINO SERVER LOGS
• Log Analysis

LOG ANALYSIS RESULTS
• Does not require your log.nsf to be full text indexed

DOMINO SERVER LOGS
• Maintaining logs at a manageable size is always an issue especially on busy servers
• Log=log.nsf,1,0,7,40000 - maintains 7 days of data with 40k document sizes
• Archiving
• You can archive a log.nsf the same as you would any other database
• Create the archive settings under database property
• Create a program document to compact log.nsf with -a option for “archive”
• Don’t maintain unread marks - why would you?
• Never Full Text Index
• Console_log_Enabled

LOG FILTERS
• Configured in events4.nsf
• Choose to only log specific entry types and error levels either for the whole server or for
individual databases

NSD
• If the server or client crashes it will generate a NSD file in the directory
IBM_TECHNICAL_SUPPORT
• Simple (very) Analysis
• open the file in a text editor
• search for “fatal” and find the thread and task that caused the crash
• search for the thread ID to find the database / document / agent

FAULT REPORTING DATABASE
• Configure in the server configuration document
• Can be configured for clients in desktop policies
• The NSD will be mailed in to the assigned fault recovery database
• from the Log Analysis view in Domino Administrator choose “open fault reports” and the database
will be created

FAULT REPORTING DATABASE
• Disposition view
• all faults when analyzed have a disposition value that categorises as
• Problem
• Possible Problem (possibly actionable )
• Possible Problem (likely NOT actionable )
• Informational
• Unknown (investigate)

POSSIBLE PROBLEM - ACTIONABLE
• Out Of Memory: Represents a crash in which the Java virtual machine (JVM) ran out of a
memory resource such as heap space.
• Launched Notes multiple times: Indicates that the user quickly launched multiple instances of
the Notes client
• Possible hang: Indicates that the Notes client was manually terminated while it appeared to be
doing useful work.
• User Kill: Indicates that the user manually terminated the client while it appeared to be waiting
for input or network timeout

DELETION LOGS
• Individual databases can be enabled to log deletions
• The detail of who deleted a document and the content of that document is written to a
delete.log file
• One for all databases that are enabled
• Enabling is done via a compact option -dl that also specifies the fields in the document to log
e.g.
• load compact mailgabdavis.nsf -dl on subject,posteddate,sendto,recipient
• enable log deletions on gabdavis.nsf and track the subject, post-date, sender and
recipient
• different databases will want to use different fields in the log
• any Text, Text_List, RFC822_Text, or Time field can be used

REVIEWING A DELETION LOG ENTRY
• “20190408T211516,06+01",
• “Mailgabdavis.nsf”,
• “80256487:00352154",
• “nserver","CN=DEWO=Turtle",
• “SOFT",
• “0001","72C0E3F8:44B53FB5DC4EDBF8:A785466D",
• "from","""New Relic” “<marketing@newrelic.com>",
• “sendto","gabriella@turtlepartnership.com",

MESSAGE TRACKING
• Configure message tracking via the
server configuration document
• it eats up resources so don’t over
configure it
• it’s not designed for auditing
• consider configuring it purely on the
outbound SMTP server or inbound
SMTP server

DDM MAIL PROBES
• Transfer queue check
• is mail backlogged to or from your servers
• In this example I’m monitoring if the SMTP mail that sends to messagelabs via our hub SMTP
server is being delayed

DDM MAIL PROBES
• Mail Reflector
• validate that a specific email address can be reached. Invaluable for a very specific use case
where routing of certain messages is business critical
• Mail Flow Statistic Check
• Monitoring the % of pending mail on each server to alert any potential delays in mail delivery
• Slack % = Mail.TotalPending - Mail.Dead - Mail.Held - Mail.Waiting) / Mail.Waiting

MAINTENANCE TASKS
• Compact
• using -B or -c as a copy style compact creates a new DBID which backup software will
recognise as an entirely new db
• -REPLICA creates a new replica instance then deletes the in use instance and renames the
copy which is useful for a database that is continually in case
• Fixup
• don’t use it too much, I like to use it monthly at most. It can be a hammer to crack a nut.
• Updall
• Updates view and full text indexes
• notes.ini Update_FullText_Thread separates the full text indexing thread from the view
indexing threads for efficiency
• Load updall - nodbmt tells updall to run but not perform the functions that DMBT already
does

DATABASE MANAGEMENT TOOLS
• DBMT Server Command - run weekly
• runs copy-style compact operations
• purges deletion stubs
• expires soft deleted entries
• updates views
• reorganises folders
• merges full-text indexes
• updates unread lists
• ensures that critical views are created for
failover
• Replaces Updall

DBMT PARAMETERS
• -compactThreads
• -updallThreads
• -ftiThreads
• -timeLimit refers to compact timeout for
DBMT
• -range starttime stoptime
• compactNdays (run Compact every x
days)
• ftiNdays (run FT Index every x days)
• force d (day Sunday =1) fixup if compact
fails for consecutive day

STATISTICS
• View all current statistics in Domino
Administrator, identify what you might
want to monitor
• Statistics Event Generators
• Set alerts / notifications / log entries
when statistic thresholds are met
• configured in events4.nsf

ENABLING STATISTICS COLLECTION
• Configured in events4.nsf - statistic collection
• Server task “Collect” gathers the statistics and writes them to the specified reporting database
• tell collector collect

STATISTIC MONITORING
• Events4.nsf

DATABASE EVENT GENERATOR
• Monitor ACL changes
• Monitor replication
• Monitor unused space *
• Monitor for user inactivity *
• * These can also be done by the DDM Database Probe both are configured in events4.nsf

DDM PROBES
• Set collection servers to group similar server types together to get the maximum context for
logs or if you don’t have many servers have them all collected together
• ddm.nsf holds errors written to the console as well as errors generated by enabled probes
• Don’t enable too many or for too long but use the probes to monitor bottlenecks, business
critical services or identify resource issues such as disk or memory performance

CONFIGURING ACTIVITY TRENDS
• In the server configuration document - configure both Activity Logging and Activity Trends

ACTIVITY TRENDS
• Once activity tracking and trends are enabled via the Server Configuration document you can
review activity trends via Domino Administrator - Performance

NEWRELIC REPORTING
• By default servers report into a Domino database that can get huge and has no built in analysis
• We can now utilise cloud services that are designed specifically to analyse statistics and can
consolidate reporting from multiple servers in multiple domains
• Newrelic is only one cloud service for publishing statistics (but it’s free)
• Configuring statistics reporting is simply a case of adding one line to the notes.ini on the server
• NewRelic_License_Key=<key> (found under your NewRelica account settings)

Domino Server Health - Monitoring and Managing

DOMINO CONFIGURATION TUNER
• Domino Configuration Tuner is an analysis tool based on a set of pre-configured best practice/
worst practice rules
• The Rules are shipped by with the Domino installs inside dct.ntf and are updated via a public
update site
• although not since 2014
• Makes recommendations on configuration changes to enhance performance and security and
reduce TCO

HOW DOES IT WORK?
• Run and installed via the Domino Configuration Tuner database
• Updated by online template updates and rule updates
• Scans
• Server documents
• notes.ini settings
• advanced database properties
• Intended to scan servers within a single domain
• It’s a good baseline to begin a healthcheck for instance or post any significant upgrade

HOW DOES IT WORK?
• Creates reports on each scanned server based on the rules you select
• Each report contains
• Issues
• recommendations for adjustments
• links to supporting documentation

PRE-REQUISITES
• v8 Notes client (standard or basic) or administrator
• dct.nsf database and dct.ntf template
• servers 7.x or higher

SETUP
• DCT.NSF
• StdDominoConfigTuner Template (dct.ntf)
• ID must have reader access to names.nsf
• ID must have ‘View Administrator’ rights
• Requires no server or domain changes

RUNNING THE TUNER
• You can manually type in the full hierarchical names of any other servers you want to scan as
part of this analysis
• Separate multiple server names with commas, semi colons or new lines
• You can only scan servers you can reach so you need a connection document to any you list
• or the server needs to be available via your passthru server in your location

UNDERSTANDING THE RESULTS
• Summary results
• Issues by criticality

• Filter results to make analysis easier
• by server
• by specific rules
• by severity

• Categorised results of recommendations, sorted by criticality and then by server name
• Each recommendation comes with an explanation so you can evaluate on a result by result
basis if you want to make the change

DATABASE REPAIR & SYMMETRICAL CLUSTERS
• Missing databases across cluster mates
• Missing or corrupt NLOs
• Fast distribution of databases
• Corrupt databases
• I have built clusters by using cluster symmetry to populate the new server faster than
accelerated replicas

CLUSTER SYMMETRY
• The servers listed on the cluster configuration document are instructed to probe other servers
in the cluster if they need to populate or replace a document
• Each server uses its own cldbdir.nsf to probe a server that it knows contains the file it needs
and pull that file
• Files must have
• the same path and filename on each cluster mate including case sensitivity relative to the
data directory
• have the same DAOS configuration for minimum attachment size
• be ODS52 or higher
• The server to be populated must be running the AutoRepair and RepairCleanup tasks (put
them in the servertasks= line)

HOW DOES IT WORK?
Folders to maintain
Fix or list missing files - to
be safe choose “list”
initially

REPAIR LIST FILES CONSOLE COMMAND
• Mailturtle.nsf, 8055555:00372848 [Present] 
TurtleRFC.NSF, 8055555:007ABEBD [Present] 
Turtledbdesign.nsf, 858055555:0076FBF6 [Present] 
Mailghedley.nsf, 802568E4:8055555 [Present] 
Missing files for server CN=Clouds/O=Turtle 
archivea_turtle.nsf, 80256A24:0053E4CF [Missing] 
archivel_turtle.nsf, 80256A24:0053F569 [Missing]
• Repair List Missing to show only “missing” files

REPAIR ALL CONSOLE COMMAND
• “ archivea_turtle.nsf needs to be repaired but Auto Repair is configured to list only”
• With the configuration document set to “List” instead of “Fix” the server will not fix the
database

REPAIR ALL CONSOLE COMMAND
• Repair end copy C:IBMDominodataarchivel_turtle.nsf : Invalid NSF version
• Requires ODS52 minimum (9.0.1) database format 
 
 
 
 
 
Repair File <filename> (case sensitive
repair File Mailturtle.nsf
Remote console command issued by Gabriella Davis/Turtle: repair File Mailturtle.nsf
File Mailturtle.nsf already exists, no repair necessary.
REPAIR: 0 file(s) queued for repair

• REPAIR: Job-00007.1 *Copying* - Repair copying C:IBMDominodataarchivea_timd.nsf
from donor CN=Oceanic/O=Turtle

• Domino’s focus on self monitoring and reporting continue to reduce its TCO
• Use events to monitor business critical activity
• Use probes to troubleshoot problems or identify bottlenecks such as when
launching a new application
• Don’t monitor more than you can regularly review
• plan to review fatal and failure events daily and the rest weekly
• have a plan for both weekly and monthly maintenance tasks, monitor database
health and leverage the automatic database repair feature to get ahead of
corruptions

QUESTIONS? 
GABRIELLA@TURTLEPARTNERSHIP.COM
@GABTURTLE
SKYPE: GABRIELLADAVIS

Domino Server Health - Monitoring and Managing

Recommended

More Related Content

What's hot (20)

Similar to Domino Server Health - Monitoring and Managing (20)

More from Gabriella Davis (20)

Recently uploaded (20)

Domino Server Health - Monitoring and Managing