SUTOL 2016: IBM Connections Deployment Best and Worst Practices

8th Sutol Conference, November 2016
Best and worst practices
deploying IBM
Connections
Christoph Stoettner
panagenda

Christoph Stoettner
• Senior Consultant at panagenda
• IBM Notes / Domino since 1999
• IBM Connections since version 2.5 /
2009
• Many years of experience in:
– Migrations
– Administration and Installation
– Performance Analysis
• Joined panagenda in 2015 focusing in:
– IBM Connections Deployment and
Optimization
– IBM Connections Monitoring
• Husband of one & father of two,
Bavarian
@stoeps
linkedin.com/in/christophstoettner
www.stoeps.de
christophstoettner
+49 173 8588719
christoph.stoettner@panagenda.com

Don’t Be Scared
4

Databases
WebSphere Application Server
IBM HTTP ServerWebSphere Plugins
Application DBsPEOPLEDB
LDAP Server
TDI
Forward to Application Server
and Port
(Load balancing and Failover)
Redirect
unknown
URL
Upload and Download
of Files, Attachments
Common: Access
Customization,
Webressources
Read and Write
Authentication
Users / Groups
Reads Userdata for Profiles
Create, Update,
Delete and
Inactivate Profiles
Shared
Directory
Link
to
Attachments
Profile changes synchronize to
Membertables through JMS Queue
Optional:
Direct Access
to Attachments
Main Components
5

System Requirements
• Follow Strictly
• Check Regularly
• All versions
– http://short.stoeps.de/vwzrv
• IBM Connections 5
– http://short.stoeps.de/mspdi
• IBM Connections 5.5
– http://short.stoeps.de/cnx55sysreq
• Check all Notes
• Download PDF
– About 100 pages
6

Installation Documentation
• Be careful with non-official documentation
documents
• Sometimes wrong dependencies mentioned
(only trust System Requirements)
• Statement if something is supported does not mean
it’s licensed
– Example (from Update strategy for IBM Connections 5.0):
DB2 10.5 FP4 in http://www-
01.ibm.com/support/docview.wss?uid=swg21683118
– Connections 5.0 License Agreement does not contain a 10.5
License! -> No download in PA possible
7

Prepare A Installation Documentation
• http://www.ibm.com/support/knowledgecenter/en/SSYGQH
– Contains mixed informations of all Operating Systems and
Database Type
– If you install often, create your own Install documentation
• Check Performance Tuning Guides
– Not available for 5.5 until now
– Check 4.0, 4.5 and 5.0 to get an idea
• Read all lists before you follow them
– e.g. “point 13: if you use Linux, don’t follow point 8, do ...
instead”
– Nothing found in the actual documentation, but had this
multiple times
8

Prepare Your Installation
• Download all needed Software Packages
• Download IBM Connections 5.5
– List with Part Numbers (but with all possible Software):
http://www-
• Don’t forget mandatory Fixes from
http://www.ibm.com/fixcentral
– Download WebSphere Fixpack 9:
http://www-
9

Save Time
• Download and Install Installation Manager separate
– Version 1.8.3 or newer 64 Bit
– Do not use the one packaged with Connections
– You can install all other products without updating IM
• Directly install Fix pack Versions
– Add Fixes and Original File to Installation Manager
Repository List
– Connections 5.5 CR1
(works with 5.0 CR2 and newer, but not 5.0 CR1)
– WebSphere Application Server ND 8.5.5.9
– DB2 10.5FP5 Universal or ESE
(without 10.5.0 package, no need to download)
10

Sizing
• Be prepared for future growth
• Do not overact
– Some hundred users mostly won't need a large deployment
• I'm not a fan of multi-instance database machines
– If I run in database performance issues I split the databases
to different machines
– Performance tuning guide
• Multi-instance is best practice, if you have enough resources
11

Minimum Requirements
• 4 | 8 GB memory is often too less,
better start with 10 or 12 GB
– Memory swapping kills ALL tuning efforts
• CPU cores
– 2 cores minimum only on small tiny
deployments
– Thumb rule:
I calculate one core for each JVM
(expensive with PVU license)
• Disk
– Using network storage or virtualized servers
– Easier to extend
12
Connections 5.5Connections 5.0

General Installation
• Download all needed software packages
– Always check System Requirements before you install
• Source and Destination Paths shouldn’t contain
spaces
– Don’t install from or to folders with a space
– e.g. Installation Manager | InstallationManager
– avoid “c:Program FilesIBM”
– Use c|d|e:IBM
• Different operating systems need special settings
• Always use the operating system where you have
the best skills
13

Network
• Name lookup / DNS
– All servers must be resolvable (hosts is not a workaround)
– Knowing the protocol
• Avoid Round Robin
• No Authentication failover in WebSphere with Round Robin!
• Network storage (file locking is important)
– NFS v4 / SMB|CIFS
– No DFS
• Reverse Proxies / Proxies
– Always test your deployment without proxies
– Activate after successful testing

Register WAS as a service
• Services for Deployment Manager and NodeAgent(s)
– wasservice.bat|sh
• Map service to a technical user
– any Active Directory User is possible
– allowed to read / write network share with Shared Content
• Service can parse commands to nodeagent
– -stopArgs "<NA commands>"
• Configure monitoring policy (if required)

WasService.bat|sh – Register service
• Create a local filebased user to stop the service
– Operator Role is enough
16
cd D:IBMCNXWebSphereAppServerbin
WASService.exe
-add CnxNode01
-serverName nodeagent
-profilePath d:ibmcnxwebsphereappserverprofilesCNXNode01
-stopArgs "-username wasadmin -password password -stopservers"
-userid cnxtec -password password
-encodeParams
-restart true
-startType automatic
parsed to nodeAgent
stops AppServer

Monitoring Policy
• Each Application Server
– Change Node restart state to "RUNNING”
• Must set this to STOPPED before
performing updates
– Or use syncNode.bat|sh after applying
an update
• Increase Timeout on Windows
– Default timeout for service shutdown
= 30 seconds
– From around 5 Application Servers
– Increase Value at:
HKEY_Local_Machine:
SYSTEMCurrentControlSetControlWaitToKillServiceTimeout

Directories & Synching
• Prepare your LDAP
– Better data within LDAP → better Profiles
– Login Items must be unique
• E-Mail-Address
• UID
• Switching Authentication directories is possible
– Need some planning
• Dependencies
– Quality of LDAP data
– Plans to activate SPNEGO
– Domino Mail Integration

Federated Repositories Best Practice
• Leave the file based wasadmin enabled
– Fallback if LDAP Bind Credentials changed
– Solving problems with Federated Repositories
• Default does not allow this
(you have to disable security
through wsadmin to
change configuration)
Check this box

Logs – adjust language WebSphere
• Change log language to English (IBM will love you for this)
• WebSphere
Add "-Duser.language=en –Duser.region=US" to Generic JVM
arguments of
– Each application server (Process definition – Java Virtual Machine)
– dmgr (System Administration – Deployment Manager – Process
Definition ...)
– nodeagents (System Administration – Node agents – nodeagent –
Process Def ...)

Logs – adjust language TDI
• TDI
– edit ibmdisrv.bat|sh
– add -Duser.language=en –Duser.region=US to LOG_4J
variable
– Linux:
– Windows:
21

Rotate Logs
• WebSphere Logs too small for Troubleshooting
– Default: 5 Logs 1 MB each (SystemOut and SystemErr)
– Better 5-10 Logs 20 MB each
• Setting for each Application Server
• remember Nodeagents and Dmgr
• Change this as soon as your servers have been
created

Rotate Logs
• IBM Connections 5.5 – SET BY DEFAULT!!
– Install.log
– Result:
– So your logs are stored 30 days, independent of size

Rotate IBM HTTP Server Logs
• Default: no max size for access_log and error_log
• Often some GB of Log files
– Open with an Editor?
– Disk size
• Search for this lines in httpd.conf:
• Comment out:
CustomLog log/access_log common
ErrorLog logs/error_log
# CustomLog log/access_log common
# ErrorLog logs/error_log

Rotate IBM HTTP Server Logs
• Add:
• Delete Log Files older than x days (examples with 3)
– Linux
– Windows (Batch through Task Scheduler or Powershell)
Linux:
CustomLog "|/opt/IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" common
ErrorLog "|/opt/IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/error_log.%Y%m%d 86400“
Windows:
CustomLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" common
ErrorLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/error_log.%Y%m%d 86400"
crontab -e
# Delete logfiles older than 3 days in logs
10 0 * * * find /opt/IBM/HTTPServer/logs/*_log.* -mtime +3 -exec
rm -rf {} ;
forfiles -p "D:IBMHTTPServerlogs" -s -m *_log.* -d -3 -c "cmd /c del @file"

Rotate Logs DB2
• db2diag.log
• Default: no maximum size
– Default: %PROGRAMDATA%IBMDB2instancenameDB2
– Full C-Partition in Windows still hard to solve
[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsize
Size of rotating db2diag & notify logs (MB) (DIAGSIZE) = 0
[db2inst1@cnx-db2 ~]$ db2 update dbm cfg using DIAGSIZE 1024
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed
successfully.
[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsize
Size of rotating db2diag & notify logs (MB) (DIAGSIZE) = 1024

HTTP Server Keystore
• Several Reviews showed
– Keystore of WebSphere Plugin used for IHS SSL Key
– Why is this worse?
– What would you do when you get SSL Errors within
Connections?
– This overwrites plugin-key.kdb on your Webserver
– SSL Key deleted
– Backup?
27

HTTP Server Key store
• When you want to reuse Plugin Key store
– Import SSL Key into CMSKeyStore
– But never seen this in the wild
28

HTTP Server Keystore
• Best Practice - Create a
separate key store for IHS
– Ikeyman will help you
– Possible to use a wildcard
• Then you can just copy it to use on
dev / test machines
– Backup the keystore before
changes
– Don’t activate “Expiration time”
– “Stash password to a file”
• Creates a cnx-key.sth file
• Used by IHS to open keystore

Harden HTTP
• Disable SSLv2 / v3
– Automatically disabled with 8.5.5.4
• Check with
hydra, nmap or ssllabs.com/ssltest/
SSLProtocolEnable TLS
SSLProtocolDisable SSLv2 SSLv3
# Disable SSLCompression -> CRIME ATTACK
SSLCompression off
#Prefer ECDHE-RSA ciphers
SSLCipherSpec ALL NONE
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec ALL TLS_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec ALL TLS_RSA_WITH_AES_256_GCM_SHA384
SSLCipherSpec ALL TLS_RSA_WITH_AES_128_CBC_SHA256
SSLCipherSpec ALL TLS_RSA_WITH_AES_256_CBC_SHA256
# Enabling this 3 ciphers mean A- rating on ssllabs, but without Connections is not working
SSLCipherSpec ALL TLS_RSA_WITH_AES_128_CBC_SHA
SSLCipherSpec ALL TLS_RSA_WITH_AES_256_CBC_SHA
SSLCipherSpec ALL SSL_RSA_WITH_3DES_EDE_CBC_SHA
# Enable HSTS
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

Harden HTTP
• If you use SSL Keys longer than 2048 bit, you must
replace Java policy
– Download and replace Java (unrestricted) policy files
• All Nodes
• https://www-01.ibm.com/marketing/iwm/iwm/web/reg/pick.do?source=jcesdk
– Also needed if any application connected use longer keys
• LDAPS
• Mailserver
• Sametime Proxy
• Remove Server Information (HTTP Header, Error
pages)
– ServerSignature Off
– ServerTokens Prod (DEFAULT)
– AddServerHeader Off
Default

Remove Index
• Remove all Files except index.html from
<IHS_ROOT>/htdocs
• Rename index.html (e.g. 0815.html)
– echo 1 > 0815.html
– For testing you can access the file
• Add robots.txt

Performance Tuning Guides
• 4.0
– http://www-
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tu
ning_Guide
• 4.5 Addendum
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tu
ning_Guide_Addendum
• 5.0 CR1
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide
• 5.5 (June 2016)
– https://www-
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_V5.5_Tuning_guide
• Read everything carefully
• Check and understand dependencies

Worst Practise Example - Tuning
• Customer showed me a system with following
infrastructure
• WebSphere
– Large deployment
– 16 GB RAM
– 4 Cores
• DB2
– 12 instances
– 8 GB RAM
– 4 Cores
• Connections restart 22 minutes
Web Server
ihs.example.local
WebSphere
was1.example.local
Db2 / TDI
db2.example.local
Freigabe
LDAP
domino1.example.local
User SynchronisationAuthentication

Solving the problem
• Large deployment means about 15 JVM on the machine
– Restart shows 15 min 100% CPU usage
– Adding 4 cores and restart time get down to 7 minutes
– Other option would be midsize deployment, but then you have
to reinstall Connections
• Java Heap Sizes set to default (256 MB and 768 MB) ->
increase to 1.5 – 2.5 GB
• Perf Guide mentions that multiple instances on DB2 only
increase performance with enough resources
– But that was not the real problem
• DataSource connectionPool Sizes are set to Default 1/10
– Increase this values to the proposals in the guide and ...
– Restart time comes down under 3 minutes
• Key point: read the complete guide

Java Heap
• Default Java Heap Sizes on Midsize Deployment: 2506 MB /
application server
• Large Deployment depends on application: 0.5 to 2.5 GB
• Main point in memory tuning
– Never exceed the system memory
– Swapping kills all your tuning efforts
• Counting the JVM Heap sizes is not enough
– Maximum heap is not the maximum amount of memory the jvm
uses!
– Libraries, jars and so on count additional to memory usage
– JVM memory usage may be 3 * JVM maximum Heap
• Initial and maximum Heap Size should be equalized
– mentioned in IBM Connections 5.0 tuning guide

IBM HTTP Server
• Enable compression
– Important !!!!!
– See Slides from BP307 - IBM Connect 2014
– Save up to 70% network traffic
– Minimal increase of CPU load
• Enable file download through IHS
– Depend on your deployment
– Often security forbids storage access from DMZ
• If you have no access to file share from IHS -> Files
should be installed in a separate Cluster

Database Reorg
• Performance of Connections is often database
related
• IBM provides runstats.sql and reorg.sql Scripts
– Wizardsconnections.sql<app>db2
– Run them after big amount of data is changed or added
– Run on a weekly/monthly basis
• No Scripts for (Create your own!)
– CCM
– Docs
– Surveys
38

What to Backup
• Using an example Connections installation guide rarely
explains backups
– These guides normally do not mention backup, or what to back
up
– Disk crash means data loss
• Database backups through file backup are not supported
and mostly not restorable
• Important!!!
– Database Backup through Online Backups can be taken when
Connections is up
– Offline backups are also possible
• Ensure the file system & DB backup are run at the same
time of day
– DB and Filesystem data will stay in sync – if you take your DB
backup at midnight and the file system at midday they will be
out of sync

Backup
• Most important (minimum daily)
– Databases (offline or online)
– Shared content
• Important
– Configuration
• WebSphere Application Server
• Connections
• IBM HTTP Server
• TDI Solution
• Test if restore is possible!!!!
– Several issues with WebSphere restores, where binaries
weren't on the tape

Install Checklist
• WebSphere Application Server
– Configure Federated Repository
– LtpaToken, enable security
• WebSphere Application Server Supplements (IHS,
Plugins)
• DB2 (or other DBM)
• TDI
• Add Webserver to Dmgr (use configurewebserver.bat)
• Enable SSL on IHS
• Import IHS Root Key within WebSphere cell trust keystore
(retrieve from port)
• Configure CCM
• Install optional Addons like Forms Experience Builder,
IBM Docs, Cognos

Documentation
Document EVERYTHING !!!
because you can remember everything you did ….

Documentation
• Everyone hates writing documentation
• BUT – make notes as you go, it doesn’t need to be a
full step by step guide with screenshots
• Document all customizations
• Any additional changes made
• Anything of note that deviates from the guides
• Lessons learnt or how you solved issues
• Use the scripts to output some of it

Useful Tools
• Editor with syntax highlighting
– Linux/Unix: vim, geany
– Windows: notepad++, pspad, UltraEdit
• Log Viewer
– tail -f
– Baretail (https://www.baremetalsoft.com/baretail/)
– Multitail (https://www.vanheusden.com/multitail/)
• (Intercept) Proxy
– Fiddler (http://www.telerik.com/fiddler)
• IBM support ask often for traces
• Windows only
– Burpsuite (https://portswigger.net/burp/)
• OS independent

Useful Tools (2)
• Browser
– Firefox (portable) / Firefox ESR
– Chrome
– IE (download Virtual Machine with different versions)
• https://www.modern.ie
• You can be sure there is no GPO applied
• Network analyzer
– Wireshark (https://www.wireshark.org/)
• SPNEGO Analysis
– tcpdump (save pcap and analyse with Wireshark)
– Nmap (Check SSL, Ports)
• Unzip / Unarchiver
– 7-zip
– WinRar
– Do not use tar delivered with AIX! Use GNU tar

Links and References
• IBM Connections System Requirements
– http://www-01.ibm.com/support/docview.wss?uid=swg27012786
• IBM Connections Family Documentation
– http://www.ibm.com/support/knowledgecenter/SSYGQH/welcome
• IBM Connections 4 Performance Tuning Guide
– https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide
• IBM Connections 4.5 Performance Tuning Guide Addendum
– https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum
• IBM Connections 5 CR1 Performance Tuning Guide
– https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide
• 5.5 (June 2016)
– https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_V5.5_Tuning_guide

47
Useful Blogs
http://ibmconnections.com
http://turtleblog.info
http://portal2portal.blogspot.de
https://www.urspringer.de
http://socialconnections.info
http://blog.robertfarstad.com
http://www.curiousmitch.com
http://techblog.gis-ag.info
http://ibmdocs.com
http://domino.elfworld.org
https://dontforgetthe0.com
http://www.ramsit.com/category/blog
https://milanmatejic.wordpress.com
http://dilf.me.uk/socialshazza
http://www.stoeps.de
http://scripting101.org
http://meisenzahl.org
http://martin.leyrer.priv.at
http://kbild.ch
http://www.notesgoddess.net
http://www.dominodiva.com
http://notesbusters.com
https://rob59blog.wordpress.com
http://connections101.info
http://brandlrainer.blogspot.de
https://collaborationben.com

SUTOL 2016: IBM Connections Deployment Best and Worst Practices

Recommended

More Related Content

What's hot (19)

Viewers also liked (14)

Similar to SUTOL 2016: IBM Connections Deployment Best and Worst Practices (20)

More from panagenda (20)

Recently uploaded (20)

SUTOL 2016: IBM Connections Deployment Best and Worst Practices

Editor's Notes