Chapter 8: Analytical System Administration

8.1 System Observation

Introduction to System Observation

 Definition: System observation refers to the ongoing monitoring of various aspects of

computer and network systems to ensure proper performance, identify potential issues,
and optimize efficiency.
 Importance: Helps in detecting bottlenecks, system failures, or irregular activities before
they affect performance.

Key Concepts in System Observation

1. Real-Time Monitoring:
o Tools and techniques for observing system status in real time (e.g., CPU usage,
disk space, memory utilization).
o Importance of observing critical system metrics (e.g., load average, response
times).
2. Monitoring Tools:
o System Resource Monitors: Task Manager, Resource Monitor (Windows), top,
htop, vmstat (Linux).
o Network Monitors: Wireshark, NetFlow, Nmap.
o Application Monitors: Logs, database query performance tools.
3. Log Files and Event Logging:
o Use of logs in troubleshooting and performance analysis.
o Types of logs: System logs, application logs, and security logs.
o Example: Reviewing logs for detecting service failures or security breaches.
4. Key Performance Indicators (KPIs):
o Metrics that help gauge system health: response time, throughput, CPU load,
memory usage.
o Establishing thresholds for acceptable performance.
5. Visualizing System Health:
o Dashboards: How to use graphical representations for monitoring (e.g., graphs,
charts for CPU, disk, and network activity).
o Real-time alerting systems for automatic responses to anomalies.
6. Automated Monitoring:
o Scripting and automation for regular system checks.
o Using cron jobs, Task Scheduler, or monitoring systems like Nagios, Zabbix, or
Prometheus for automated reporting.
Examples of System Observation in Action

 Case studies on how real-time monitoring helped identify performance degradation or

hardware failure.
 Example: Monitoring network latency over time and detecting packet loss in real-time.

8.2 Evaluation Methods and Problems

Introduction to Evaluation Methods

 Evaluation is a critical part of system administration that involves measuring the

performance, reliability, and efficiency of systems and services.
 Understanding evaluation methods enables admins to make informed decisions for
capacity planning, system upgrades, and performance improvements.

Common Evaluation Methods

1. Benchmarking:
o The process of measuring the performance of a system by running a set of
standardized tests.
o Popular benchmarks: SPEC, Geekbench, SysBench.
o Using benchmarking results to compare systems or versions.
2. Stress Testing:
o Testing a system under heavy load to determine its breaking point or maximum
capacity.
o Tools for stress testing: Stress, Apache JMeter, or custom scripts.
3. Load Testing:
o Simulating high levels of traffic or user requests to evaluate the behavior under
load.
o Tools: LoadRunner, JMeter, Siege.
4. Performance Profiling:
o Tools and techniques for identifying the most resource-hungry processes (e.g.,
CPU profiling with perf in Linux, Windows Performance Toolkit).
o Analyzing performance bottlenecks at both application and system levels.
5. Availability and Reliability Testing:
o Ensuring the system is available for use and identifying downtime causes.
o Measuring uptime percentages (e.g., 99.99% uptime).
6. Security Evaluation:
o Evaluating the security of systems by performing vulnerability scans (e.g., using
tools like OpenVAS or Nessus).
o Security audits: Checking for compliance with best practices or industry
standards.
Problems in System Evaluation

1. Data Inconsistencies:
o Evaluation may be affected by inconsistent data inputs or poor logging practices.
o The importance of reliable data collection and maintenance.
2. System Complexity:
o Evaluating large-scale or distributed systems may not always yield
straightforward results due to complexity.
o Difficulty in replicating production environments for testing purposes.
3. Time Constraints:
o Testing and evaluation can be time-consuming, especially when involving stress
or load testing.
o Balancing thorough evaluation with operational requirements.
4. False Positives/Negatives:
o Risks associated with misinterpreting test results or misconfigurations leading to
inaccurate evaluation outcomes.
o The importance of clear and repeatable evaluation methods.

Real-World Evaluation Examples

 How evaluating a website’s traffic during peak hours can help scale infrastructure.
 Case study on improving database query performance through profiling.

8.3 Evaluating a Hierarchical System

Introduction to Hierarchical Systems

 Definition: Hierarchical systems are multi-tiered structures where various components

(e.g., hardware, software, network layers) interact in a specific hierarchy, with higher
levels controlling or providing resources to lower levels.
 Examples: Client-server architecture, cloud computing models, and enterprise network
designs.

Evaluating Performance in Hierarchical Systems

1. Top-Down vs. Bottom-Up Approaches:

o Top-Down: Start by evaluating the system as a whole and then drill down to find
bottlenecks or failures.
o Bottom-Up: Evaluate individual components or subsystems and then scale the
analysis to the entire hierarchy.
2. Evaluating Layers:
o Assessing each layer of a hierarchical system: Physical layer (network cables),
data link layer (switches), transport layer (protocols), and application layer
(services).
 o Example: In a client-server architecture, evaluating the server load and client
response time.
3. Dependency Analysis:
o Understanding how performance at one level (e.g., network layer) affects the
overall system.
o Using tools like ping, tracert, or SNMP monitoring to evaluate network
dependencies.
4. Hierarchical Performance Metrics:
o Defining performance metrics for each layer: Throughput, latency, packet loss,
error rates.
o Creating a performance baseline for comparison.
5. Fault Isolation in Hierarchical Systems:
o Identifying which layer in the hierarchy is causing a fault or degradation in
performance.
o Using the OSI model for systematic fault isolation.

Challenges in Hierarchical System Evaluation

 Distributed Nature: Complex dependencies between layers can make it difficult to

pinpoint problems.
 Scalability Issues: Hierarchical systems may not scale well, leading to performance
degradation at higher levels of traffic.

Evaluating Cloud-Based Hierarchical Systems

 Evaluating cloud infrastructure for performance (e.g., AWS, Azure).

 Understanding the unique challenges posed by virtualized, multi-tenant cloud
environments.

8.4 Faults
Types of Faults

1. Hardware Faults:
o Failures in physical components (e.g., hard drives, network cards).
o Impact of hardware failures on system availability and performance.
2. Software Faults:
o Bugs, memory leaks, misconfigurations that cause system crashes or slowdowns.
o Identifying and troubleshooting software failures through logs and system
diagnostics.
3. Network Faults:
o Loss of connectivity, slow network speeds, DNS issues, or routing failures.
o Troubleshooting network faults with tools like Wireshark, ping, and tracert.
4. Environmental Faults:
o Power outages, overheating, environmental factors that affect system reliability.
 o Ensuring systems are housed in controlled environments with adequate power
backup.

Fault Diagnosis

1. Symptom Analysis:
o Understanding the signs and symptoms of various faults (e.g., high CPU usage,
application crashes).
o Using systematic approaches to isolate the fault.
2. Root Cause Analysis (RCA):
o The process of determining the underlying cause of faults.
o Tools and techniques for RCA: Fishbone diagrams, 5 Whys, log analysis.

Preventing Faults

1. Redundancy and Failover Systems:

o Using RAID for storage redundancy, clustered servers for high availability.
o Backup power systems (UPS) and automatic failover to prevent service
interruption.
2. Proactive Monitoring:
o Setting up continuous monitoring to identify potential faults early.
o Automating alerts based on thresholds.

8.5 Deterministic and Stochastic Behaviors

Introduction to Deterministic and Stochastic Systems

 Deterministic Behaviors: Systems where outputs are predictable from inputs, with no
randomness involved.
 Stochastic Behaviors: Systems with inherent randomness, where outputs can vary even
with the same inputs.

Deterministic Systems in Network Administration

1. Characteristics of Deterministic Systems:

o Predictable outcomes, controlled environments.
o Example: A server's performance under consistent, non-variable load.
2. Applications:
o Real-time systems, embedded systems where consistent performance is critical.

Stochastic Systems in Network Administration

1. Characteristics of Stochastic Systems:

o Randomness in system behavior due to external factors (e.g., network congestion,
varying workloads).
 o Impact of stochastic behavior on system performance and troubleshooting.
2. Applications:
o Network performance (latency, jitter) during periods of congestion or high traffic.

Modeling and Analyzing Stochastic Systems

1. Queuing Theory:
o Application of queuing models in networking (e.g., how packets queue in routers
under load).
o Basic concepts: arrival rate, service rate, waiting times.
2. Simulations:
o Using Monte Carlo simulations to model network traffic and predict system
performance under stochastic conditions.
3. Statistical Methods for Performance Analysis:
o Tools like Markov Chains, probability distributions for analyzing system
behavior.
o Example: Evaluating server response times under different traffic patterns.

End of Course