APIGW - SOP

Maintenance Overview

1 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 General Administration and Maintenance

Topics to cover:

 Starting and Stopping the Apigee System

 Router/Message Processor Maintenance
 Recurring Analytics Services Maintenance Tasks
 Apache Cassandra Maintenance Tasks
 Apache Zookeeper Maintenance Tasks
 OpenLDAP Maintenance Tasks
 Setting the log level for an Edge component

2 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Starting and Stopping the Apigee System

The order of stopping and starting the subsystems is important. Start and stop scripts
are provided that take care of that for you for Edge components running on the same
node.

Stop order: If you install Edge on multiple nodes, then you should stop Edge
components on those nodes in the following order:
 Management Server (edge-management-server)
 Message Processor (edge-message-processor)
 Postgres Server (edge-postgres-server)
 Qpid Server (edge-qpid-server)
 Router (edge-router)
 Edge UI (edge-ui)
 Cassandra (apigee-cassandra)
 OpenLDAP (apigee-openldap)
 PostgreSQL database (apigee-postgresql)
 Qpidd (apigee-qpidd)
3  ZooKeeper (apigee-zookeeper)
© 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL
 Starting and Stopping the Apigee System

Start order: If you install Edge on multiple nodes, then you should start Edge
components on those nodes in the following order:
 Cassandra (apigee-cassandra)
 OpenLDAP (apigee-openldap)
 PostgreSQL database (apigee-postgresql)
 Qpidd (apigee-qpidd)
 ZooKeeper (apigee-zookeeper)
 Management Server (edge-management-server)
 Message Processor (edge-message-processor)
 Postgres Server (edge-postgres-server)
 Qpid Server (edge-qpid-server)
 Router (edge-router)
 Edge UI (edge-ui)

4 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Starting and Stopping the Apigee System

 The following scripts detect the Apigee components configured to run on the system
on which the script is executed, and will start or stop only those components in the
correct order for that node.

 To stop Apigee Edge, Apache Cassandra, and Apache ZooKeeper:

/opt/apigee/apigee-service/bin/apigee-all stop
 To start Apache ZooKeeper, Apache Cassandra, and Apigee Edge:
/opt/apigee/apigee-service/bin/apigee-all start
 To check if the server is running:
/opt/apigee/apigee-service/bin/apigee-all status
 To check the status of httpd(Dev portal)
sudo systemctl status httpd
 To start/stop/restart httpd
sudo systemctl start/stop/restart httpd.service

5 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Starting and Stopping the Apigee System
Starting, Stopping, Restarting, and Checking the Status of an Individual Service:

 You can use the following tool to start/stop/restart or check the status of an individual
Apigee service on any specific server.

 /opt/apigee/apigee-service/bin/apigee-service <service> <command>

where:
 <service> is one of the following: edge-management-server, edge-ui, edge-router, edge-
message-processor, edge-qpid-server, edge-postgres-server, apigee-zookeeper, apigee-
cassandra, apigee-openldap, apigee-qpidd, apigee-postgresql
 <command> is one of the following: start, stop, restart, status

 For example, to start, stop, or restart the Management Server, run the following
commands:
 apigee-service edge-management-server start
 apigee-service edge-management-server stop
 apigee-service edge-management-server restart
6 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL
 Router/Message Processor Maintenance

 It is a good practice to disable reachability on a server during maintenance, such as

for a server restart or upgrade. When reachability is disabled, no traffic is directed to
the server. For example, when reachability is disabled on a Message Processor,
Routers will not direct any traffic to that Message Processor.

For example, to upgrade a Message Processor, you can use the following procedure:
 Disable reachability on the Message Processor.
 Upgrade the Message Processor.
 Enable reachability on the Message Processor.

7 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Disabling/enabling reachability on a Message Processor

 To disable reachability on Message Processor, you can just stop the Message
Processor:
 > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop

 The Message Processor first processes any pending messages before it shuts down.
Any new requests are routed to other available Message Processors.

 To restart the Message Processor, use the following commands:

 > /opt/apigee/apigee-service/bin/apigee-service edge-message-processor start

 To Check if the message-processor is ready to process

 ​> /opt/apigee/apigee-service/bin/apigee-service edge-message-processor
wait_for_ready
The wait_for_ready command returns the following message when the Message
Processor is ready to process messages:
Checking if message-processor is up: message-processor is up.

8 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Disabling/enabling reachability on a Router

 In a production environment, you typically have a load balancer in front of the Edge
Routers. Load balancers monitor port 15999 on the Routers to ensure that the Route
is available.

 Configure the load balancer to perform an HTTP or TCP health check on the Router
using the following URL:
 curl –v http://router_IP:15999/​v1/servers/self/reachable

 This URL returns an HTTP 200 response code if the Router is reachable.
 Make the router un reachable from Load balancer level and then try the same
command to check unreachability

9 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Checking the status of Router or Message Processor

 To get the status of a Router, make a request to port 8081 on the Router:
 > curl -v http://<routerIP>:8081/v1/servers/self/up or self/reachable

 If the Router is up, the request returns "true" in the response and HTTP 200. Note
that this call only checks if the Router is up and running. Control of the Router's
reachability from a load balancer is determined by port 15999

 To get the status of a Message Processor:

 > curl http://<mpIP>:8082/v1/servers/self/up

10 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Apache Cassandra Maintenance Tasks

 The Apache Cassandra ring nodes require periodic maintenance to ensure

consistency across all nodes. To perform this maintenance, use the Cassandra
"nodetool -h localhost repair" command.
 This maintenance must be run on every Cassandra node at least every seven days in
order to eliminate problems related to Cassandra "forgotten deletes". Running
"nodetool -h localhost repair " imposes a significant load on the system, so Apigee
recommends that this process only be run during periods of relatively low workload.

11 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Cassandra Log File Maintenance

 Cassandra logs are stored in the /opt/apigee/var/log/cassandra directory on

each node. By default, a maximum of 50 log files, each with a maximum size of
20 MB, can be created; once this limit is reached older logs are deleted when
newer logs are created.

 If you should find that Cassandra log files are taking up excessive space, you can
modify the amount of space allocated for log files by editing the log4j settings.
 Edit /opt/apigee/customer/application/cassandra.properties to set the following
properties. If that file does not exist, create it:

conf_log4j-server_log4j.appender.r.maxfilesize=20MB # max file size

conf_log4j-server_log4j.appender.r.maxbackupindex=50 # max open files

 Restart Cassandra by using the commands:

$ /opt/apigee/apigee-service/bin/apigee-service apigee-cassandra restart

12 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Recurring Analytics Services Maintenance
Tasks
 Many Apigee Analytics Services tasks can be performed using standard Postgres
utilities. The routine maintenance tasks you would perform on the Analytics
database – such as database reorganization using VACUUM, reindexing and log
file maintenance - are the same as those you would perform on any PostgreSQL
database.
 Postgres autovacuum is enabled by default, so you do not have to explicitly run
the VACUUM command. Autovacuum reclaims storage by removing obsolete
data from the PostgreSQL database.
 No pruning is required for Cassandra. Expired tokens are automatically purged
after 180 days.
 As the amount of analytics data available within the Apigee repository increases,
you may find it desirable to "prune" the data beyond your required retention
interval. Run the following command to prune data for a specific organization
and environment:

13 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Organization and Environment
Maintenance
Analytics file pruning script:

#!/bin/bash
# main
if [ $# -lt 3 ]; then
echo "Usage: $0 <Organization> <Env> <NoOfDaysToPurgeBackFromCurrentDay>";
exit 1;
fi

org=$1;
env=$2;
NoOfDaysToPurgeBackFromCurrentDay=$3;

for i in `psql -U postgres -d apigee -c "select tablename from analytics.childfactables where customer='$org~$env' and
to_timestamp(starttime/1000) < (SELECT CURRENT_TIMESTAMP -INTERVAL '$NoOfDaysToPurgeBackFromCurrentDay
DAYS')" |grep fact`
do
psql -U postgres -d apigee -c "drop table if exists analytics.\"$i\" cascade ";
echo "$i partition is dropped"
done ;

psql -U postgres -d apigee -c "delete from analytics.childfactables where customer='$org~$env' and to_timestamp(starttime/1000) <
(SELECT CURRENT_TIMESTAMP -INTERVAL '$NoOfDaysToPurgeBackFromCurrentDay DAYS')";

14 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Apache Zookeeper Maintenance Tasks

 Apache ZooKeeper has a number of "four-letter commands" that can be helpful

in determining the current status of ZooKeeper voter and observer nodes. These
commands can be invoked using "nc", "telnet" or another utility that has the
ability to send commands to a specific port.

Use the “four-letter commands” to determine Zookeeper status

 http://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkCommand
s
has more information

Old snapshot files can be periodically removed

 http://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_maintenance
has more information

15 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Zookeeper Log File Maintenance

 Apache Zookeeper log files are kept in /opt/apigee/var/log/zookeeper.

Normally, log file maintenance should not be required, but if you find that
there are an excessive number of ZooKeeper logs or that the logs are very
large you can modify ZooKeeper’s log4j properties to set the maximum file
size and file count.
 Edit /opt/apigee/customer/application/zookeeper.properties to set the
following properties. If that file does not exist, create it.
 Set the following properties in zookeeper.properties:
conf_log4j_log4j.appender.rollingfile.maxfilesize=10MB # max file size
conf_log4j_log4j.appender.rollingfile.maxbackupindex=50 # max open files
 Make sure the file is owned by the "apigee" user:
> chown
apigee:apigee /opt/apigee/customer/application/zookeeper.properties
 Restart ZooKeeper by using the command:
$ /opt/apigee/apigee-service/bin/apigee-service apigee-zookeeper restart

16 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Tools at a glance

17 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 OpenLDAP Maintenance Tasks

 Log file location

 OpenLDAP log files are contained in the directory /opt/apigee/var/log. These
files can be periodically archived and removed in order to ensure that they do
not take up excessive disk space.
 Information on maintaining, archiving and removing OpenLDAP logs can be
found in Section 19.2 of the OpenLDAP manual at
http://www.openldap.org/doc/admin24/maintenance.html.

18 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Setting the log level for an Edge component

 By default, Edge components use a logging level of INFO. However, you

can set the logging level for each Edge component. For example, you
might want to set it to DEBUG for the Message Processor, or to ERROR
for the Management Server.
 The available log levels are: ALL, DEBUG, ERROR, FATAL, INFO, OFF,
TRACE, WARN.
 To set the log level for the component, you have to edit the component's
properties file to set a token, then restart the components:
 For the Edge UI, the tokens
are conf_logger_settings_application_log_leveland conf_logger_settings
_play_log_level. Set them to the same value.
 For all other Edge components, the token is conf_system_log.level

19 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Setting the log level for an Edge
component(Cont.)
 To set the log level for the Edge UI:
 Open /opt/apigee/customer/application/ui.properties in an editor. If the
file does not exist, create it.
 Set the following properties in ui.properties to the desired log level. For
example, to set it to DEBUG:
conf_logger_settings_application_log_level=DEBUG
conf_logger_settings_play_log_level=DEBUG
 Save the file.
 Make sure the properties file is owned by the 'apigee' user:
> chown apigee:apigee /opt/apigee/customer/application/ui.properties
 Restart the Edge UI:
/opt/apigee/apigee-service/bin/apigee-service edge-ui restart

20 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 ORG and ENV maintenance

21 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Thank You

22 © 2016 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL