Exercise 02.

06: OpsCenter
In this exercise, you will accomplish the following:
• Use OpsCenter to create a cluster
• Use OpsCenter to collect metrics about the cluster
• Use OpsCenter to keep the cluster synchronized
• Use OpsCenter to backup and restore the cluster

Step 1: Enabling OpsCenter

1) Remove the previous cluster in order to demonstrate how easy it is to create a new
cluster using OpsCenter. Start by shutting down each of the nodes. On both the
ds210-node1 and ds210-node3 instances, run the following command:

sudo service dse stop

2) Next, remove the associated data files. In each of the three (3) terminal windows
(including ds210-node2), execute the following command:

sudo rm -rf /var/lib/cassandra /var/lib/spark /var/log/cassandra

/var/log/spark /etc/cassandra /etc/dse /var/lib/opscenter
/var/log/opscenter /etc/opscenter /var/lib/datastax-agent
/var/log/datastax-agent

3) Also, remove the DSE packages. In each of the three (3) terminal windows, execute the
following command:

sudo apt-get purge --auto-remove dse-full dse dse-libcassandra

dse-libspark dse-libgraph dse-libhadoop2-client dse-liblog4j
dse-libhadoop2-client-native dse-libsolr dse-libtomcat

4) Install OpsCenter on the ds210-node3 node. Remember, we have already performed

the following commands, so it is not required to run these as well. The following
commands are included for completeness and to summarize, but DO NOT EXECUTE:

echo "deb https://debian.datastax.com/enterprise stable main" | sudo tee -

a /etc/apt/sources.list.d/datastax.sources.list

curl -L https://debian.datastax.com/debian/repo_key | sudo apt-key add –

sudo apt-get update

5) In the ds210-node3 window, execute the following:

sudo apt-get install opscenter

6) It is now possible to start OpsCenter. In the ds210-node3 window, run the following
command:

sudo service opscenterd start

7) It may take several minutes for OpsCenter to initialize, but once it does, OpsCenter
should be accessible via a web browser. Enter the following URL in your browser’s
address field:

http://<EXTERNAL IP ADDRESS OF ds210-node3>:8888

8) Substitute the IP address for the node (the one used to SSH connect to the node). Also,
if attempting to access OpsCenter before it is ready, the browser will display an error.
Wait a minute or two and try again.

9) Remember from previous exercises how much work it is to set up a cluster; installing
the software on each of the nodes, configuring the settings, and then starting the
service. OpsCenter makes this process much easier. We will use OpsCenter to create a
single configuration and then replicate it across nodes in our cluster. In this exercise,
we will only create a two-node cluster, but it is just as easy to create a cluster with
several nodes. From the OpsCenter browser, select the option Create a new
cluster as shown below, and then click the Get Started button.
10) Notice that Lifecycle Manager, as shown in the screenshot below, opens in a
separate tab in the browser. To create the cluster, follow the steps outlined in the
Lifecycle Manager. Start by clicking on the SSH Credentials link:

11) Credentials allow OpsCenter to access the machines within the cluster. In the SSH
Credentials screen, click on the plus (+) sign to add credentials:

12) Fill out the Add Credential form. Supply a name for the credential; this will be used
later when setting up the node. Also fill in the Login User field with ubuntu. Click on
the Password radio button, and enter the password datastax.

13) After clicking the Save button, the Lifecycle Manager screen appears again. On the
left side of the window the screen highlights >_ SSH Credentials. On the right side is
the credential just created (highlighted with the green oval in the image below). If you
were to create more credentials, you would see them in the list to the right as well.
Click on the Clusters link to view the steps of the Lifecycle Manager again:
14) From the Lifecycle Manager Clusters view, click on Step 2: Adding a Config
Profile and then click on the plus (+) sign to add a profile.

15) A profile is a specific configuration for a node. Add a profile by clicking on the plus (+)
sign.
16) This configuration includes setting up the cassandra.yaml configuration file and other
configuration files. Enter a name for this profile and select the desired DSE version.
Modify various configuration files, including cassandra.yaml, by selecting the file from
the list on the left and changing the fields as needed. For this exercise, retain the
default values and click the Save button:

17) Click on the Clusters link to return to the list of cluster configuration steps and then
click on Step 3, Adding a Repository. Inform OpsCenter where to access the
software to load onto the various nodes; this is the purpose of adding a repository.

18) From the Repositories selection, click on the plus (+) sign to add a new repository.
Fill out the name of the repository and then click Save:

19) Having created the credentials, profile, and repository, we are ready to create the
cluster. Click on the Clusters link again to see the Lifecycle Manager’s steps for
creating a cluster.

20) Click on Step 4, Adding a Cluster, populating it with datacenters and

nodes, and starting an install job to get to the Add Cluster dialog box. Fill in
the Name field, which will include the name of the cluster. Select the credential
previously configured from the drop-down list. Similarly, select the profile and
repository. You will also need to fill in the password you will use for the cassandra
user. The Old Password is the default cassandra password, which is cassandra. You will
need to enter the old password twice for verification purposes. Enter a new password
of your choosing, again this needs to be done twice. This new password will be the
 password for accessing Cassandra as the cassandra user. After completing these fields,
click Save:

21) Returning to the Clusters view of the Lifecycle Manager, a new cluster should now be
present in the Clusters list. Add nodes to this cluster by first clicking on the cluster in
the list:
22) After clicking on the newly created cluster an additional column labeled Datacenters
appears. These are the datacenters associated with the currently selected cluster.
Since we have not yet defined any datacenters for the cluster we are building, there
are no datacenters in the list. Create one by clicking on the plus (+) sign above the
Datacenters list:

23) In the Add Datacenter dialog, fill in the datacenter name. Notice that the profile and
credentials inherit the cluster settings, so these do not require completion. Click the
Save button:

24) Returning to the Clusters view of the Lifecycle Manager, the newly created cluster as
well as the new datacenter should both be visible. Click on the datacenter. This
 datacenter does not yet have any nodes; add nodes by clicking on the plus (+) sign
above the Nodes list.

25) In the Add Node dialog, give the node a name, assign a rack (use the default), and fill in
the IP address for the node. The IP address you will use should be an IP address
accessible by the machine running OpsCenter, typically an internal IP. For this exercise,
it is the internal IP address rendered when you run the command hostname -i on the
target node. Click Save:
26) Repeat the previous steps to add a second node to the cluster. Give the second node
its own unique name as well as its own IP address. Use the same or the default rack
number.

27) By this point OpsCenter should know how the cluster should look based on the
provided description, but the creation of the cluster is still pending. That is, we have
not yet installed, configured, and started the Cassandra software on the nodes. Click
on the ellipses to the right of the cluster name to reveal a drop-down menu. Then,
click Install.
28) A Run Installation Job dialog will appear. Click the Submit button.

29) Next, a pop-up window appears. Click on the View Job Summary link to monitor the
cluster as it installs and starts Cassandra.
 30) The Job Summary view should be visible. Click on View Job Events to review the
steps the job has completed.

31) If the cluster comes up cleanly, a successful log window will appear. This may take a
few minutes to complete, so be patient:

If the cluster has any issues or problems coming online, use the log to investigate the problem
and perform any needed adjustments.
Step 2: Monitoring
With the cluster up and running under OpsCenter control, life just became a lot easier. In this
section of the exercise we will examine how to monitor the behavior of the cluster.

1) Click on the top-left corner of the Lifecycle Manager where it says OpsCenter
Monitoring. This will switch your browser back to the original OpsCenter tab. If the
OpsCenter tab still has the Create a new cluster dialog showing, reload the page.
Once the page reloads, click on the dashboard icon as indicated by the red arrow in the
following screenshot:

This dashboard view consists of several widgets and graphs. Customize the dashboard as
needed.

2) Click on Add Graph on right-hand side to add new graphs to the dashboard.
 3) In the Add Metric dialog, select Heap Used for the Metric, and All Nodes from the
Node drop-down list. Then, click Add Metric.

4) In the Metrics on this Graph dialog, click Save Graph:

An additional graph will appear on the dashboard (you may have to scroll down to see it):
Step 3: Services
OpsCenter provides several useful cluster-wide services to save users from operation
headaches. Let’s investigate these services.

1) To view the list of services in the OpsCenter tab, click on the cluster name and then
select the Services icon from the column of icons on the left:

2) The description column explains the function of each service.

Start by looking at the Best Practice Service. This service uses rules to check the
configuration and functioning of the cluster. Click on the Details link on the right side of the
Best Practice Service row:
 3) The Best Practice service view shows a list of rules’ categories. Each category
contains several rules. To review the rules in more depth, click on the category name.
Look at the Config Advisor set of rules, and then click on Config Advisor:

4) Notice the NodeSync Not Running rule near the bottom of the Config Advisor
section. This rule verifies the NodeSync feature is running on your cluster (You may
recall that NodeSync keeps replicas synchronized. NodeSync also needs to be enabled
on the keyspace/table).

By default, these rules run once a day at 5:00AM GMT. Let’s see if we can get one of these rules
to run now. Click on the Configure link on the right side of the NodeSync not running row.
 5) Set the rule to run in just a minute or so from the present time. Determine what the
current time is in GMT (Google can provide this information). Configure the rule to run
in the next few minutes by setting the Schedule fields to two minutes from the
present (be careful to set the correct time and date). Having completed the
configuration, click Save Rule:

Note: If the rule does not run as expected, check the date. It is possible the rule is configured to
run a day later than intended.

6) Back at the Best Practice service view, wait for the rule to run. Once the rule
completes it displays a Rule Succeeded message, meaning NodeSync is running on the
cluster:
7) Next, determine how easy it is to create a backup using OpsCenter. Start by creating
some data. In the ds210-node1 window, start cqlsh and execute the statements
below. To run cqlsh on nodes configured by OpsCenter, just type the command cqlsh
-u cassandra -p <CASSANDRA USER’S PASSWORD> `hostname –i`.

CREATE KEYSPACE bedrock WITH replication = {'class':

'NetworkTopologyStrategy', 'Datacenter1': '2'} AND durable_writes =
true;

CREATE TABLE bedrock.actors (

id uuid PRIMARY KEY,
name text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
INSERT INTO bedrock.actors (id, name)
VALUES ( uuid(), 'Fred Flinstone');

8) Confirm the row was successfully inserted by executing the following command in
cqlsh:

SELECT * FROM bedrock.actors;

9) Next, perform the back up. Returning to the OpsCenter web interface, click on the
Services button, and then click on the Details link associated with the Backup
service.

10) To schedule regular backups, click on the SCHEDULED BACKUPS tab. For this exercise
we will perform only a single backup. Click on the Create Backup button.

11) Fill out the Create Backup dialog by selecting the Run Now radio button and selecting
All Keyspaces. Then, click Create Backup:
12) Because the backup is just the creation of hard-links, it only takes a few seconds. The
backup will appear in the list. Click on the row to review the backup report:

13) Click on the X in the top-right corner to close the report:

14) Consider what needs to be done to perform a restore from the backup. Truncate the
actors table to simulate the loss of data:

TRUNCATE TABLE bedrock.actors;

15) Again, confirm the data is lost by executing the following command and verifying there
is no data:

SELECT * FROM bedrock.actors;

16) Back in OpsCenter, click on Restore Backup:

17) Select the backup copy to restore and click Next:

Notice the other available options; restore to a point in time or from other locations.

18) Choose the keyspace(s) to restore. For this exercise select All Keyspaces and click
Restore Backup:
19) Finally, confirm restore should be run. Click Start Restore:
20) Watch the progress of the restore. The restore consumes more time than the backup
because the restore needs to rebuild the SSTables. Wait for the backup to complete.

21) Confirm the restore worked. Back in cqlsh, perform the following query and confirm
the record is back:
 SELECT * FROM bedrock.actors;

END OF EXERCISE