Hog user manual v3

2016 copyrighted by the KEYW Corp. All rights reserved.
Integrated Development Environment
[ HOG ]
User Manual
Developed under CCIS IRaD
Project Owner: Rick Gould
Author: Connor J. Drew

2016 copyrighted by the KEYW Corp. All rights reserved. 2
Table of Contents
Introduction ....................................................................................................................3
Installation ......................................................................................................................5
Design .............................................................................................................................5
Landing Page................................................................................................................5
Simple Page .................................................................................................................5
New...........................................................................................................................5
Edit............................................................................................................................5
List.............................................................................................................................6
Complex Page ..............................................................................................................6
New...........................................................................................................................6
Edit............................................................................................................................6
List.............................................................................................................................6
Outputs .........................................................................................................................7
Show Graphs................................................................................................................7
Node Functions ..............................................................................................................8
Relational Operators.....................................................................................................8
Eval Functions ..............................................................................................................8
Load Store Functions....................................................................................................9
Load Types...................................................................................................................9
Storage Types ..............................................................................................................9
Tuple Bag Map Functions...........................................................................................10
Output.........................................................................................................................10
Versioning.....................................................................................................................10
Change Log ................................................................................................................10
Bugs............................................................................................................................10
FAQ................................................................................................................................10
Can I view exported data in Excel? ............................................................................11
What browsers support Hog? .....................................................................................11
Links..............................................................................................................................11
Developers....................................................................................................................11

Introduction
This Interactive Development Environment (codename: Hog) allows for the creation of
Apache Pig scripts.
The Simple interface allows users to create scripts without the knowledge of Pig syntax
using drag and drop functions. The Complex interface is an all-inclusive development
environment to write, store and run scripts that the user has created – even if those
scripts were not made using the Simple interface. Hog can display log, numerical and
graphical output.
This manual will provide the user with detailed instructions of Hog’s various features
and how to leverage them.
Hog was developed in 2016 under CCIS IRaD.

Web Design

Installation
To install Hog, follow the instructions laid out in the Hog Installation Guide.
Design
Landing Page
The landing page shows the user the last five complex and the last five simple scripts
that were edited and the time they were edited. From this page the user can click on a
script to view it in edit mode, list view, or create a new script.
Simple Page
New
The Simple New page allows the user to create new scripts by using a drag and drop
interface. The user can select from a number of pre-entered Pig functions to create a
script without in-depth knowledge of the language; this page enters all the syntax in
automatically.
To begin, there are two fields located at the top of the page: Name and Pig Arguments.
The Name field will be the title of the new script (note that any space characters will be
replaced by underscores once saved). The Pig Arguments field allows the user to pass
in command line arguments to the Pig binary.
The ‘+’ sign will enable the user to add Nodes and begin creating their Pig script. For a
list of available Nodes and their functions, see “Node Functions”.
Once nodes have been created, the user may connect them how they desire, or
immediately can hit the Save button, which will save the new script and bring the user to
the Simple Edit page. Note that the Save button will only be available once all the fields
have been supplemented.
Edit
The Simple Edit page allows the user to edit, run and save settings associated with the
script strictly by adding or removing nodes. The user can edit all areas of the script
including the command line arguments, the name as well as the script itself.
The Edit page retains the information that was input in the New page when the script
was created. The Name, Pig Arguments, Nodes will all be available for editing.
From this page, users may run their script by clicking the Run button. By running the
script, Outputs will be generated (see “Outputs”). To stop the compiler, click the Kill
button.
Another feature on the Simple Edit page is the Complex Script Viewer tab. This tab will
allow scripts created by the Simple page to be viewed in code form. It can take users to
the Complex Edit page (see Complex Page), as well as give the user the option to
delete their script entirely.

List
The Simple List page shows all the scripts the user has created with the Simple
interface. They are listed on the page with two buttons, Run and Edit.
The user is able to quickly and conveniently run the scripts from this page. All output
from running scripts will show in a modal as it does on other pages.
Attempting to edit the script will take the user to the Simple Edit page.
In the bottom right hand corner of the page is a plus sign. When clicked the user will be
taken to the new script page to create a new simple script.
The drop down menu will allow scripts to be viewed in their Complex view, as well as
allowing the user to delete the scripts.
Scripts are sorted alphabetically. The user may also use the filter to find the desired
script.
Complex Page
New
The Complex New page allows the user to create new Pig scripts in a traditional
environment. Similar to the Simple New page, this page contains Name and Pig
Argument fields, along with a Save button that will take you to the Complex Edit page.
Unique to this New page is the ability to upload a script to Hog.
Once the script is saved, the Complex Edit page will appear.
Edit
The Complex Edit page allows the user to edit all branches of the script, including the
name, command line arguments, settings associated with the output and the script
itself. It is designed for more experienced users, as there is no drag and drop node
editing available on this page.
Features of the Edit page include autocomplete functions and colored font for usability.
The script can be run and outputs are displayed in a modal from this page. The user
may also download the script they created by selecting the Download/Export button.
By clicking the Settings button, users may toggle which graphs they would like to see
when the Output is produced. They may also delete the script from this modal.
List
The Complex List page displays all scripts the user has written on both the Simple and
Complex interfaces. From this page, the user can select a script and be taken to the
Complex Edit page to either run or edit it.
Scripts are sorted alphabetically. The user may also use the filter to find the desired
script.
Similar to the Simple List page, there is a plus sign in the bottom right corner that, when
clicked, the user will be taken to the new script page to create a new complex script.

Outputs
There are 5 different types of outputs displayed by running a Pig script on Hog:
Results
All of the outputs that have been successfully output by the script
Info Logs
Indicators of the progress that is being made while the script is running
Warnings
Abnormalities that were detected while running the script, but will not prevent the
script from completing
Errors
Issues that were detected while running the script that causes its compilation to
cease
Graphs
Creates various graphs from the Results outputs (see ‘Show Graphs’)
Additionally, users may opt to “Show All”, which shows a color-coded display of all of
the various types of log outputs in one view.
Show Graphs
Hog has the ability to display various graphs, using information from the results output
and displaying the ones that are able to be displayed graphically.
Graphs may be filtered by the number of outputs. Via the Settings Tab, the Data Layout
field also provides the ability to assign the x and y indices to specific values, as well as
switching between number of outputs filters.
While viewing the graph, the slider allows for alterations to the number of outputs to be
displayed.

Node Functions
Descriptions of the different Pig Node Functions utilized in Hog:
Relational Operators
Cross
Computes the cross product of two or
more relations
Initial Cube
Performs cube/rollup operations
Cube
Cube operation computes aggregates
for all possible combinations of specified
group by dimensions. The number of
group combinations generated by cube
for n dimensions will be 2^n
Rollup
Rollup operations computes multiple
levels of aggregates based on
hierarchical ordering of specified group
by dimensions. Rollup is useful when
there is hierarchical ordering on the
dimensions.
Distinct
Removes duplicate tuples in a relation
Filter
Selects tuples from a relation based on
some condition
Group
Takes in an input and groups by a type
Join
Performs an inner join of two or more
relations based on common field values
Limit
Limits the number of outputs tuples
OrderBy
Sorts a relation based on one or more
fields
Rank
Returns each tuple with the rank within
a relation
Sample
Selects a random sample of data based
on the specified sample size
Stream
Sends data to an external script or
program
Union
Computes the union of two or more
relation
Eval Functions
Sum
Take in an input and group and returns
the sum of the type
Average
the average of the type
Count
the count of the type
Diff
Compares two fields in a tuple

Max
Computes the maximum of the numeric
values or chararrays in a single-column
bag. Requires a preceding GROUP ALL
statement for global maximums and a
GROUP BY statement for group
maximums
Min
Computes the minimum of the numeric
values or chararrays in a single-column
bag. Requires a preceding GROUP ALL
statement for global minimums and a
GROUP BY statement for group
minimums
Size
Computes the number of elements
based on any Pig data type
Tokenize
Splits a string and outputs a bag of
words
BagToString
Concatenate the elements of a Bag into
a chararray string, placing an optional
delimiter between each value
Concat
Concatenates two or more expressions
of identical type
Count Star
Computes the number of elements in a
bag
Load Store Functions
Load
Loads data from the file system
Storage
Store variable into location, requires a
storage type
Load Types
JSONLoader
Load JSON data
TextLoader
Loads unstructured data in UTF-8
format
Storage Types
BinStorage
Loads and stores data in machine-
readable format
JSONStorage
Store JSON data
PigDump
Stores data in UTF-8 format
PigStorage
Loads and stores data as structured text
files
HBaseStorage
Loads and stores data from an HBase
table
AvroStorage
Loads and stores data from Avro files
TrevniStorage
Loads and stores data from Trevni files
OrcStorage
Loads from or stores data to Orc file

AccumuloStorage
Loads or stores data from an Accumulo
table. The first element in a Tuple is
equivalent to the row form the Accumulo
be grouped in various static or
wildcarded ways. Basic wildcarding Key
while the columns in that row can
functionality exists to group various
columns families/qualifiers into a Map
for LOADs, or serializes a Map into
some group of column families/
qualifiers on STOREs
Tuple Bag Map Functions
ToTuple
Converts one or more expressions to
type tuple
ToBag
Converts one or more expressions to
type bag
ToMap
Converts key/value expression pairs into
a map
Top
Returns the top-n tuples from a bag of
tuples
Output
Dump
Takes in an input and outputs to standard out
Versioning
1.0.0 – Released on 6/30/16
Change Log
N/A
Bugs
None in version 1.0.0
FAQ
Are all PIG functions available for drag and drop in the Simple Page?
No. Apache Pig has a plethora of functions available for scripting. The HOG simple
page covers only functions and functionality that we have deemed the most common.
Can I add functions to the drag and drop page?
Not at this time. The included functions and their functionality are the only functions
available in the Simple page. You made add any of Pig’s functions to your script by
editing by hand in the Edit page.

Proprietary © 2016 KEYW Corporation. All rights reserved. 11
I see that I can export the Output. Can I export a graph, too?
Only Output data can be exported.
Can I view exported data in Excel?
Yes. The output data exports in CSV format. This can then be ingested into Excel or
viewed with any text editor.
What browsers support Hog?
It is recommended that you use Google Chrome.
Links
Pig Documentation: https://pig.apache.org
Developers
Rick Gould – Project Owner
Kevin Coxe – Full Stack Developer
Steve Emmerich – Full Stack Developer

Hog user manual v3

Recommended

More Related Content

What's hot (20)

Similar to Hog user manual v3 (20)

Recently uploaded (20)

Hog user manual v3