ABSTRACT

Big Data concern large-volume, complex, growing data sets with multiple,
autonomous sources. With the fast development of networking, data storage,
and the data collection capacity, Big Data are now rapidly expanding in all
science and engineering domains, including physical, biological and biomedical
sciences. This paper presents a HACE theorem that characterizes the features of
the Big Data revolution, and proposes a Big Data processing model, from the
data mining perspective. This data-driven model involves demand-driven
aggregation of information sources, mining and analysis, user interest
modelling, and security and privacy considerations. We analyse the challenging
issues in the data-driven model and also in the Big Data revolution.

1 .INTRODUCTION
Introduction Data Mining

Structure of Data Mining

Generally, data mining (sometimes called data or knowledge discovery) is the
process of analyzing data from different perspectives and summarizing it into
useful information - information that can be used to increase revenue, cuts costs,
or both. Data mining software is one of a number of analytical tools for
analyzing data. It allows users to analyze data from many different dimensions
or angles, categorize it, and summarize the relationships identified. Technically,
data mining is the process of finding correlations or patterns among dozens of
fields in large relational databases.
Data Mining Works
While large-scale information technology has been evolving separate
transaction and analytical systems, data mining provides the link between the
two. Data mining software analyzes relationships and patterns in stored
transaction data based on open-ended user queries. Several types of analytical
software are available: statistical, machine learning, and neural networks.
Generally, any of four types of relationships are sought:
Classes: Stored data is used to locate data in predetermined groups. For
example, a restaurant chain could mine customer purchase data to
determine when customers visit and what they typically order. This
2

information could be used to increase traffic by having daily specials.

Clusters: Data items are grouped according to logical relationships or
consumer preferences. For example, data can be mined to identify market
segments or consumer affinities.
Associations: Data can be mined to identify associations. The beer-diaper
example is an example of associative mining.
Sequential patterns: Data is mined to anticipate behavior patterns and
trends. For example, an outdoor equipment retailer could predict the
likelihood of a backpack being purchased based on a consumer's purchase
of sleeping bags and hiking shoes.
Data mining consists of five major elements:
1) Extract, transform, and load transaction data onto the data warehouse
system.
2) Store and manage the data in a multidimensional database system.
3) Provide data access to business analysts and information technology
professionals.
4) Analyze the data by application software.
5) Present the data in a useful format, such as a graph or table.

Different levels of analysis are available:

Artificial neural networks: Non-linear predictive models that learn
3

through training and resemble biological neural networks in structure.

Genetic algorithms: Optimization techniques that use process such as
genetic combination, mutation, and natural selection in a design based on
the concepts of natural evolution.
Decision trees: Tree-shaped structures that represent sets of decisions.
These decisions generate rules for the classification of a dataset. Specific
decision tree methods include Classification and Regression Trees
(CART) and Chi Square Automatic Interaction Detection (CHAID).
CART and CHAID are decision tree techniques used for classification of
a dataset. They provide a set of rules that you can apply to a new
(unclassified) dataset to predict which records will have a given outcome.
CART segments a dataset by creating 2-way splits while CHAID
segments using chi square tests to create multi-way splits. CART
typically requires less data preparation than CHAID.
Nearest neighbor method: A technique that classifies each record in a
dataset based on a combination of the classes of the k record(s) most
similar to it in a historical dataset (where k=1). Sometimes called the knearest neighbor technique.
Rule induction: The extraction of useful if-then rules from data based on
statistical significance.
Data visualization: The visual interpretation of complex relationships in
multidimensional data. Graphics tools are used to illustrate data
relationships.
Characteristics of Data Mining:
4

 Large quantities of data: The volume of data so great it has to be

analyzed by automated techniques e.g. satellite information, credit card
transactions etc.
Noisy, incomplete data: Imprecise data is the characteristic of all data
collection.
Complex data structure: conventional statistical analysis not possible
Heterogeneous data stored in legacy systems
Benefits of Data Mining:
1) Its one of the most effective services that are available today. With the
help of data mining, one can discover precious information about the
customers and their behavior for a specific set of products and evaluate
and analyze, store, mine and load data related to them
2) An analytical CRM model and strategic business related decisions can be
made with the help of data mining as it helps in providing a complete
synopsis of customers
3) An endless number of organizations have installed data mining projects
and it has helped them see their own companies make an unprecedented
improvement in their marketing strategies (Campaigns)
4) Data mining is generally used by organizations with a solid customer
focus. For its flexible nature as far as applicability is concerned is being
used vehemently in applications to foresee crucial data including industry
analysis and consumer buying behaviors
5) Fast paced and prompt access to data along with economic processing
techniques have made data mining one of the most suitable services that a
company seek
5

Advantages of Data Mining:

1. Marketing / Retail:
Data mining helps marketing companies build models based on historical
data to predict who will respond to the new marketing campaigns such as direct
mail, online marketing campaignetc. Through the results, marketers will have
appropriate approach to sell profitable products to targeted customers.
Data mining brings a lot of benefits to retail companies in the same way as
marketing. Through market basket analysis, a store can have an appropriate
production arrangement in a way that customers can buy frequent buying
products together with pleasant. In addition, it also helps the retail companies
offer certain discounts for particular products that will attract more customers.
2. Finance / Banking
Data mining gives financial institutions information about loan information
and credit reporting. By building a model from historical customers data, the
bank and financial institution can determine good and bad loans. In addition,
data mining helps banks detect fraudulent credit card transactions to protect
credit cards owner.
3. Manufacturing
By applying data mining in operational engineering data, manufacturers can
detect faulty equipments and determine optimal control parameters. For
example semi-conductor manufacturers has a challenge that even the conditions
of manufacturing environments at different wafer production plants are similar,
the quality of wafer are lot the same and some for unknown reasons even has
defects. Data mining has been applying to determine the ranges of control
parameters that lead to the production of golden wafer. Then those optimal
6

control parameters are used to manufacture wafers with desired quality.

4. Governments
Data mining helps government agency by digging and analyzing records of
financial transaction to build patterns that can detect money laundering or
criminal activities.
5. Law enforcement:
Data mining can aid law enforcers in identifying criminal suspects as well as
apprehending these criminals by examining trends in location, crime type,
habit, and other patterns of behaviors.
6. Researchers:
Data mining can assist researchers by speeding up their data analyzing
process; thus, allowing those more time to work on other projects.

LITERATURE SURVEY

1) Algorithms for Mining the Evolution of Conserved Relational States in

Dynamic Networks,
AUTHORS: R. Ahmed and G. Karypis
Dynamic networks have recently being recognized as a powerful abstraction to
model and represent the temporal changes and dynamic aspects of the data
underlying many complex systems. Significant insights regarding the stable
relational patterns among the entities can be gained by analyzing temporal
evolution of the complex entity relations. This can help identify the transitions
from one conserved state to the next and may provide evidence to the existence
of external factors that are responsible for changing the stable relational patterns
in these networks. This paper presents a new data mining method that analyzes
the time-persistent relations or states between the entities of the dynamic
networks and captures all maximal non-redundant evolution paths of the stable
relational states. Experimental results based on multiple datasets from realworld applications show that the method is efficient and scalable.
2) Novel Approaches to Crawling Important Pages Early
AUTHORS: M.H. Alam, J.W. Ha, and S.K. Lee
Web crawlers are essential to many Web applications, such as Web search
engines, Web archives, and Web directories, which maintain Web pages in their
local repositories. In this paper, we study the problem of crawl scheduling that
biases crawl ordering toward important pages. We propose a set of crawling
algorithms for effective and efficient crawl ordering by prioritizing important
pages with the well-known PageRank as the importance metric. In order to
score URLs, the proposed algorithms utilize various features, including partial
link structure, inter-host links, page titles, and topic relevance. We conduct a
large-scale experiment using publicly available data sets to examine the effect
of each feature on crawl ordering and evaluate the performance of many
algorithms. The experimental results verify the efficacy of our schemes. In
particular, compared with the representative Rank Mass crawler, the FPR-titlehost algorithm reduces computational overhead by a factor as great as three in
8

running time while improving effectiveness by 5 % in cumulative PageRank

3) Identifying Influential and Susceptible Members of Social Networks
AUTHORS: S. Aral and D. Walker
Identifying social influence in networks is critical to understanding how
behaviors spread. We present a method that uses in vivo randomized
experimentation to identify influence and susceptibility in networks while
avoiding the biases inherent in traditional estimates of social contagion.
Estimation in a representative sample of 1.3 million Facebook users showed that
younger users are more susceptible to influence than older users, men are more
influential than women, women influence men more than they influence other
women, and married individuals are the least susceptible to influence in the
decision to adopt the product offered. Analysis of influence and susceptibility
together with network structure revealed that influential individuals are less
susceptible to influence than noninfluential individuals and that they cluster in
the network while susceptible individuals do not, which suggests that influential
people with influential friends may be instrumental in the spread of this product
in the network.

4) Big Privacy: Protecting Confidentiality in Big Data

AUTHORS: A. Machanavajjhala and J.P. Reiter
A tremendous amount of data about individuals e.g., demographic
information, internet activity, energy usage, communication patterns and social
9

interactions are being collected and analyzed by many national statistical

agencies, survey organizations, medical centers, and Web and social networking
companies. Wide dissemination of microdata (data at the granularity of
individuals) facilitates advances in science and public policy, helps citizens to
learn about their societies, and enables students to develop skills at data
analysis. Often, however, data producers cannot release microdata as collected,
because doing so could reveal data subjects' identities or values of sensitive
attributes. Failing to protect confidentiality (when promised) is unethical and
can cause harm to data subjects and the data provider. It even may be illegal,
especially in government and research settings. For example, if one reveals
confidential data covered by the U. S. Confidential Information Protection and
Statistical Efficiency Act, one is subject to a maximum of $250,000 in fines and
a five year prison term.
5) Analyzing Collective Behavior from Blogs Using Swarm Intelligence
AUTHORS: S. Banerjee and N. Agarwal
With the rapid growth of the availability and popularity of interpersonal and
behavior-rich resources such as blogs and other social media avenues, emerging
opportunities and challenges arise as people now can, and do, actively use
computational intelligence to seek out and understand the opinions of others.
The study of collective behavior of individuals has implications to business
intelligence, predictive analytics, customer relationship management, and
examining online collective action as manifested by various flash mobs, the
Arab Spring (2011) and other such events. In this article, we introduce a natureinspired theory to model collective behavior from the observed data on blogs
using swarm intelligence, where the goal is to accurately model and predict the
future behavior of a large population after observing their interactions during a
training phase. Specifically, an ant colony optimization model is trained with
behavioral trend from the blog data and is tested over real-world blogs.
Promising results were obtained in trend prediction using ant colony based
10

pheromone classier and CHI statistical measure. We provide empirical

guidelines for selecting suitable parameters for the model, conclude with
interesting observations, and envision future research directions.

2. SYSTEM STUDY

FEASIBILITY STUDY
The feasibility of the project is analyzed in this phase and business
11

proposal is put forth with a very general plan for the project and some cost
estimates. During system analysis the feasibility study of the proposed system is
to be carried out. This is to ensure that the proposed system is not a burden to
the company.

For feasibility analysis, some understanding of the major

requirements for the system is essential.

Three key considerations involved in the feasibility analysis are
ECONOMICAL FEASIBILITY
TECHNICAL FEASIBILITY
SOCIAL FEASIBILITY
ECONOMICAL FEASIBILITY
This study is carried out to check the economic impact that the system
will have on the organization. The amount of fund that the company can pour
into the research and development of the system is limited. The expenditures
must be justified. Thus the developed system as well within the budget and this
was achieved because most of the technologies used are freely available. Only
the customized products had to be purchased.
TECHNICAL FEASIBILITY
This study is carried out to check the technical feasibility, that is, the
technical requirements of the system. Any system developed must not have a
high demand on the available technical resources. This will lead to high
demands on the available technical resources. This will lead to high demands
being placed on the client. The developed system must have a modest
requirement, as only minimal or null changes are required for implementing this
system.
SOCIAL FEASIBILITY
The aspect of study is to check the level of acceptance of the system by
the user. This includes the process of training the user to use the system
12

efficiently. The user must not feel threatened by the system, instead must accept
it as a necessity. The level of acceptance by the users solely depends on the
methods that are employed to educate the user about the system and to make
him familiar with it. His level of confidence must be raised so that he is also
able to make some constructive criticism, which is welcomed, as he is the final
user of the system.

3. SYSTEM REQUIREMENTS:
HARDWARE REQUIREMENTS:
System

Pentium IV 2.4 GHz.

Hard Disk

40 GB.

Floppy Drive

1.44 Mb.

13

 Monitor

15 VGA Colour.

Mouse

Logitech.

Ram

512 Mb.

SOFTWARE REQUIREMENTS:
Operating system :

Windows XP/7.

Coding Language :

JAVA/J2EE

IDE

Netbeans 7.4

Database

MYSQL

4. Software Environment
The Java Programming Language
The Java programming language is a high-level language that can be
characterized by all of the following buzzwords:
Simple
Architecture neutral
14

 Object oriented
Portable
Distributed
High performance
Interpreted
Multithreaded
Robust
Dynamic
Secure
With most programming languages, you either compile or interpret a program
so that you can run it on your computer. The Java programming language is
unusual in that a program is both compiled and interpreted. With the compiler,
first you translate a program into an intermediate language called Java byte
codes the platform-independent codes interpreted by the interpreter on the
Java platform. The interpreter parses and runs each Java byte code instruction
on the computer. Compilation happens just once; interpretation occurs each time
the program is executed. The following figure illustrates how this works.

You can think of Java byte codes as the machine code instructions for the
Java Virtual Machine (Java VM). Every Java interpreter, whether its a
development tool or a Web browser that can run applets, is an implementation
of the Java VM. Java byte codes help make write once, run anywhere
15

possible. You can compile your program into byte codes on any platform that
has a Java compiler. The byte codes can then be run on any implementation of
the Java VM. That means that as long as a computer has a Java VM, the same
program written in the Java programming language can run on Windows 2000,
a Solaris workstation, or on an iMac.

The Java Platform

A platform is the hardware or software environment in which a
program runs. Weve already mentioned some of the most popular
platforms like Windows 2000, Linux, Solaris, and MacOS. Most
platforms can be described as a combination of the operating system and
hardware. The Java platform differs from most other platforms in that its
a software-only platform that runs on top of other hardware-based
platforms.
The Java platform has two components:
The Java Virtual Machine (Java VM)
The Java Application Programming Interface (Java API)
Youve already been introduced to the Java VM. Its the base for the
Java platform and is ported onto various hardware-based platforms.
16

The Java API is a large collection of ready-made software

components that provide many useful capabilities, such as graphical user
interface (GUI) widgets. The Java API is grouped into libraries of related
classes and interfaces; these libraries are known as packages. The next
section, What Can Java Technology Do? Highlights what functionality
some of the packages in the Java API provide.
The following figure depicts a program thats running on the Java
platform. As the figure shows, the Java API and the virtual machine
insulate the program from the hardware.

Native code is code that after you compile it, the compiled code runs on a
specific hardware platform. As a platform-independent environment, the Java
platform can be a bit slower than native code. However, smart compilers, welltuned interpreters, and just-in-time byte code compilers can bring performance
close to that of native code without threatening portability.
What Can Java Technology Do?

The most common types of programs written in the Java programming

language are applets and applications. If youve surfed the Web, youre
probably already familiar with applets. An applet is a program that
adheres to certain conventions that allow it to run within a Java-enabled
browser.
However, the Java programming language is not just for writing cute,
entertaining applets for the Web. The general-purpose, high-level Java
17

programming language is also a powerful software platform. Using the

generous API, you can write many types of programs.
An application is a standalone program that runs directly on the Java
platform. A special kind of application known as a server serves and
supports clients on a network. Examples of servers are Web servers,
proxy servers, mail servers, and print servers. Another specialized
program is a servlet. A servlet can almost be thought of as an applet that
runs on the server side. Java Servlets are a popular choice for building
interactive web applications, replacing the use of CGI scripts. Servlets
are similar to applets in that they are runtime extensions of applications.
Instead of working in browsers, though, servlets run within Java Web
servers, configuring or tailoring the server.
How does the API support all these kinds of programs? It does so with
packages of software components that provides a wide range of
functionality. Every full implementation of the Java platform gives you
the following features:
The essentials: Objects, strings, threads, numbers, input and
output, data structures, system properties, date and time, and so on.
Applets: The set of conventions used by applets.
Networking: URLs, TCP (Transmission Control Protocol), UDP
(User Data gram Protocol) sockets, and IP (Internet Protocol)
addresses.
Internationalization: Help for writing programs that can be
localized for users worldwide. Programs can automatically adapt to
specific locales and be displayed in the appropriate language.
Security: Both low level and high level, including electronic
signatures, public and private key management, access control, and
18

certificates.
Software components: Known as JavaBeansTM, can plug into
existing component architectures.
Object

serialization:

Allows

lightweight

persistence

and

communication via Remote Method Invocation (RMI).

Java Database Connectivity (JDBCTM): Provides uniform access
to a wide range of relational databases.
The Java platform also has APIs for 2D and 3D graphics, accessibility,
servers, collaboration, telephony, speech, animation, and more. The
following figure depicts what is included in the Java 2 SDK.

How Will Java Technology Change My Life?

We cant promise you fame, fortune, or even a job if you learn the Java
programming language. Still, it is likely to make your programs better and
requires less effort than other languages. We believe that Java technology will
help you do the following:
Get started quickly: Although the Java programming language is
a powerful object-oriented language, its easy to learn, especially
for programmers already familiar with C or C++.
19

 Write less code: Comparisons of program metrics (class counts,

method counts, and so on) suggest that a program written in the
Java programming language can be four times smaller than the
same program in C++.
Write better code: The Java programming language encourages
good coding practices, and its garbage collection helps you avoid
memory leaks. Its object orientation, its JavaBeans component
architecture, and its wide-ranging, easily extendible API let you
reuse other peoples tested code and introduce fewer bugs.
Develop programs more quickly: Your development time may be
as much as twice as fast versus writing the same program in C++.
Why? You write fewer lines of code and it is a simpler
programming language than C++.
Avoid platform dependencies with 100% Pure Java: You can
keep your program portable by avoiding the use of libraries written
in other languages. The 100% Pure JavaTM Product Certification
Program has a repository of historical process manuals, white
papers, brochures, and similar materials online.
Write once, run anywhere: Because 100% Pure Java programs
are compiled into machine-independent byte codes, they run
consistently on any Java platform.
Distribute software more easily: You can upgrade applets easily
from a central server. Applets take advantage of the feature of
allowing new classes to be loaded on the fly, without
recompiling the entire program.
ODBC
Microsoft Open Database Connectivity (ODBC) is a standard
programming interface for application developers and database systems
20

providers. Before ODBC became a de facto standard for Windows programs to

interface with database systems, programmers had to use proprietary languages
for each database they wanted to connect to. Now, ODBC has made the choice
of the database system almost irrelevant from a coding perspective, which is as
it should be. Application developers have much more important things to worry
about than the syntax that is needed to port their program from one database to
another when business needs suddenly change.
Through the ODBC Administrator in Control Panel, you can specify the
particular database that is associated with a data source that an ODBC
application program is written to use. Think of an ODBC data source as a door
with a name on it. Each door will lead you to a particular database. For
example, the data source named Sales Figures might be a SQL Server database,
whereas the Accounts Payable data source could refer to an Access database.
The physical database referred to by a data source can reside anywhere on the
LAN.
The ODBC system files are not installed on your system by Windows 95.
Rather, they are installed when you setup a separate database application, such
as SQL Server Client or Visual Basic 4.0. When the ODBC icon is installed in
Control Panel, it uses a file called ODBCINST.DLL. It is also possible to
administer your ODBC data sources through a stand-alone program called
ODBCADM.EXE. There is a 16-bit and a 32-bit version of this program and
each

maintains

separate

list

of

ODBC

data

sources.

From a programming perspective, the beauty of ODBC is that the

application can be written to use the same set of function calls to interface with
any data source, regardless of the database vendor. The source code of the
application doesnt change whether it talks to Oracle or SQL Server. We only
21

mention these two as an example. There are ODBC drivers available for several
dozen popular database systems. Even Excel spreadsheets and plain text files
can be turned into data sources. The operating system uses the Registry
information written by ODBC Administrator to determine which low-level
ODBC drivers are needed to talk to the data source (such as the interface to
Oracle or SQL Server). The loading of the ODBC drivers is transparent to the
ODBC application program. In a client/server environment, the ODBC API
even handles many of the network issues for the application programmer.
The advantages of this scheme are so numerous that you are probably
thinking there must be some catch. The only disadvantage of ODBC is that it
isnt as efficient as talking directly to the native database interface. ODBC has
had many detractors make the charge that it is too slow. Microsoft has always
claimed that the critical factor in performance is the quality of the driver
software that is used. In our humble opinion, this is true. The availability of
good ODBC drivers has improved a great deal recently. And anyway, the
criticism about performance is somewhat analogous to those who said that
compilers would never match the speed of pure assembly language. Maybe not,
but the compiler (or ODBC) gives you the opportunity to write cleaner
programs, which means you finish sooner. Meanwhile, computers get faster
every year.

JDBC
In an effort to set an independent database standard API for Java; Sun
Microsystems developed Java Database Connectivity, or JDBC. JDBC offers a
generic SQL database access mechanism that provides a consistent interface to a
variety of RDBMSs. This consistent interface is achieved through the use of
22

plug-in database connectivity modules, or drivers. If a database vendor wishes

to have JDBC support, he or she must provide the driver for each platform that
the database and Java run on.
To gain a wider acceptance of JDBC, Sun based JDBCs framework on
ODBC. As you discovered earlier in this chapter, ODBC has widespread
support on a variety of platforms. Basing JDBC on ODBC will allow vendors to
bring JDBC drivers to market much faster than developing a completely new
connectivity solution.
JDBC was announced in March of 1996. It was released for a 90 day
public review that ended June 8, 1996. Because of user input, the final JDBC
v1.0 specification was released soon after.
The remainder of this section will cover enough information about JDBC for
you to know what it is about and how to use it effectively. This is by no means a
complete overview of JDBC. That would fill an entire book.

JDBC Goals
Few software packages are designed without goals in mind. JDBC is one
that, because of its many goals, drove the development of the API. These goals,
in conjunction with early reviewer feedback, have finalized the JDBC class
library into a solid framework for building database applications in Java.
The goals that were set for JDBC are important. They will give you some
insight as to why certain classes and functionalities behave the way they do. The
eight design goals for JDBC are as follows:

23

1. SQL Level API

The designers felt that their main goal was to define a SQL interface for
Java. Although not the lowest database interface level possible, it is at a low
enough level for higher-level tools and APIs to be created. Conversely, it is
at a high enough level for application programmers to use it confidently.
Attaining this goal allows for future tool vendors to generate JDBC code
and to hide many of JDBCs complexities from the end user.
2. SQL Conformance
SQL syntax varies as you move from database vendor to database vendor.
In an effort to support a wide variety of vendors, JDBC will allow any query
statement to be passed through it to the underlying database driver. This
allows the connectivity module to handle non-standard functionality in a
manner that is suitable for its users.
3. JDBC must be implemental on top of common database interfaces
The JDBC SQL API must sit on top of other common SQL level
APIs. This goal allows JDBC to use existing ODBC level drivers by the
use of a software interface. This interface would translate JDBC calls to
ODBC and vice versa.
4. Provide a Java interface that is consistent with the rest of the Java system
Because of Javas acceptance in the user community thus far, the
designers feel that they should not stray from the current design of the core
Java system.
5. Keep it simple
This goal probably appears in all software design goal listings. JDBC is
no exception. Sun felt that the design of JDBC should be very simple,
allowing for only one method of completing a task per mechanism. Allowing
duplicate functionality only serves to confuse the users of the API.
6. Use strong, static typing wherever possible
Strong typing allows for more error checking to be done at compile time;
24

also, less error appear at runtime.

7. Keep the common cases simple
Because more often than not, the usual SQL calls used by the
programmer are simple SELECTs, INSERTs, DELETEs and UPDATEs, these
queries should be simple to perform with JDBC. However, more complex
SQL statements should also be possible.
Java ha two things: a programming language and a platform. Java is a
high-level programming language that is all of the following
Simple

Architecture-neutral

Object-oriented

Portable

Distributed

High-performance

Interpreted

multithreaded

Robust

Dynamic

Secure

Java is also unusual in that each Java program is both compiled and
interpreted. With a compile you translate a Java program into an
intermediate language called Java byte codes the platform-independent
code instruction is passed and run on the computer.
Compilation happens just once; interpretation occurs each time the
program is executed. The figure illustrates how this works.

25

Interpreter

Java Program

Compilers

My Program

You can think of Java byte codes as the machine code instructions
for the Java Virtual Machine (Java VM). Every Java interpreter,
whether its a Java development tool or a Web browser that can run
Java applets, is an implementation of the Java VM. The Java VM can
also be implemented in hardware.
Java byte codes help make write once, run anywhere possible.
You can compile your Java program into byte codes on my platform
that has a Java compiler. The byte codes can then be run any
implementation of the Java VM. For example, the same Java program
can run Windows NT, Solaris, and Macintosh.
Networking

TCP/IP stack
The TCP/IP stack is shorter than the OSI one:

26

TCP is a connection-oriented protocol; UDP (User Datagram

Protocol) is a connectionless protocol.

IP datagrams
The IP layer provides a connectionless and unreliable delivery
system. It considers each datagram independently of the others. Any
association between datagram must be supplied by the higher layers.
The IP layer supplies a checksum that includes its own header. The
header includes the source and destination addresses. The IP layer
handles routing through an Internet. It is also responsible for breaking
up large datagram into smaller ones for transmission and reassembling
them at the other end.
UDP
UDP is also connectionless and unreliable. What it adds to IP is a
checksum for the contents of the datagram and port numbers. These are
27

used to give a client/server model - see later.

TCP
TCP supplies logic to give a reliable connection-oriented protocol
above IP. It provides a virtual circuit that two processes can use to
communicate.
Internet addresses
In order to use a service, you must be able to find it. The Internet
uses an address scheme for machines so that they can be located. The
address is a 32 bit integer which gives the IP address. This encodes a
network ID and more addressing. The network ID falls into various
classes according to the size of the network address.
Network address
Class A uses 8 bits for the network address with 24 bits left over for
other addressing. Class B uses 16 bit network addressing. Class C uses
24 bit network addressing and class D uses all 32.
Subnet address
Internally, the UNIX network is divided into sub networks. Building
11 is currently on one sub network and uses 10-bit addressing, allowing
1024 different hosts.
Host address
8 bits are finally used for host addresses within our subnet. This
28

places a limit of 256 machines that can be on the subnet.

Total address

The 32 bit address is usually written as 4 integers separated by dots.

Port addresses
A service exists on a host, and is identified by its port. This is a 16
bit number. To send a message to a server, you send it to the port for
that service of the host that it is running on. This is not location
transparency! Certain of these ports are "well known".

Sockets
A socket is a data structure maintained by the system to handle
network connections. A socket is created using the call socket. It returns
an integer that is like a file descriptor. In fact, under Windows, this
handle can be used with Read File and Write File functions.

29

#include <sys/types.h>
#include <sys/socket.h>
int

socket(int family, int type, int protocol);

Here "family" will be AF_INET for IP communications, protocol will

be zero, and type will depend on whether TCP or UDP is used. Two
processes wishing to communicate over a network create a socket each.
These are similar to two ends of a pipe - but the actual pipe does not yet
exist.
JFree Chart
JFreeChart is a free 100% Java chart library that makes it easy for
developers to display professional quality charts in their applications.
JFreeChart's extensive feature set includes:
A consistent and well-documented API, supporting a wide range of
chart types;
A flexible design that is easy to extend, and targets both server-side
and client-side applications;
Support for many output types, including Swing components,
image files (including PNG and JPEG), and vector graphics file formats
(including PDF, EPS and SVG);
JFreeChart is "open source" or, more specifically, free software. It
is distributed under the terms of the GNU Lesser General Public Licence
(LGPL), which permits use in proprietary applications.
1. Map Visualizations
Charts showing values that relate to geographical areas. Some
examples include: (a) population density in each state of the United
States, (b) income per capita for each country in Europe, (c) life
expectancy in each country of the world. The tasks in this project include:
Sourcing freely redistributable vector outlines for the countries of
30

the world, states/provinces in particular countries (USA in particular, but

also other areas);
Creating

an

appropriate

dataset

interface

(plus

default

implementation), a rendered, and integrating this with the existing

XYPlot class in JFreeChart;
Testing, documenting, testing some more, documenting some
more.
2. Time Series Chart Interactivity

Implement a new (to JFreeChart) feature for interactive time series charts --to display a separate control that shows a small version of ALL the time
series data, with a sliding "view" rectangle that allows you to select the
subset of the time series data to display in the main chart.
3. Dashboards

There is currently a lot of interest in dashboard displays. Create a flexible

dashboard mechanism that supports a subset of JFreeChart chart types (dials,
pies, thermometers, bars, and lines/time series) that can be delivered easily
via both Java Web Start and an applet.
4. Property Editors

The property editor mechanism in JFreeChart only handles a small subset

of the properties that can be set for charts. Extend (or reimplement) this
mechanism to provide greater end-user control over the appearance of the
charts.
What is a Java Web Application?
A Java web application generates interactive web pages containing various
31

types of markup language (HTML, XML, and so on) and dynamic content. It is
typically comprised of web components such as JavaServer Pages (JSP),
servlets and JavaBeans to modify and temporarily store data, interact with
databases and web services, and render content in response to client requests.
Because many of the tasks involved in web application development can be
repetitive or require a surplus of boilerplate code, web frameworks can be
applied to alleviate the overhead associated with common activities. For
example, many frameworks, such as JavaServer Faces, provide libraries for
templating pages and session management, and often promote code reuse.
What is Java EE?
Java EE (Enterprise Edition) is a widely used platform containing a set of
coordinated technologies that significantly reduce the cost and complexity of
developing, deploying, and managing multi-tier, server-centric applications.
Java EE builds upon the Java SE platform and provides a set of APIs
(application programming interfaces) for developing and running portable,
robust, scalable, reliable and secure server-side applications.
Some of the fundamental components of Java EE include:
Enterprise JavaBeans (EJB): a managed, server-side component
architecture used to encapsulate the business logic of an application. EJB
technology enables rapid and simplified development of distributed,
transactional, secure and portable applications based on Java technology.
Java Persistence API (JPA): a framework that allows developers to
manage data using object-relational mapping (ORM) in applications built
on the Java Platform.
JavaScript and Ajax Development
JavaScript is an object-oriented scripting language primarily used in client-side
interfaces for web applications. Ajax (Asynchronous JavaScript and XML) is a
32

Web 2.0 technique that allows changes to occur in a web page without the need
to perform a page refresh. JavaScript toolkits can be leveraged to implement
Ajax-enabled components and functionality in web pages.
Web Server and Client
Web Server is a software that can process the client request and send the
response back to the client. For example, Apache is one of the most widely used
web server. Web Server runs on some physical machine and listens to client
request on specific port.
A web client is a software that helps in communicating with the server. Some of
the most widely used web clients are Firefox, Google Chrome, Safari etc. When
we request something from server (through URL), web client takes care of
creating a request and sending it to server and then parsing the server response
and present it to the user.
HTML and HTTP
Web Server and Web Client are two separate softwares, so there should be some
common language for communication. HTML is the common language between
server and client and stands for HyperText Markup Language.
Web server and client needs a common communication protocol, HTTP
(HyperText Transfer Protocol) is the communication protocol between server
and client. HTTP runs on top of TCP/IP communication protocol.
Some of the important parts of HTTP Request are:
HTTP Method action to be performed, usually GET, POST, PUT etc.
URL Page to access
Form Parameters similar to arguments in a java method, for example
user,password details from login page.
Sample HTTP Request:
33

1 GET /FirstServletProject/jsps/hello.jsp HTTP/1.1

2 Host: localhost:8080
3 Cache-Control: no-cache
Some of the important parts of HTTP Response are:
Status Code an integer to indicate whether the request was success or
not. Some of the well known status codes are 200 for success, 404 for Not
Found and 403 for Access Forbidden.
Content Type text, html, image, pdf etc. Also known as MIME type
Content actual data that is rendered by client and shown to user.

MIME Type or Content Type: If you see above sample HTTP response header,
it contains tag Content-Type. Its also called MIME type and server sends it
to client to let them know the kind of data its sending. It helps client in
rendering the data for user. Some of the mostly used mime types are text/html,
text/xml, application/xml etc.

Understanding URL
URL is acronym of Universal Resource Locator and its used to locate the
server and resource. Every resource on the web has its own unique address.
Lets see parts of URL with an example.
http://localhost:8080/FirstServletProject/jsps/hello.jsp

http:// This is the first part of URL and provides the communication protocol
to be used in server-client communication.

34

localhost The unique address of the server, most of the times its the hostname
of the server that maps to unique IP address. Sometimes multiple hostnames
point to same IP addresses and web server virtual host takes care of sending
request to the particular server instance.

8080 This is the port on which server is listening, its optional and if we dont
provide it in URL then request goes to the default port of the protocol. Port
numbers 0 to 1023 are reserved ports for well known services, for example 80
for HTTP, 443 for HTTPS, 21 for FTP etc.

FirstServletProject/jsps/hello.jsp Resource requested from server. It can be

static html, pdf, JSP, servlets, PHP etc.

Why we need Servlet and JSPs?

Web servers are good for static contents HTML pages but they dont know how
to generate dynamic content or how to save data into databases, so we need
another tool that we can use to generate dynamic content. There are several
programming languages for dynamic content like PHP, Python, Ruby on Rails,
Java Servlets and JSPs.
Java Servlet and JSPs are server side technologies to extend the capability of
web servers by providing support for dynamic response and data persistence.

35

Web Container
Tomcat is a web container, when a request is made from Client to web server, it
passes the request to web container and its web container job to find the
correct resource to handle the request (servlet or JSP) and then use the
response from the resource to generate the response and provide it to web
server. Then web server sends the response back to the client.
When web container gets the request and if its for servlet then container
creates two Objects HTTPServletRequest and HTTPServletResponse. Then it
finds the correct servlet based on the URL and creates a thread for the request.
Then it invokes the servlet service() method and based on the HTTP method
service() method invokes doGet() or doPost() methods. Servlet methods
generate the dynamic page and write it to response. Once servlet thread is
complete, container converts the response to HTTP response and send it back to
client.
Some of the important work done by web container are:
Communication

Support

Container

provides

easy

way

of

communication between web server and the servlets and JSPs. Because of
container, we dont need to build a server socket to listen for any request
from web server, parse the request and generate response. All these
important and complex tasks are done by container and all we need to
focus is on our business logic for our applications.
Lifecycle and Resource Management Container takes care of
managing the life cycle of servlet. Container takes care of loading the
servlets into memory, initializing servlets, invoking servlet methods and
destroying them. Container also provides utility like JNDI for resource
36

pooling and management.

Multithreading Support Container creates new thread for every
request to the servlet and when its processed the thread dies. So servlets
are not initialized for each request and saves time and memory.
JSP Support JSPs doesnt look like normal java classes and web
container provides support for JSP. Every JSP in the application is
compiled by container and converted to Servlet and then container
manages them like other servlets.
Miscellaneous Task Web container manages the resource pool, does
memory optimizations, run garbage collector, provides security
configurations, support for multiple applications, hot deployment and
several other tasks behind the scene that makes our life easier.
Web Application Directory Structure
Java Web Applications are packaged as Web Archive (WAR) and it has a
defined structure. You can export above dynamic web project as WAR file and
unzip it to check the hierarchy. It will be something like below image.

Deployment Descriptor
web.xml file is the deployment descriptor of the web application and contains
37

mapping for servlets (prior to 3.0), welcome pages, security configurations,

session timeout settings etc.
Thats all for the java web application startup tutorial, we will explore Servlets
and JSPs more in future posts.

MySQL:

MySQL, the most popular Open Source SQL database management system, is
developed, distributed, and supported by Oracle Corporation.
The MySQL Web site (http://www.mysql.com/) provides the latest information
about MySQL software.

MySQL is a database management system.

A database is a structured collection of data. It may be anything from a
simple shopping list to a picture gallery or the vast amounts of
information in a corporate network. To add, access, and process data
stored in a computer database, you need a database management system
such as MySQL Server. Since computers are very good at handling large
amounts of data, database management systems play a central role in
computing, as standalone utilities, or as parts of other applications.

MySQL databases are relational.

38

A relational database stores data in separate tables rather than putting

all the data in one big storeroom. The database structures are organized
into physical files optimized for speed. The logical model, with objects
such as databases, tables, views, rows, and columns, offers a flexible
programming environment. You set up rules governing the relationships
between different data fields, such as one-to-one, one-to-many, unique,
required or optional, and pointers between different tables. The
database enforces these rules, so that with a well-designed database,
your application never sees inconsistent, duplicate, orphan, out-of-date,
or missing data.
The SQL part of MySQL stands for Structured Query Language.
SQL is the most common standardized language used to access
databases. Depending on your programming environment, you might
enter SQL directly (for example, to generate reports), embed SQL
statements into code written in another language, or use a languagespecific API that hides the SQL syntax.
SQL is defined by the ANSI/ISO SQL Standard. The SQL standard has
been evolving since 1986 and several versions exist. In this manual,
SQL-92 refers to the standard released in 1992, SQL:1999 refers to
the standard released in 1999, and SQL:2003 refers to the current
version of the standard. We use the phrase the SQL standard to mean
the current version of the SQL Standard at any time.

MySQL software is Open Source.

Open Source means that it is possible for anyone to use and modify the
software. Anybody can download the MySQL software from the Internet
39

and use it without paying anything. If you wish, you may study the source
code and change it to suit your needs. The MySQL software uses the GPL
(GNU General Public License), http://www.fsf.org/licenses/, to define
what you may and may not do with the software in different situations. If
you feel uncomfortable with the GPL or need to embed MySQL code into
a commercial application, you can buy a commercially licensed version
from us. See the MySQL Licensing Overview for more information
(http://www.mysql.com/company/legal/licensing/).

The MySQL Database Server is very fast, reliable, scalable, and easy to
use.
If that is what you are looking for, you should give it a try. MySQL Server
can run comfortably on a desktop or laptop, alongside your other
applications, web servers, and so on, requiring little or no attention. If
you dedicate an entire machine to MySQL, you can adjust the settings to
take advantage of all the memory, CPU power, and I/O capacity
available. MySQL can also scale up to clusters of machines, networked
together.
You can find a performance comparison of MySQL Server with other
database managers on our benchmark page.

MySQL Server was originally developed to handle large databases much

faster than existing solutions and has been successfully used in highly
demanding production environments for several years. Although under
constant development, MySQL Server today offers a rich and useful set of
40

functions. Its connectivity, speed, and security make MySQL Server

highly suited for accessing databases on the Internet.

MySQL Server works in client/server or embedded systems.

The MySQL Database Software is a client/server system that consists of a
multi-threaded SQL server that supports different backends, several
different client programs and libraries, administrative tools, and a wide
range of application programming interfaces (APIs).
We also provide MySQL Server as an embedded multi-threaded library
that you can link into your application to get a smaller, faster, easier-tomanage standalone product.

A large amount of contributed MySQL software is available.

MySQL Server has a practical set of features developed in close
cooperation with our users. It is very likely that your favorite application
or language supports the MySQL Database Server.

The official way to pronounce MySQL is My Ess Que Ell (not my

sequel), but we do not mind if you pronounce it as my sequel or in some
other localized way.

41

SYSTEM DESIGN & DEVELOPMENT

SYSTEM ARCHITECTURE:

DATA FLOW DIAGRAM:

1. The DFD is also called as bubble chart. It is a simple graphical formalism

that can be used to represent a system in terms of input data to the
system, various processing carried out on this data, and the output data is
42

generated by this system.

2. The data flow diagram (DFD) is one of the most important modeling
tools. It is used to model the system components. These components are
the system process, the data used by the process, an external entity that
interacts with the system and the information flows in the system.
3. DFD shows how the information moves through the system and how it is
modified by a series of transformations. It is a graphical technique that
depicts information flow and the transformations that are applied as data
moves from input to output.

4. DFD is also known as bubble chart. A DFD may be used to represent a

system at any level of abstraction. DFD may be partitioned into levels
that represent increasing information flow and functional detail.

Manager Master
Manager Master

Location

Forming cluster reports

tweeteraccount

Verify account
1.2.1

1.2.2

Location info
a

Tweet count
1.2.4

t
Managers Master

43

Hash tags
1.2.3
2.3

UML DIAGRAMS
UML stands for Unified Modeling Language. UML is a standardized
general-purpose modeling language in the field of object-oriented software
engineering. The standard is managed, and was created by, the Object
Management Group.
The goal is for UML to become a common language for creating models
of object oriented computer software. In its current form UML is comprised of
two major components: a Meta-model and a notation. In the future, some form
of method or process may also be added to; or associated with, UML.
The Unified Modeling Language is a standard language for specifying,
Visualization, Constructing and documenting the artifacts of software system, as
well as for business modeling and other non-software systems.
The UML represents a collection of best engineering practices that have
proven successful in the modeling of large and complex systems.
The UML is a very important part of developing objects oriented
software and the software development process. The UML uses mostly
graphical notations to express the design of software projects.
GOALS:
The Primary goals in the design of the UML are as follows:
1. Provide users a ready-to-use, expressive visual modeling Language so
that they can develop and exchange meaningful models.
2. Provide extendibility and specialization mechanisms to extend the core
concepts.
3. Be independent of particular programming languages and development
process.
44

4. Provide a formal basis for understanding the modeling language.

5. Encourage the growth of OO tools market.
6. Support higher level development concepts such as collaborations,
frameworks, patterns and components.
7. Integrate best practices.

USE CASE DIAGRAM:

A use case diagram in the Unified Modeling Language (UML) is a type
45

of behavioral diagram defined by and created from a Use-case analysis. Its

purpose is to present a graphical overview of the functionality provided by a
system in terms of actors, their goals (represented as use cases), and any
dependencies between those use cases. The main purpose of a use case diagram
is to show what system functions are performed for which actor. Roles of the
actors in the system can be depicted.

CLASS DIAGRAM:
In software engineering, a class diagram in the Unified Modeling Language
(UML) is a type of static structure diagram that describes the structure of a
system by showing the system's classes, their attributes, operations (or
methods), and the relationships among the classes. It explains which class
contains information.

46

SEQUENCE DIAGRAM:
A sequence diagram in Unified Modeling Language (UML) is a kind of
interaction diagram that shows how processes operate with one another and in
what order. It is a construct of a Message Sequence Chart. Sequence diagrams
are sometimes called event diagrams, event scenarios, and timing diagrams.

47

ACTIVITY DIAGRAM:
Activity diagrams are graphical representations of workflows of stepwise
activities and actions with support for choice, iteration and concurrency. In the
Unified Modeling Language, activity diagrams can be used to describe the
business and operational step-by-step workflows of components in a system. An
activity diagram shows the overall flow of control.

48

5. SCREEN LAYOUT

49

50

51

52

53

6. SYSTEM TESTING
The purpose of testing is to discover errors. Testing is the process of
trying to discover every conceivable fault or weakness in a work product. It
provides a way to check the functionality of components, sub assemblies,
assemblies and/or a finished product It is the process of exercising software
with the intent of ensuring that the
Software system meets its requirements and user expectations and does not fail
in an unacceptable manner. There are various types of test. Each test type
addresses a specific testing requirement.
TYPES OF TESTS
Unit testing
Unit testing involves the design of test cases that validate that the internal
54

program logic is functioning properly, and that program inputs produce valid
outputs. All decision branches and internal code flow should be validated. It is
the testing of individual software units of the application .it is done after the
completion of an individual unit before integration. This is a structural testing,
that relies on knowledge of its construction and is invasive. Unit tests perform
basic tests at component level and test a specific business process, application,
and/or system configuration. Unit tests ensure that each unique path of a
business process performs accurately to the documented specifications and
contains clearly defined inputs and expected results.

Integration testing
Integration tests are designed to test integrated software components to
determine if they actually run as one program. Testing is event driven and is
more concerned with the basic outcome of screens or fields. Integration tests
demonstrate that although the components were individually satisfaction, as
shown by successfully unit testing, the combination of components is correct
and consistent. Integration testing is specifically aimed at

exposing the

problems that arise from the combination of components.

Functional test
Functional tests provide systematic demonstrations that functions tested are
available as specified by the business and technical requirements, system
documentation, and user manuals.
Functional testing is centered on the following items:
55

Valid Input

: identified classes of valid input must be accepted.

Invalid Input

: identified classes of invalid input must be rejected.

Functions

: identified functions must be exercised.

Output

: identified classes of application outputs must be

exercised.
Systems/Procedures: interfacing systems or procedures must be invoked.
Organization and preparation of functional tests is focused on requirements,
key functions, or special test cases. In addition, systematic coverage pertaining
to identify Business process flows; data fields, predefined processes, and
successive processes must be considered for testing. Before functional testing is
complete, additional tests are identified and the effective value of current tests is
determined.
System Test
System testing ensures that the entire integrated software system meets
requirements. It tests a configuration to ensure known and predictable results.
An example of system testing is the configuration oriented system integration
test. System testing is based on process descriptions and flows, emphasizing
pre-driven process links and integration points.
White Box Testing
White Box Testing is a testing in which in which the software tester has
knowledge of the inner workings, structure and language of the software, or at
least its purpose. It is purpose. It is used to test areas that cannot be reached
from a black box level.
Black Box Testing
Black Box Testing is testing the software without any knowledge of the
56

inner workings, structure or language of the module being tested. Black box
tests, as most other kinds of tests, must be written from a definitive source
document, such as specification or requirements document, such as
specification or requirements document. It is a testing in which the software
under test is treated, as a black box .you cannot see into it. The test provides
inputs and responds to outputs without considering how the software works.
6.1 Unit Testing:
Unit testing is usually conducted as part of a combined code and unit test
phase of the software lifecycle, although it is not uncommon for coding and unit
testing to be conducted as two distinct phases.
Test strategy and approach
Field testing will be performed manually and functional tests will be
written in detail.

Test objectives
All field entries must work properly.
Pages must be activated from the identified link.
The entry screen, messages and responses must not be delayed.

Features to be tested
Verify that the entries are of the correct format
No duplicate entries should be allowed
All links should take the user to the correct page.

57

6.2 Integration Testing

Software integration testing is the incremental integration testing of two
or more integrated software components on a single platform to produce failures
caused by interface defects.
The task of the integration test is to check that components or software
applications, e.g. components in a software system or one step up software
applications at the company level interact without error
Test Results: All the test cases mentioned above passed successfully. No
defects encountered.
6.3 Acceptance Testing
User Acceptance Testing is a critical phase of any project and requires
significant participation by the end user. It also ensures that the system meets
the functional requirements.
Test Results: All the test cases mentioned above passed successfully. No
defects encountered.

Test Case Report1

(Use one template for each test case)

GENERAL INFORMATION
Test Stage:

Test Date:

Unit
Interface

Functionality

Performance

Acceptance

09/09/2010

System Date, if

58

09/03/2015

Tester:
Test Case
Description:

Results:
Requirement(s)
to be tested:
Roles and
Responsibilities
:
Set Up
Procedures:
Hardware:
Software:
Test Items and
Features:
Procedural
Steps:
Expected
Results of
Case:

applicable:
Test Case Number:

Janardhan

Unit testing focuses on verifying the effort on the

smallest unit of software-module. The local data
structure is examined to ensure that the date stored
temporarily maintains its integrity during all steps in the
algorithms execution. Boundary conditions are tested to
ensure that the module operates properly at boundaries
established to limit or restrict processing.
Pass(OK)

Fail

INTRODUCTION
Getting Twitter Account
Gathering the Requirements of the Project Designing and Testing.
By Installing Eclipse.
ENVIRONMENTAL NEEDS
PC with Minimum 20GB Hard Disk and 1GB RAM.
Windows XP/2000, Oracle, Eclipse.
TEST
LocationID and Retweet Count.
If the User enters the Location id it will be redirected to another appropriate
page so that we can confirm test is accepted.
If the page is redirected we can confirm the result of this Test case is
succeeded.

Test Case Report2

(Use one template for each test case)

GENERAL INFORMATION
Test Stage:

Test Date:
Tester:
Test Case
Description:

Results:

Unit
Interface

Functionality

Performance

Acceptance

09/09/2010

System Date, if
applicable:
Test Case Number:

Janardhan

09/02/2015
2

Unit testing focuses on verifying the effort on the

smallest unit of software-module. The local data
structure is examined to ensure that the date stored
temporarily maintains its integrity during all steps in the
algorithms execution. Boundary conditions are tested to
ensure that the module operates properly at boundaries
established to limit or restrict processing.
Pass(OK)

Fail
INTRODUCTION

59

Requirement(s)
to be tested:
Roles and
Responsibilities
:
Set Up
Procedures:
Hardware:
Software:
Test Items and
Features:
Procedural
Steps:
Expected
Results of
Case:

After getting View the Details

The responsibilities are text.
By Installing Eclipse
ENVIRONMENTAL NEEDS
PC with Minimum 20GB Hard Disk and 1GB RAM.
Windows XP/2000, Oracle, Net Beans. Windows XP/2000
TEST
After login view the transactions corresponding user id password
Authentication SMTP
If the page is redirected we can confirm the result of this Test case is
succeeded.

CONCLUSION
Driven by real-world applications and key industrial stakeholders and initialized
by national funding agencies, managing and mining Big Data have shown to be
a challenging yet very compelling task. While the term Big Data literally
concerns about data volumes, our HACE theorem suggests that the key
60

characteristics of the Big Data are 1) huge with heterogeneous and diverse data
sources, 2) autonomous with distributed and decentralized control, and 3)
complex and evolving in data and knowledge associations. Such combined
characteristics suggest that Big Data require a big mind to consolidate data for
maximum values [27].
To explore Big Data, we have analyzed several challenges at the data, model,
and system levels. To support Big Data mining, high-performance computing
platforms are required, which impose systematic designs to unleash the full
power of the Big Data. At the data level, the autonomous information sources
and the variety of the data collection environments, often result in data with
complicated conditions, such as missing/uncertain values. In other situations,
privacy concerns, noise, and errors can be introduced into the data, to produce
altered data copies. Developing a safe and sound information sharing protocol is
a major challenge. At the model level, the key challenge is to generate global
models by combining locally discovered patterns to form a unifying view. This
requires carefully designed algorithms to analyze model correlations between
distributed sites, and fuse decisions from multiple sources to gain a best model
out of the Big Data. At the system level, the essential challenge is that a Big
Data mining framework needs to consider complex relationships between
samples, models, and data sources, along with their evolving changes with time
and other possible factors. A system needs to be carefully designed so that
unstructured data can be linked through their complex relationships to form
useful patterns, and the growth of data volumes and item relationships should
help form legitimate patterns to predict the trend and future.
We regard Big Data as an emerging trend and the need for Big Data mining is
arising in all science and engineering domains. With Big Data technologies, we
will hopefully be able to provide most relevant and most accurate social sensing
feedback to better understand our society at real time. We can further stimulate
the participation of the public audiences in the data production circle for societal
61

and economical events. The era of Big Data has arrived.

BIBLIOGRAPHY
[1] R. Ahmed and G. Karypis, Algorithms for Mining the Evolution of
Conserved Relational States in Dynamic Networks, Knowledge and
Information Systems, vol. 33, no. 3, pp. 603-630, Dec. 2012.
[2] M.H. Alam, J.W. Ha, and S.K. Lee, Novel Approaches to Crawling
Important Pages Early, Knowledge and Information Systems, vol. 33, no. 3, pp
707-734, Dec. 2012.
[3] S. Aral and D. Walker, Identifying Influential and Susceptible Members of
Social Networks, Science, vol. 337, pp. 337-341, 2012.
[4] A. Machanavajjhala and J.P. Reiter, Big Privacy: Protecting Confidentiality
in Big Data, ACM Crossroads, vol. 19, no. 1, pp. 20-23, 2012.
[5] S. Banerjee and N. Agarwal, Analyzing Collective Behavior from Blogs
Using Swarm Intelligence, Knowledge and Information Systems, vol. 33, no.
62

3, pp. 523-547, Dec. 2012.

[6] E. Birney, The Making of ENCODE: Lessons for Big-Data Projects,
Nature, vol. 489, pp. 49-51, 2012.
[7] J. Bollen, H. Mao, and X. Zeng, Twitter Mood Predicts the Stock Market,
J. Computational Science, vol. 2, no. 1, pp. 1-8, 2011.
[8] S. Borgatti, A. Mehra, D. Brass, and G. Labianca, Network Analysis in the
Social Sciences, Science, vol. 323, pp. 892-895, 2009.
[9] J. Bughin, M. Chui, and J. Manyika, Clouds, Big Data, and Smart Assets:
Ten Tech-Enabled Business Trends to Watch. McKinSey Quarterly, 2010.
[10] D. Centola, The Spread of Behavior in an Online Social Network
Experiment, Science, vol. 329, pp. 1194-1197, 2010.
[11] E.Y. Chang, H. Bai, and K. Zhu, Parallel Algorithms for Mining LargeScale Rich-Media Data, Proc. 17th ACM Intl Conf. Multimedia, (MM 09,)
pp. 917-918, 2009.
[12] R. Chen, K. Sivakumar, and H. Kargupta, Collective Mining of Bayesian
Networks from Distributed Heterogeneous Data, Knowledge and Information
Systems, vol. 6, no. 2, pp. 164-187, 2004.
[13] Y.-C. Chen, W.-C. Peng, and S.-Y. Lee, Efficient Algorithms for Influence
Maximization in Social Networks, Knowledge and Information Systems, vol.
33, no. 3, pp. 577-601, Dec. 2012.

63

[14] C.T. Chu, S.K. Kim, Y.A. Lin, Y. Yu, G.R. Bradski, A.Y. Ng, and K.
Olukotun, Map-Reduce for Machine Learning on Multicore, Proc. 20th Ann.
Conf. Neural Information Processing Systems (NIPS 06), pp. 281-288, 2006.
[15] G. Cormode and D. Srivastava, Anonymized Data: Generation, Models,
Usage, Proc. ACM SIGMOD Intl Conf. Management Data, pp. 1015-1018,
2009.
[16] S. Das, Y. Sismanis, K.S. Beyer, R. Gemulla, P.J. Haas, and J. McPherson,
Ricardo: Integrating R and Hadoop, Proc. ACM SIGMOD Intl Conf.
Management Data (SIGMOD 10), pp. 987-998. 2010.
[17] P. Dewdney, P. Hall, R. Schilizzi, and J. Lazio, The Square Kilometre
Array, Proc. IEEE, vol. 97, no. 8, pp. 1482-1496, Aug. 2009.
[18] P. Domingos and G. Hulten, Mining High-Speed Data Streams, Proc.
Sixth ACM SIGKDD Intl Conf. Knowledge Discovery and Data Mining (KDD
00), pp. 71-80, 2000.
[19] G. Duncan, Privacy by Design, Science, vol. 317, pp. 1178-1179, 2007.
[20] B. Efron, Missing Data, Imputation, and the Bootstrap, J. Am. Statistical
Assoc., vol. 89, no. 426, pp. 463-475, 1994.
[21] A. Ghoting and E. Pednault, Hadoop-ML: An Infrastructure for the Rapid
Implementation of Parallel Reusable Analytics, Proc. Large-Scale Machine
Learning: Parallelism and Massive Data Sets Workshop (NIPS 09), 2009.
[22] D. Gillick, A. Faria, and J. DeNero, MapReduce: Distributed Computing
64

for Machine Learning, Berkley, Dec. 2006.

[23] M. Helft, Google Uses Searches to Track Flus Spread, The New York
Times,

http://www.nytimes.com/2008/11/12/technology/internet/12flu.html.

2008.
[24] D. Howe et al., Big Data: The Future of Biocuration, Nature, vol. 455,
pp. 47-50, Sept. 2008.
[25] B. Huberman, Sociology of Science: Big Data Deserve a Bigger
Audience, Nature, vol. 482, p. 308, 2012.
[26] IBM What Is Big Data: Bring Big Data to the Enterprise, http:// www01.ibm.com/software/data/bigdata/, IBM, 2012.
[27] A. Jacobs, The Pathologies of Big Data, Comm. ACM, vol. 52, no. 8, pp.
36-44, 2009.
[28] I. Kopanas, N. Avouris, and S. Daskalaki, The Role of Domain
Knowledge in a Large Scale Data Mining Project, Proc. Second Hellenic Conf.
AI: Methods and Applications of Artificial Intelligence, I.P. Vlahavas, C.D.
Spyropoulos, eds., pp. 288-299, 2002.
[29] A. Labrinidis and H. Jagadish, Challenges and Opportunities with Big
Data, Proc. VLDB Endowment, vol. 5, no. 12, 2032-2033, 2012.
[30] Y. Lindell and B. Pinkas, Privacy Preserving Data Mining, J. Cryptology,
vol. 15, no. 3, pp. 177-206, 2002.

65

[31] W. Liu and T. Wang, Online Active Multi-Field Learning for Efficient
Email Spam Filtering, Knowledge and Information Systems, vol. 33, no. 1, pp.
117-136, Oct. 2012.
[32] J. Lorch, B. Parno, J. Mickens, M. Raykova, and J. Schiffman, Shoroud:
Ensuring Private Access to Large-Scale Data in the Data Center, Proc. 11th
USENIX Conf. File and Storage Technologies (FAST 13), 2013.
[33] D. Luo, C. Ding, and H. Huang, Parallelization with Multiplicative
Algorithms for Big Data Mining, Proc. IEEE 12th Intl Conf. Data Mining, pp.
489-498, 2012.
[34] J. Mervis, U.S. Science Policy: Agencies Rally to Tackle Big Data,
Science, vol. 336, no. 6077, p. 22, 2012.
[35] F. Michel, How Many Photos Are Uploaded to Flickr Every Day and
Month? http://www.flickr.com/photos/franckmichel/6855169886/, 2012.
[36] T. Mitchell, Mining our Reality, Science, vol. 326, pp. 1644-1645, 2009.
[37] Nature Editorial, Community Cleverness Required, Nature, vol. 455, no.
7209, p. 1, Sept. 2008.
[38] S. Papadimitriou and J. Sun, Disco: Distributed Co-Clustering with MapReduce: A Case Study Towards Petabyte-Scale End-to-End Mining, Proc.
IEEE Eighth Intl Conf. Data Mining (ICDM 08), pp. 512-521, 2008.
[39] C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis,
Evaluating MapReduce for Multi-Core and Multiprocessor Systems, Proc.
66

IEEE 13th Intl Symp. High Performance Computer Architecture (HPCA 07),
pp. 13-24, 2007.
[40] A. Rajaraman and J. Ullman, Mining of Massive Data Sets. Cambridge
Univ. Press, 2011.
[41] C. Reed, D. Thompson, W. Majid, and K. Wagstaff, Real Time Machine
Learning to Find Fast Transient Radio Anomalies: A Semi-Supervised Approach
Combining Detection and RFI Excision, Proc. Intl Astronomical Union Symp.
Time Domain Astronomy, Sept. 2011.
[42] E. Schadt, The Changing Privacy Landscape in the Era of Big Data,
Molecular Systems, vol. 8, article 612, 2012.
[43] J. Shafer, R. Agrawal, and M. Mehta, SPRINT: A Scalable Parallel
Classifier for Data Mining, Proc. 22nd VLDB Conf., 1996.
[44] A. da Silva, R. Chiky, and G. Hebrail, A Clustering Approach for
Sampling Data Streams in Sensor Networks, Knowledge and Information
Systems, vol. 32, no. 1, pp. 1-23, July 2012.
[45] K. Su, H. Huang, X. Wu, and S. Zhang, A Logical Framework for
Identifying Quality Knowledge from Different Data Sources, Decision Support
Systems, vol. 42, no. 3, pp. 1673-1683, 2006.
[46]

Twitter

Blog,

Dispatch

from

the

Denver

Debate,

http://

blog.twitter.com/2012/10/dispatch-from-denver-debate.html,Oct. 2012.
[47] D. Wegener, M. Mock, D. Adranale, and S. Wrobel, Toolkit-Based High67

Performance Data Mining of Large Data on MapReduce Clusters, Proc. Intl

Conf. Data Mining Workshops (ICDMW 09), pp. 296-301, 2009.
[48] C. Wang, S.S.M. Chow, Q. Wang, K. Ren, and W. Lou, PrivacyPreserving Public Auditing for Secure Cloud Storage IEEE Trans. Computers,
vol. 62, no. 2, pp. 362-375, Feb. 2013.
[49] X. Wu and X. Zhu, Mining with Noise Knowledge: Error-Aware Data
Mining, IEEE Trans. Systems, Man and Cybernetics, Part A, vol. 38, no. 4, pp.
917-932, July 2008.
[50] X. Wu and S. Zhang, Synthesizing High-Frequency Rules from Different
Data Sources, IEEE Trans. Knowledge and Data Eng., vol. 15, no. 2, pp. 353367, Mar./Apr. 2003.
[51] X. Wu, C. Zhang, and S. Zhang, Database Classification for MultiDatabase Mining, Information Systems, vol. 30, no. 1, pp. 71- 88, 2005.
[52] X. Wu, Building Intelligent Learning Database Systems, AI Magazine,
vol. 21, no. 3, pp. 61-67, 2000.
[53] X. Wu, K. Yu, W. Ding, H. Wang, and X. Zhu, Online Feature Selection
with Streaming Features, IEEE Trans. Pattern Analysis and Machine
Intelligence, vol. 35, no. 5, pp. 1178-1192, May 2013.
[54] A. Yao, How to Generate and Exchange Secretes, Proc. 27th Ann. Symp.
Foundations Computer Science (FOCS) Conf., pp. 162-167, 1986.
[55] M. Ye, X. Wu, X. Hu, and D. Hu, Anonymizing Classification Data Using
68

Rough Set Theory, Knowledge-Based Systems, vol. 43, pp. 82-94, 2013.
[56] J. Zhao, J. Wu, X. Feng, H. Xiong, and K. Xu, Information Propagation in
Online Social Networks: A Tie-Strength Perspective, Knowledge and
Information Systems, vol. 32, no. 3, pp. 589-608, Sept. 2012.
[57] X. Zhu, P. Zhang, X. Lin, and Y. Shi, Active Learning From Stream Data
Using Optimal Weight Classifier Ensemble, IEEE Trans. Systems, Man, and
Cybernetics, Part B, vol. 40, no. 6, pp. 1607- 1621, Dec. 2010.

69