Risk Assessment

ABSTRACT
Developing a system to aid analysts in anticipating

and Horizon Scanning

emerging strategic issues is a challenging engineering feat. The
Risk Assessment and Horizon Scanning (RAHS) Experimentation

Experimentation Centre
Centre (REC) has successfully delivered a cognitive-based system
to achieve this. This article describes three key challenges for
RAHS. First, analyses are based mostly on discovery instead
of deduction. Second, there is a lack of suitable commercial
solutions. Third, the processes and technology landscape of the
analysts are evolving rapidly. To address these challenges, two
approaches have been undertaken by the REC to identify and
develop RAHS system capabilities to meet the unique analytical
needs and to create an REC process to manage the changing
operational environment. This article also describes the REC’s
vision of how the system can be further enhanced.

Tan Kwan Chong

John Kho Yew Hann
Joseph Tan Tow Hua
Wong Rong Fah
Seng Ai Lim
Choo Chee Seng
 Risk Assessment
and Horizon Scanning
6 Experimentation Centre
The Government also recognises that organisations such as SPSS Inc. and SAS The modules and tools augment the
BACKGROUND effective exploitation of technology is Institute Inc. support business intelligence analyst’s work in collecting and classifying
critical in ensuring the success of the and data mining functions. Through information, conducting analysis and
We live in a dynamic and unpredictable RAHS programme. Consequently, the REC discussions with organisations doing RAHS reasoning, as well as constructing models
world where many elements are was inaugurated on 16 October 2007 to work, the REC also learnt that only a few of and running simulations to anticipate
interconnected and interdependent. This spearhead technological exploration, them have dedicated teams of analysts to and discover trends and patterns. This
environment presents both opportunities experimentation and capability development anticipate and monitor emerging strategic is done in a collaborative environment
and challenges. in support of the RAHS programme. The issues. Among the organisations that have that promotes the diffusion of knowledge
REC is currently staffed and managed by such analyst teams, most do not use human across the team of analysts3. Some of
Those who can make sense of the chaos will DSTA under the National Security arm of the augmentation systems in their work. The the individual modules and tools are
secure an enduring advantage. As Peter Ho, Networked Systems Programme Centre. REC realised that a new cognitive-based explained in relation to the analyst’s
former Head Civil Service said at the launch system was needed to exploit new concepts workflow.
of the Risk Assessment and Horizon Scanning in RAHS.
(RAHS) Experimentation Centre (REC) in THE ENGINEERING
Advanced Data Analytics
2007: “We were no longer operating just CHALLENGE Third, there is a constant impetus to update
in the simple or complicated spaces, where the RAHS system. The RAHS concept is still There is a wealth of data available today
events were more predictable, and the causes Delivering a system that supports RAHS evolving as methodologies are continually and the analyst is challenged to extract the
and effects were clearer. Instead, we were analysis does not simply involve identifying integrated into operational processes, relevant data and make sense of it.
operating in complex and chaotic domains, and acquiring commercial software. In resulting in new functional requirements.
where the first to discern patterns out of chaos fact, complex and challenging engineering RAHS also uses a broad range of technology A typical analyst using the RAHS system starts
would have the competitive advantage.” problems are involved. areas such as text analytics and modelling with a database populated with articles of
(Ho, 2007) which are developing rapidly. Therefore, interest. These articles have been trawled in
First, RAHS analysts operate in a different a robust process is needed to manage the by robots or uploaded manually by fellow
Governments around the world have been paradigm that requires systems to support system changes and continually validate the analysts. The analyst can use advanced search
challenged with unforeseen situations such the mental processes of discovery rather than system against the analyst‘s operations which tools e.g. concept, pattern and Boolean
as the Severe Acute Respiratory Syndrome deduction. Traditional human augmented are constantly changing. search to navigate the database effectively
pandemic, the September 11 attacks, and command and control systems are based on
and efficiently. For example, concept search
the 1997 Asian financial crisis. These are deductive models of reasoning. For instance, Conceptualising, architecting and developing enables the analyst to find related words
just some of the more prominent examples in air traffic control or maritime surveillance an effective RAHS system requires engineers or concepts that may be relevant to a
of how traditional analytic and forecasting systems, the operator is clear about the who are creative and competent in a range of search query by utilising a knowledge base
techniques have failed to identify emerging entities to monitor and their signatures. interdisciplinary subjects including cognitive containing word meanings, syntax, word
strategic threats. Deductive systems are suitable in these science and software architecture. Members variations, and relationships between words.
mature operating environments where of the REC are highly innovative and open to In a pattern search, query terms are expanded
The Government places strong emphasis on there is deep domain knowledge and clear experimenting. to include terms with similar spellings.
planning for the future. Scenario planning understanding of the threats. However,
was institutionalised in the mid-1990s and the threats identified through RAHS are
has since been embedded in annual strategic evolving rapidly. A RAHS system needs to DEVELOPING SYSTEM Article summarisation and timeline tools in
the RAHS system provide the analyst with a
planning and budget cycles (Ho, 2009). While assist analysts in identifying patterns from CAPABILITIES quick overview of the search results. Thus,
scenario planning still serves as a vital tool for seemingly disparate data. This can only be
the analyst can easily filter relevant articles
forward planning, it is inadequate for dealing done using discovery-based methods which The RAHS system is designed based on a
to focus on. The system also has an engine
with the inherent dynamism of current machine systems are weak at. cognitive model that is representative of the
that can extract and identify entities such
complex situations. This is because scenario human thought process. Modules and tools in
as organisations, people and locations
planning is essentially a linear process that Second, there are no existing RAHS systems the system are categorised under three main
referenced across the search material. This
extrapolates possible outcomes from existing to refer to. The REC conducted an initial capability areas:
helps the analyst to quickly identify the key
trends (Ho, 2008). The RAHS programme market survey but did not find any
entities associated with a particular domain
was initiated in 2004 to build up capability suitable commercial solution. Among the • Advanced data analytics
without having to plough through the
within the Government to better anticipate systems that the REC has looked at, some • Modelling
content manually. Network analysis tools are
emerging strategic issues, also referred to as support analytics but they are optimised • Perspective sharing
‘black swans’1 and ‘wild cards’2. This capability for specific areas. Systems from Palantir
also available to provide the analyst with a 7
complements the existing scenario planning Technologies, for instance, target finance
process. and intelligence clients while systems from
 Risk Assessment
and Horizon Scanning
Experimentation Centre
8

Figure 2. Screenshots of RAHS modelling and perspective sharing tools

Figure 1. Screenshots of RAHS advanced data analytics tools

visual representation of how these entities be investigated and define relationships The RAHS system is designed to promote phases need not occur sequentially. In fact,
correlate with one another (see Figure 1). between the parameters on the basis real-time collaboration across agencies and they often occur concurrently.
of internal consistency” (Ritchey, 2009). analysts. Documents, folders and models are
Modelling Computational tools allow the analyst to updated immediately to ensure each analyst Explore
collapse numerous combinations of possible using the system has the latest information.
After reviewing the relevant literature, the scenarios into a handful of combinations that The REC keeps abreast of new technologies
are consistent internally.
analyst will externalise his mental models. THE TECHNOLOGY that have been developed and applied
The Systems Map module allows the analyst globally. However, it is difficult to be at
to build network diagrams of variables which Finally, the Ranking Builder module allows INNOVATION the forefront of every field. Hence, the REC
are represented as interconnected nodes, the analyst to monitor scenarios or issues FRAMEWORK identifies and focuses on technologies that
and to model the causal dependencies among that have been identified. The orchestrator are likely to have high payoffs. This was
them. The module also helps the analyst to tool automatically filters articles based The work of the REC goes beyond the difficult initially because the analysts’
identify endogenous reinforcing and on keyword terms into various nodes of delivery of the RAHS system. To develop operations were not clear. Over time,
balancing loops within a system, and the the ranking model. The analyst can also and deliver effective human augmentation through the use of DSTA Enterprise
factors that have the greatest impact on the define thresholds to monitor the nodes. systems for RAHS analysts, the REC has to Architecture modelling techniques, the REC
dynamics in the system (Senge, 1994). The This automation allows the analyst to track continually harness and exploit available was able to elicit and externalise critical
RAHS system automates the identification multiple scenarios or issues across different technologies. Thus, the REC created its own analyst workflows. This enabled a better
of these loops and the calculation of their domains. unique E3 Technology Innovation Framework focus on technology domains which have
relative magnitudes. It is important for (see Figure 3). been narrowed down from seven to three:
analysts to build and share their models Perspective Sharing
to enable better collaboration with their The E3 Technology Innovation Framework • Data analytics
colleagues. The Perspective-based Pattern Detection is an iterative process consisting of three • Text analytics
module allows analysts to tap the insights phases: Explore, Engage and Enhance. The • Computational modelling
Once the dynamics existing in the area of of external stakeholders. This module allows three phases complement and reinforce one
concern have been understood, the analyst stakeholders to provide their perspectives another. For instance, technologies identified The REC also charted technology roadmaps
can start to formulate future scenarios. The on specific articles of interest. A powerful in the Explore phase drive the formulation for each domain to drive efforts in
of experiments, which would involve user
Scenario Option Space module supports the visualisation tool then consolidates these
engagement. Requirements identified during
technology scanning. This enables the REC 9
application of morphological analysis. In perspectives to assist the analyst in detecting to be proactive in recommending new ideas
this method, analysts develop a “discrete weak signals and trends (see Figure 2). the Engage phase serve to build the next- when engaging users. These roadmaps are
parameter space of future scenarios to generation RAHS system and also provide dynamic and they are refined over time.
direction for future exploration. The three
 Risk Assessment
and Horizon Scanning
Experimentation Centre
10

oil and natural gas supply chains would affect techniques to enhance situational awareness.
local industries and the economy. Thus, the A combined experiment was conducted
REC and DSO were led to conduct experiments in 2008 to validate the effectiveness of an
and investigate if advanced computational integrated RAHS and Impactorium4 system in
modelling techniques could improve analysts’ a tactical setting (see Figure 6). A joint paper
understanding of strategic issues. based on this experiment was presented at
the Fusion 2010 conference (Svenson et al.,
An illustration of a highly successful 2010). The REC is working with FOI to further
international collaboration that the REC has enhance and refine the use and application
maintained is that with the Swedish Defence of these modelling techniques for strategic
Research Agency, FOI. Both agencies are analysis.
interested in using probabilistic modelling

Figure 3. E3 Technology Innovation Framework

Engage The experiment was completed in 2010 with

participants from the Ministry of Information,
The REC focuses on operations-technology Communication and the Arts as well as the
integration. Through cognitive task analysis, Ministry of National Development.
the REC elicits mental models and decision
making challenges from the analysts. Sentiment analysis, which applies
Based on the identified challenges, the REC computational linguistics to determine an
conducts experiments on the relevant author’s attitude towards a topic of interest,
technologies that have the potential to is highly relevant in today’s Internet age
simplify the job of the analyst. New work where a large amount of information is
processes are often developed with the available online.
analysts to exploit these latest technologies.
Figure 4. Sentiment analysis prototype
The REC identified the growing use and
The REC conducts experiments using importance of sentiment analysis tools at an
two key methods: in-house prototyping early stage through technology scanning, and
and development, as well as external quickly proposed a concept of use which was
collaborations. Given the small size of the endorsed by the RAHS analysts. The prototype
REC, collaborations often involve leveraging developed by the REC is able to highlight and
the ongoing research of other organisations. recommend important articles for the analyst
Thus, more experiments can be conducted to focus on, saving valuable time. The REC has
simultaneously. The risks are also diversified successfully demonstrated the effectiveness
when the REC invests resources and of sentiment analysis through the prototype
effort in multiple technologies. The REC and it will be integrated into the next version
leverages DSTA’s strong ties with local and of the RAHS system.
foreign defence research institutions for
collaboration. However, if a particular An example of a collaborative effort is the
technology capability is required urgently, experiment conducted with DSO National
the REC has the ability to conduct Laboratories (DSO) to investigate the
prototyping in-house. concept of using interdependency modelling 11
on energy supply chains (see Figure 5). In
An example of an in-house experiment this experiment, analysts from the Energy
conducted by the REC is the development of Market Authority were interested to Figure 5. Global dependency modelling (energy) project
a sentiment analysis prototype (see Figure 4). understand how disruptions to global crude
 Risk Assessment
and Horizon Scanning
Experimentation Centre
12

Figure 6. Swedish-Singapore studies of Bayesian modelling techniques for tactical intelligence analysis

Enhance Figure 7. RAHS system vision

issues regarding the robustness and usability
Operating in a dynamic field, the RAHS system of the system which were addressed in and the use of that picture in directing further
has to adapt to fast-changing user operations subsequent upgrades. perception and anticipating future events”. REFERENCES
and requirements. Enhancing the situational awareness of the
analyst is a goal of the RAHS system. Dominguez, C., Vidulich, M., Vogel, E. and
CONCLUSION McMillan, G. 1994. Situation Awareness:
The RAHS system started out as a web-
The REC has identified three main areas Papers and Annotated Bibliography.
based system for research and analysis. The current RAHS system provides analysts
of experimentation to achieve situational Armstrong Laboratory, Human System
As the analysts began using the system, with a suite of tools and modules that are
awareness in the domain of RAHS (see Centre, ref. AL/CF-TR-1994-0085.
the open-ended nature of RAHS fuelled a more than sufficient to meet their
requirement for a flexible layout to facilitate Figure 7). First, an improvement is required
requirements. The next generation of the Foo, K.J., Wong, R.F., Ni, D.E., Leong, M.W.
data comparison. Thus, the REC decided to in the selection of important articles that the
RAHS system by the REC promises additional and Leong W.H. 2007. Developing a Horizon
explore portlet technology. However, as the analyst should focus on. Next, key themes
analytical and modelling tools as well as Scanning System for Early Warning. Paper
modelling requirements grew, it became and hypotheses of the available data streams
enhanced usability features. presented at the 12th International Command
apparent that a Thick Client was more have to be identified. Finally, there is a
need to study computational models that and Control Research and Technology
suitable to manage the increased resources Looking ahead, the REC will focus on
automatically fuse fragments of information Symposium, Newport, Rhode Island, US,
needed for model editing. technologies to further augment the analysts’
into a cognitive knowledge base. This June 19-21.
experience by automating processes. The
Anticipating the evolving requirements is will enable the analyst to concentrate on
REC will also establish systemic approaches Ho, P. 2007. Future Government: Thinking
a challenging task. The REC has learnt to verifying the accuracy of these automated
to tackle common human analyst failures about the Future. Speech presented at the
use systems architecture to help manage representations and selecting the areas of
due to cognitive bias and linear thinking. Launch of the Risk Assessment and Horizon
this uncertainty. Identifying gaps between focus. For each of these areas, the REC is
Finally, the REC will address the challenges of Scanning Experimentation Centre, Singapore,
the systems architecture and exploration constructing technology and engineering
information overload given the vast quantity October 16.
roadmaps can flag out difficult enhancements roadmaps to identify gaps between current
and diversity of data available to analysts.
at an early stage. and envisioned capabilities. These roadmaps
Ho, P. 2008. Thinking about the Future –
Cynthia Dominguez et al. (1994) describe
will guide the REC in future experimentation
Strategic Anticipation and RAHS. National
13
In 2008, the RAHS system was fielded work.
situational awareness as the “continuous Security Coordination Secretariat.
successfully in an exercise involving extraction of environmental information,
participants from various national security integration of this information with previous
agencies. The exercise helped to identify knowledge to form a coherent mental picture,
 Risk Assessment
and Horizon Scanning
Experimentation Centre
14

Ho, P. 2009. Future Government: Thinking

about the Future. Speech at the Fifth ENDNOTES
Joseph Tan Tow Hua is Chief Executive Officer of Cap Vista Pte Ltd which
Strategic Perspectives Conference, Singapore,
1
Black swans are described in Nassim Nicholas invests in innovative technologies that serve the defence and security
November 23.
Taleb’s book “The Black Swan” as events with needs of Singapore. Prior to joining Cap Vista, Joseph was Head of the REC.
the following three attributes. First, it is an He was a key member of the National Security Engineering Centre and
Petersen, J. 1997. Out of The Blue - How to
outlier as it lies outside the realm of regular was involved in developing strategic technology roadmaps and providing
Anticipate Big Future Surprises. Arlington
expectations, because nothing in the past can technological advice to the National Security Coordination Secretariat
Institute. First Edition.
convincingly point to its possibility. Second, and national security agencies. He held key appointments in the Ministry
it carries an extreme impact. Third, in spite of Defence (MINDEF) Chief Information Officer Office and was involved
Ritchey, T. 2009. Future Studies using
of its outlier status, human nature makes in managing MINDEF’s IT investment portfolio, spearheading the SAP
Morphological Analysis. http://www.
us concoct explanations for its occurrence Enterprise Resource Planning implementation and championing IT
swemorph.com/pdf/futures.pdf (accessed 15
after the fact, making it explainable and governance initiatives such as portfolio management and enterprise
November 2010)
predictable. architecture. Under the Public Service Commission Scholarship, Joseph
obtained a Master of Engineering (Electrical Engineering and Information
Senge, P.M. 1994. The Fifth Discipline: The Art
2
Wild cards are described in futurologist Science) degree with Distinction from the Cambridge University, UK,
and Practice of the Learning Organization.
John Petersen’s book “Out of the Blue – in 1998.
Doubleday Business.
How to Anticipate Big Future Surprises” as
low probability and high impact events that
Svenson, P., Forsgren, R., Kylesten, B.,
would severely impact the human condition.
Berggren, P., Wong, R.F., Choo, M.S. and
Wong Rong Fah is a Senior Engineer (Networked Systems). He oversaw
Kho, J.Y.H. 2010. Swedish-Singapore Studies 3
A more detailed technical discussion the RAHS programme and led a team to implement the RAHS system. He
of Bayesian Modelling Techniques for Tactical
about these modules and tools can be also played a key role in the establishment and management of the REC.
Intelligence Analysis. Paper presented at the
found in the paper entitled “Developing a As a recipient of the DSTA Postgraduate Scholarship, Rong Fah is currently
13th International Conference on Information
Horizon Scanning System for Early Warning” pursuing a Master of Science (Cognitive and Decision Sciences) degree from
Fusion, EICC Edinburgh, UK, 26-29 July.
(Foo et al., 2007). Birkbeck University of London, UK. He obtained his Bachelor of Science
(Computer and Information Sciences) degree from the National University
Taleb, N.N. 2010. The Black Swan. Random 4
Suite of software tools developed by FOI to of Singapore (NUS) in 2000.
House Trade paperbacks. Second Edition. create models representing analyst beliefs.

BIOGRAPHY Seng Ai Lim is a Senior Engineer (Networked Systems). She leads experiments
in developing prototypes of information extraction and analysis capabilities.
Ai Lim also works on trend analysis and sentiment analysis of open source
Tan Kwan Chong is an Engineer (Networked Systems). He is in charge of
unstructured data. She was involved in exploratory projects in data fusion
designing and implementing modelling and simulation experiments at the
systems as well as designing and implementing sensemaking technologies
Risk Assessment and Horizon Scanning (RAHS) Experimentation Centre (REC).
for situational awareness operations. Ai Lim received her Bachelor of
A recipient of the DSTA Overseas Scholarship, Kwan Chong graduated with
Computing degree from NUS in 2002.
a Bachelor of Science (Electrical Engineering) degree from the University of
Michigan, US in 2007. He further obtained a Master of Science (Technology
and Policy) degree from the Massachusetts Institute of Technology, US in
2009.
Choo Chee Seng is a Senior Engineer (Networked Systems) and he is part
of the RAHS pioneer team. He is currently overseeing the development of
a new system that incorporates advancement of commercial off-the-shelf
John Kho Yew Hann is a Senior Engineer (Networked Systems). He oversees
analytics tools and evolution in RAHS processes. He obtained his Bachelor of
the technical architecture of the REC and ensures that it is adaptable to
Science (Computational Science and Physics) degree with Combined Honours
exploit new technologies. He is also responsible for scanning and applying
and a Master of Technology (Knowledge Engineering) degree from NUS in
technologies in the data mining domain as well as fusing data across varied
sources ranging from newspapers to maps and databases. Under the Public
1998 and 2003 respectively.
15
Service Commission Overseas Merit Scholarship, John graduated with a
Master of Engineering (Electrical and Computer Engineering) degree from
Cornell University, US in 2005.