Coevolving Build-Orders for RTS Games

Christopher A. Ballinger, Member, IEEE, Sushil J. Louis, Member, IEEE

AbstractReal-time strategy games pose a difcult challenge
for computational intelligence research. Good players need to
be able to simultaneously manage a complex economy in order
to produce an army capable of defeating the opponent all the
while defending themselves from enemy attack. Often, early
game resource allocation decisions determine whether a player
wins or loses the game. This paper investigates a coevolutionary
approach for generating build-orders for making those initial
early game choices. Using the WaterCraft real-time strategy
game environment developed in our lab, our preliminary results
show that we can coevolve competent build orders that beat
hand crafted opponents. In the future, our current work will
serve as a basis for investigating the coevolution of complete
real-time strategy game AI players.
I. INTRODUCTION
Real-Time Strategy (RTS) games have become a popular
area of computational and articial intelligence research be-
cause of the many challenges they pose. RTS games involve
gathering different resources (such as gas, food, or steel)
in order to build different types of units (such as soldiers,
tanks and planes), which in turn may also have prerequisites
(such as barracks or a factory) that need to be built before
the desired unit can be produced and used. RTS players
usually combat each other until only one player remains.
To be successful a player must gather enough resources and
build units that can successfully attack and defeat opponents
while defending against opponent attacks. However, unlike
board games like Chess, Checkers, Go or Tic-Tac-Toe that
have discrete turns for each player to make a decision and
take an action, RTS games have continuous game play;
players can take as many actions as they possibly can at
any given moment. Also unlike most board games, a fog of
war obfuscates areas of the board so players do not have
access to the game state. Thus, players will not initially know
their opponents locations or what opponents are building.
However, players can choose to dedicate some resources and
units to scout their opponents and learn this information.
RTS game players need two different types of skills:
micro management of game units, often abbreviated to just
micro and macro management of the economy (macro).
Specically, micro deals with unit control and upgrades while
macro involves planning and managing resources and the
economy. Macro play decides when and where to expand
to new resource areas and what and how many unit types
to build. Though players need to make many good macro
and micro decisions to tradeoff economic and military might,
Christopher A. Ballinger is with the Evolutionary Computing Systems
Lab (ECSL), in the Department of Computer Science and Engineering,
University of Nevada, Reno. USA; email: [email protected]
Sushil J. Louis directs the Evolutionary Computing Systems Lab (ECSL),
in the Department of Computer Science and Engineering, University of
Nevada, Reno. USA; email: [email protected]
what units the player decides to build and the order in which
these units get built can have a huge (non-linear) determining
inuence on the outcome of a game. As such, our research
goal is to create build-orders that lead to successful strategies
in an RTS game. We used WaterCraft, a game we developed,
as the testbed for our research. Specically, we investigated
a co-evolutionary approach to generating build orders. Our
coevolutionary approach used tness sharing and a hall of
fame to reliably generate competent, viable build orders that
beat hand-crafted opponents.
The rest of this paper decribes our work and is organized
as follows. The next section describes previous work in
developing game AI for RTS games and more speci-
cally, prior work in generating build-orders int RTS games.
Section III describes WaterCraft, the RTS environment we
developed and used for our research, as well as other similar
research RTS frameworks. We describe our representation
and coevolutionary GA operations in section IV. Section ??
presents preliminary results and shows how well the GA
progresses in coevolving better solutions. We also provide
some qualitative analysis of the build-orders produced by
our algorithm. Finally, the last section provides conclusions
and discusses possible future work.
II. RELATED WORK
Much research has been done in developing game AI for
playing RTS games. One approach uses case based planning
taking a database of expert demonstrations and modifying
them as similar problems are encountered. Mishra et. al.
created one example of such a system which they used
to quickly produce good results. However, such a system
not only requires recording the actions of an expert that
played the game, but also having the expert explain to the
system what goal was being accomplished with each action
taken[?]. A similar method to case-based planning is case-
injected genetic algorithms. This method was used in Miles
and Louis CIGAR study. Unlike case-based planning, a case-
injected genetic algorithm does not require a case-base from
the start, because the GA will learn new cases as the GA
works on the problem. Using case injection increased the
rate at with the GA learned new strategies[?]. Avery and
Louis also did work on coevolving team AI using inuence
maps allowing a group of entities to adapt to opponent
moves[?]. Genetic Algorithms (GAs) have also been used
with reinforcement learning by Tsapanos et. al., in what is
called a zeroth-level classier. The GA in this case found
new rules that helped reinforcement learning algorithms nd
solutions[?]. Spronck et al. used a method called dynamic
scripting to create game AI for a different type of game.
Dynamic scripting involves selecting rules from a knowledge
base with a certain probability for each individual unit that
the game spawns. The probability of a rule being selecting
during this process was set using reinforcement learning as
well. Initially, the rules used in the knowledge base had to
be coded by hand[?]. However, Spronck et. al. later used a
GA to generate rules for the dynamic scripting knowledge
base, which further improved their results[?].
While these previous studies focused on the development
of game AI to manage and adapt to all aspects of RTS game
play, we focus solely on creating build-orders. Case-based
reasoning has been applied to build-orders, as opposed to a
complete game [?]. A depth rst branch and bound algorithm
was used by Buro to adapt a build-order in real time [?]. Tong
et. al. used a GA to tune an articial neural network (ANN)
for determining what units the ANN should use to counter the
opponents attack force [?]. Sandberg used a similar approach,
but used the GA to tune potential eld parameters for small-
scale combat [?]. Others work in developing some aspect of
game AI for RTS games can be found in [?], [?], [?], [?]. In
addition to the variety of RTS game AI approaches, there are
also a variety of RTS environments in which game AIs can
be developed. The next section describes our environment as
well as other existing environments available for RTS game
research.
III. WATERCRAFT
WaterCraft is an RTS game research environment devel-
oped by the Evolutionary Computing Systems Lab (ECSL)
at the University of Nevada, Reno. We wrote Watercraft
primarily in Python with some C/C++ to speed up physics,
and used the Python-OGRE graphic engine for the GUI and
graphics display. We modeled the game play in WaterCraft
around the popular commercial game, StarCraft [?]. While
Watercraft is not as fully featured as a commercial game
and was built for evolutionary computing research, we have
implemented some of the core features of all RTS games.
Players can build several types of buildings and units, with
the objective of destroying their opponents base. The player
UI is very much like other RTS games and a player can
interact with one or more units simultaneously by dragging
the mouse over the units they wish to select, and then clicking
a button in the graphical user interface (GUI) or using a
keyboard hotkey to issue a command. The player also has
the option of playing against a game AI or another player
over the network.
Other similar projects have been used in the past for
research in RTS games. The Brood-War API (BWAPI) allows
developers retrieve information about the game state and
interact with units in StarCraft, one of the most popular
commercial RTS games [?]. Stratagus is a free, cross-
platform RTS engine that has been used directly in several
research projects [?]. One popular game based on Stratagus
is called WARGUS, which uses data from the commercial
game WarCraft II but uses the Stratagus engine as the
back-end for game play and controlling the AI. The same
thing has been done with STARGUS, which uses data from
StarCraft. ORTS is another RTS engine that provides a
total programming environment for AI research, including a
graphical client [?]. Watercraft was built specically for ease
of programming and designed to be used with evolutionary
computing approaches. In the next section, we describe how
we congured Watercraft for our research as well as our
approach to coevolution.
IV. METHODOLOGY
We made several changes to the default Watercraft cong-
uration. First since we evaluate the tness of an individual in
our evolving (coevolving) population by decoding the indi-
vidual into a build order and simulating it in Watercraft, we
need to run watercraft for every evaluation. With thousands
of evaluations necessary, we turned off the graphics display
and GUI during evaluation to simplify our simulations and
reduce the amount of wall-clock time a game takes to run.
By disabling the OGRE graphics engine we greatly re-
duced the amount of processing time required to run a
simulation. Since the same build-order should have the same
effectiveness against the same opponent, we needed our
results to be deterministic. By default, WaterCraft places
buildings in a random location nearby the players base if
a location was not specied. This can cause the outcome of
a game to change since units will have to travel different
distances, and may attack buildings that are in their path. To
make the game play deterministic, we changed this behavior
so that buildings are always built in the same location.
A. Page Layout
The IEEE now only accepts 100% Xplore compliant
papers prepared in PDF format. Please make sure that
you follow these guidelines in preparing your PDF les.
Violations of any of these specications may result in
the rejection of your paper.
Paper size: US letter format (8.5 11 in or 216 278
mm).
Paper length: maximum 8 pages, including gures,
tables, and references. In exceptional circumstances, up
to two additional pages will be permitted for a charge
of USD$100 per additional page.
Paper format: double column, single spaced, 10pt font.
Text width: 7.0 in (178 mm).
Text height: 9.375 in (240 mm). All text and gures
must be contained in the 178 240 mm image area.
The left/right/bottom margin must be 0.75 in (19 mm).
The top margin must be 0.75 in (19 mm), except for
the title page, where it must be 1 in (25 mm).
Text should appear in two columns, each 3.4 in (86.5
mm) wide, with 0.2 in (5 mm) space between columns.
Do NOT number the pages in the manuscript.
Unix LaTeX users should use the following commands
to compile their paper:
latex mypaper
dvips -Ppdf -G0 -tletter mypaper.dvi
ps2pdf -dEmbedAllFonts=true mypaper.ps mypa-
per.pdf
The page size and margin settings in IEEEtran.cls are set
for IEEE Transactions papers. Some adjustments have been
made to produce this sample paper.
Also, please note, IEEE PDF eXpress will be made avail-
able to assist in creating IEEE Xplore compliant PDF les
for camera-ready submission.
V. RESULTS
The main results and ndings go here.
Do not number an equation if it will not be directly cited
in the paper. In order to avoid numbered equations, use
\begin{equation*}\end{equation*}, \[ \], or $$$$. For
example:
a = b + c,
x = f(x, u) + g(x, u),
or
s = G(s, t)
where f, g, and G are functions.
Note that Equation (1) below is numbered! It is produced
using \begin{equation}\end{equation}:
F
i
(P
i
) = a
i
+ b
i
P
i
+ c
i
P
2
i
(1)
where a
i
, b
i
, and c
i
are coefcients of unit i, and P
i
represents some value for unit i.
Aligning equations can be done with either the align or eq-
narray commands. Recently, \begin{align}\end{align} has
gained popularity over \begin{eqnarray}\end{eqnarray}.
Equation (2) is produced using \begin{align}
\end{align}:
x
l
=
m

i=1
c
Px
i
e
kx
i
xi
+ c
Nx
i
e
kx
i
xi
e
kx
i
xi
+ e
kx
i
xi
+
1
2
q

j
(c
Puj
+ c
Nu
j
)
y = A
0
+ A
1
tanh(K
x
x) + B tanh(K
u
u)
= F(x), (2)
where F(x) is a function.
Equation (3) represents the same equation produced using
\begin{eqnarray}\end{eqnarray}:
x
l
=
m

i=1
c
Px
i
e
kx
i
xi
+ c
Nx
i
e
kx
i
xi
e
kx
i
xi
+ e
kx
i
xi
+
1
2
q

j
(c
Puj
+ c
Nu
j
)
y = A
0
+ A
1
tanh(K
x
x) + B tanh(K
u
u)
= F(x), (3)
where F(x) is a function. You get the idea!
TABLE I
PAGE LIMIT
Page limit: 8
Excess page charge: USD$100/page
A. Example of a Figure
Below is an example of a oating gure using the
graphicx package. Note that \label must occur AFTER (or
within) \caption. For gures, \caption should occur after the
\includegraphics. To reference a gure, use the word Figure
followed by the gure number. Here is an example: Figure 1.
Fig. 1. The rst author and a member of her team
B. Figures and Tables
Please follow the style in this sample paper when gener-
ating your gures and tables.
C. What Sections to Include
Usually, your paper should contain an Introduction, Back-
ground, Methodology, Results, and Conclusions section. You
may also add Acknowledgments if you like. After that, you
should list your references in a References section.
D. Page Limit and Overlength Page Charges
A paper submitted to this conference should be prepared in
a single-spaced, two-column format. Its length must be kept
to 8 pages or less. In exceptional circumstances, up to two
additional pages will be permitted for a charge of USD$100
per additional page. Table I shows the page limit and page
charge schedule.
Another example of a table is shown in Table II.
TABLE II
A SECOND TABLE
Method
Mean Best Mean Maximum Minimum
time time cost cost cost
A 928.36 926.20 124793.5 126902.9 123488.3
B 646.16 644.28 124119.4 127245.9 122679.7
C 1056.8 1054.2 123489.7 124356.5 122647.6
D 632.67 630.36 123382.0 125740.6 122624.4
Citations are included like so [1]. Multiple citations appear
like this [2], [3].
VI. CONCLUSIONS
The conclusion goes here.
This template is for LaTeX users. This CIG12 LaTeX
template is based on the CIG08 template written by Luigi
Barone and the CEC07 template written by K.C. Tan. Many
thanks go to them for their hard work. Authors should use
this sample paper as a guide in the production of their
paper. Word users should download and use the template
les posted on the conference website.
APPENDIX
Put your appendix here if you have any.
ACKNOWLEDGMENTS
The authors would like to thank Mr. XYZ for his/her help.
This work was supported in part by the National Science
Foundation under grant no. XXXXX, etc.
REFERENCES
[1] A. Great, This is the book title. This is the name of the publisher,
2006.
[2] F. Author, S. Author, and T. NonRelatedAuthor, This is the paper title,
in This is the proceedings title, 2008, pp. 18.
[3] B. Myself, This is the title of the journal article, This is the name of
the journal, pp. 130, 2007.