See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/265820853

Determining the Position and Motion of a Vessel from Celestial Observations

Article in Navigation - Journal of The Institute of Navigation · December 1995

DOI: 10.1002/j.2161-4296.1995.tb01911.x

CITATIONS READS
27 3,136

1 author:

George H. Kaplan
United States Naval Observatory
123 PUBLICATIONS 1,255 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Evolution and activity of binary and mulriple stars View project

Naval Observatory Vector Astrometry Software (NOVAS) View project

All content following this page was uploaded by George H. Kaplan on 05 February 2019.

The user has requested enhancement of the downloaded file.

 Determining the Position and Motion of a Vessel
from Celestial Observations

George H. Kaplan
U.S. Naval Observatory

Abstract Although many mathematical approaches to the celestial fix problem have been pub-
lished, all of them fundamentally assume a stationary observer. Since this situation seldom occurs
in practice, methods have been developed that effectively remove the observer’s motion from the
problem before a fix is determined. As an alternative, this paper presents a development of celestial
navigation that incorporates a moving observer as part of its basic construction. This development
allows recovery of the information on the vessel’s course and speed contained in the observations.
Thus, it provides the means for determining, from a suitable ensemble of celestial observations, the
values of all four parameters describing a vessel’s rhumb-line track across the Earth: latitude and
longitude at a specific time, course, and speed. In many cases, this technique will result in better
fixes than traditional methods.
Key Words celestial navigation, celestial fix, motion of observer

Introduction
Celestial navigation is the oldest and most fundamental means of determining position at sea. Although
now seldom employed as the primary means of navigation, it is still widely used. For example, despite the
ubiquity of GPS receivers in modern U.S. defense systems, celestial navigation is practiced every day on
ships of the U.S. Navy. Navy policy requires two independent means of position determination, and celestial
navigation backs up GPS as an independent, absolute, world-wide, low-cost, self-contained system. Although
the demise of celestial navigation has been predicted many times, reports of its death are, in the words of
Mark Twain, greatly exaggerated.
The object of celestial navigation, as traditionally practiced, is the determination of the latitude and
longitude of a vessel at a specific time, through the use of observations of the altitudes of celestial bodies.
The chart-based approach, developed in the last century by Sumner and St.-Hilaire, translates each celestial
altitude observation into a line of position (LOP) on the surface of the Earth. In principle, a series of
observations defines a group of intersecting LOPs and this intersection represents the observer’s position—
the fix. Even before computers became small and cheap enough for routine shipboard use, specially designed
navigational almanacs and precomputed sight-reduction tables made the procedure of finding position at
sea relatively simple and free of laborious hand calculations. Chart-based celestial navigation employs some
extremely clever schemes whereby a problem that is simple in concept, but tedious in practice, has been
made relatively easy to solve. The method does involve some approximations (such as the use of straight-line
LOPs), but given the accuracy of the observations made with hand-held sextants (of order ±10 ), these errors
are usually regarded as being insignificant.
The availability of inexpensive compact computing power over the last two decades motivated a re-
examination of the celestial fix problem. For most authors, the objective was to develop robust mathematical
solutions to the problem so that the algorithms could be readily converted into reliable computer or calculator
programs. A number of innovative mathematical approaches to celestial navigation were formulated.
Two independent papers [1, 2] present an exact mathematical solution to a two-body fix. In [1], the
development is based on the celestial coordinates of the two observed bodies, while in [2], the development
is based on the geometry of the two LOPs. Both solutions are suitable for hand-held calculators. A less
elegant iterative scheme for solving a two-body fix is given in [3].
When there are more than two observations, the problem is overdetermined and least-squares techniques
are appropriate. A least-squares approach for a multi-star fix is presented in [4] based on the plane geometry
and straight lines formed by LOPs near the estimated position. The method is a direct mathematical
translation of chart-based navigation. It was developed independently at the Royal Greenwich Observatory

1
(RGO) and has been adopted by RGO in the publication Compact Data for Navigation and Astronomy [5].
The next issue of Compact Data will include a PC diskette with software that implements it. On page 282
of the Nautical Almanac [6], the section titled “Position from intercept and azimuth using a calculator” uses
this algorithm.
Several other approaches to the multi-body problem have been published. A least-squares method based
directly on the equation for a body’s altitude as a function of latitude and longitude is developed in [7]. A
vector-matrix approach to multi-body fixes has been published [8]. An alternative least-squares algorithm
for sight reduction problems, using a Lagrange multiplier for normalization, is presented in [9], and applied
to a vector-matrix formulation of the problem similar to that in [8]. The vector-matrix formulations, while
elegant, have the disadvantage that the solution is obtained for a set of three parameters that have to be
transformed to latitude and longitude. The transformations for this are straightforward and are given in
the papers cited. However, the corresponding transformations of the statistical uncertainties (including the
covariance needed to construct an error ellipse) are much less tidy and have not been published.
As varied as these algorithms are, the fundamental developments for all of them assume two or more
co-located observations, which requires either a stationary observer or simultaneous sights. Normally, of
course, the observations are taken sequentially from a moving ship. Therefore, something has to be done
to account for the change in the observer’s position during the time required to make the observations.
Conventional schemes for doing so are reviewed in [10]; they effectively transform a moving-observer problem
to a fixed-observer problem. For example, a commonly used procedure is the addition of motion-of-observer
corrections to the observed altitudes. The most important weakness of such procedures is well known: they
require data on the motion of the observer’s ship over bottom, and the course and speed values used may
not be accurate. The accuracy of these quantities is usually limited by our inexact knowledge of the local
current. (The problems of running fixes are described in [11], pp. 136–137.) The errors involved are such
that, for sights made with ordinary hand-held sextants, difficulties may arise for observations spread over
more than about 0.5 h. If automated star trackers or similar high-accuracy devices were used for shipboard
celestial navigation, observations spread over only a few minutes might be problematic.
Fortunately, the observations themselves contain information on the actual track of the vessel, so the
possibility exists that the sight-reduction procedure can be made self-correcting. In principle, given enough
observations, suitably distributed in time and azimuth, we should be able to obtain an estimate of the
average over-bottom track of the vessel as part of the solution for the fix. The next two sections of this
paper, together with the Appendix, present a development of celestial navigation that incorporates a moving
observer as part of its basic construction. Such an approach correctly represents the propagation of positional
error along the observer’s track, here considered to be a rhumb line (loxodrome). Furthermore, the procedure
allows us, under certain conditions, to recover information on the vessel’s actual course and speed from the
observations. This new algorithm includes the observer’s motion as an essential part of the mathematics of
celestial navigation, rather than as an add-on.

Celestial Navigation as an Orbit Correction Problem

Suppose we are given a series of observations taken over an extended period of time from a moving
vessel. Is there a way to mathematically develop celestial navigation that includes the vessel’s motion in the
problem from the outset? Further, can such a development allow us to exploit the observations to correct
our initial estimates of the course and speed of the vessel, as well as to provide a fix for a given time?
Our problem is quite similar to “orbit correction” problems faced by astronomers who deal with the
dynamics of solar system bodies. Given a series of observations of some moving object in the solar system—an
artificial satellite, a deep-space probe, an asteroid, or a planet—we want to be able to compute the position
of the object at any given time. We know the laws of motion that the body obeys, but there are an infinite
set of physically possible trajectories. Therefore, we must use the observations to determine the initial
conditions, or orbital elements. The orbital elements are six parameters that specify the object’s position
and motion, in three dimensions, at a designated time. Once these six parameters have been determined,
the object’s position at any other time can be computed. The problem is the same regardless of what kind
of observations are available. The observations may consist of simultaneous measurements of both celestial

2
coordinates, or the observations may be only of range (distance). In the latter case we use a series of one-
dimensional observations to solve a six-dimensional problem. As long as we have at least six observations
(suitably distributed) the problem is solvable.

The running fix problem in celestial navigation is analogous. The moving object of interest is the
observer’s ship. The fact that the observations are taken from the moving object rather than of the moving
object does not change the nature of the problem. However, in celestial navigation, the problem is four-
dimensional rather than six-dimensional because ships are constrained to move over the two-dimensional
surface of the Earth. The sailing formulas for rhumb-line tracks are the “laws of motion.” We have a series
of one-dimensional observations—sextant altitudes—from which we wish to determine, for a given time,
the four “orbital elements” of the vessel: latitude, longitude, course, and speed. Once these have been
determined, the vessel’s position at any other time can be computed. As long as we have four or more
observations, well distributed in azimuth and time, the problem is solvable.

The orbit correction problem is usually dealt with through differential correction, utilizing least squares.
This requires that we have some initial knowledge of the trajectory of the object of interest, which is almost
always the case in both astronomy and navigation. This allows us to make a reasonably accurate estimate
of the value of any observed quantity (e.g., declination or altitude) for any given time. The small difference
between the observed value and the computed estimate can be accounted for by corrections to selected
parameters in our a priori model of the object’s motion. The problem is said to be linearized because
mathematically we expand the formula for the observed quantity to first order in the parameters that we
want to correct. That expansion becomes the conditional equation for a least-squares solution.

The solution to the problem is straightforward in theory, but the algebra can be messy. Suppose we
have a sextant altitude, hs, measured at time t. We can obtain an observed altitude, Ho, by applying the
usual corrections for index error, dip, and refraction to hs. We can also generate a computed altitude, Hc,
which is based on our initial estimates of the vessel’s location at the time of the observation. The formula
for Hc as a function of the observer’s location and the celestial coordinates of the observed body is well
known. Each observation’s altitude intercept, a = Ho − Hc, corresponds to what astronomers refer to as an
O–C, an “observed minus computed” value. We assume that our initial knowledge of the vessel’s track is
reasonably good and that the observed body’s celestial coordinates are exactly known, so that Hc ≈ Ho and
a is small. If the observation were perfect, a non-zero value for a would represent an error in Hc due to an
incorrect estimated position. Expanding Hc in terms of small increments of the instantaneous latitude, φ,
and longitude, λ:
∂Hc ∂Hc
a = ∆Hc = ∆φ + ∆λ (1)
∂φ ∂λ

Equation (1) can be readily evaluated for each observation, an approach taken in [7]. In the relation above,
∆φ and ∆λ are corrections to the coordinates of the vessel at the observation time, t. However, what we
really want are corrections to the latitude, longitude, course, and speed of the vessel at some reference time,
t0 . Obviously, having correct values for these four parameters would allow us to compute the instantaneous
latitude and longitude of the vessel at any other time. Symbolically,

φ = f (φ0 , λ0 , C, S, t − t0 )
(2)
λ = g(φ0 , λ0 , C, S, t − t0 )

where φ0 , λ0 , C, and S are, respectively, the vessel’s latitude, longitude, course, and speed at time t0 . We
do not place subscripts on C or S because we assume the vessel is sailing a rhumb-line track at a fixed speed
so these parameters have constant values. The functions f and g represent the sailing formulas for latitude
and longitude, respectively, for rhumb-line tracks (see [12]). (There is no reason this development could not
also be made to apply to great-circle tracks.) Assume that initially we have estimates for the values of φ0 ,
λ0 , C, and S, but that these are in need of correction from the observations. That is, t0 , our reference time,
is also the time for which we want a fix. Therefore, we want to rewrite equation (1) as a function of these
four quantities. To do so we need to express ∆φ and ∆λ, which are themselves small quantities, in terms of

3
small increments in the four parameters of interest:

∂f ∂f ∂f ∂f
∆φ = ∆φ0 + ∆λ0 + ∆C + ∆S
∂φ0 ∂λ0 ∂C ∂S
(3)
∂g ∂g ∂g ∂g
∆λ = ∆φ0 + ∆λ0 + ∆C + ∆S
∂φ0 ∂λ0 ∂C ∂S

Substituting equations (3) into equation (1) and rearranging terms we obtain
   
∂Hc ∂f ∂Hc ∂g ∂Hc ∂f ∂Hc ∂g
a= + ∆φ0 + + ∆λ0
∂φ ∂φ0 ∂λ ∂φ0 ∂φ ∂λ0 ∂λ ∂λ0
    (4)
∂Hc ∂f ∂Hc ∂g ∂Hc ∂f ∂Hc ∂g
+ + ∆C + + ∆S
∂φ ∂C ∂λ ∂C ∂φ ∂S ∂λ ∂S

Equation (4) expresses the altitude intercept in terms of small increments (errors) in the estimated values of
φ0 , λ0 , C, and S, denoted ∆φ0 , ∆λ0 , ∆C, and ∆S, respectively. These latter four quantities are unknowns
to be determined, and once their values are known, we can correct our estimate of the vessel’s track. The
quantities within parentheses in equation (4) are the coefficients of the unknowns, which, along with the
altitude intercept a, can be numerically evaluated for each observation. We originally postulated a perfect
observation, and evaluating equation (4) for four such observations would allow a direct solution for the four
unknowns. Of course, we never have perfect observations, so we obtain more than four and use the above
equation as the conditional equation in a least-squares solution for the unknowns.
Equation (4) can also be expressed in matrix form:

 ∂f ∂f ∂f ∂f   ∆φ0 
  ∂φ0 ∂λ0 ∂C ∂S ∆λ0 
a= ∂ Hc ∂ Hc   (5)
∂φ ∂λ ∆C
 
∂g ∂g ∂g ∂g
∂φ0 ∂λ0 ∂C ∂S ∆S

To numerically evaluate equation (4) for each observation, we need to expand the partial derivatives in
equation (4) into algebraic expressions. There are ten partials required:

∂Hc ∂Hc ∂f ∂f ∂f ∂f ∂g ∂g ∂g ∂g
, , , , , , , , ,
∂φ ∂λ ∂φ0 ∂λ0 ∂C ∂S ∂φ0 ∂λ0 ∂C ∂S

These are obtained from three formulas: the equation for Hc as a function of φ and λ, and the sailing
formulas represented by the functions f and g. In the Appendix, the expressions for the partials needed in
equation (4) are evaluated.
Note that we can obtain a traditional fix by simply not solving for ∆C and ∆S, that is, by ignoring the
second line of equation (4). That would be advisable if there were only a few observations, or the observations
were poorly distributed in azimuth or tightly grouped in time.
Least-squares methods are based on the assumption that the errors of observation have a Gaussian
distribution. Ideally, a non-zero value for a = Ho − Hc arises from the incorrect values of the parameters
used to compute Hc and from measurement noise in Ho. However, there are several other contributors to the
value of a that can affect the solution. First, Ho is not a direct measurement, but is derived from the sextant
altitude hs by applying several corrections. The most uncertain of these is atmospheric refraction, and small
systematic errors (of order 0.1 arcmin) cannot be ruled out under some conditions. Second, a ship does not
follow an exact rhumb-line course at constant speed, but is subject to random variations in wind, current,
and steering. As illustrated in [11], a vessel’s path over bottom is a somewhat irregular line. These deviations
from a rhumb-line track are, of course, unknown and cannot be included in the calculation of Hc. The effect
of these variations on the values of a is random if the typical time between observations is greater than the
time scale of the variations themselves. (To make a more definitive statement, a quantitative model of the

4
wind, current, and steering variations would be needed.) Given a sufficient number of such observations,
then, the algorithm described in this paper yields the parameters for a kind of average rhumb-line track over
bottom, which is, presumably, what is desired. More problematic are systematic changes in the current or
wind that occur over time scales of hours. In such circumstances the ship’s track may not be well represented
by a single constant-speed rhumb line, yet the algorithm described in this paper will force fit the data to
such a model. If the ship’s track can be modelled as a series of connected rhumb lines, then a generalization
of the algorithm described here can be used; see [13].

Procedure
An outline of the recommended computational procedure follows. Equations (A1) through (A9), referred
to below, are given in the Appendix.
(1) Determine the time, t0 , for which the fix is to be computed. Usually t0 will be within the time span
represented by the observations. Estimate the values of the latitude φ0 , longitude λ0 , course C, and speed
S for time t0 . The course and speed should include the set and drift of the current, that is, they should
refer to the vessel’s motion over bottom. If necessary, use the sailing equations (A7) (or equations (A5) for
east-west tracks) to propagate the latitude and longitude forward from an earlier fix.
(2) Decide whether course and speed corrections are to be determined. These corrections should not be
attempted unless there are at least eight observations with a good distribution of azimuths. Furthermore, for
the determination to be reliable, the observations must span a significant period of time. What constitutes a
significant period of time depends on the uncertainties in the vessel’s course and speed over bottom and the
accuracy of the observations. For hand-held sextant observations, the required period of time will generally
be several hours or more, and twilight observations would have to be combined with day or night observations
(see next two sections).
(3) Correct each observation for index error, dip, refraction, and other effects in the usual manner,
converting each sextant altitude, hs, to an observed altitude, Ho.
(4) Take each of the observations in turn and perform the following operations on it:
(a) Use the sailing formulas to compute the estimated position (φ, λ) of the vessel at the time t
of the observation. For east-west courses, use equations (A5), then set φ = φ0 and λ = λ0 . For other
courses, first compute φ0 from equations (A5), then use equations (A7) to obtain φ and λ.
(b) From the appropriate ephemeris (almanac routine), determine the celestial coordinates (GHA,
Dec) of the body observed at time t. Compute Hc using equation (A1), then compute the altitude
intercept a = Ho − Hc.
(c) Compute the partial derivatives of the sailing formulas. For east-west courses, use equations
∂φ0 0 0
(A6). For other courses, compute ∂φ 0
, ∂∂φC , and ∂φ
∂S
∂λ
from equations (A6), compute ∂φ from equa-
tion (A8), then use equations (A9).
Hc and
(d) Evaluate equation (4), the conditional equation for the observation. First compute ∂∂φ
∂ Hc
∂λ from equations (A2). Then combine these with the partial derivatives previously computed to
evaluate the quantities in parentheses in equation (4), i.e., the four coefficients of the unknowns. If
corrections to course and speed are not to be determined, only the two coefficients in the first line of
equation (4) need to be evaluated.
(e) Include the conditional equation for the observation in the least-squares solution. This may
involve a routine that accepts as input the altitude intercept and the coefficients of the unknowns for
the current observation. The coefficients for the current observation become a new row in what is called
the design matrix of the least-squares solution. (For a description of least-squares techniques, see any
text on scientific data analysis, for example, [14].)
(5) Compute the least-squares solution, based on all the observations. In the most general case, the
solution will yield values of ∆φ0 , ∆λ0 , ∆C, and ∆S. If corrections to course and speed are not being

5
determined, however, we will obtain only ∆φ0 and ∆λ0 from the solution and must consider ∆C and ∆S to
be zero. The corrections ∆φ0 , ∆λ0 , and ∆C obtained from the solution will be expressed in radians, and ∆S
will be expressed in km/h. Least-squares procedures also provide the uncertainties in the computed values of
the unknowns as elements of a variance-covariance matrix. If needed, an error ellipse in the latitude-longitude
plane can be constructed using the variances of latitude and longitude together with the latitude-longitude
covariance (see, for example, [14]).
(6) Using the results of the least-squares solution, correct the initial estimates of φ0 , λ0 , C, and S:
φ0 → φ0 + ∆φ0
λ0 → λ0 + ∆λ0
C → C + ∆C
S → S + ∆S

(7) Decide whether another iteration of the procedure is necessary. Although good results are often
obtained in a single pass, a second is recommended to ensure that the results have converged. Generally,
more than two iterations are needed only if the initial estimates of φ0 , λ0 , C, and S used in step (1) were
badly in error. A general iteration decision can be based on the values of ∆φ0 , ∆λ0 , ∆C, and ∆S; if they are
all smaller than pre-established criteria, or significantly smaller than their own uncertainties, then another
iteration is unnecessary. If another iteration is required, take the new values of φ0 , λ0 , C, and S, and redo
the procedure above beginning at step (4).
The new values of φ0 and λ0 define the fix for time t0 . These, combined with the new values of C and
S, define an estimate of the vessel’s average rhumb-line track, over bottom, during the time the observations
were taken.

Sample Calculation
A numerical example using this algorithm is illustrative. In the sample calculation below, a set of eight
artificial sextant observations (hs values), spanning eight hours, was reduced using the above procedure,
resulting in a position fix and course and speed estimates. The artificial observations were generated by a
computer program that numerically integrated a hypothetical ship’s rhumb-line track based on the data in
Table 1 (north latitudes and east longitudes are positive).
Table 1—Ship’s Track Used in Sample Calculation
Date: 10 Apr 1995
Time: 14:00:00 UTC
Latitude: +45.0000◦
Longitude: −50.0000◦
Course: 330.00◦
Speed: 20.00 kn
The data in Table 1, and any positions derived from them, constitute “truth” for this case. Astronomical
positions of the celestial bodies chosen were computed that are consistent with those in the 1995 Nautical
Almanac and transformed to local altitudes. Atmospheric refraction was included but the height of eye
was considered to be zero, and the time scale UT1 was assumed to be identical to UTC (that is, DUT1=
UT1–UTC=0). Random errors with a 0.7 arcmin standard deviation were added to the calculated hs values.
The artificial observations, all for 10 April 1995 (UTC), are given in Table 2.

Table 2—Artificial Observations for Sample Calculation

UTC Object hs UTC Object hs
◦ ◦
15:19:22.4 Sun (L) 52.2903 22:30:15.6 Mars 59.3888
18:45:05.9 Moon (U) 19.1563 22:46:27.5 Kochab 44.0681
20:01:13.8 Sun (L) 20.2358 23:02:39.4 Capella 54.3544
21:01:10.0 Moon (U) 39.4857 23:20:28.5 Rigel 14.3925

6
The data in the left column of Table 2 represent daytime observations, and those in the right column represent
evening twilight observations (sunset was at 22:09). The Sun data apply to the lower limb (L) and the Moon
data apply to the upper limb (U), as would be appropriate for observations from the hypothetical ship at
the times listed. The Moon’s phase was 0.75 (age 11 days).
Once computed and stored in a file, these artificial observations were used as input to a separate
computer program that implemented the new algorithm described in this paper. This second program was
given the incorrect data on the ship’s position and motion listed in Table 3.
Table 3—Incorrect Data on Ship’s Track
Supplied to New Algorithm
Date: 10 Apr 1995
Time: 14:00:00 UTC
Latitude: +45.4000◦
Longitude: −49.5000◦
Course: 332.00◦
Speed: 18.50 kn

The test of the algorithm is whether, given only the eight observations (Table 2), it can determine the true
track of the vessel (Table 1), limited only by the error in the observations. The date and time for the
desired fix, t0 , were chosen to be 10 Apr 1995 at 23:00:00. Latitude, longitude, course, and speed were to be
determined. Based on the incorrect data on the ship’s track it was given, the second program computed the
estimated position for time t0 to be φ0 = +47.8493◦ , λ0 = −51.3907◦ . It also computed the data in Table 4
for the eight observations.

Table 4—Estimates of Sight-Reduction Quantities for Observations

UTC Est φ Est λ Object Ho Hc a Zn
◦ ◦ ◦ ◦ 0 ◦
15:19:22.4 45.7601 −49.7726 Sun (L) 52.2774 51.9007 +22.61 179.5
18:45:05.9 46.6932 −50.4876 Moon (U) 19.1091 19.5297 −25.24 96.8
20:01:13.8 47.0386 −50.7553 Sun (L) 20.1913 19.6929 +29.90 260.1
21:01:10.0 47.3104 −50.9672 Moon (U) 39.4656 39.6996 −14.04 126.2
22:30:15.6 47.7144 −51.2843 Mars 59.3790 59.3585 + 1.23 154.9
22:46:27.5 47.7879 −51.3422 Kochab 44.0510 44.4809 −25.80 22.4
23:02:39.4 47.8613 −51.4002 Capella 54.3425 53.9105 +25.92 287.7
23:20:28.5 47.9421 −51.4641 Rigel 14.3293 13.7379 +35.48 240.6

In Table 4, the altitude intercept, a, is Ho − Hc and positive values indicate “toward.” The Ho and Hc values
are topocentric and, for the Sun and Moon, they refer to the limb observed. The computed azimuth values,
Zn, are not used by the algorithm and are listed in the table only for reference.
As it processed each observation, the program computed the values of the coefficients of the four un-
knowns in equation (4). It sent these coefficient values, along with the value of a, to the least-squares routine,
where a new row in the design matrix was constructed. All the observations were given equal weight. After
all eight observations had been processed in this way, the least-squares solution was formed. The result, for
time t0 = 23:00:00 UTC, is given in Table 5.

Table 5—Least-Squares Solution—Iteration 1

Parameter Values Parameter Correlations
∆φ0 −0.249370 ± 0.008981 ◦ ∆φ0 +1.000 −0.038 +0.108 +0.526
∆λ0 −0.787800 ± 0.012634 ◦ ∆λ0 −0.038 +1.000 +0.669 −0.492
∆C −2.175234 ± 0.579230 ◦ ∆C +0.108 +0.669 +1.000 −0.397
∆S +1.505770 ± 0.150014 kn ∆S +0.526 −0.492 −0.397 +1.000
∆φ0 ∆λ0 ∆C ∆S

7
In Table 5, the values and errors of the unknowns (1σ mean errors) have been converted to degrees and knots
from radians and kilometers per hour, respectively. The parameter correlations were derived directly from
the variance-covariance matrix from the solution and indicate the degree to which pairs of parameters were
statistically coupled in the solution. The solution was well behaved and the mean error of unit weight was
computed to be 0.8 arcmin. The program then added the parameter values from Table 5 to the estimated
(incorrect) data on the ship’s track it had been using. The results for 23:00:00 UTC are summarized in
Table 6.

Table 6—Results of Solution for Ship’s Track—Iteration 1

Truth Estimate Solution Truth–Estimate Truth–Solution
φ0 +47.5972 +47.8493 +47.5999 −0.2521 −0.0027
λ0 −52.1640 −51.3907 −52.1785 −0.7733 +0.0145
C 330.00 332.00 329.82 −2.00 +0.18
S 20.00 18.50 20.01 +1.50 −0.01

The units used in Table 6 are the same as in previous tables. Obviously a great improvement in the data on
the ship’s track was made. Of course, in a real situation, “Truth” would not be known; the question at this
point would be whether another iteration of the procedure would be advisable, using the updated values of
φ0 , λ0 , C, and S now available. We note that in Table 5, the absolute values of all the unknowns are greater
than their errors, a condition that suggests that another iteration might be fruitful. Another pass through
the procedure was therefore made, using the values listed under the “Solution” column in the Table 6 as
the new data on the ship’s track. For the second iteration the program obtained the data given in Tables 7
and 8.

Table 7—Least-Squares Solution—Iteration 2

Parameter Values Parameter Correlations
◦
∆φ0 −0.004465 ± 0.009198 ∆φ0 +1.000 −0.034 +0.131 +0.509
∆λ0 +0.004870 ± 0.012817 ◦ ∆λ0 −0.034 +1.000 +0.662 −0.510
∆C +0.076011 ± 0.543305 ◦ ∆C +0.131 +0.662 +1.000 −0.402
∆S +0.021225 ± 0.154628 kn ∆S +0.509 −0.510 −0.402 +1.000
∆φ0 ∆λ0 ∆C ∆S

Table 8—Results of Solution for Ship’s Track—Iteration 2

Truth Estimate Solution Truth–Estimate Truth–Solution
φ0 +47.5972 +47.5999 +47.5954 −0.0027 +0.0018
λ0 −52.1640 −52.1785 −52.1737 +0.0145 +0.0097
C 330.00 329.82 329.90 +0.18 +0.10
S 20.00 20.01 20.03 −0.01 −0.03

Table 8 indicates that the second iteration was an improvement over the first iteration. Note that the
“Solution” from the first iteration appears as the “Estimate” for the second iteration.
The new algorithm did recover the ship’s true track quite well. For this case, the error ellipse in the
latitude-longitude plane is nearly a circle, with a circular probable error (CEP) of 0.63 nmi. The ship’s true
position for 23:00 is well within the circle; the actual error of the fix is 0.41 nmi.
Other numerical experiments were conducted to provide a broader assessment of the algorithm. The
first was to repeat the above calculations without solving for course and speed. The fix position obtained
(for 23:00) was φ0 = +47.5471◦ , λ0 = −52.1115◦ , an error of 3.67 nmi. Clearly, in this case, the entire set
of observations should not be used for a fix without solving for course and speed. However, a more common
practice would be to use just the twilight observations. When the four twilight observations were used to
obtain a fix without solving for course and speed, the resulting position was φ0 = +47.5867◦ , λ0 = −52.1752◦ ,
an error of 0.78 nmi. So the new algorithm, used with the entire set of eight observations, improved the

8
23:00 fix by about a factor of two. The algorithm becomes progressively more advantageous as the accuracy
of the observations increase, as Table 9, computed for the case being considered, shows.

Table 9—Algorithm Comparison

Random Error — New Algorithm — — Standard Fix —
of Observations CEP Actual Error CEP Actual Error
0 nmi nmi nmi nmi
0.7 0.63 0.41 0.80 0.78
0.3 0.27 0.17 0.51 0.53
0.1 0.090 0.058 0.36 0.45
0.02 0.018 0.013 0.31 0.42
0 0.001 0.001 0.29 0.42

In Table 9, “New Algorithm” refers to a solution for φ0 , λ0 , C, and S from all eight observations, while
“Standard Fix” refers to a solution for φ0 and λ0 from the four twilight observations. (If the standard fix
had included all eight observations, its results would have been considerably worse.) Two iterations were
performed in all cases. As the errors in the observations decrease, the two methods separately converge, but
to different answers—one correct, the other not. For the standard fix, neither the CEP nor the actual error
decreases in proportion to the random error of the observations because of the underlying systematic error
from the incorrect course and speed. Geometrically, the incorrect course and speed prevent the advanced
LOPs from converging toward a single point. Table 9 neatly summarizes the advantages of the new algorithm
for accurate observations. Of course, we have been assuming that the ship’s track is well modelled by a
constant-speed rhumb line over the eight hours that the observations span. If that is not the case, the new
algorithm will be less successful.

Discussion
The algorithm described in this paper has been incorporated into a new software package for celestial
navigation. The package was developed by the U.S. Naval Observatory in response to a Navy requirement for
a modern computer implementation of various celestial navigation procedures. Although now implemented
only on PCs, this software will eventually become a module in the Navy’s Navigation Sensor System Interface
(NAVSSI) [15].
The algorithm has been extensively tested using artificially generated data, both with and without
random errors. Many examples found in navigation texts have been reduced again using it, and the results
compared with other sight-reduction algorithms. Additionally, because the software that implements the
algorithm has been beta tested on board deployed Navy vessels, the method has been checked with real
data. The algorithm works well and is robust. Often, adequate answers are obtained in a single iteration,
although two iterations are the norm. Covariances among the parameters being solved for are usually low
with reasonable observational data. The tests with perfect artificial observations have demonstrated the
mathematical correctness of the algorithm.
However, the full power of the algorithm has probably not been used so far in practice. As previously
noted, the most common navigation procedure is a round of sights, in which a small number of observations
(usually three to five) are taken within a short period of time, in twilight, and reduced to determine a fix.
The accuracies of hand-held sextant observations from a moving ship are typically of order one arcminute
and the accuracy of the resulting fix may range from a half to several nautical miles. For these reductions
almost any sight-reduction procedure is adequate and the algorithm described in this paper does not have
significant practical advantages over others. Course and speed corrections cannot be determined with such
a limited observation set (and the software that implements the new algorithm will not attempt to do so).
For such cases, all reasonable algorithms give essentially the same answer.
The advantages of the algorithm presented in this paper emerge when navigational practice is extended
beyond the usual twilight round of sights or noon Sun line. It would be especially suited for an automated
star tracker with an artificial horizon that could observe all night (or, in the near infrared, during the day).

9
Star trackers in various forms are currently used on aircraft, missiles, and spacecraft, and accuracies of a few
arcseconds are attainable. But even with ordinary hand-held marine sextants, more flexibility in navigational
procedures is possible than is usually practiced. For example, at high latitudes, long periods of twilight allow
for extended sets of observations. Sun-Moon fixes are geometrically possible during about half of all days.
(Navigators have avoided using the Moon as a target because of the extra corrections that Moon observations
require and the false horizons that can appear under the Moon at night. Sight reduction software such as
that described in this paper eliminates the first problem, and the second does not arise during the day.) Sun
or Moon observations from early or late in the day can be combined with twilight observations. Observations
of the stars and planets are possible at night near full Moon when the sky is bright enough to make the
horizon visible. It is these cases—involving an extended sequence of observations, very accurate observations,
or both—that the algorithm described in this paper was designed for. Furthermore, the algorithm can be
generalized to observation sets taken over multiple voyage legs [13].

Conclusion
A variety of algorithms for celestial navigation have been published, but all are fundamentally based
on the concept of a fixed observer. For these algorithms, the motion of the observer is a nuisance external
to the basic development. Either explicitly or implicitly, the moving-observer case must be transformed to
a fixed-observer case, a process that requires data on the observer’s course and speed. Errors in the course
and speed values used can propagate into the celestial fix.
As an alternative, this paper presents an algorithm for celestial navigation based upon the concept of a
moving observer. This algorithm can use the information contained in a set of celestial observations on the
track of the observer’s ship across the surface of the Earth. For a suitable set of observations, the algorithm
allows for the determination of the course and speed of the observer’s vessel, as well as its position at a given
time. Fully exploiting this algorithm would require some changes in navigational practice or equipment.
However, the algorithm also yields a correct fix from a traditional round of sights. It has been extensively
tested. The algorithm has been incorporated into a new software package that will soon be a standard
navigation tool on ships of the U.S. Navy.

References
1. A’Hearn, M. F., and Rossano, G. S., “Two Body Fixes by Calculator,” Navigation Vol. 24, No. 1, Spring
1977, pp. 59–66.
2. Van Allen, J. A., “An Analytical Solution of the Two Star Sight Problem of Celestial Navigation,”
Navigation Vol. 28, No. 1, Spring 1981, pp. 40–43.
3. Ogilvie, R. E., “A New Method of Celestial Navigation,” Navigation Vol. 24, No. 1, Spring 1977, pp. 67–
71.
4. De Wit, C., “Optimal Estimation of a Multi-Star Fix,” Navigation Vol. 21, No. 4, Winter 1974–75,
pp. 320–325.
5. Yallop, B. D., and Hohenkerk, C. Y., Compact Data for Navigation and Astronomy for the Years 1991–
1995, Cambridge University Press, Cambridge, 1990.
6. The Nautical Almanac for the Year 1995, U.S. Government Printing Office, Washington, 1994.
7. Severance, R. W., “Overdetermined Celestial Fix by Iteration,” Navigation Vol. 36, No. 4, Winter 1989–
90, pp. 373–378.
8. Watkins, R., and Janiczek, P. M., “Sight Reduction with Matrices” (Forum), Navigation Vol. 25, No. 4,
Winter 1978–79, pp. 447–448.
9. Metcalf, T. R., and Metcalf, F. T., “On the Overdetermined Celestial Fix,” Navigation Vol. 38, No. 1,
Spring 1991, pp. 79–89.
10. Kaplan, G. H., “The Motion of the Observer in Celestial Navigation,” unpublished technical report,
U.S. Naval Observatory, 1995.

10
11. The American Practical Navigator, Pub. No. 9, Defense Mapping Agency Hydrographic/Topographic
Center, Bethesda, MD, 1995.
12. Kaplan, G. H., “Practical Sailing Formulas for Rhumb-Line Tracks on an Oblate Earth,” Navigation
Vol. 42, No. 2, Summer 1995, pp. 313–326.
13. Kaplan, G. H., “A Celestial Navigation Solution Involving Changes to Course and Speed,” unpublished
technical report (to be submitted to Navigation), U.S. Naval Observatory, 1995.
14. Meyer, S. L., Data Analysis for Scientists and Engineers, John Wiley & Sons, Inc., New York, 1975.
15. Bangert, J.,“STELLA: New Life for an Old Art,” in Proceedings of the Annual DoD Applied Astrometry
Forum, edited by F. S. Gauss and B. Hicks, U.S. Naval Observatory, 1995.

11
Appendix

Expressions for Terms in the Conditional Equation

To use equation (4) as a conditional equation for a least-squares solution for various unknowns, we need
to be able to numerically evaluate the quantities within parentheses in the equation. These quantities are the
coefficients of the unknowns. For each observation, the following ten partial derivatives must be computed:
∂Hc ∂Hc ∂f ∂f ∂f ∂f ∂g ∂g ∂g ∂g
, , , , , , , , ,
∂φ ∂λ ∂φ0 ∂λ0 ∂C ∂S ∂φ0 ∂λ0 ∂C ∂S
The first two will be derived from the usual spherical-trig equation for Hc, the next four are obtained from
the sailing formula for latitude (the function f ), while the last four are obtained from the sailing formula for
longitude (the function g). The equation for Hc is elementary:

Hc = arcsin [sin Dec sin φ + cos Dec cos φ cos(GHA + λ)] (A1)

from which we immediately obtain

∂Hc
= sec Hc [sin Dec cos φ − cos Dec sin φ cos(GHA + λ)]
∂φ
(A2)
∂Hc
= − sec Hc [cos Dec cos φ sin(GHA + λ)]
∂λ

For rhumb-line tracks, the sailing formulas for latitude and longitude, including terms for the oblateness
of the Earth, are given in [12]. The sailing formulas contain the quantities M and N , which represent the
radii of curvature of the Earth’s surface in the meridian and the prime vertical, respectively, in kilometers.
(The prime vertical is the plane containing the normal to the Earth’s surface at the location of interest,
perpendicular to the local meridian.) The curvatures M and N at a given latitude, φ, are computed from
the Earth’s equatorial radius a, in kilometers, and the dimensionless eccentricity of the Earth’s ellipsoid, e:

a(1 − e2 )
M=
(1 − e2 sin2 φ)3/2
(A3)
a
N=
(1 − e2 sin2 φ)1/2

Here, e is related to the Earth’s flattening, f, by e2 = 2f − f2 (e is sometimes referred to in geodesy texts

as the “first eccentricity”). The WGS-84 values are a = 6378.137 km and f = 1/298.257223563. In this
appendix we will use metric units for distance and speed in order to avoid confusion between nautical miles
and arcminutes—on an oblate Earth, nautical miles and arcminutes do not, in general, correspond. One
International Nautical Mile is exactly 1.852 km, so 1 kn = 1.852 km/h. Angular quantities will be expressed
in radians.
The two curvatures M and N occur in the sailing formulas, and we will be needing the partial derivatives
of M , M −1 , and N −1 with respect to φ:

∂M 3ae2 (1 − e2 ) sin 2φ
=
∂φ 2(1 − e2 sin2 φ)5/2

∂M −1 3e2 sin 2φ (1 − e2 sin2 φ)1/2

= − (A4)
∂φ 2a(1 − e2 )

∂N −1 e2 sin 2φ
= −
∂φ 2a(1 − e2 sin2 φ)1/2

12
To evaluate the partial derivatives we need for equation (4), we will start with the simplified sailing formulas
given in [12]. These formulas are approximations, but they are accurate to within about 10 m over tracks
where the excursion in latitude does not exceed about 15 km (≈10 nmi). They are much less accurate over
larger latitude ranges. Thus, these formulas could be used to accurately project the motion of the vessel over
one-half to one hour of sailing, or longer periods if the course is east-west or very nearly so. These formulas,
which give the latitude and longitude of the vessel as a function of the time, t, are:

S (t − t0 ) cos C
φ0 = φ0 +
M0
S (t − t0 ) sin C
λ0 = λ0 +
N0 cos φavg

where φavg = 12 (φ0 + φ0 ) (A5)

a(1 − e2 )
M0 = M (φ0 ) =
(1 − e2 sin2 φ0 )3/2
a
N0 = N (φ0 ) =
(1 − e2 sin2 φ0 )1/2
Here, φ0 and λ0 are the latitude and longitude of the reference (initial or known) position and t0 is the time
at which the vessel crosses that position. The quantities M0 and N0 are the values of the curvatures M and
N at the at the reference latitude φ0 . The vessel’s speed, S, is expressed in km/h = kn × 1.852, and the
time difference, t − t0 , is expressed in hours. The primes on φ0 and λ0 are simply meant to indicate that these
are approximations that are accurate only over certain (short or east-west) tracks. More generally, these
formulas can be used to provide a first approximation to the latitude and longitude at time t. The above
sailing formulas are modifications of the usual “middle latitude” sailing formulas given on pp. 364–365 of
Bowditch [11].
We can identify the expressions for φ0 and λ0 given above with the functions f and g referred to in
equations (2), (3), and (4). That puts us in a position to use equations (A5) to derive the partial derivatives
needed for equation (4):
∂f ∂φ0 ∂M0−1
= = 1 + S (t − t0 ) cos C
∂φ0 ∂φ0 ∂φ0
∂f ∂φ0
= =0
∂λ0 ∂λ0
∂f ∂φ0 S (t − t0 ) sin C
= =−
∂C ∂C M0
∂f ∂φ0 (t − t0 ) cos C
= =
∂S ∂S M0

∂λ0 ∂N0−1 ∂φ0

  
∂g S (t − t0 ) sin C tan φavg
= = + 1+ (A6)
∂φ0 ∂φ0 cos φavg ∂φ0 2N0 ∂φ0

∂g ∂λ0
= =1
∂λ0 ∂λ0
∂λ0 1 ∂φ0
  
∂g S (t − t0 )
= = cos C + sin C tan φavg
∂C ∂C N0 cos φavg 2 ∂C

∂λ0 1 ∂φ0
  
∂g (t − t0 ) sin C
= = 1 + S tan φavg
∂S ∂S N0 cos φavg 2 ∂S

13
 ∂M −1 −1 ∂N −1 −1
In the above expressions, ∂φ00 is simply ∂M∂φ evaluated for φ0 ; similarly, ∂φ00 is ∂N
∂φ evaluated for φ0
(use equations (A4)). Equations (A6), along with equations (A2), can now be used to construct equation (4).
That is, we now have algebraic formulas for all ten of the partial derivatives needed for equation (4), which
was our goal.
Unfortunately, the sailing formulas (A5), and the partial derivatives (A6) derived from them, work
properly only for short tracks, or tracks that are nearly east-west. They are not normally adequate for the
case we are really interested in: combining observations made over many hours of sailing. For that situation,
we need to use more general—and considerably more complicated—sailing formulas. These are given in [12]:

M0
1 − 34 e2 (φ0 − φ0 ) + 38 e2 (sin 2φ0 − sin 2φ0 )


φ = φ0 + 2
a(1 − e )
    
π φ e 1 − e sin φ
λ = λ0 + tan C ln tan + + ln (A7)
4 2 2 1 + e sin φ
    !
π φ0 e 1 − e sin φ0
− ln tan + − ln
4 2 2 1 + e sin φ0

Note that to use the equation for longitude, λ, we must first evaluate the equation for latitude, φ, which,
in turn, requires pre-computation of the approximate latitude, φ0 . Thus, to obtain the required partial
derivatives from equations (A7), it is necessary to make liberal use of the chain rule. Since longitude is given
as a function of latitude, one of the partial derivatives we will obviously need is

e2 cos φ
 
∂λ 1
= tan C − (A8)
∂φ cos φ 1 − e2 sin2 φ

This expression has been considerably simplified from its original form.
The partial derivatives that we need for equation (4) then are:

∂f ∂φ 1 ∂M0 
1 − 43 e2 (φ0 − φ0 ) + 38 e2 (sin 2φ0 − sin 2φ0 )


= =1+ 2
∂φ0 ∂φ0 a(1 − e ) ∂φ0

∂φ0
    0 
M0 3 2 3 2 ∂φ 0
+ 1 − 4e − 1 + 4e cos 2φ − cos 2φ0
a(1 − e2 ) ∂φ0 ∂φ0
∂f ∂φ
= =0
∂λ0 ∂λ0
∂f ∂φ M0  ∂φ0
1 − 34 e2 + 34 e2 cos 2φ0

= = 2
∂C ∂C a(1 − e ) ∂C

∂f ∂φ M0  3 2 3 2
 0
0 ∂φ
= = 1 − 4 e + 4 e cos 2φ
∂S ∂S a(1 − e2 ) ∂S

e2 cos φ0
 
∂g ∂λ 1 ∂λ ∂φ
= = − tan C − 2 2 + (A9)
∂φ0 ∂φ0 cos φ0 1 − e sin φ0 ∂φ ∂φ0
∂g ∂λ
= =1
∂λ0 ∂λ0

14
     
∂g ∂λ 1 π φ e 1 − e sin φ
= = ln tan + + ln
∂C ∂C cos2 C 4 2 2 1 + e sin φ
    !
π φ0 e 1 − e sin φ0 ∂λ ∂φ
− ln tan + − ln +
4 2 2 1 + e sin φ0 ∂φ ∂C

∂g ∂λ ∂λ ∂φ
= =
∂S ∂S ∂φ ∂S

In the first of these expressions, ∂M ∂M

∂φ0 is simply ∂φ evaluated for φ0 —see the first of equations (A4). Formulas
0

have already been given for all of the other partial derivatives that are shown unexpanded on the right sides
of the expressions above. (This kind of presentation is meant to facilitate the use of these expressions in
computer programs. Certain partials and other terms occur repeatedly, and they can be evaluated once,
stored, and used wherever needed.) Note that the expression within the first set of brackets in the formula
∂f
for ∂φ 0
is the same as that shown in brackets in the first of equations (A7), and the expression within
parentheses in the formula for ∂∂gC is the same as that shown in parentheses in the second of equations (A7).
These formulas can be evaluated numerically for each observation, using the best estimates available
for the values of all of the quantities involved. Combining these partials with those from equations (A2), we
can evaluate the expressions in parentheses in equation (4). That is, we can now construct the conditional
equation for each observation, which then enters into the least-squares solution for ∆φ0 , ∆λ0 , ∆C, and ∆S.
The tan C factor in the second of equations (A7) prevents these sailing formulas from being used for
∂λ
east-west courses, or those close to east-west. Equation (A8) shows that the same problem is present in ∂φ ,
which then goes on to serve as a factor in three of the partial derivatives in equations (A9). Therefore, for
courses that are within about a half-degree (0.01 radian) of azimuth 90◦ or 270◦ , equations (A5) and (A6)
should be used. For all other cases, equations (A7), (A8), and (A9) should be used. Note that even in the
∂φ0
latter cases, the approximate latitude φ0 from equation (A5) is needed, as well as the partial derivatives ∂φ 0
,
∂φ0 0

∂C
and ∂φ
, ∂S
from equations (A6).
If all of the angles are expressed in radians, then all of the partial derivatives are dimensionless except
0 0
those with respect to speed, that is, ∂φ∂S
, ∂λ
∂S ∂S
, ∂φ , and ∂∂λ
S . These have dimension h/km (that is, inverse
speed). That means that the corrections ∆φ0 , ∆λ0 , and ∆C that come out of the least-squares solution will
be expressed in radians, and ∆S will be expressed in km/h.
The validity of the formulas for the partials was checked by numerically evaluating them for various
points along a large number of rhumb-line tracks and comparing the results at each point with numerical
differentiations of the rhumb-line formulas. Despite the crudeness of the numerical differentiation, the two
sets of values agreed well; the worse disagreements were in the fifth significant digit. Furthermore, the
values of the partials computed from equations (A6) closely matched those from equations (A9) in their
region of overlap—that is, for courses with azimuths about a half-degree different from 90◦ or 270◦ . There
is a certain amount of “overkill” in the above expressions; the region of parameter space within which
the problem is linear enough to justify such accuracy is relatively small. We could have approached the
entire problem assuming a spherical Earth and obtained much simpler expressions (without terms in e),
which would have been quite adequate for celestial navigation as it is now usually practiced. However, with
the computing power currently available, there is virtually no penalty in approaching the problem with a
better geometric model, and obtaining results that could be used with more accurate automated observing
techniques. Furthermore, the increased accuracy does help the problem to converge more rapidly to the
correct answer.

15

View publication stats