B-402339.3, Sig Sauer, Inc.

, July 23, 2010

DOCUMENT FOR PUBLIC RELEASE

The decision issued on the date below was subject to a GAO Protective Order. This redacted
version has been approved for public release.

DOCUMENT FOR PUBLIC RELEASE

The decision issued on the date below was subject to a GAO Protective Order. This redacted
version has been approved for public release.

Decision

Matter of: Sig Sauer, Inc.

File: B-402339.3

Date: July 23, 2010

Lee P. Curtis, Esq., Troy E. Hughes, Esq., and Maggie L.C. Greenlee, Esq., Perkins Coie LLP,
for the protester.
Dorn C. McGrath III, Esq., and William M. Jack, Esq., Greenberg Traurig, LLP, for Smith &
Wesson Corp., and John F. Renzulli, Esq., and Michael R. Patrick, Esq., Renzulli Law Firm,
LLP, for Glock, Inc., the intervenors.
Ralph G. Bittelari, Esq., and Nathan E. Mires, Esq., Department of Justice, Bureau of Alcohol,
Tobacco, Firearms and Explosives, for the agency.
Christina Sklarew, Esq., and Guy R. Pietrovito, Esq., Office of the General Counsel, GAO,
participated in the preparation of the decision.

DIGEST

In a negotiated procurement for a handgun system that provided for a three-phased evaluation,
the rejection of the protester's proposal after phase II was unobjectionable where the agency
reasonably assessed the protester's handgun to be unsatisfactory following live-fire testing and
did not treat offerors unequally in conducting the testing.

DECISION

Sig Sauer, Inc., of Exeter, New Hampshire, protests the rejection of its proposal under request for
proposals (RFP) No. DJA09S000028, issued by the Bureau of Alcohol, Tobacco, Firearms and
Explosives (ATF), for semi-automatic handguns.

We deny the protest.

BACKGROUND

ATF, a principal law enforcement agency within the Department of Justice, seeks to replace its
current metal-frame handguns with a new .40 caliber, polymer-frame handgun system. This
system consists of two handguns, which are issued to each ATF agent: a standard-size duty
weapon for enforcement operations and a compact model for backup and auxiliary needs; the
two models are in all other respects required to be identical in function and design. RFP,
Statement of Work (SOW), at 22. The SOW stated detailed design and performance
specifications, describing the exact features and performance characteristics required.

Offerors were instructed to submit written proposals and sample handguns, in both standard and
compact sizes, to be evaluated in three phases. RFP at 58-61. Under phase I, the handguns were
to be evaluated on a pass/fail basis for compliance with the RFP's specifications; all submissions
assessed as satisfying all specifications moved to the phase II evaluation. Under phase II, ATF
planned to perform a live-fire assessment of the guns, consisting of the following four parts: an
accuracy test, a sight evaluation test, an evaluation by a cross-section of ATF agents, and an
assessment of the interchangeability of parts. Id. at 62. Offerors were informed that offers
"deemed most suitable for performance" would continue to phase III. Id. The RFP provided for
the award of a single or multiple indefinite-delivery, indefinite-quantity (IDIQ) contract(s) to the
offeror(s) whose proposal(s) were found to represent the best value after the phase III evaluation.
Id. at 66-67.

ATF received proposals from Sig Sauer, Smith & Wesson Corp., and Glock, Inc., each of whose
proposals were evaluated as passing phase I. In phase II, 20 ATF agents performed live-fire
assessments of the offerors' sample handguns, and completed rating forms. See, e.g., Agency
Report (AR), B-402393, Tabs 17, Evaluation of Sig Sauer Full-size Handgun, and Tab 18,
Evaluation of Sig Sauer Compact Handgun; Declaration of Live-Fire Lead Proctor, B-402339,
Jan. 2, 2010, at 1.

At issue here is the live-fire testing by a cross-section of ATF agents, which was identified in the
RFP as the most important among the four parts of the phase II testing, described above. RFP at
63. Under this test, 20 ATF agents with diverse skill levels, physical size, and right- or left-
handedness were to evaluate each of the handguns that were submitted. Each agent evaluated the
sample handguns first by firing 100 rounds of familiarity shooting, then twice firing 50 rounds
for the pistol qualification course (PQC) test, and finally field stripping, cleaning and
reassembling each type of handgun. For each handgun, the agent completed a questionnaire
assigning adjectival ratings (excellent, very good, good, fair, or unsatisfactory) and providing
narrative under the following categories: safety, reliability, accuracy, ergonomics and
adaptability, suitability of size or weight, ease of cleaning and assembly, and overall assessment
as a pistol for general issue to ATF agents.

The offerors' handgun systems received the following number of adjectival ratings in the areas of
reliability and general assessment from the ATF agents who fired them:

RELIABILITY
Excellent Very Good Good Fair Unsatisfactory
 Sig Sauer 10 12 10 6 2
Smith & 21 12 2 2 3
Wesson
Glock 19 13 3 4 1

AR, B-402339.3, Tab 4, Sig Sauer Evaluation, Tab 5, Smith & Wesson Evaluation, and Tab 6,
Glock Evaluation.

GENERAL ASSESSMENT
Excellent Very Good Fair Unsatisfactor No
Good y Answer
Sig Sauer 6 17 8 5 1 3
Smith & 13 14 6 3 2 2
Wesson
Glock 12 11 9 6 1 1

Id.

In addition to the agents' rating sheets, a record was kept of any stoppages or malfunctions that
occurred during the live fire testing. These records show that ATF's agents recorded 58
stoppages with Sig Sauer's full-size and compact pistols, 13 of which were considered to be gun-
induced and 45 shooter-induced. Id., Tab 2, Competitive Range Determination, at 3. In contrast,
the agents recorded a total of 16 shooter-induced stoppages for Smith & Wesson's guns and 7
shooter-induced stoppages for Glock's guns. There were no gun-induced stoppages recorded for
the Smith & Wesson or Glock guns. Id., Tab 8, Contracting Officer's Smith & Wesson
Evaluation Summary, and Tab 9, Contracting Officer's Glock Evaluation Summary.

The shooters' phase II evaluations of the offerors' handgun systems were provided to the agency's
source selection board (SSB). See Declaration of SSB Chair, Feb. 2, 2010, at 2. According to the
SSB chair, "[i]t was determined that any weapon that displayed numerous gun-induced or
shooter-induced malfunctions was not suitable for agents to depend on to protect their lives." Id.
at 3. Based on the results of the shooting tests, the SSB recommended to the contracting officer
that Sig Sauer's handgun be excluded from further consideration as unacceptable with respect to
reliability. The contracting officer reviewed the SSB's consensus reports and agreed with the
recommendation that Sig Sauer's proposal should be excluded from phase III, based upon the
number of stoppages recorded for Sig Sauer's handguns. AR, B-402339.3, Tab 2, Competitive
Range Determination, at 1.

Following notification of the rejection of its proposal and a debriefing, Sig Sauer protested to our
Office, challenging the agency's phase II live-fire assessment and complaining that Smith &
Wesson's and Glock's handguns should have been rejected during phase I for failing to comply
with one aspect of the specifications. ATF decided to take corrective action in response to this
earlier protest. Specifically, ATF stated that it would reconsider its phase II evaluations,
including reviewing the live-fire assessment results, consensus summary reports, and individual
ratings, and make a new decision about which offers should progress from phase II to the final
phase of the evaluation. In response, we dismissed Sig Sauer's protest earlier protest as academic.

During the course of its corrective action, ATF did not reconvene the SSB or re-evaluate
proposals. Instead, the contracting officer reviewed the live-fire assessment record; the phase II
incident reports; and the declarations made during the original protest, concerning how stoppages
were classified as either shooter-induced or gun-induced. At the end of this review, the
contracting officer created new spreadsheets to organize the information generated during the
various stages of the phase II assessment "for ease of review." Contracting Officer's Statement,
B-402339.3, at 2. The contracting officer and SSA reviewed the spreadsheets and concluded that
Sig Sauer's proposal was appropriately excluded from the final phase of the procurement. AR, B-
402339.3, Tab 2, Competitive Range Determination. Specifically, the contracting officer and
SSA noted that the incident reports confirmed that the reliability of Sig Sauer's handguns was a
major concern, given that Sig Sauer's full and compact models experienced a total of 13 gun-
induced stoppages and 45 shooter-induced stoppages during the phase II testing. It was
determined that "the significant number of malfunctions was . . . unacceptable." Id. at 2.

Following a written debriefing, Sig Sauer again protested to our Office, arguing that the
evaluation methodology that ATF used in the phase II assessments deviated from the procedures
announced in the RFP and that the evaluation reflected unequal treatment of offerors.

DISCUSSION

Sig Sauer challenges the ATF's live-fire testing assessment, arguing that the SSB substituted its
own judgment for that of the agents who performed the testing. The crux of Sig Sauer's argument
is that the SSB members focused on the number of stoppages the shooters encountered, instead
of the actual ratings and comments provided by the shooters, and equated a high number of
stoppages with unreliability--even where the shooters had not made this connection.

The protester argues that this was inconsistent with the evaluation methodology "established in
the solicitation and the underlying Source Selection Plan," because that plan explicitly provided
that the SSB would only tally the ratings on the agents' questionnaires, but would base its
assessment upon the questionnaires' ratings and narrative comments. Protest at 4, 15. Sig Sauer
notes that, notwithstanding the number of stoppages, only two agents assigned unsatisfactory
ratings for reliability for Sig Sauer's handguns, whereas three agents rated Smith & Wesson's
handguns unsatisfactory in this category. Similarly, only one agent rated Sig Sauer's handguns as
unsatisfactory overall under the general assessment, while Smith & Wesson's handguns received
two such ratings. Sig Sauer contends that, contrary to the evaluators' adjectival ratings, the SSB
improperly rated Sig Sauer's handguns as unsatisfactory overall, and, in contrast, rated Smith &
Wesson's handguns as excellent overall.

In reviewing protests challenging the evaluation of proposals, we do not conduct a new

evaluation or substitute our judgment for that of the agency but examine the record to determine
whether the agency's judgment was reasonable and in accord with the RFP evaluation criteria.
Abt Assocs., Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4. Our Office affords
particular deference to the technical expertise of agency personnel regarding judgments that
involve matters of human life and safety. PEMCO World Air Servs., B-284240.3 et al., Mar. 27,
2000, 2000 CPD para. 71 at 7. A protester's mere disagreement with the agency's judgment
concerning the adequacy of its proposal is not sufficient to establish that the agency acted
unreasonably. Realty Executives, B-237537, Feb. 16, 1990, 90-1 CPD para. 288 at 3.

Here, as we explain below, we find that the agency's assessment of Sig Sauer's proposal was
reasonable and consistent with the RFP's evaluation criteria. In this regard, we do not agree with
Sig Sauer that the SSB, contracting officer, and SSA were required to follow the shooters'
judgments with respect to the reliability and acceptability of the offerors' handguns. Source
selection officials and higher-level agency evaluators may reasonably disagree with the
evaluation ratings and results of lower-level evaluators. See, e.g., Verify, Inc., B-244401.2, Jan.
24, 1992, 92-1 CPD para. 107 at 6-8. In this regard, an SSA has broad discretion in determining
the manner and extent to which technical and cost evaluation results are used, is permitted to
make an independent evaluation of offerors' proposals, and may disagree with or expand upon
the findings of lower-level evaluators provided the basis for the evaluation is reasonable and
documented in the contemporaneous record. KPMG Consulting LLP, B-290716, B-290716.2,
Sept. 23, 2002, 2002 CPD para. 196 at 13-14; Brisk Waterproofing Co., Inc., B-276247, May 27,
1997, 97-1 CPD para. 195 at 2 n.1. The issue is not whether the agency's final assessments are
consistent with earlier assessments, but whether they reasonably reflect the relative merits of the
proposals, consistent with the solicitation.

The RFP provided that "[r]atings on the proposed PQC [the shooters' tests], any stoppages or
parts breakages during testing, and the written evaluations will be recorded and used to evaluate
each model." RFP at 62-63. The record here supports ATF's assertion that it considered these
things when it affirmed the SSB's earlier determination to exclude Sig Sauer. While each shooter
completed a questionnaire on the basis of his or her own individual experience, the SSB
conducted a broader review of the cumulative effect of all the stoppages recorded. The shooters'
questionnaires, as a whole, show that 11 of the 20 shooters experienced stoppages with Sig
Sauer's handguns--in some cases, as many as 10 stoppages. In total, the shooters experienced 58
stoppages with Sig Sauer's handguns, 13 of which were considered to be gun-induced. Given the
relatively high number of stoppages by Sig Sauer's guns as compared to Smith & Wesson's and
Glock's guns, we think that the SSB reasonably concluded that the stoppages provided a more
significant measure of the guns' reliability than the adjectival ratings for reliability assigned by
individual shooters. In addition, notwithstanding the low number of "unsatisfactory" ratings for
reliability recorded on the shooters' scoresheets, the questionnaires included a number of
negative comments that the SSB reasonably found questioned the reliability of Sig Sauer's
handguns.

Sig Sauer also contends that ATF placed too great an emphasis upon reliability in determining
which offers should continue to phase III. In this regard, Sig Sauer argues that reliability was
only one of a number of elements to be considered in the live-fire assessment, and notes that
reliability was not identified as having any more importance than the other elements. We
disagree.

As noted above, the RFP provided that only those offers which were "deemed most suitable for
performance to the Government" would be included in the phase III evaluation. See RFP at 63.
An agency is accorded considerable discretion in making such a subjective judgment as to what
best meets its needs, see Handheld Sys., Inc., B-288036, Aug. 10, 2001, 2001 CPD para. 142 at
3, particularly where, as here, the judgment involves matters of human life and safety. PEMCO
World Air Servs., supra. In making this judgment, the solicitation allowed the agency to assess a
number of considerations, including reliability. We find that the agency could reasonably rely
upon its concerns about the reliability of Sig Sauer's handguns to exclude Sig Sauer's proposal
from the final phase of the procurement, even though there were other elements of Sig Sauer's
proposed handgun system that were found to be attractive.

Sig Sauer also argues that the ratings assigned by the shooters resulted from disparate treatment
that occurred during the live-fire assessments. Specifically, the protester contends that, in
incidents involving two separate shooters, when Smith & Wesson's handguns encountered a
stoppage during the live fire evaluation, the agency intervened to identify the cause of the
stoppage and to correct it. In contrast, the protester complains the agency did not take similar
steps when stoppages occurred during Sig Sauer's live-fire assessment.

The ATF denies that it treated offerors unequally in its live-fire testing. With respect to one of
the two shooters, the agency's lead proctor states that he and another special agent had installed
optional extended base pads that Smith & Wesson had supplied for use with their handguns. In
documenting the stoppage of this shooter's handgun, the proctor discovered he had improperly
installed the optional extended base pad, which caused the stoppage. Because the stoppage was
"proctor-induced," and not gun-induced or shooter-induced, the proctor allowed correction of
this problem. Declaration of Live-Fire Lead Proctor, B-402339, Jan. 2, 2010, 4-5.

With respect to the second shooter, for whom the record contains multiple incident reports, the
lead proctor states that he did not intervene; rather, the shooter herself requested a change in grip
size after experiencing "a high volume of stoppages." Id. at 5. Sig Sauer challenges this account
of events, noting that after an initial entry by a proctor on two of the incident reports for this
shooter, the lead proctor's handwriting appears on the report, commenting on the stoppage and its
possible cause. Sig Sauer deduces from these report entries that the lead proctor intervened and
attempted to influence this shooter to make a change, and that she then did so and experienced no
further stoppages.

In our view, the record does not support Sig Sauer's conjecture in this regard. The lead proctor
explained in a sworn statement that he only responded to the shooter's request for a grip change
and the shooter's written comments. We find that the contemporaneous record shows that the
shooter herself recognized the cause of the slide problem, and that the proctor's actions involved
only a change in grip. We do not see support for the protester's theory that the proctor noticed the
problem and then intervened.

In sum, the record shows that ATF reasonably evaluated the firms' proposals in accordance with
the solicitation's evaluation criteria and concluded that Sig Sauer's offer was not one of the "most
suitable for performance to the Government" to continue to phase III.

The protest is denied.

Lynn H. Gibson
Acting General Counsel

Sig Sauer provides the handgun currently used by ATF's agents.

Although not at issue in this protest, Phase III consisted of four test segments which were
designed to further test the reliability of the handguns.

The agency's incident report form defines a "stoppage" as an incident that a shooter can solve in
less than 30 seconds without using tools, and a "malfunction" as an incident that takes longer
than 30 seconds to resolve or requires the use of tools, but not replacement parts. AR, B-402339,
Tab 13a, Phase II and Phase III Pistol Incident Forms. Although the protest filings refer
occasionally to malfunctions, the contemporaneous evaluation record shows only stoppages. We
therefore refer in our decision to any incidents as stoppages.

The agency describes gun-induced stoppages to be stoppages where the shooter's use of the
weapon did not play a role in the stoppage. Legal Memorandum, B-402339, at 6.

ATF describes its judgment as to which offers would continue to phase III to be a competitive
range determination.

The source selection plan provided that the SSB would tally the shooters' surveys and assess the
shooters' ratings and narrative comments. AR, B-402339, Tab 8, Source Selection Plan, at 32.

The purpose of the spreadsheets was to organize the phase II evaluation data to permit the
contracting officer and source selection authority (SSA) to determine whether the shooters'
evaluations and comments and the SSB's individual evaluations and consensus summaries were
consistent. Contracting Officer's Statement, B-402339.3, at 2.

Sig Sauer dropped its earlier challenge to the acceptability of Smith & Wesson's and Glock's
handguns.

Sig Sauer's arguments in this area arise from its reading of the source selection plan, rather than
the solicitation's evaluation criteria. An agency's source selection plan, however, is an internal
guide that does not give rights to parties; it is the RFP's evaluation scheme, not internal agency
documents such as source selection plans, to which an agency is required to adhere in evaluating
proposals and making the award selection. Sayres & Assocs. Corp., B-295946, B-295946.2, Apr.
25, 2005, 2005 CPD para. 90 at 6 n.9. In any event, we do not find that the SSB's review was
inconsistent with the spirit of the approach set out in the source selection plan. Although the
agency did not limit itself to a mechanical tallying of scores, as Sig Sauer suggests, it drew
conclusions based on its overview of the shooters' assessments. For example, the SSB concluded
that the shooter-induced stoppages appeared to be exacerbated by the ergonomic design of Sig
Sauer's gun.

It is true, as Sig Sauer notes, that the RFP did not define what constituted a stoppage; establish
any basis for categorizing stoppages as either shooter-induced or gun-induced; establish any
standards for an acceptable level of stoppages; or define reliability in terms of stoppages. Protest
at 17. This does not demonstrate, however, that these observations about the testing results led to
an inconsistency between the evaluation and the RFP's evaluation criteria.

The RFP provided that the live-fire assessment would consider accuracy, reliability, comfort,
ergonomics, controllability, maintenance, and suitability and provided for an overall assessment
of the handgun as a "potential new service handgun for ATF." RFP at 63.

These comments indicated that the shooter "[s]tarted with medium size palm swell and had
problems with slide not locking back. Switched to large and fit (with my grip) was perfect. No
further problems and fit was excellent." Protest, B-402339.3, Tab 15, Shooter #3's Technical
Rating Sheet for Smith & Wesson full-size weapon, at 1.