Automatic Model Selection with Large Language Models for Reasoning

Xu Zhao1 , Yuxi Xie1 , Kenji Kawaguchi1 , Junxian He2 , Qizhe Xie1

1
National University of Singapore 2 Shanghai Jiao Tong University
{xu.zhao}@u.nus.edu

Abstract problem through a Python function. CoT, which re-

lies on natural language, is more general and offers
Chain-of-Thought and Program-Aided Lan- clearer explanations, reducing ambiguity. Mean-
guage Models represent two distinct reasoning
arXiv:2305.14333v1 [cs.CL] 23 May 2023

while, PAL, employing a programming language,

methods, each with its own strengths and weak-
enjoys guaranteed computation correctness. Intu-
nesses. We demonstrate that it is possible to
combine the best of both worlds by using dif- itively, choosing between CoT and PAL based on
ferent models for different problems, employ- their reasoning results to a specific problem would
ing a large language model (LLM) to perform prove beneficial. However, without access to the
model selection. Through a theoretical analy- ground-truth, choosing a better method itself be-
sis, we discover that the performance improve- comes a machine learning problem.
ment is determined by the differences between In order to select among multiple results, previ-
the combined methods and the success rate
ous studies have suggested training a ranker (Ue-
of choosing the correct model. On eight rea-
soning datasets, our proposed approach shows sato et al., 2022). While training a dedicated
significant improvements. Furthermore, we model generally results in improved accuracy, it
achieve new state-of-the-art results on GSM8K can also be somewhat cumbersome and entail sig-
and SVAMP with accuracies of 96.5% and nificant costs. Conversely, large language models
93.7%, respectively. Our code is publicly avail- (LLMs) have demonstrated good calibration and
able at https://github.com/XuZhao0/Model- have been used to assess the accuracy of their own
Selection-Reasoning.
outputs (Guo et al., 2017; Shinn et al., 2023; Xie
et al., 2023). Therefore, in this study, we propose
1 Introduction utilizing LLMs to perform model selection with
Large language models (LLMs) have made impres- few-shot learning. Additionally, we ask LLMs to
sive progresses in numerous fields (Devlin et al., provide explanations for their choices, ensuring
2019; Brown et al., 2020; OpenAI, 2023; Chowd- they consider deliberate comparisons in the pro-
hery et al., 2022; Bubeck et al., 2023; Wei et al., cess.
2022a) and are often powerful enough to solve Through a theoretical analysis, we discover that
problems through a single unified method. While the effectiveness of the proposed method is mainly
convenient, this approach tends to ignore the dis- influenced by two factors: (1) the significance of
tinct structures and variations among the problems, the difference between two models, and (2) the
which would benefit from using different methods. probability of selecting the correct model. More
On the other hand, in human society, individuals specifically, we can attain a higher overall perfor-
with different skill sets excel in various roles, lead- mance when there is a substantial difference be-
ing to a thriving world as a whole. tween the models being considered and when the
In the case of reasoning, Chain-of-Thought correct model is more likely to be chosen. However,
(CoT) (Wei et al., 2022b) and Program-Aided we also observe that even without an exceptional
Language Models (PAL) (Gao et al., 2022) have model selection, we can still achieve improvement
emerged as two effective methods that offer dif- in certain cases. This validates our decision to
ferent strengths and weaknesses. Essentially, CoT simply employ an LLM for the purpose of model
decomposes a reasoning problem into a series of selection.
intermediate steps using natural language. In con- We evaluate the proposed method on eight
trast, PAL represents the solution to a reasoning reasoning tasks, with CoT and PAL serving as
Figure 1: We propose to perform model selection to combine two distinct methods, CoT and PAL . The
figure illustrates an example where PAL makes mistakes about crucial information and therefore fails to answer
the question correctly. In contrast, CoT manages to correctly answer the same question. Our selection model
successfully chooses the correct solution and provides a brief explanation to support its choice.

the baseline methods. On these eight reason- program.

ing benchmarks, our method demonstrates sub- Considering the unique strengths and weak-
stantial performance improvements when employ- nesses of these two approaches, it would be advan-
ing Codex, ChatGPT, and GPT-4 as the back- tageous to select between the two methods based
bone LLMs. Moreover, our approach attains new on the problem at hand. Furthermore, model selec-
state-of-the-art accuracies of 96.5% and 93.7% on tion could be more precise if we incorporate the
GSM8K (Cobbe et al., 2021) and SVAMP (Patel reasoning chains from CoT and PAL, as the model
et al., 2021), respectively. could potentially identify errors, gaps, or ambigu-
ities within the chains and determine the correct
2 Automatic Model Selection with Large one.
Language Models
We employ an LLM to make the model selec-
2.1 Method tion. Specifically, we present the LLM with a few
In this study, we examine reasoning tasks using examples of choosing between PAL or CoT based
two baseline models: CoT and PAL. To tackle com- on their reasoning chains. In all the included exam-
plex reasoning tasks, CoT employs an LLM to ples, either CoT or PAL is correct, i.e., we do not
generate several intermediate reasoning steps be- include instances where both are correct or both are
fore arriving at a final conclusion. This approach incorrect. In our prompt, we present the choices as
has proven to be highly effective for various tasks. two options: (A) or (B), with the results of (A) and
Due to its use of natural language, the reasoning (B) being generated by PAL and CoT, respectively.
steps are clearly explained, resulting in less am- By providing a few model selection examples in the
biguity. Furthermore, natural language supports a context, an LLM learns to output either (A) or (B)
broad range of reasoning that may involve common to make a decision. However, there are instances
sense and confidence of the reasoning steps. In con- where the model fails to generate (A) or (B); in
trast, PAL breaks down a reasoning problem into these cases, we randomly choose a method.
computations using Python. Naturally, employing Owing to the in-context learning capabilities of
Python ensures a high level of computational ac- LLMs, we find that they exhibit reasonable accu-
curacy. However, similar to how Python code can racy in selecting the appropriate method. Addition-
be challenging to comprehend, this approach also ally, as shown in our theoretical analysis, a highly
introduces ambiguity regarding the meaning of the accurate method selection is not actually necessary
for the algorithm to perform well. We discover On the accuracy of selection Define ρ to be the
that the performance improvement is jointly deter- overall probability of selecting a better method:
mined by the significance of the difference between ρ = Ex [ρx ]. Theorem 1 shows that ρ can be much
the two combined methods and the effectiveness worse than that of a random guess to achieve the
of the model selection. Furthermore, there are sit- improvement over the base methods m1 and m2 ;
uations where the model selection might be poor, i.e., err < err1 and err1 ≤ err2 can happen with
yet the overall improvement remains substantial. In ρ < 0.5:
addition to requesting that the model selection iden- Theorem 1. For any ϵ > 0, there exist data dis-
tifies the correct method, we also ask it to provide tribution over x, two base methods (m1 , m2 ), and
explanations, which proves to be effective in our combining probability (ρx ) such that err < err1 ,
experiments. The proposed algorithm is illustrated err1 ≤ err2 , and ρ < ϵ.
in Figure 1.
We provide a stronger version of Theorem 1 in
2.2 Theoretical Analysis Appendix A (that implies Theorem 1) and its proof
In this section, we conduct a theoretical analysis in Appendix B.
to determine under which condition the proposed Theorem 1 supports our proposition that, despite
method could work (and fail). not training a new model for the selection process
and with the in-context learning limited to a few-
Quantifying error rates Let us denote the error shot prompt, it is possible to achieve improvement,
rates of the two base methods, m1 and m2 , by err1 even if we do not achieve ρx > 0.5 in some in-
and err2 , respectively. Without loss of general- stances. This theoretical analysis offers deeper
ity, let m1 be a better base method in the overall insights into the conditions and strategies for effec-
performance: i.e., err1 ≤ err2 . For a given ques- tive performance improvement with the proposed
tion x, we define ρx as the probability of choos- methodology.
ing a more accurate method, either m1 or m2 , for
the given x using the proposed approach. Define 3 Experiments
R(x) = p(correct | x, m2 ) − p(correct | x, m1 )
where p(correct | x, mi ) represents the probabil- 3.1 Setup
ity of outputting a correct prediction given input x Datasets and backbones We conduct experiments
with method mi . Then we can quantify the final on eight datasets that span a range of arithmetic and
error rate err of the proposed method as follows: symbolic reasoning tasks. 7 of these datasets, in-
Proposition 1. For any methods m1 and m2 with cluding GSM8K (Cobbe et al., 2021), SVAMP (Pa-
any combining probability function ρx , tel et al., 2021), ASDIV (Miao et al., 2020), Sin-
gleOP, SingleEQ, AddSub and MultiArith (Koncel-
err = err1 − Ex [|R(x)| (ρx − 1{R(x) < 0})] Kedziorski et al., 2016), are about arithmetic
We refer readers to Appendix B for the full proof. reasoning, while Date Understanding (Srivastava
Proposition 1 decomposes the possible improve- et al., 2022) focuses on symbolic reasoning. To
ment (or deterioration) over base methods in terms comprehensively evaluate the effectiveness of our
of R(x) and ρx . It quantitatively shows when and approach, we employ three LLMs as backbone
how we can expect improvement (or deterioration) systems: Codex (code-davinci-002), ChatGPT
based on these two factors. For example, to im- (gpt-3.5-turbo) and GPT-4 (gpt-4). In our ex-
prove over the best base method m1 , Proposition 1 periments, we always use the same LLM for the
suggests us to choose another base method m2 such two base models and the model selection LLM. 1 .
that |R(x)| is not too small and ρx is high when Prompt design To effectively exploit the in-
|R(x)| is large. In other words, it discourages us context learning abilities of the LLMs, we create
from choosing too similar methods as base meth- a set of few-shot examples by manually creating
ods, because for similar methods, |R(x)| tends to an error in one model’s reasoning chain. For arith-
be small and it is challenging to increase ρx even metic reasoning tasks, we employ an 8-shot exam-
when |R(x)| is large due to the similarity. This ple for Codex and 5-shot example for ChatGPT and
provides a theoretical motivation for us to use CoT GPT-4. For date understanding task, we use 6-shot
and PAL, instead of combining CoT with another 1
Codex results are obtained in February and March 2023,
CoT. ChatGPT in April and May, and GPT-4 in May 2023.
 Arithmetic Symbolic
Backbones Methods
GSM8K SVAMP ASDIV SingleOP SingleEQ AddSub MultiArith Date
CoT 64.4 77.6 80.2 92.7 93.5 88.4 95.7 64.5
Codex PAL 71.5 79.6 79.1 95.4 96.5 91.9 99.7 77.5
Ours 74.7 82.2 81.6 96.3 96.9 91.6 99.7 79.4
CoT 80.8 83 89.3 94.8 97.4 90.4 98.7 69.1
ChatGPT PAL 79.2 80.3 83 90.7 97.6 89.4 96.3 68.3
Ours 82.6 84.3 89.4 94.8 97.8 90.6 98.7 70.2
CoT 94.6 91.9 92.7 97.2 97.2 93.9 98 90
GPT-4 PAL 94.0 92.2 90.2 95.2 98.8 94.9 98.5 88.1
Ours 95.6 93.7 93.5 97.3 98.6 95.7 99 90.5

Table 1: Results comparison (Accuracy %) on 7 arithmetic datasets and 1 symbolic dataset with greedy decoding.
We evaluate our methods on three backbone LLMs.

Backbones Methods SC@5 SC@10 3.2 Main Results

CoT 85.4 86.7 The results of our experiments with greedy de-
ChatGPT PAL 80.9 82.6
Ours 88.2 (+2.8) 88.9 (+2.2) coding are shown in Table 1. First, we find that
our proposed method effectively and robustly en-
CoT 95.6 95.5
GPT-4 PAL 94.7 95.4 hances performance on most settings with different
Ours 96.5 (+0.9) 96.5 (+1.0) datasets of different difficulties and different back-
bone LLMs, simply by combining two base models.
Table 2: Results comparison (Accuracy %) on GSM8K For example, with GPT-4, we achieve an accuracy
with the integration of the Self-Consistency (SC). SC@5
of 95.6% on GSM8K and 93.7% on SVAMP with-
and SC@10 represents 5 and 10 sampled paths respec-
tively. The previous state-of-the-art on GSM8K is 95.5, out self-consistency.
achieved by Zheng et al. (2023). Second, our results show a considerable improve-
ment even when one of the base models performs
much worse than the other. For instance, we ob-
serve a significant 3.2% improvement over PAL’s
71.5% accuracy on GSM8K, even though CoT has
for all backbones. The used prompts can be found a lower accuracy of 64.4%.
in Appendix C. Third, our model’s general applicability is fur-
ther underscored by its 1.9% improvement on the
Hyperparameters In the solution generation symbolic date understanding task when we use
and selection process, we set the temperature at Codex, illustrating its applicability not only to
0 for greedy decoding. Some experiments also mathematical reasoning but also to other reasoning
use self-consistency (Wang et al., 2022b) to further tasks. In fact, on this task, the accuracy difference
improve performance. In this context, we set the between two base models is as large as 13%, our
temperature for CoT at 0.5 and PAL at 0.8 during proposed method still improves the accuracy from
answer generation. For model selection, the tem- 77.5% to 79.4%.
perature is set to 0 so that we always select the It is worth noting that the improvement on Codex
method that has a higher probability of being cor- and GPT-4 is larger than the improvement on Chat-
rect. However, it is worth noting that ChatGPT and GPT. As we will show in the analysis, this is due to
GPT-4 do not always follow the few-shot instruc- ChatGPT has limited in-context learning capabil-
tions. As such, when these models output "Both" or ities and cannot choose the correct method based
"Neither" to a task, we randomly select one of the on the presented reasoning chains.
two choices as the final result for simplicity. In our
initial experiments, we discovered that our method Experiments with self-consistency We aim
is highly robust with respect to hyperparameters. to investigate the relationship between self-
For instance, simply having the model selection consistency and model selection and whether they
LLM output the selection result leads to significant complement each other. To do this, we run our algo-
improvements, and there is no need to fine-tune the rithm 5 or 10 times and use self-consistency to per-
threshold for the selection probability. form majority voting, arriving at the final answer.
 Arithmetic Symbolic
Backbones Metrics
GSM8K SVAMP ASDIV SingleOP SingleEQ AddSub MultiArith Date
∆UpperBound 10 8.1 6.5 1.6 0.9 2.0 0.3 4.6
Codex Success Rate 74.8 72.5 63.7 87.9 88 70 92.9 87.8
Improvement +3.2 +2.6 +1.4 +0.9 +0.4 -0.3 0 +1.9
∆UpperBound 8.6 6.0 3.4 2.4 1.4 2.5 0.6 9.8
ChatGPT Success Rate 60.4 66.4 69.8 58 57.1 60.9 75 53.6
Improvement +1.8 +1.3 +0.1 0 +0.4 +0.2 0 +1.1
∆UpperBound 2.5 3.6 2.2 2.4 0.6 1.3 0.5 1.1
GPT-4 Success Rate 72.6 68.7 64.2 57 69.2 85.7 100 86.7
Improvement +1.0 +1.8 +0.8 +0.1 -0.2 +0.8 +0.5 +0.5

Table 3: We define the following terms: ∆UpperBound = AccUpper Bound − Accm1 , where AccUpper Bound is the
upper bound accuracy where we assume a perfect model selection and m1 is the stronger one of the two base models.
∆UpperBound reflects the expected performance difference between the two base models. Success Rate calculates
the correct selection rates when either CoT is correct or PAL is correct, i.e., we ignore the cases where both methods
are either correct or wrong. Improvement is the performance improvement achieved over the performance of the
stronger base model.

As demonstrated in Table 2, we can achieve sub- This echoes with the theoretical analysis shown
stantial improvements over self-consistency alone. in Section 2.2. Essentially, to achieve better per-
With GPT-4, even though both base models al- formance, we need a different behavior of model
ready score around 95%, combining them leads to results, i.e., a high |R(x)| and a high success rate ρ,
a new state-of-the-art performance on GSM8K at which jointly contribute more substantial improve-
96.5%. The performance of using 5 samples and ments. Indeed, the study on ∆Upper Bound and suc-
10 samples turn out to be the same with GPT-4. Ad- cess rate explains the significant performance im-
ditionally, using ChatGPT, we attain an accuracy provement on these datasets.
of 88.9% when sampling 10 times.
4 Analysis
3.3 Influencing Factors In this section, we provide a few analysis to see
We have shown that our model performs well across when and how the method works.
different datasets. To better understand the rea- 4.1 Combination between Similar Methods
sons for the performance improvement on differ-
ent datasets and backbone LLMs, we present the Backbone ChatGPT GPT-4
performance improvement, and the associated in- m1 CoT CoT
fluencing factors in Table 3. We can find that m2 PAL CoT′ PAL CCoT
there is a high expected performance difference Accm1 80.8 80.8 94.6 94.6
Accm2 79.2 79.2 94.0 95.1
between the two base methods, reflected in the Ours 82.6 80.8 95.6 95.2
high ∆Upper Bound , which means how different the Improvement (+1.8) (+0) (+1.0) (+0.1)
two base models behave across questions. A larger ∆Upper Bound 8.6 7.5 2.5 1.7
Success Rate 60.4 52.2 72.6 58.8
∆Upper Bound indicates a larger room for potential
improvement. Specifically, notice that on GSM8K, Table 4: Other model combinations results on GSM8K.
with ChatGPT, ∆Upper Bound is 8.6% although the CoT′ denotes the base CoT model when the temperature
accuracy of CoT and PAL are very similar (80.8% is set to 0.1. CCoT denotes ComplexCoT (Fu et al.,
and 79.2% respectively). Similarly, with GPT-4, 2022).
∆Upper Bound is 2.5% while both the accuracy of
the two base models are close (94.6% vs 94.0%). We choose CoT and PAL as our two base mod-
In addition, we find that the success rate of model els due to the motivation of combining different
selection is relatively high with Codex and GPT-4. strengths of distinct models. We conduct exper-
For example, with GPT-4, the success rate is 72.6% iments to examine whether the performance im-
on GSM8K. In contrast, ChatGPT suffers on the proves when we combine two similar base mod-
success rate, which explains the relatively weaker els. We use two variants of CoT in this experi-
performance improvement on ChatGPT. ment: CoT′ where we set the temperature at 0.1,
ComplexCoT (Fu et al., 2022) where we use more Success
complex examples in the prompt. Both of these Backbone Explanation Acc
Rate
methods’ accuracies are similar or higher than the
accuracy of PAL. The results are shown in Table 4. w/o exp 74.7 74.9
Codex
We have the following observations: w/ exp 74.6 74.2
w/o exp 81.8 55.9
• In our experiments, we found that model se- ChatGPT
w/ exp 82.6 60.4
lection between CoT and CoT′ , or CoT and
ComplexCoT, does not lead to substantial per- w/o exp 95.5 69.9
GPT-4
formance gains, even though the accuracy of w/ exp 95.6 72.6
CoT′ and ComplexCoT is on par with PAL.
Table 5: Accuracy and success rate with and without
On the other hand, model selection between
explanation on GSM8K.
CoT and PAL results in consistent perfor-
mance improvements. To understand the rea-
sons behind these outcomes, we further in- higher success rate compared to selecting between
vestigate the ∆Upper Bound and the success two similar base models like CoT-CoT′ . This holds
selection rate. true even when different prompts or temperature
settings are used.
• ∆Upper Bound of CoT-PAL exceeds that
of other combinations, CoT-CoT′ and 4.2 Ablation Study on Explanation
ComplexCoT-CoT, despite their employing To perform model selection, we provide expla-
two stronger or equivalent two base models. nations in the prompt and also ask the model to
This observation suggests a larger absolute generate explanations after making a choice, as
value of the accuracy difference per question we expect to improve the model’s selection ability
for CoT-PAL. It indicates that CoT and PAL by pointing out why the other choice is incorrect.
perform more dissimilarly than other model To investigate the potential role of explanations
combinations. Theoretically, it represents a in enhancing the model’s selection capability, we
larger |R(x)|. As Proposition 1 highlights, conduct experiments on GSM8K by excluding ex-
without a substantial |R(x)|, it is unlikely to planations from the answer.
achieve significant performance gain since The results shown in Table 5 reveal that for back-
the improvement component is factored by bone LLMs with more powerful in-context learning
|R(x)|. abilities, such as Codex and GPT-4, the inclusion
• The success selection rate of CoT-PAL sur- of explanations within the answer seems to have
passes that of other model combinations. It negligible influence on the model’s performance.
means that the selection model is more likely In other words, for Codex and GPT-4, they have a
to select the correct choice when one solution relatively high success selection rate even when no
derives from CoT and the other from PAL. In explanation is given. However, for ChatGPT, which
theory, this higher success rate implies that possesses relatively low in-context learning capa-
when |R(x)| is high for a given question x, bility, the integration of explanations yields a larger
the success selection probability ρx for CoT- improvement in the model’s success selection rate.
PAL is higher than others. This observation suggests that for backbones with
limited in-context learning abilities, explanations
These findings support our initial motivation and can serve as a valuable tool to improve the success
hypothesis. We choose CoT and PAL as our two selection rate.
base models because they represent distinct rea-
soning approaches using natural language and pro- 4.3 Qualitative Studies
gramming language. We expect these models to We are also interested in gaining an intuitive un-
exhibit a significant difference in errors and accura- derstanding of the algorithm, so we conduct some
cies, indicated by a high ∆Upper Bound . Moreover, qualitative analysis. As displayed in Table 6, in
the considerable disparity in errors for a particular the first example, both ChatGPT and GPT-4 can
question makes it easier for large language models accurately identify that PAL is the correct solu-
(LLMs) to select the correct option, leading to a tion. However, ChatGPT provides an incorrect
 Question: Courtney attended a concert and reported that the audience was 48 in number. However, Kelly went to the
same concert and said that Courtney had made the mistake of overstating the number of people in attendance by 20%.
If Kelly was right, how many people really attended the concert?

(A) (B)
Answer:
Courtney reported 48 people in attendance. def solution () :
If this number is overstated by 20%, then we need to find courtney_report = 48
the actual number of people. overstatement_percentage = 0.20
First, we need to find 20% of 48. actual_attendance =
20% of 48 is (20/100) * 48 = 9.6. courtney_report /(1 +
Now, we subtract this number from Courtney’s reported overstatement_percentage )
number to find the actual number of people. result = int ( actual_attendance )
48 - 9.6 = 38.4. return result
Since we cannot have a fraction of a person, we round the
number to the nearest whole number.
So, the actual number of people in attendance was approxi-
mately 38.
So the answer is 38.

Which of the above two choices can correctly answer the math problem?

ChatGPT: (B) can correctly answer the math problem. Because (A) rounds up the result to the nearest whole number,
which is not necessary.

GPT-4: (B) can correctly answer the math problem. Because (A) incorrectly calculated the actual number of people
by subtracting the overstatement percentage from the reported number instead of dividing it.

Question: How much does it cost you for lunch today at Subway if you pay $40 for a foot-long fish sub and thrice as
much for a six-inch cold-cut combo sub?

(A) (B)

Answer: def solution () :

If the foot-long fish sub costs $40, then the six-inch cost_footlong_fish = 40
cold-cut combo sub costs 3 * $40 = $120. cost_sixinch_coldcut =
So the total cost for both subs is $40 + $120 = $160. cost_footlong_fish / 3
Therefore, it costs $160 for lunch today at Subway. total_cost = cost_footlong_fish +
cost_sixinch_coldcut
result = total_cost
return result

Which of the above two choices can correctly answer the math problem?

ChatGPT: (A) can correctly answer the math problem. Because (B) calculates the cost of the six-inch cold-cut combo
sub as one-third of the cost of the foot-long fish sub instead of three times the cost of the foot-long fish sub.

GPT-4: (A) can correctly answer the math problem. Because (B) incorrectly calculates the cost of the six-inch cold-cut
combo sub as one-third of the foot-long fish sub instead of thrice as much.

Table 6: In the first example, ChatGPT performs model selection correctly, but gives a wrong explanation, while
GPT-4 gives a correct explanation. The second example shows where both ChatGPT and GPT-4 select correctly and
give the correct explanation.

explanation, while GPT-4 offers a valid one. In the 5 Related Work

second example, which is relatively simpler, both
Ensemble learning. In machine learning, the strat-
ChatGPT and GPT-4 successfully perform accurate
egy of combining various models to address a sin-
model selection and provide valid explanations. In
gle problem is exemplified in techniques such as
the first example, we can see that GPT-4 actually
bagging (Breiman, 1996), boosting (Freund and
possesses exceptional reasoning capabilities and
Schapire, 1997; Chen and Guestrin, 2016; Ke et al.,
provide reliable explanations.
2017), and random forest (Ho, 1995; Breiman,
2001). The underlying idea in these methods is that
a group of weak learners can collectively manifest
as a strong learner. This concept has also found for verification. Shinn et al. (2023) proposes an
its place in deep learning through the use of en- approach to provide an agent with dynamic mem-
sembles. For reasoning, Self-consistency samples ory and self-reflection capabilities. Madaan et al.
diverse reasoning paths and chooses the most con- (2023) proposes a method to generate outputs from
sistent answer through majority voting, which can LLMs and refine its previously generated output
be regarded as ensembling (Wang et al., 2022b). given its own feedback. Different from these works
Wang et al. (2022a) takes it a step further by in- where the underlying method is the same, in this
troducing rationale-augmented ensembles, empha- work, we are interested in combining systems with
sizing rationale sampling in the output space as different strengths and weaknesses.
a critical component for enhancing performance
robustly. However, typically, ensembling places 6 Conclusion
equal weights on models through majority voting, This study introduces a method that effectively
which may restrict the full potential of the diverse combines two distinct base models, CoT and PAL,
strengths that each model offers. by enabling an LLM to make the correct selec-
Reasoning. The research community has made tion. We provide a theoretical analysis that sup-
tremendous progress in the field of reasoning. ports the feasibility of such a model combination,
Apart from CoT (Wei et al., 2022b) and PAL (Gao which is further validated through empirical results.
et al., 2022; Chen et al., 2022), Zhou et al. (2022) Our method achieves performance improvements
simplifies complex problems by breaking them across eight datasets with various backbone LLMs.
down into a series of subproblems solved in se- This research represents a significant step towards
quence. It has also been shown that prompts with tapping into the potential of diversity and collabo-
higher complexity lead to superior performance on ration among models in LLMs.
multi-step reasoning tasks (Fu et al., 2022). Kojima In our future work, we aim to expand this frame-
et al. (2022) shows that by simply adding "Let’s work to other domains. Another intriguing concept
think step by step" before each answer, LLMs involves exploring the use of diverse system instruc-
can be competent zero-shot reasoners. Selection- tions to elicit varying model behaviors for model
Inference (Creswell et al., 2022) alternates between combinations. For example, we could prompt a
selection and inference stages, generating causal model to provide reasoning steps in as much detail
reasoning steps to the final answer. Kazemi et al. as possible or as succinctly as possible.
(2022) proposes a backward chaining algorithm
that breaks reasoning down into four sub-models. 7 Limitation
REFINER (Paul et al., 2023) generates intermedi- • We focus on reasoning tasks in this work, but
ate reasoning steps under the scrutiny of a critic we posit that exploring model selection in
model. Xie et al. (2023) delves into the explo- other domains also presents a worthwhile di-
ration of the reasoning search space and gener- rection for future research.
ates reasoning chains by a self-evaluation guided
stochastic beam search. Yao et al. (2023) intro- • As our approach solely relies on in-context
duces Tree of Thoughts (ToT), enabling exploration learning capabilities of LLMs, it is sensitive
over "thoughts". It is important to mention that the to the prompts, which is a common issue with
contributions of these methods are distinct from our in-context learning.
approach, and the progress made by them could po-
tentially be seamlessly integrated using the method • Our approach combines different models
we propose. through a selection mechanism, which may
not wholly leverage the unique strengths of
Self-Evaluation. LLM calibration studies reveal each model. Future work can explore alternate
that the probabilistic predictions made by current methodologies for model collaboration.
LLMs closely align with the actual frequencies of
token occurrences, hence producing well-calibrated
predictions for certain tasks (Guo et al., 2017; Ka-
davath et al., 2022; Jiang et al., 2020). As LLMs ex- References
hibit reliable calibration, there is a growing number Leo Breiman. 1996. Bagging predictors. Machine
of research emphasizing the use of self-evaluation learning, 24:123–140.
Leo Breiman. 2001. Random forests. Machine learning, Antonia Creswell, Murray Shanahan, and Irina Higgins.
45:5–32. 2022. Selection-inference: Exploiting large language
models for interpretable logical reasoning. ArXiv,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie abs/2205.09712.
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Kristina Toutanova. 2019. Bert: Pre-training of deep
Gretchen Krueger, T. J. Henighan, Rewon Child, bidirectional transformers for language understand-
Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens ing. ArXiv, abs/1810.04805.
Winter, Christopher Hesse, Mark Chen, Eric Sigler,
Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Yoav Freund and Robert E Schapire. 1997. A decision-
Clark, Christopher Berner, Sam McCandlish, Alec theoretic generalization of on-line learning and an
Radford, Ilya Sutskever, and Dario Amodei. 2020. application to boosting. Journal of computer and
Language models are few-shot learners. ArXiv, system sciences, 55(1):119–139.
abs/2005.14165.
Yao Fu, Hao-Chun Peng, Ashish Sabharwal, Peter Clark,
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- and Tushar Khot. 2022. Complexity-based prompt-
dan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter ing for multi-step reasoning. ArXiv, abs/2210.00720.
Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg,
Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon,
and Yi Zhang. 2023. Sparks of artificial general Pengfei Liu, Yiming Yang, Jamie Callan, and Gra-
intelligence: Early experiments with gpt-4. ArXiv, ham Neubig. 2022. Pal: Program-aided language
abs/2303.12712. models. ArXiv, abs/2211.10435.

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein-
scalable tree boosting system. In Proceedings of berger. 2017. On calibration of modern neural net-
the 22nd acm sigkdd international conference on works. In International Conference on Machine
knowledge discovery and data mining, pages 785– Learning.
794.
Tin Kam Ho. 1995. Random decision forests. In Pro-
Wenhu Chen, Xueguang Ma, Xinyi Wang, and ceedings of 3rd international conference on docu-
William W. Cohen. 2022. Program of thoughts ment analysis and recognition, volume 1, pages 278–
prompting: Disentangling computation from rea- 282. IEEE.
soning for numerical reasoning tasks. ArXiv,
abs/2211.12588. Zhengbao Jiang, J. Araki, Haibo Ding, and Graham
Neubig. 2020. How can we know when language
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, models know? on the calibration of language models
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul for question answering. Transactions of the Associa-
Barham, Hyung Won Chung, Charles Sutton, Sebas- tion for Computational Linguistics, 9:962–977.
tian Gehrmann, Parker Schuh, Kensen Shi, Sasha
Tsvyashchenko, Joshua Maynez, Abhishek Rao, Saurav Kadavath, Tom Conerly, Amanda Askell, T. J.
Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- Henighan, Dawn Drain, Ethan Perez, Nicholas
kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-
Hutchinson, Reiner Pope, James Bradbury, Jacob Johnson, Scott Johnston, Sheer El-Showk, Andy
Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Jones, Nelson Elhage, Tristan Hume, Anna Chen,
Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep
Sunipa Dev, Henryk Michalewski, Xavier García, Ganguli, Danny Hernandez, Josh Jacobson, John
Vedant Misra, Kevin Robinson, Liam Fedus, Denny Kernion, Shauna Kravec, Liane Lovitt, Kamal
Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Ndousse, Catherine Olsson, Sam Ringer, Dario
Barret Zoph, Alexander Spiridonov, Ryan Sepassi, Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph,
David Dohan, Shivani Agrawal, Mark Omernick, An- Benjamin Mann, Sam McCandlish, Christopher Olah,
drew M. Dai, Thanumalayan Sankaranarayana Pillai, and Jared Kaplan. 2022. Language models (mostly)
Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- know what they know. ArXiv, abs/2207.05221.
won Child, Oleksandr Polozov, Katherine Lee, Zong-
wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia,
Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Xinyuan Xu, and Deepak Ramachandran. 2022. Lam-
Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, bada: Backward chaining for automated reasoning in
and Noah Fiedel. 2022. Palm: Scaling language mod- natural language. ArXiv, abs/2212.13894.
eling with pathways. ArXiv, abs/2204.02311.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang,
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.
Jacob Hilton, Reiichiro Nakano, Christopher Hesse, 2017. Lightgbm: A highly efficient gradient boost-
and John Schulman. 2021. Training verifiers to solve ing decision tree. Advances in neural information
math word problems. ArXiv, abs/2110.14168. processing systems, 30.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- Singh, Charles Rathkopf, Chenlin Meng, Chitta
taka Matsuo, and Yusuke Iwasawa. 2022. Large Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites,
language models are zero-shot reasoners. ArXiv, Christian Voigt, Christopher D. Manning, Christo-
abs/2205.11916. pher Potts, Cindy Tatiana Ramirez, Clara Rivera,
Clemencia Siro, Colin Raffel, Courtney Ashcraft,
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Cristina Garbacea, Damien Sileo, Daniel H Garrette,
Kushman, and Hannaneh Hajishirzi. 2016. Mawps: Dan Hendrycks, Dan Kilman, Dan Roth, Daniel
A math word problem repository. In North Ameri- Freeman, Daniel Khashabi, Daniel Levy, Daniel
can Chapter of the Association for Computational Gonz’alez, Danny Hernandez, Danqi Chen, Daphne
Linguistics. Ippolito, Dar Gilboa, David Dohan, D. Drakard,
David Jurgens, Debajyoti Datta, Deep Ganguli, De-
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler nis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen,
Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dil-
Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, yar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-
Sean Welleck, Bodhisattwa Prasad Majumder, Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk,
Shashank Gupta, Amir Yazdanbakhsh, and Peter Elad Segal, Eleanor Hagerman, Elizabeth Barnes,
Clark. 2023. Self-refine: Iterative refinement with Elizabeth P. Donoway, Ellie Pavlick, Emanuele
self-feedback. ArXiv, abs/2303.17651. Rodolà, Emma FC Lam, Eric Chu, Eric Tang,
Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. Dyer, Ethan J. Jerzak, Ethan Kim, Eunice Engefu
2020. A diverse corpus for evaluating and devel- Manyasi, Evgenii Zheltonozhskii, Fan Xia, Fate-
oping english math word problem solvers. ArXiv, meh Siar, Fernando Mart’inez-Plumed, Francesca
abs/2106.15772. Happ’e, François Chollet, Frieda Rong, Gaurav
OpenAI. 2023. Gpt-4 technical report. ArXiv, Mishra, Genta Indra Winata, Gerard de Melo, Ger-
abs/2303.08774. mán Kruszewski, Giambattista Parascandolo, Gior-
gio Mariani, Gloria Wang, Gonzalo Jaimovitch-
Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021. L’opez, Gregor Betz, Guy Gur-Ari, Hana Galija-
Are nlp models really able to solve simple math word sevic, Han Sol Kim, Hannah Rashkin, Hanna Ha-
problems? In North American Chapter of the Associ- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin,
ation for Computational Linguistics. Hinrich Schütze, Hiromu Yakura, Hongming Zhang,
Hubert Wong, Ian Aik-Soon Ng, Isaac Noble, Jaap
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- Jumelet, Jack Geissinger, John Kernion, Jacob Hilton,
riz Borges, Antoine Bosselut, Robert West, and Boi Jaehoon Lee, Jaime Fernández Fisac, J. Brooker
Faltings. 2023. Refiner: Reasoning feedback on in- Simon, James Koppel, James Zheng, James Zou,
termediate representations. ArXiv, abs/2304.01904. Jan Koco’n, Jana Thompson, Jared Kaplan, Jarema
Radom, Jascha Narain Sohl-Dickstein, Jason Phang,
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Jason Wei, Jason Yosinski, Jekaterina Novikova,
Reflexion: an autonomous agent with dynamic mem- Jelle Bosscher, Jenni Marsh, Jeremy Kim, Jeroen
ory and self-reflection. ArXiv, abs/2303.11366. Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Ji-
acheng Xu, Jiaming Song, Jillian Tang, Jane W
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Waweru, John Burden, John Miller, John U. Balis,
Abu Awal Md Shoeb, Abubakar Abid, Adam Jonathan Berant, Jorg Frohberg, Jos Rozen, José
Fisch, Adam R. Brown, Adam Santoro, Aditya Hernández-Orallo, Joseph Boudeman, Joseph Jones,
Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua,
Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik
Alex Ray, Alex Warstadt, Alexander W. Kocurek, Gopalakrishnan, Katerina Ignatyeva, Katja Markert,
Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng’
rish, Allen Nie, Aman Hussain, Amanda Askell, Omondi, Kory Wallace Mathewson, Kristen Chia-
Amanda Dsouza, Ameet Annasaheb Rahane, Anan- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc-
tharaman S. Iyer, Anders Andreassen, Andrea San- Donell, Kyle Richardson, Laria Reynolds, Leo Gao,
tilli, Andreas Stuhlmuller, Andrew M. Dai, An- Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-
drew D. La, Andrew Kyle Lampinen, Andy Zou, Ochando, Louis-Philippe Morency, Luca Moschella,
Angela Jiang, Angelica Chen, Anh Vuong, Ani- Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng
mesh Gupta, Anna Gottardi, Antonio Norelli, Anu He, Luis Oliveros Col’on, Luke Metz, Lutfi Kerem
Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, cSenel, Maarten Bosma, Maarten Sap, Maartje ter
Arul Menezes, Arun Kirubarajan, Asher Mullokan- Hoeve, Madotto Andrea, Maheen Saleem Farooqi,
dov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Manaal Faruqui, Mantas Mazeika, Marco Baturan,
Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Marco Marelli, Marco Maru, M Quintana, Marie
Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Tolkiehn, Mario Giulianelli, Martha Lewis, Martin
Batuhan Ozyurt, Behnam Hedayatnia, Behnam Potthast, Matthew Leavitt, Matthias Hagen, M’aty’as
Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- Schubert, Medina Baitemirova, Melissa Arnaud,
mekci, Bill Yuchen Lin, Blake Stephen Howald, Melvin Andrew McElrath, Michael A. Yee, Michael
Cameron Diao, Cameron Dour, Catherine Stinson, Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Star-
Cedrick Argueta, C’esar Ferri Ram’irez, Chandan
ritt, Michael Strube, Michal Swkedrowski, Michele Jonathan Uesato, Nate Kushman, Ramana Kumar, Fran-
Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike cis Song, Noah Siegel, L. Wang, Antonia Creswell,
Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mo- Geoffrey Irving, and Irina Higgins. 2022. Solving
hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh math word problems with process- and outcome-
Gheini, T MukundVarma, Nanyun Peng, Nathan based feedback. ArXiv, abs/2211.14275.
Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas
Cameron, Nicholas S. Roberts, Nicholas Doiron, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc
Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Le, Ed Huai hsin Chi, and Denny Zhou. 2022a.
Nitish Shirish Keskar, Niveditha Iyer, Noah Con- Rationale-augmented ensembles in language mod-
stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar els. ArXiv, abs/2207.00747.
Agha, Omar Elbaghdadi, Omer Levy, Owain Evans,
Pablo Antonio Moreno Casares, Parth Doshi, Pas- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le,
cale Fung, Paul Pu Liang, Paul Vicol, Pegah Ed Huai hsin Chi, and Denny Zhou. 2022b. Self-
Alipoormolabashi, Peiyuan Liao, Percy Liang, Pe- consistency improves chain of thought reasoning in
ter W. Chang, Peter Eckersley, Phu Mon Htut, Pi- language models. ArXiv, abs/2203.11171.
Bei Hwang, P. Milkowski, Piyush S. Patil, Pouya Jason Wei, Yi Tay, Rishi Bommasani, Colin Raf-
Pezeshkpour, Priti Oli, Qiaozhu Mei, QING LYU, fel, Barret Zoph, Sebastian Borgeaud, Dani Yo-
Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, gatama, Maarten Bosma, Denny Zhou, Donald Met-
Raefer Gabriel, Rahel Habacker, Ram’on Risco zler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol
Delgado, Raphaël Millière, Rhythm Garg, Richard
Vinyals, Percy Liang, Jeff Dean, and William Fedus.
Barnes, Rif A. Saurous, Riku Arakawa, Robbe
2022a. Emergent abilities of large language models.
Raymaekers, Robert Frank, Rohan Sikand, Roman
Trans. Mach. Learn. Res., 2022.
Novak, Roman Sitelew, Ronan Le Bras, Rosanne
Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhut- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
dinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and
Teehan, Rylan Yang, Sahib J. Singh, Saif M. Mo- Denny Zhou. 2022b. Chain of thought prompting
hammad, Sajant Anand, Sam Dillavou, Sam Shleifer, elicits reasoning in large language models. ArXiv,
Sam Wiseman, Samuel Gruetter, Sam Bowman, abs/2201.11903.
Samuel S. Schoenholz, Sanghyun Han, Sanjeev
Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-
Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Yen Kan, Junxian He, and Qizhe Xie. 2023. De-
Gehrmann, Sebastian Schuster, Sepideh Sadeghi, composition enhances reasoning via self-evaluation
Shadi S. Hamdan, Sharon Zhou, Shashank Srivas- guided decoding. ArXiv, abs/2305.00633.
tava, Sherry Shi, Shikhar Singh, Shima Asaadi,
Shixiang Shane Gu, Shubh Pachchigar, Shubham Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran,
Toshniwal, Shyam Upadhyay, Shyamolima Deb- Thomas L. Griffiths, Yuan Cao, and Karthik
nath, Siamak Shakeri, Simon Thormeyer, Simone Narasimhan. 2023. Tree of thoughts: Deliberate
Melzi, Siva Reddy, Sneha Priscilla Makini, Soo problem solving with large language models.
hwan Lee, Spencer Bradley Torene, Sriharsha Hat-
war, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo
Stella Rose Biderman, Stephanie C. Lin, S. Prasad, Li, and Yu Li. 2023. Progressive-hint prompting
Steven T. Piantadosi, Stuart M. Shieber, Summer improves reasoning in large language models. ArXiv,
Misherghi, Svetlana Kiritchenko, Swaroop Mishra, abs/2304.09797.
Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A.
Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes, Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei,
Theodore Rothschild, Thomas Phan, Tianle Wang, Nathan Scales, Xuezhi Wang, Dale Schuurmans,
Tiberius Nkinyili, Timo Schick, T. N. Kornev, Tim- Olivier Bousquet, Quoc Le, and Ed Huai hsin
Chi. 2022. Least-to-most prompting enables com-
othy Telleen-Lawton, Titus Tunduny, Tobias Ger-
plex reasoning in large language models. ArXiv,
stenberg, Trenton Chang, Trishala Neeraj, Tushar
abs/2205.10625.
Khot, Tyler O’Brien Shultz, Uri Shaham, Vedant
Misra, Vera Demberg, Victoria Nyamai, Vikas Rau-
nak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu,
Vishakh Padmakumar, Vivek Srikumar, William Fe-
dus, William Saunders, William Zhang, W Vossen,
Xiang Ren, Xiaoyu Tong, Xinyi Wu, Xudong Shen,
Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song,
Yasaman Bahri, Ye Ji Choi, Yichi Yang, Yiding Hao,
Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yun-
tao Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao,
Zi Fu Wang, Zijie J. Wang, Zirui Wang, Ziyi Wu,
Sahib Singh, and Uri Shaham. 2022. Beyond the imi-
tation game: Quantifying and extrapolating the capa-
bilities of language models. ArXiv, abs/2206.04615.
A A detailed version of Theorem 1
In this appendix, we provide a detailed version of Theorem 1. Whereas Theorem 1 only states the
existence of problem instances, Theorem 2 constructs such instances concretely: i.e., Theorem 2 implies
Theorem 1. Define µx [X ] to be the distribution for the expected errors: i.e., an expected error can be
written by Ex∼µx [X ],y,f [1[y ̸= f (x)]] for some function f . Define S[X ] = {x ∈ X : R(x) < 0}. Let us
denote U [X ] as the uniform distribution over X . Given any X , we write n = |X |, T = |S[X ]|, α = T /n.
Assume that 1 ≤ T < n.
Theorem 2. Let µx [X ] = U [X ] and X be given such that |X | < ∞. Let ϵ, δ ∈ (0, 1) and λ ∈ (0, 1]
ϵT β
such that β = n−T ∈ (0, 1) and λ ≥ 1 − ϵT (n −PT − δ). Let R and ρx be set suchP that R(x) = −ϵ
for x ∈ S[X ], R(x) = β for x ∈ X \ S[X ], (1/T ) x∈S[X ] ρx = λ, and (1/(n − T )) x∈X \S[X ] ρx =
ϵ(T /(n − T ))(1 − λ)β −1 + δ/(n − T ). Then, we have that err < err1 , err1 ≤ err2 , and

δ
ρ = 1 − α + λ[2α − 1] + .
n

In particular, when α ≥ 0.5, we have ρ → 0 as α → 1 and (δ/(n−T )) → 0 (with λ = 1− Tβϵ (n−T −δ));
when α < 0.5, we have ρ → 0 as α → 0 and (δ/n) → 0 (with λ = 1).
The proof of Theorem 2 is presented in Appendix B. Theorem 2 shows that the overall success
probability of the selection process can be much worse than a random guess to achieve the improvement
over the base methods m1 and m2 ; i.e., err < err1 and err1 ≤ err2 can happen with ρ < 0.5. Indeed, it
is possible to have ρ → 0 with the improvement (err < err1 and err1 ≤ err2 ) when the size of X is large:
when α ≥ 0.5, we can choose λ = 1 − Tβϵ (n − T − δ) with which err < err1 , err1 ≤ err2 , and ρ → 0 as
α → 1 and (δ/(n − T )) → 0. When α < 0.5, we can choose λ = 1 with which err < err1 , err1 ≤ err2 ,
and ρ → 0 as α → 0 and (δ/n) → 0. This supports our proposition that, despite not training a new model
for the selection process and with the in-context learning limited to a few-shot prompt, it is possible to
achieve improvement, even if we do not achieve ρx > 0.5 in some instances.
Theorem 2 also suggests that if the overall performance of two base methods is similar, captured
by ϵ, the overall selection process can be weak to achieve some improvement, as long as the success
selection probability is relatively high when the two methods have very different expected errors (or
accuracies) for a given question. In essence, Theorem 2 suggests a trade-off: we want |R(x)| to be larger
when deciding which two base methods m1 and m2 to choose, implying that we prefer base methods to
perform dissimilarly on X . On the other hand, if two base methods exhibit a substantial expected accuracy
difference, then the selection process needs to be stronger to improve the performance (i.e., ρ needs to
be larger). However, if the expected accuracy difference between two base methods is relatively small,
increasing the power of the selection process is not that necessary to boost performance.

B Proofs
B.1 Proof of Proposition 1
Proof. Define acc = 1 − err and acci = 1 − erri for i ∈ {1, 2}. Since expected error =
E[1[incorrect prediction]] = P (incorrect prediction) = 1 − P (correct prediction), we have that

acci = p(correct|mi ) = Ex [p(correct|x, mi )]

where correct represents the event of the correct prediction. Similarly,

" 2 #
X
acc = Ex p(mi |x)p(correct|x, mi )
i=1

where p(mi |x) represents the probability of selecting method mi given x via the proposed method. Thus,

acc − acc1 = Ex [p(m1 |x)p(correct|x, m1 ) + p(m2 |x)p(correct|x, m2 ) − p(correct|x, m1 )]

 = Ex [(p(m1 |x) − 1)p(correct|x, m1 ) + p(m2 |x)p(correct|x, m2 )]
= Ex [(p(m2 |x)p(correct|x, m2 ) − (1 − p(m1 |x))p(correct|x, m1 )]

Since 1 − p(m1 |x) = p(m2 |x),

acc − acc1 = Ex [(p(m2 |x)p(correct|x, m2 ) − p(m2 |x)p(correct|x, m1 )]

= Ex [(p(m2 |x)R(x)] .

Here, we notice that

(
ρx if R(x) ≥ 0
p(m2 |x) =
1 − ρx if R(x) < 0
= 1{R(x) ≥ 0}ρx + 1{R(x) < 0}(1 − ρx ).

By plugging this into the above equation,

acc − acc1 = Ex [(1{R(x) ≥ 0}ρx + 1{R(x) < 0}(1 − ρx ))R(x)]

= Ex [R(x)1{R(x) ≥ 0}ρx ] + Ex [R(x)1{R(x) < 0}(1 − ρx )]

Since R(x)1{R(x) ≥ 0} = |R(x)|1{R(x) ≥ 0} and R(x)1{R(x) < 0} = −|R(x)|1{R(x) < 0}, we

have that

acc − acc1 = Ex [|R(x)|1{R(x) ≥ 0}ρx ] − Ex [|R(x)|1{R(x) < 0}(1 − ρx )]

= Ex [|R(x)|((1{R(x) ≥ 0} + 1{R(x) < 0})ρx − 1{R(x) < 0})]

Since (1{R(x) ≥ 0} + 1{R(x) < 0}) = 1 for any x,

acc − acc1 = Ex [|R(x)|(ρx − 1{R(x) < 0})].

B.2 Proof of Theorem 2

Proof. We first confirm that R(x) and ρx define valid probabilities under the condition of this statement.
For R(x), since ϵ ∈ (0, 1) and β ∈ (0, 1), it defines valid probabilities for methods m1 and m2 . For ρx ,
since λ ∈ [0, 1], it also defines valid probabilities for the case of x ∈ S[X ]. For the case of x ∈ X \ S[X ],
since ϵ(T /(n − T ))(1 − λ)β −1 + δ/n ≥ 0, we need to show that ϵ(T /(n − T ))(1 − λ)β −1 + δ/n ≤ 1.
That is,

ϵ(T /(n − T ))(1 − λ)β −1 + δ/(n − T ) ≤ 1

⇐⇒ϵT (1 − λ)β −1 ≤ n − T − δ
β(n − T − δ)
⇐⇒1 − ≤ λ,
ϵT
β
which is satisfied by the condition on λ that λ ≥ 1 − ϵT (n − T − δ). Thus, the condition on ρx defines
the valid probabilities for both cases of x ∈ S[X ] and x ∈ X \ S[X ].
We now show that err < err1 . Invoking Proposition 1,

err = err1 − Ex [|R(x)| (ρx − 1{R(x) < 0})] .

Thus, we have err < err1 if Ex [|R(x)| (ρx − 1{R(x) < 0})] > 0. This condition can be rewritten as

Ex [|R(x)| (ρx − 1{R(x) < 0})] > 0

1 X
⇐⇒ [|R(x)| (ρx − 1{R(x) < 0})] > 0
n
x∈X
 X X X
⇐⇒ |R(x)|ρx + |R(x)|ρx > |R(x)|
x∈X \S[X ] x∈S[X ] x∈S[X ]
X
⇐⇒ |R(x)|ρx + ϵT λ > ϵT
x∈X \S[X ]
X
⇐⇒β ρx > ϵT − ϵT λ = ϵT (1 − λ)
x∈X \S[X ]
X ϵT (1 − λ)
⇐⇒ ρx >
β
x∈X \S[X ]

This is satisfied by the condition on ρ that (1/(n − T )) x∈X \S[X ] ρx = ϵ(T /(n − T ))(1 − λ)β −1 +
P
δ/(n − T ) for some δ > 0: i.e.,
X ϵT (1 − λ)
ρx = + δ.
β
x∈X \S[X ]

Therefore, we have that err < err1 .

We now show that err1 ≤ err2 . Similarly to the proof of Proposition 1, we define acci = 1 − erri for
i ∈ {1, 2}. Then, the inequality err1 ≤ err2 holds if acc1 ≥ acc2 . By using correct to represent the event
of the correct prediction, this condition can be rewritten as
acc1 ≥ acc2
X X
⇐⇒ p(correct|x, m1 )] ≥ p(correct|x, m2 )]
x∈X x∈X
X
⇐⇒0 ≥ R(x) = (n − T )β − ϵT
x∈X
ϵT
⇐⇒ ≥β
n−T
ϵT
This is satisfied by β = n−T . Thus, we have that err1 ≤ err2 .
Using these, we now compute the ρ as
ρ = Ex∼µx [X ] [ρx ]
1 X 1 X
= ρx + ρx
n n
x∈X \S[X ] x∈S[X ]
 
1 ϵT (1 − λ)
= + δ + αλ
n β
ϵT (1 − λ) δ
= + αλ +
βn n
ϵT (1 − λ)(n − T ) δ
= + αλ +
ϵT n n
δ
= (1 − λ) − α(1 − λ) + αλ +
n
δ
= 1 − α + λ[2α − 1] + .
n
Finally, we prove the asymptotic behavior using this equation. When α < 0.5, by setting λ = 1, we have
that
δ δ δ
ρ = 1 − α + λ[2α − 1] + = 1 − α + 2α − 1 + = α + → 0
n n n
as α → 0 and (δ/n) → 0. When α ≥ 0.5, by setting λ = 1 − Tβϵ (n − T − δ), we have that
 
β δ
ρ=1−α+ 1− (n − T − δ) [2α − 1] +
Tϵ n
 β δ
= 1 − α + 2α − 1 − [2α − 1] (n − T − δ) +
Tϵ n
β δ
= α − [2α − 1] (n − T − δ) +
Tϵ n
β
By defining Q = T ϵ (n − T − δ), we have

δ
ρ = α − [2α − 1]Q + .
n
Here,
β ϵT 1 1 δ
Q= (n − T − δ) = (n − T − δ) = (n − T − δ) = 1 −
Tϵ n − T Tϵ n−T n−T
Thus,
 
δ δ
ρ = α − [2α − 1] 1 − +
n−T n
2α − 1 δ
= α − 2α + 1 + δ +
 n − T n
2α − 1 1
=1−α+δ + →0
n−T n

as α = T /n → 1 and (δ/(n − T )) → 0: e.g., by setting δ = ζ(n − T ) and take ζ → 0, with which

(δ/(n − T )) = ζ → 0.
C Prompts
We show examples for model selection prompts used on different tasks with different backbones. We only
show a few examples for each case. Full prompts can be found in our code.

Math Problem: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
Question: Which of the following two choices can correctly answer the math problem?

(A) (B)
def solution () : Answer:
money_initial = 23 Olivia had 23 dollars.
bagels = 5 5 bagels for 3 dollars each will be 5 * 3 = 15 dollars.
bagel_cost = 3 So she has 23 - 5 = 18 dollars left.
money_spent = bagels * bagel_cost The answer is 18.
money_left = money_initial -
money_spent
result = money_left
return result

Answer: (A)

Table 7: An example of 8-shot model selection prompts used on 7 arithmetic datasets with Codex.

Date Understanding Problem: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
Question: Which of the following two choices can correctly answer the date understanding problem?

(A) (B)
def solution () : A:
# If 2015 is coming in 36 hours , If 2015 is coming in 36 hours, then it is coming in 2 days.
then today is 36 hours before . 2 days before 01/01/2015 is 12/30/2014, so today is
today = datetime (2015 , 1, 1) - 12/30/2014.
relativedelta ( hours =36) So one week from today will be 01/06/2015.
# One week from today , So the answer is 01/06/2015.
one_week_from_today = today +
relativedelta ( weeks =1)
# The answer formatted with %m /% d
/% Y is
result = one_week_from_today .
strftime ( '%m /% d /% Y ')
return result

Answer: (A)

Table 8: An example of 6-shot model selection prompts used on Date Understanding task with Codex.
There are two choices to the same math problem. One uses natural language to answer the question, while the other
uses Python program to answer it. Either of them can correctly answer the math problem. You need to identify which
choice can correctly answer the math problem. Here is one example how to do it,
Math problem: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

(A) (B)

Answer: def solution () :

Olivia had 23 dollars. money_initial = 23
5 bagels for 3 dollars each will be 5 * 3 = 15 dollars. bagels = 5
So she has 23 - 15 = 8 dollars left. bagel_cost = 3
So the answer is 8. money_spent = bagels + bagel_cost
money_left = money_initial -
money_spent
result = money_left
return result

Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) adds the number of bagels to the cost of each bagel instead of
multiplying them.

Now it’s your turn. Here is another math problem and two choices.
Math Problem: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How
many golf balls did he have at the end of wednesday?

(A) (B)

Answer: def solution () :

Michael started with 58 golf balls. golf_balls_initial = 58
Then after losing 23 on tuesday, he had 58 -23 = 35. golf_balls_lost_tuesday = 23
After losing 2 more, he had 35 + 2 = 37 golf balls. golf_balls_lost_wednesday = 2
So the answer is 37. golf_balls_left =
golf_balls_initial - \
golf_balls_lost_tuesday -
\ golf_balls_lost_wednesday
result = golf_balls_left
return result

Which of the above two choices can correctly answer the math problem?
(B) can correctly answer the math problem. Because (A) adds 2 more balls after losing 2 more on Wednesday instead
of subtracting them.

Table 9: Two examples of 5-shot model selection prompts used on 7 arithmetic datasets with ChatGPT.
 There are two choices to the same date understanding problem. One uses natural language to answer the question,
while the other uses Python program to answer it. Either of them can correctly answer the date understanding problem.
You need to identify which choice can correctly answer the problem. Here is one example how to do it,
Date Understanding Problem: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?

(A) (B)

Answer: def solution () :

If 2015 is coming in 36 hours, then it is coming in 2 days. # If 2015 is coming in 36 hours ,
And 2 days before 01/01/2015 is 12/30/2014, so today is then today is 36 hours before .
12/30/2014. today = datetime (2015 , 1, 1) +
So one week from today will be 01/05/2015. relativedelta ( hours =36)
So the answer is 01/05/2015. # One week from today ,
one_week_from_today = today +
relativedelta ( weeks =1)
# The answer formatted with %m /% d
/% Y is
result = one_week_from_today .
strftime ( '%m /% d /% Y ')
return result

Which of the above two choices can correctly answer the date understanding problem?
(A) can correctly answer the date understanding problem. Because (B) incorrectly calculates the date 36 hours later
instead of 36 hours before.

Now it’s your turn. Here is another date understanding problem and two choices.
Date Understanding Problem: The first day of 2019 is a Tuesday, and today is the first Monday of 2019. What is the
date today in MM/DD/YYYY?

(A) (B)

Answer: def solution () :

If the first day of 2019 was Tuesday, then 01/01/2019 was # If the first day of 2019 is a
a Tuesday. Tuesday , and today is the first
And today is the first monday, would be 5 days later. Monday of 2019 , then today is
So today is 01/06/2019. 6 days later .
So the answer is 01/06/2019. today = datetime (2019 , 1, 1) +
relativedelta ( days =6)
# The answer formatted with %m /% d
/% Y is
result = today . strftime ( '%m /% d /% Y ')
return result

Which of the above two choices can correctly answer the date understanding problem?
(B) can correctly answer the problem. Because (A) missed the fact that there are 6 days between the first day of 2019
and the first Monday of 2019.

Table 10: Two examples of 6-shot model selection prompts used on Date Understanding task with ChatGPT and
GPT-4.
There are two choices to the same math problem. One uses natural language to answer the question, while the other
uses Python code to answer it. Either of them can correctly answer the math problem. You need to identify which
choice can correctly answer the math problem. Here is one example how to do it,
Math problem: There were nine computers in the server room. Five more computers were installed each day, from
monday to thursday. How many computers are now in the server room?

(A) (B)

Answer: def solution () :

There were originally 9 computers. computers_initial = 9
For each of 4 days from monday to thursday, 5 more computers_added = 5
computers were added. computers_total =
So 5 * 4 = 20 computers were added. computers_initial +
So there are 9 + 20 = 29 computers now. computers_added
So the answer is 29. result = computers_total
return result

Which of the above two choices can correctly answer the math problem?
(A) can correctly answer the math problem. Because (B) missed the fact that computers were added each day from
monday to thursday.

Now it’s your turn. Here is another math problem and two choices.
Math Problem: A piece of square paper has a perimeter of 32 centimeters. Nicky’s dog, Rocky, tore off 1/4 of the
paper. What is the area of the remaining paper?

(A) (B)

Answer: def solution () :

A square has 4 equal sides. perimeter = 32
The perimeter of the square paper is 32 centimeters. fraction_torn = 1 / 4
So each side of the square is 32 / 4 = 8 centimeters. area_total = ( perimeter / 4) ** 2
The area of the whole square paper is side * side = 8 * 8 = area_remaining = (1 -
64 square centimeters. fraction_torn ) * area_total
Rocky tore off 1/4 of the paper. result = area_remaining
So The area of the remaining paper is 1/4 * 64 = 16 square return result
centimeters.
So the answer is 16.

Which of the above two choices can correctly answer the math problem?
(B) can correctly answer the math problem. Because (A) incorrectly calculated the area of the torn-off portion instead
of the remaining portion.

Table 11: Two examples of 5-shot model selection prompts used on 7 arithmetic datasets with GPT-4.