Scribd
Upload
15 views1 page

Hello 2

Uploaded by

poybmverwsevzngkqi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views1 page

Hello 2

Uploaded by

poybmverwsevzngkqi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

1 Introduction

Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly
flexible and task-agnostic ways for downstream transfer. First, single-layer representations were learned using word
vectors [MCCD13, PSM14] and fed to task-specific architectures, then RNNs with multiple layers of representations
and contextual state were used to form stronger representations [DL15, MBXS17, PNZtY18] (though still applied to
task-specific architectures), and more recently pre-trained recurrent or transformer language models [VSP+ 17] have
been directly fine-tuned, entirely removing the need for task-specific architectures [RNSS18, DCLT18, HR18].
This last paradigm has led to substantial progress on many challenging NLP tasks such as reading comprehension,
question answering, textual entailment, and many others, and has continued to advance based on new architectures
and algorithms [RSR+ 19, LOG+ 19, YDY+ 19, LCG+ 19]. However, a major limitation to this approach is that while
the architecture is task-agnostic, there is still a need for task-specific datasets and task-specific fine-tuning: to achieve
strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands
of examples specific to that task. Removing this limitation would be desirable, for several reasons.
First, from a practical perspective, the need for a large dataset of labeled examples for every new task limits the
applicability of language models. There exists a very wide range of possible useful language tasks, encompassing
anything from correcting grammar, to generating examples of an abstract concept, to critiquing a short story. For many
of these tasks it is difficult to collect a large supervised training dataset, especially when the process must be repeated
for every new task.
Second, the potential to exploit spurious correlations in training data fundamentally grows with the expressiveness
of the model and the narrowness of the training distribution. This can create problems for the pre-training plus
fine-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then
fine-tuned on very narrow task distributions. For instance [HLW+ 20] observe that larger models do not necessarily
generalize better out-of-distribution. There is evidence that suggests that the generalization achieved under this paradigm
can be poor because the model is overly specific to the training distribution and does not generalize well outside it
[YdC+ 19, MPL19]. Thus, the performance of fine-tuned models on specific benchmarks, even when it is nominally at
human-level, may exaggerate actual performance on the underlying task [GSL+ 18, NK19].
Third, humans do not require large supervised datasets to learn most language tasks – a brief directive in natural
language (e.g. “please tell me if this sentence describes something happy or something sad”) or at most a tiny number
of demonstrations (e.g. “here are two examples of people acting brave; please give a third example of bravery”) is often

Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad
set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize
the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within
the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a
model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded
within a single sequence.

You might also like