向沐神学习笔记：GPT，GPT-2，GPT-3 论文精读【论文精读】GPT部分

系列文章目录

例如：

一、GPT

1. 同样模型大小，比如一个亿模型大小的时候，bert的性能表现优于gpt，也就是未来的工作更愿意用bert这篇文章，因为我咬咬牙还能跑起来，但是gpt的实验实在跑不起来。

1、Abstract

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve