Alpacas evolve into whales, Meta "automates" alignment, and Humpback defeats all existing LLaMa models

Editors: Xiaozhou, Chen Ping

**Source:**The Heart of the Machine

In the past year, the large language model (LLM) represented by ChatGPT and GPT-4 has developed rapidly, followed by Meta's open source LLaMa and Llama 2 series models, which have also caused quite a stir in the AI world. But what followed was constant controversy. Some people believed that LLM had some uncontrollable risks, posing some potential threats to human survival.

In order to deal with these challenges, the research on LLM alignment has become more and more important. Some researchers have proposed instruction following (instruction following), but this method requires a lot of manual annotation. However, annotating such high-quality instruction-following datasets is costly.

In this paper, researchers from Meta AI propose a scalable method called instruction backtranslation, which builds a high-quality instruction-following language model by automatically annotating corresponding instructions.

Paper address:

Specifically, the study starts with a language model as a seed model, which is fine-tuned on a small amount of seed data as well as web corpora. The role of the seed model is to build training samples, and then some high-quality samples from these samples will be screened out, and then these data are used to fine-tune a more powerful model.

After two rounds of iterative dataset fine-tuning LLaMa, the resulting model Humpback outperforms other existing non-distilled models such as LIMA, Claude, Guanaco, etc. on the Alpaca leaderboard.

Humpback originally meant a humpback whale, also known as a humpback whale. Meta named the model Humpback, so there is no deep meaning.

The reason why it is called instruction back translation, the researchers said, is that it draws on the classic back translation method in machine translation, in which the target sentence written by humans is automatically annotated with the source sentence in another language generated by the model.

Turing Award winner Yann LeCun gave a high-level overview of the study's methodology and praised Meta's work as an important contribution to alignment research:

Some netizens made a good summary of this research: data quality is really important for large models. During the research process, they used different levels of filtered data to fine-tune a model. The results showed that only the best samples In order to obtain a model that performs better than other samples.

This paper proposes a new data augmentation paradigm that needs to be completed in two steps. First, it is necessary to have a set of seed (instruction, output) pairs and a corpus to generate more good instruction data.

The figure below compares Humpback with some open source and proprietary models.

Table 4 below shows that our method performs best among non-distilled models on both 65B and 33B model scales.

Let's look at the specific method below.

Method Introduction

The study proposes a self-training approach that generally assumes access to a basic language model, a small amount of seed data, and an unlabeled sample set (such as a web corpus). Unlabeled data is often a large collection of documents of various shapes, written by humans, including content on various topics of human interest, but most importantly, it has not been paired with instructions.

There are two key assumptions here. The first assumption is that there are some subsets of this very large text set (unlabeled sample set) that are suitable as generated samples for some user instructions. The second hypothesis is that the instructions of these candidate answers can be predicted, which can be used to form high-quality sample pairs to train instruction-following models.

As shown in Figure 1 below, the study proposes that the instruction back-translation process includes two core steps:

• Self-augmentation: Generate instructions for unlabeled data (i.e. web corpus) to generate training data pairs (instruction-output) for instruction tuning.
• Self-management: Independently select high-quality sample data as training data to fine-tune the basic model to follow instructions. This method is done iteratively.

Among them, the self-management steps adopted are shown in Table 1 below:

Experiment and Results

The data set in this paper mainly includes seed data and enhanced data. The specific information is shown in Table 2 and Figure 2:

Figure 3 shows that augmented data without self-curation used to train the model does not improve instruction following performance despite increased data size.

The figure below compares the data efficiency of different instruction tuning datasets.

Joint expansion of data and models: The study finds that the data expansion trends observed in the 7B model also apply to larger models. For example, adding high-quality augmentation data to the 65B seed model will bring further improvements.

Commonsense reasoning: The study was tested on five commonsense reasoning benchmarks, SIQA, PIQA, Arc-Easy, Arc-Challenge and Openbook QA (OBQA), and the results are summarized in Table 5. The results show that compared with the base model, the performance of our model has been improved in several aspects such as social reasoning.

MMLU: Table 6 summarizes the results of different models in MMLU (massive multitask language understanding). Our fine-tuned model improves zero-shot accuracy compared to the base model, but performs poorly on the 5-sample context example.