Self-Rewarding Language Models(arxiv.org)

    Harnessing the power of human-annotated data through 
    Supervised Fine-Tuning (SFT) is pivotal for advancing 
    Large Language Models (LLMs). In this paper, we 
    delve into the prospect of growing a strong LLM out of a 
    weak one without the need for acquiring additional 
    human annotated data. We propose a new fine-tuning 
    method called Self-Play fIne-tuNing (SPIN),
    which starts from a supervised fine-tuned model. At the 
    heart of SPIN lies a self-play mechanism,
    where the LLM refines its capability by playing against 
    instances of itself. More specifically, the
    LLM generates its own training data from its previous 
    iterations, refining its policy by discerning
    these self-generated responses from those obtained from 
    human-annotated data. Our method
    progressively elevates the LLM from a nascent model to a 
    formidable one, unlocking the full
    potential of human-annotated demonstration data for SFT. 
    Theoretically, we prove that the global optimum to the 
    training objective function of our method is achieved 
    only when the LLM policy aligns with the target data 
    distribution. Empirically, we evaluate our method on
    several benchmark datasets including the HuggingFace 
    Open LLM Leaderboard, MT-Bench, and datasets from Big- 
    Bench. 

    Our results show that SPIN can significantly improve the 
    LLM’s performance across a variety of benchmarks and 
    even outperform models trained through direct
    preference optimization (DPO) supplemented with extra 
    GPT-4 preference data. This sheds light on the promise 
    of self-play, enabling the achievement of human-level 
    performance in LLMs without the need for expert 
    opponents.