🥇Top ML Papers of the Week

The top ML Papers of the Week (Feb 20 - Feb 26)

Feb 26

This issue highlights the top ML Papers of the Week (Feb 20 - Feb 26).

1). LLaMA - a 65B parameter foundation model released by Meta AI; relies on publicly available data and outperforms GPT-3 on most benchmarks despite being 10x smaller. (paper)

Guillaume Lample @GuillaumeLample

Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters. LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B. The weights for all models are open and available at research.facebook.com/publications/l… 1/n

2) Composer - a 5B parameter creative and controllable diffusion model trained on billions (text, image) pairs. (paper)

AK @_akhaliq

Composer is a large (5 billion parameters) controllable diffusion model trained on billions of (text, image) pairs github: github.com/damo-vilab/com… paper: arxiv.org/abs/2302.09778 project page: damo-vilab.github.io/composer-page/

3) Hindsight Instruction Relabeling - an alternative algorithm to train LLMs from feedback; the feedback is converted to instruction by relabeling the original one and training the model, in a supervised way, for better alignment. (paper)

Tianjun Zhang @NeurIPS 2022 @tianjun_zhang

Can we use purely supervised learning for RLHF using large language models? We introduce HIR (Hindsight Instruction Relabeling), which achieves impressive results using FLAN-T5 on hard BigBench tasks!

4). Active-Prompt - a prompting technique to adapt LLMs to different task-specific example prompts (annotated with human-designed chain-of-thought reasoning); this process involves finding where the LLM is most uncertain and annotating those. (paper)

John Nay @johnjnay

Active Prompting for LLMs -Most Chain-of-Thought examples are pulled from a fixed set -Instead, to adapt to diff tasks 1) Find where LLM is most uncertain 2) Annotate those -State-of- the-art on complex reasoning tasks Paper arxiv.org/abs/2302.12246 Code github.com/shizhediao/act…

5). Modular Deep Learning - a survey offering a unified view of the building blocks of modular neural networks; it also includes a discussion about modularity in the context of scaling LMs, causal inference, and other key topics in ML. (paper)

Sebastian Ruder @seb_ruder

In our new survey “Modular Deep Learning”, we provide a unified taxonomy of the building blocks of modular neural nets and connect disparate threads of research. 📄 arxiv.org/abs/2302.11529 📢 ruder.io/modular-deep-l… 🌐 modulardeeplearning.com w/ @PfeiffJo @licwu @PontiEdoardo

6). Recitation-Augmented LMs - an approach that recites passages from the LLM’s own memory to produce final answers; shows high performance on knowledge-intensive tasks. (paper)