Efficient Natural Language and Speech Processing

(Models, Training and Inference)

This workshop aims at introducing some fundamental problems in the field of natural language and speech processing which can be of interest to the general machine learning and deep learning community to improve the efficiency of the models, their training and inference. The workshop program offers an interactive platform for gathering experts and talents from academia and industry through different invited keynote talks, panel discussions, paper submissions and reviews, poster and oral presentations and a mentorship program. This will provide an opportunity to discuss and learn from each other, exchange ideas, build connections, and brainstorm on potential solutions and future collaborations. The topics of this workshop can be of interest for people working on general machine learning, deep learning, optimization, theory and NLP & Speech applications.

Overview

Despite the great success of deep neural networks due to huge over-parameterization and using very large amount of data in different tasks of natural language processing (NLP) and speech processing, training or deploying these networks on devices or even cloud services with limited memory and computational power can be very expensive and challenging. For instance, pre-trained language models (PLMs) such as GPT-3 have led to a great breakthrough in NLP; but running GPT-3 with more than 170 billion parameters trained with more than 500 GB of data requires more than 10 Tesla V-100 GPUs. That being said, still improving the NLP and Speech models by increasing their number of parameters and incorporating more data is deemed a very common practice in the NLP and Speech domains. Therefore, it is of vital importance to invest on enhancing the efficiency of these models in terms of model architectures, training and inference from different perspectives highlighted in this workshop. In this regard, we would like to share some unique and fundamental challenges with the NeurIPS community to be considered in their future investigations.



Call for Papers

We encourage the NeurIPS community to submit their solutions, ideas, and ongoing work concerning data, model, training, and inference efficiency for NLP and speech processing. The scope of this workshop includes, but not limited to, the following topics.

Efficient Pre-Training and Fine-Tuning. Pre-training is a very expensive process. Even a small modification to the configuration of the models requires the user to redo pre-training:
  • Fast pre-training techniques, avoiding pre-training from scratch
  • Multi-domain pre-training/fine-tuning and fast domain adaptation for pre-trained/fine tuned models
  • Multimodal pre-trained (e.g., text--speech) models
  • Avoiding task-specific fine tuning of pre-trained models
  • New efficient architectures for pre-trained models

Model Compression. Neural model compression techniques such as quantization, pruning, layer decomposition and knowledge distillation (KD) aim at reducing the number of parameters of the models, improving their memory requirements or running efficiency:
  • Impact of different compression techniques on the inductive biases learned by the original models
  • Combined compression techniques for more efficient NLP and speech models
  • Efficient KD for NLP and speech, efficient intermediate layer distillation, and teacher-free distillation
  • Improving KD for large classification problems (e.g., text generation and machine translation with a very large number of output classes)
  • Solving the Capacity Gap problem and the Search Problem associated with finding the best checkpoint of the teacher
  • Theory of KD (e.g., how does KD work?)

Efficient Training. How to improve the training speed of the NLP and speech models:
  • Improving the optimizer for faster training
  • Accelerated training of different tasks in NLP and speech
  • Distributed training, federated learning and continual learning for NLP and speech tasks

Data Efficiency. Pre-trained models rely on a huge amount of unlabeled data which makes the training very sample inefficient:
  • Sample efficient training, training with less data, few-shot and zero-shot learning
  • Sample efficient data-augmentation, identifying which training samples should be augmented
  • Low-resource NLP and speech, considering training tasks with limited available data

Edge Intelligence. Running the trained models on edge devices will require a conversion process to match the network with hardware specifications:
  • TinyML for NLP and speech on embedded systems
  • Efficient conversion versus hardware-aware training
  • Training on device

Submission Instructions

You are invited to submit your papers in our CMT submission portal. All the submitted papers have to be anonymous for double-blind review. We expect each paper will be reviewed by at least three reviewers. The content of the paper (excluding the references and supplementary materials) should not be longer than 6 pages, strictly following the NeurIPS template style (which can be found here).

Authors can submit up to 100 MB of supplementary materials separately. Authors are highly encouraged to submit their codes for reproducibility purposes. Although original submissions are preferred, submitted papers can be among your already published or ArXiv papers, and your under submission works. Please make sure to indicate the complete list of conflict of interests for all the authors of your paper. To encourage higher quality submissions, our sponsors are offering the Best Paper Award to qualified outstanding original oral and poster presentations (upon nomination of the reviewers). Bear in mind that our workshop is not archival, but the accepted papers will be hosted on the workshop website.



Important Dates:

  • Submission Deadline: September 18, 2021 September 22, 2021 AOE
  • Acceptance Notification: October 23, 2021 AOE
  • Camera-Ready Submission: November 1, 2021 AOE
  • Workshop Date: December 13, 2021


Confirmed Spearkers

Mirella Lapata

Prof.
Mirella Lapata

University of Edinburgh

Luke Zettlemoyer

Prof.
Luke Zettlemoyer

University of Washington (Facebook)

Kevin Duh

Prof.
Kevin Duh

Johns Hopkins University

Boxing Chen

Dr.
Boxing Chen

Alibaba

Saneer Singh

Prof.
Sameer Singh

University of California

Danqi Chen

Prof.
Danqi Chen

Princeton University

Mohammad Norouzi

Dr.
Mohammad Norouzi

Google Brain

Yejin Choi

Prof.
Yejin Choi

University of Washington (Allen Institute for AI)

Xin Jiang

Dr.
Xin Jiang

Huawei Noah's Ark Lab

Xu Sun

Prof.
Xu Sun

Peking University

Barbara Plank

Prof.
Barbara Plank

IT University of Copenhagen

Samira Ebrahimi Kahou

Prof.
Samira Ebrahimi Kahou

ETS & MILA



Schedule (coming soon!)



Organizers

Mehdi Rezagholizadeh

Mehdi Rezagholizadeh
Huawei Noah's Ark Lab

Lili Mou

Lili Mou
University of Alberta

Yue Dong

Yue Dong
McGill University & MILA

Pascal Poupart

Pascal Poupart
University of Waterloo

Ali Ghodsi

Ali Ghodsi
University of Waterloo

Qun Liu

Qun Liu
Huawei Noah's Ark Lab



Technical Committee

  • Kevin Duh (Johns Hopkins University)
  • Bang Liu (University of Montreal (UDM))
  • Wulong Liu (Huawei Noah's Ark Lab)
  • Vahid Partovi Nia (Huawei Noah's Ark Lab)
  • Peyman Passban (Amazon)
  • Ivan Kobyzev (Huawei Noah's Ark Lab)
  • Jad Kabbara (McGill University & MILA)
  • Aref Jafari (University of Waterloo)
  • Ahmad Rashid (Huawei Noah's Ark Lab)
  • Shailza Jolly (TU Kaiserslautern)
  • Md. Akmal Haidar (Huawei Noah's Ark Lab)
  • Jingjing Xu (ByteDance)
  • Vasileios Lioutas (University of British Colombia (UBC))
  • Anderson R. Avila (Huawei Noah's Ark Lab)
  • Malik H. Altakrori (McGill University & MILA)
  • Ali Vahdat (Thomson Reuters)
  • Prasanna Parthasarathi (McGill University & MILA)
  • Fattane Zarrinkalam (Thomson Reuters)
  • Makesh S Narsimhan (McGill University & MILA)
  • Nasrin Taghizadeh (University of Tehran)
  • Borna Jafarpour (Thomson Reuters)
  • Shohreh Shaghaghian (Thomson Reuters)
  • Ehsan Kamalloo (University of Alberta)
  • Ali Saheb Pasand (University of Waterloo)
  • Abbas Ghaddar (Huawei Noah's Ark Lab)
  • Mehrdad Ganjeh (Ernst & Young (EY))
  • Mingxuan Wang (ByteDance)
  • Tanya Roosta (Amazon)
  • Soheila Samiee (BASF)
  • Hamidreza Mahyar (McMaster University)
  • Yimeng Wu (Huawei Noah's Ark Lab)
  • Omar Mohamed Awad (Huawei Technologies)
  • Marzieh Tahaei (Huawei Noah's Ark Lab)
  • Habib Hajimolahoseini (Huawei Technologies)
  • Mohammad Salameh (Huawei Technologies)
  • Kira Aveline Selby (University of Waterloo)