go-v.orgdoi

Towards accurate low-bitwidth BERT

Le, Phuoc-Hoan Charles · eScholarship@McGill (McGill) · 2023

  • DOI 10.82308/17519
  • Publisher McGill University
  • Type dissertation
  • Issued 2023

Not cached yet. This record was fetched live for this request and is stored after the lookup. Reopening the page should serve the cached copy.

eScholarship@McGill (McGill) · McGill University · dissertation · 2023

Abstract

Transformer-based models or BERT (Bidirectional Encoder Representations from Transformers) models have seen a lot of success in natural language processing tasks in recent years. However, these networks are very difficult to deploy on edge devices where memory and compute resources are limited. To lower their computational and memory cost, pruning and quantization can be applied. However, naively applying these techniques to compress transformers, can cause a significant reduction in the accuracy of these models.

This thesis will focus on improving the performance of quantization-aware training or improving the final performance or accuracy of a quantized transformer-based model without increasing the computational complexity of the forward pass of the network or slightly changing the network's architecture to increase the model capacity.In this thesis, we give one example of such and propose a more efficient quantization-aware training pipeline, called Efficient Two Stage Progressive Quantization in which the model with quantized weights is fine-tuned first, and at the same time, the bitwidth of the weights is lowered. Then, the model with quantized weights and activations will be fine-tuned.

At the same time, we strategically choose which bitwidth to fine-tune on and to initialize from, and which bitwidth to fine-tune under augmented data. We are able to ternarize the weights of BERT, compressing it by 14.9x while outperforming the full-precision model performance across nearly all GLUE datasets, and without data augmentation, it can outperform existing BERT ternarization methods. Also, we outperform the state-of-the-art BERT binarization in performance without adding an extra module, saving 18% more memory than BinaryBERT

Open the resolved locator · 10.82308/17519

Metadata licence: OpenAlex metadata (CC0)

Registry
OpenAlex DOI index + API
Fetched
2026-11-20 03:19 UTC
Cache state
not cached yet (bulk-index+openalex)
Locator
https://escholarship.mcgill.ca/concern/theses/np193g24c
Metadata licence
OpenAlex metadata (CC0)
Scope
Metadata as returned by the resolver; abstracts are never generated here.