Intel® Extension for Transformers v1.0a Release

• Highlights
• Features
• Productivity
• Examples

Highlights

• Intel® Extension for Transformers provides a rich set of model compression techniques and a leading sparsity-aware libraries and neural engine to accelerate the inference of Transformer-based models on Intel platforms. We published 2 papers on NeurIPS’2022 with the source code released:
  • Fast DistilBERT on CPUs: outperform existing state-of-the-art Neural Magic's DeepSparse runtime performance by up to 50%, and deliver 7x better performance on c6i.12xlarge (Ice Lake) than c6a.12xlarge (AMD Milan)
  • QuaLA-MiniLM: outperform BERT-base with ~3x reduced size and demonstrate up to 8.8x speedup with <1% accuracy loss on SQuAD1.1 task

Features

• Pruning/Sparsity
  • Support Distributed Pruning on PyTorch
  • Support Distributed Pruning on TensorFlow
• Quantization
  • Support Distributed Quantization on PyTorch
  • Support Distributed Quantization on TensorFlow
• Distillation
  • Support Distributed Distillation on PyTorch
  • Support Distributed Distillation on TensorFlow
• Compression Orchestration
  • Support Distributed Orchestration on PyTorch
• Neural Architecture Search (NAS)
  • Support auto distillation with NAS and flash distillation on PyTorch
• Length Adaptive Transformer (LAT)
  • Support Dynamic Transformer on SQuAD1.1 on PyTorch
• Transformers-accelerated Neural Engine
  • Support inference with sparse GEMM fusion patterns
  • Support automatic benchmarking of sparse and dense mixed model
• Transformers-accelerated Libraries
  • Support 1x4 block-wise sparse VNNI-INT8 GEMM kernels with post-ops
  • Support 1x16 block-wise sparse AMX-BF16 GEMM kernels with post-ops

Productivity

• Support seamless Transformers-extended APIs
• Support experimental model conversion from PyTorch INT8 model to ONNX INT8
• Support VTune performance tracing for sparse GEMM kernels

Examples

• LAT examples for MiniLM (NeurIPS’2022)
• Fast DistilBert on CPUs (NeurIPS’2022)
• PyTorch distributed compression orchestration examples
• Post-training quantization for Transformers non-trainer API
• PyTorch auto distillation (NAS based) examples
• Multiple examples of Quantization/Pruning/Distillation on PyTorch and TensorFlow
• Post-training static quantization via Intel® Extension for PyTorch examples and deployment examples

Validated Configurations

• Centos 8.4 & Ubuntu 20.04
• Python 3.7, 3.8, 3.9, 3.10
• TensorFlow 2.9.1, 2.10.0, Intel® Extension for TensorFlow 2.9.1, 2.10.0
• PyTorch 1.12.0+cpu, 1.13.0+cpu, Intel® Extension for PyTorch 1.12.0