Optimizing Bark using 🤗 Transformers

Hugging Face has detailed a method for optimizing the Bark text-to-speech (TTS) model to reduce memory usage and increase inference speed. By combining three specific optimization techniques—Better Transformer, half-precision floating point (fp16), and CPU offload—users can achieve up to an 80% reduction in memory footprint and a 23% increase in speed.

Bark Model Architecture

Bark is a transformer-based TTS model developed by Suno AI, capable of generating speech, music, background noise, and nonverbal sounds (e.g., laughter and sighs). The architecture consists of four primary components that operate sequentially:

  • BarkSemanticModel: A causal auto-regressive transformer that predicts semantic text tokens.
  • BarkCoarseModel: A causal autoregressive transformer that predicts the first two audio codebooks for EnCodec.
  • BarkFineModel: A non-causal autoencoder transformer that iteratively predicts the remaining codebooks.
  • EncodecModel: Decodes the final codebook channels into the output audio array.

Optimization Techniques

Optimization is achieved through the 🤗 Optimum and 🤗 Accelerate libraries with minimal code changes.

Better Transformer

Better Transformer utilizes kernel fusion and Flash Attention to optimize GPU operations. Flash Attention reduces memory usage from quadratic to linear relative to sequence length, which increases speed without degrading model performance.

Key Benefit: Provides a 20% to 30% speed increase with no loss in output quality.

Half-Precision (fp16)

By switching from single-precision floating point (fp32, 32 bits per number) to half-precision (fp16, 16 bits per number), the model's storage requirements are halved.

Key Benefit: Reduces the memory footprint by 50% and provides a modest speed gain (approximately 5%), though it may introduce slight performance degradation.

CPU Offload

Because Bark's four sub-models are called sequentially, only one is active at a time while the others remain idle. CPU offload unloads inactive sub-models from the GPU to the CPU to free up precious GPU memory.

Key Benefit: Reduces the memory footprint by approximately 60% with a slight (10%) increase in latency.

Performance Benchmarks

Benchmarks were conducted using the large version of Bark on an NVIDIA TITAN RTX 24GB with a maximum of 256 new tokens, averaged over 100 samples.

Single Sample Generation (Batch Size = 1)

When generating a single sample, the combination of Better Transformer, CPU offload, and fp16 yields the most significant gains:

Optimization Latency Change Memory Change
No Optimization 0% 0%
Better Transformer Only -27% -1%
Offload + Better Transformer -15% -59%
Offload + Better Transformer + fp16 -23% -80%

Batch Generation (Batch Size = 8)

Batching multiple samples together significantly increases throughput. When Better Transformer is enabled by default, the following results were observed:

Optimization Latency Change Memory Change Throughput Change
Base Case (Better Transformer) 0% 0% 0%
+ fp16 -46% -50% +87%
+ Offload +6% -38% -6%
+ Offload + fp16 -43% -69% +77%

Summary of Optimal Configurations

Depending on the use case, different combinations of optimizations are recommended:

  • For Memory Efficiency: Use Better Transformer and CPU offload to run the large Bark model with a 2GB footprint instead of 5GB.
  • For High Throughput: Use batching (size 8) combined with Better Transformer and half-precision (fp16).
  • For Balanced Performance: Combine fp16, Better Transformer, and CPU offload for the best overall reduction in both latency and memory usage.

Sources