Personal Copilot: Train Your Own Coding Assistant

Hugging Face has introduced a methodology for creating personalized coding assistants, exemplified by HugCoder, a model fine-tuned on the public repositories of the Hugging Face GitHub organization. This approach allows developers and enterprises to tailor large language models (LLMs) to proprietary codebases, improving the accuracy of code completions for internal APIs and libraries.

Data Collection Workflow

To build a personalized assistant, Hugging Face implemented a data collection pipeline that extracts code from GitHub repositories while avoiding API rate limits by cloning repositories locally.

Key aspects of the workflow include:

  • Parallel Cloning: Using Python's multiprocessing module to download repositories in parallel.
  • Filtering: Non-code files (images, presentations) and non-code paths (.git, __pycache__, xcodeproj) are excluded via a predefined list of extensions.
  • Parsing: Code files are parsed using "utf-8" encoding, and for Jupyter Notebooks, only code cells are extracted.
  • Storage: Data is serialized using chunking and the feather format for memory efficiency.

For the HugCoder project, the team focused on the top 10 Hugging Face public repositories based on stargazers, including transformers, datasets, diffusers, and peft.

Fine-Tuning Strategies: QLoRA vs. Full Fine-Tuning

Hugging Face compared two primary training methods for the bigcode/starcoder (15.5B parameters) model using a Fill-In-the-Middle (FIM) training objective.

Parameter-Efficient Fine-Tuning (PEFT) with QLoRA

QLoRA significantly reduces hardware requirements by freezing the base model and training a small set of adapter weights.

  • Memory Usage: A single A100 40GB GPU is sufficient, with total memory occupancy of approximately 26 GB (batch size 4) when using Flash Attention V2 and Gradient Checkpointing.
  • Cost and Time: Training took 12.5 hours at an estimated cost of $13.75 (based on $1.10/hr).

Full Fine-Tuning

Full fine-tuning updates all model parameters but requires substantial hardware.

  • Memory Usage: Requires a minimum of 248GB of GPU memory for the 15.5B model (excluding activations), necessitating at least 4x A100 80GB GPUs. Using PyTorch Fully Sharded Data Parallel (FSDP), memory per GPU ranged from 70 GB to 77.6 GB (per_gpu_batch_size 1).
  • Cost and Time: Training took 9 hours on 8x A100 80GB GPUs at an estimated cost of $108 (based on $12.00/hr).

Performance Comparison

Full fine-tuning converged faster and achieved slightly lower loss than QLoRA. However, the QLoRA model maintained comparable performance on the humaneval-python benchmark (Pass@1 of 33.37 vs. 33.57 for the base model), indicating no significant catastrophic forgetting.

Qualitative Results and Code Infilling

In manual analysis, fine-tuned models (both QLoRA and full) outperformed GitHub Copilot in scenarios involving recent libraries. For example, when tasked with infilling code for the 🤗 PEFT library—which may not have been in Copilot's training data—GitHub Copilot provided no completion, while the HugCoder variants correctly filled the function calls with necessary parameters.

Advanced LoRA Techniques

Mix-and-Match LoRAs

Hugging Face experimented with combining different LoRA adapters using the add_weighted_adapter utility in PEFT. By creating a code_buddy adapter (combining a chatting/QA adapter and a code-completion adapter with equal weights), the model could simultaneously perform code completion and answer technical questions about the specific codebase.

Transferring LoRAs

LoRA adapters trained on one base model can be transferred to others. The team applied a StarCoder-trained adapter to the Octocoder model, resulting in a model that could correctly answer detailed questions about LoraConfig and PEFT model creation, which the base Octocoder model failed to do.

Deployment and Local Execution

Remote Deployment

Models can be deployed via 🤗 Inference Endpoints and integrated into VS Code using the llm-vscode extension by pointing the extension to the deployed endpoint URL.

Local Execution

For consumer hardware (e.g., Mac M1), Hugging Face utilized the mlc-llm library to run a smaller starcoderbase-1b model. The process involves:

  1. Compiling the model for the target hardware (e.g., Metal for Mac).
  2. Configuring the mlc-chat-config.json for optimal generation length and temperature.
  3. Running a local REST server that connects to the llm-vscode extension.

Sources