BaruqueRodrigues/llm-legislativo

🏛️ LLM Legislativo Brasileiro – Quick Overview

What it is: A research‑grade, open‑source causal language model (decoder‑only transformer) trained from scratch on the entire public legislative corpus of Brazil’s Congress (both Chamber of Deputies and Senate). The project builds the data pipeline, tokenizer, model architecture, training loop, and evaluation entirely in‑house, using only PyTorch (no Hugging Face Trainer) and a few auxiliary tools.

Key components

• Data: >360 GiB of raw TAR archives (≈930 k documents) covering bills, amendments, reports, votes, etc. The data are downloaded, OCR‑ed (Swift + Apple Vision), deduplicated, linked into “families” of related legislative texts, and split deterministically (80/10/10) with zero leakage between train/val/test.
• Tokenizer: Byte‑level BPE with an 8 192‑token vocabulary, trained only on the training split.
• Model: A pure‑PyTorch transformer decoder (6 layers, 6 heads, d‑model = 384, context = 512 tokens). Implements weight‑tying, GELU, and an analytically‑scaled cross‑entropy loss to handle padding correctly.
• Training: Conducted on an NVIDIA A100 (Google Colab) for 173 k steps, processing 1.25 billion tokens. Checkpoints are saved with SHA‑256 hashes for cryptographic auditability.
• Evaluation: Uses a custom byte_nll_v1 metric (≈0.73 nats/byte, 1.05 bits/byte) on a held‑out test set of 71 k documents, reporting perplexities around 8–9 per token.
• Artifacts: All datasets (raw, extracted, curated) and the final model (safetensors, tokenizer, config) are publicly hosted on the Hugging Face Hub.

How to use it

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from legislm.model.gpt import GPTModel
from legislm.model.config import ModelConfig
from tokenizers import Tokenizer

# download model & tokenizer
weights = hf_hub_download('BaruqRodrigues/llm-legislativo', 'model.safetensors')
tokenizer_file = hf_hub_download('BaruqRodrigues/llm-legislativo', 'tokenizer.json')

# set up model
cfg = ModelConfig(vocab_size=8192, context_length=512, d_model=384,
                  n_layers=6, n_heads=6, tie_embeddings=True)
model = GPTModel(cfg)
model.load_state_dict(load_file(weights))
model.eval()

tokenizer = Tokenizer.from_file(tokenizer_file)
prompt = "PROJETO DE LEI Nº , DE 2026\n\nDispõe sobre"
ids = torch.tensor([tokenizer.encode(prompt).ids])

with torch.no_grad():
    out = model.generate(ids, max_new_tokens=64, temperature=0.7, top_k=40)
print(tokenizer.decode(out[0].tolist()))

The code loads the pre‑trained weights directly, without any extra libraries, and can generate Portuguese legislative‑style text.

Why it matters

• Demonstrates a full‑stack, reproducible pipeline for large‑scale domain‑specific language modeling.
• Provides a rare, openly available model trained on a national legislative corpus, useful for research on legal language, policy analysis, or building downstream tools (e.g., document summarisation, question answering) – not for legal advice.
• Emphasises scientific rigor: cryptographic integrity checks, deterministic splits, and transparent evaluation.

Ethical note The model captures statistical patterns of legislative drafting but can hallucinate facts. It should never replace qualified legal counsel.

---

License: MIT for code; the source documents are public‑domain Brazilian legislative texts.