PSRben/VisionHOPE

Official PyTorch implementation of VisionHOPE: Visual Backbones as Self-Modifying Learning Systems.

What it solves

VisionHOPE is a visual backbone architecture designed to address the limitations of static neural network layers. It introduces a "self-modifying" learning system where the model's internal memory and the rules governing how it updates that memory evolve dynamically as it processes an image, allowing for more adaptive feature extraction compared to traditional CNNs or Transformers.

How it works

At its core is the Self-Referential Nested Learning (SRNL) operator. This operator uses five coupled memories that store content and generate key/value representations while simultaneously controlling learning rates and retention. To ensure stability during these recursive updates, the model employs a soft injection cap and spectral clamping. The architecture processes images using four directional scans that align with rows and columns, followed by spatial restoration and channel-wise fusion to produce the final output.

Who it’s for

This project is for computer vision researchers and engineers looking for high-performance, hierarchical backbones for tasks like image classification, object detection, and semantic segmentation. It provides modular components that can be integrated into existing vision pipelines or used as standalone backbones.

Highlights

  • Implements a novel self-referential learning mechanism that adapts memory and update rules per image.
  • Provides hierarchical models (Tiny, Small, Base) suitable for diverse vision tasks.
  • Includes stable, non-expanding memory recurrence via spectral clamping and soft injection caps.
  • Offers ready-to-use PyTorch implementations for classification (ImageNet), detection (COCO), and segmentation (ADE20K).
  • Supports fast inference modes through parameter fusion and optimized CUDA kernels.

Related

  • Project
    facebookresearch/dinov2

    A self-supervised learning framework that produces robust visual features from 142M images, enabling high-performance computer vision tasks without fine-tuning.

  • Project
    voxelmorph/voxelmorph

    A learning-based image registration library that uses neural networks to align images and model deformations, featuring contrast-agnostic training via SynthMorph.

  • Project
    UWMILab/UniWM

    UniWM is a research codebase for a memory‑augmented world‑model that jointly predicts future egocentric images and navigation actions, using a unified transformer backbone (Anole). It provides a Hugging Face dataset, training/evaluation scripts, and supports multi‑GPU runs.

  • Dispatch
    The State of Computer Vision at Hugging Face

    Hugging Face has expanded its ecosystem to support 8 core computer vision tasks, over 3,000 models, and 100+ datasets, integrating both Transformer and convolutional architectures.

  • Dispatch
    Google DeepMind Visual Alignment Research

    Google DeepMind has developed a three-step alignment method that reorganizes AI visual representations to match human conceptual hierarchies, improving model robustness and generalization.