CUNY-CL/wikipron

Massively multilingual pronunciation mining

What it solves

WikiPron is designed to automate the collection of pronunciation data from Wiktionary. It solves the problem of manually gathering word-to-pronunciation pairs (G2P data) for a wide variety of languages, which is essential for building speech synthesis or speech recognition systems.

How it works

The tool operates as both a command-line interface (CLI) and a Python API. It scrapes Wiktionary for International Phonetic Alphabet (IPA) transcriptions, allowing users to specify the language (via ISO 639-3 codes), target specific dialects, and choose between broad or narrow transcriptions. It also handles the segmentation of IPA symbols to ensure that diacritics and modifiers are correctly grouped with their parent symbols for better machine learning modeling.

Who it’s for

This tool is for researchers and developers working in speech technology, linguistics, and natural language processing (NLP) to create pronunciation dictionaries.

Highlights

  • Multilingual Support: Supports mining data for 348 languages.
  • Customizable Scraping: Options to filter by dialect and transcription level (broad vs. narrow).
  • IPA Segmentation: Uses the segments library to correctly handle IPA diacritics for modeling purposes.
  • Ready-to-use Data: Provides a database of over 5 million word-pronunciation pairs.

Related

  • Project
    goodmami/wn

    A Python library for exploring and querying multilingual wordnets, providing tools for semantic similarity, taxonomic exploration, and lemmatization.

  • Project
    bootphon/phonemizer

    A Python library and command-line tool for converting text into phonetic transcriptions across multiple languages using various backends.

  • Project
    JeremyCCHsu/Python-Wrapper-for-World-Vocoder

    A Python wrapper for the WORLD Vocoder that allows for high-quality speech analysis and resynthesis by decomposing audio into pitch, spectral envelope, and aperiodicity.

  • Project
    deepdoctection/deepdoctection

    A Python library for document understanding that orchestrates layout analysis, OCR, and token classification to extract structured data from PDFs and scans.

  • Project
    oxylabs/AI-Search-py

    A Python-based search agent that allows developers to search the web using natural language and automatically scrape content from the results, including support for JavaScript rendering.