Mojibake 0.2.7 – Zero‑Dependency Unicode 17 Library for C11/C++17

Mojibake 0.2.7 – Zero‑Dependency Unicode 17 Library for C11/C++17

Mojibake 0.2.7 delivers full Unicode 17 support in a single‑file C library

Mojibake 0.2.7 implements the entire Unicode 17.0.0 specification—including normalization, case folding, segmentation, bidirectional layout, emoji handling, collation, and security checks—while requiring zero external dependencies and no runtime. This makes it ideal for low‑level projects that cannot afford heavyweight libraries.

Why a zero‑dependency Unicode library matters

Most C/C++ projects rely on large third‑party Unicode libraries (e.g., ICU) that pull in massive codebases, complex build systems, and runtime data tables. Mojibake sidesteps these issues by concatenating all source and Unicode data tables into two files (mojibake.c and mojibake.h). The result is a single‑header/implementation that can be dropped into any project, compiled with any C11 or C++17 compiler, and linked without additional libraries.

Core capabilities at a glance

Capability API function(s) Standard reference
Normalization (NFC, NFD, NFKC, NFKD) mjb_normalize, mjb_string_is_normalized UAX #15
Case folding & conversion (upper, lower, title, NFKC casefold) mjb_case, mjb_nfkc_casefold UAX #31, UAX #15
Segmentation (grapheme clusters, words, sentences, line breaks) mjb_break_grapheme_cluster, mjb_break_word, mjb_break_sentence, mjb_break_line UAX #29, UAX #14
Bidirectional algorithm (paragraph resolution, line reordering) mjb_bidi_resolve, mjb_bidi_reorder_line UAX #9
Emoji analysis (properties, sequence detection, RGI check) mjb_string_emoji_sequence, mjb_string_is_rgi_emoji UTS #51
Collation (UCA compare, sort‑key generation) mjb_string_compare, mjb_collation_key UTS #10
Security (confusable detection, identifier validation) mjb_confusable_skeleton, mjb_string_is_confusable, mjb_string_is_identifier UTS #39, UAX #31
Encoding conversion (UTF‑8, UTF‑16LE/BE, UTF‑32LE/BE) mjb_string_convert_encoding, mjb_string_encoding —
Display width & truncation (terminal columns, width‑aware truncation) mjb_display_width, mjb_truncate_width UAX #11

All functions return a detailed mjb_status enum, allowing callers to distinguish success, invalid arguments, memory failures, and overflow conditions.

Minimal integration steps

  1. Download the amalgamation – mojibake-amalgamation-027.zip contains mojibake.c and mojibake.h.
  2. Add the files to your project source tree.
  3. Include the header where Unicode handling is needed:
    #include "mojibake.h"
  4. Compile with any C11 or C++17 compiler; no linker flags are required.

Because the library has no runtime, you can also embed it in static libraries, firmware, or WebAssembly modules without pulling in additional binaries.

Performance considerations (community questions)

"How does it compare with utf8proc?" – @lifthrasiir

Mojibake offers a broader feature set than utf8proc (which focuses on UTF‑8 validation, normalization, and case folding). It adds full bidirectional processing, Unicode Collation, emoji sequence detection, and security‑oriented confusable checks. The author has not published benchmark numbers, so direct speed comparisons remain anecdotal.

"What's performance like compared to Python's ftfy module?" – @tjwebbnorfolk

ftfy is a high‑level Python library that performs extensive error‑correction on malformed Unicode. Mojibake operates at a lower level, exposing raw algorithms without automatic error repair. Consequently, raw throughput is likely higher, but a side‑by‑side benchmark is not provided in the source.

Build‑time feature toggles for size‑constrained environments

Mojibake allows compile‑time exclusion of optional tables to shrink binary size. For example, disabling character names reduces the data footprint by ~30 %:

# CMake
cmake -S . -B build-no-name -DMJB_FEATURE_CHARACTER_NAMES=OFF
cmake --build build-no-name

# Makefile shortcut
make build BUILD_DIR=build-no-name FEATURE_CHARACTER_NAMES=OFF

Other tables (e.g., emoji properties, script extensions) can be trimmed similarly via MJB_FEATURE_* macros.

Security‑focused utilities

  • Confusable detection – mjb_confusable_skeleton builds a language‑independent skeleton; mjb_string_is_confusable compares two strings for visual similarity, protecting against homograph attacks.
  • Identifier validation – mjb_string_is_identifier enforces Unicode identifier rules (ID_Start/ID_Continue or XID variants), useful for compilers and interpreters.

Both utilities rely on the official Unicode security data files (confusables.txt, IdentifierStatus.txt).

Testing, fuzzing, and conformance guarantees

Mojibake is validated against the official Unicode test suites for every supported algorithm. The project uses the Attractor test harness to execute 1.5 million+ assertions, including:

  • Full Unicode Normalization test vectors.
  • Bidirectional Algorithm conformance.
  • Collation test data.
  • Emoji sequence validation.

Additionally, the library is continuously fuzzed with libFuzzer, and all builds are run under AddressSanitizer and UBSan to catch memory safety issues.

Community reception (selected HN comments)

"Love the amalgamation approach—the C/C++ ecosystem desperately needs cleaner, lightweight Unicode support without pulling in massive dependencies... thanks for sharing" – @digg99

The comment highlights the practical value of a single‑file library for projects that cannot afford heavyweight dependencies.

"I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome" – @avadodin

A reminder that even minimal Unicode handling (e.g., proper length counting, case folding) often requires more than raw byte arrays; Mojibake fills that gap.

"Not to bikeshed, but isn’t the word ‘mojibake’ synonymous with ‘when character encoding breaks’?" – @CharlesW

The library’s name intentionally references the classic problem of garbled text, while providing the tools to prevent such issues.

License and legal notices

Mojibake is released under the MIT License. It incorporates Unicode Character Database and CLDR data, which retain their original Unicode licenses (see license.txt and CLDR LICENSE).

Getting started

  1. Download the latest release (v0.2.7) from the GitHub releases page.
  2. Read the API reference (API.md) for detailed function signatures.
  3. Run the CLI (src/shell/mojibake) for quick experiments, e.g.:
    mojibake nfc $'Cafe\u0301'   # → NFC: Café
mojibake emoji "☺️"        # → shows emoji classification
  4. Contribute – see CONTRIBUTING.md for guidelines on adding new Unicode versions or fixing bugs.

Bottom line

Mojibake 0.2.7 provides a complete, standards‑compliant Unicode 17 implementation in a tiny, dependency‑free package. Its single‑file distribution, extensive feature set, and thorough testing make it a compelling choice for any C or C++ project that needs reliable Unicode handling without the overhead of larger libraries.

Sources