ReadSight (Rust) — Multilingual Readability Engine

readsight measures text readability across 86 languages. It implements 17 readability formulas with language-specific coefficients and uses the Frank M. Liang (TeX) hyphenation algorithm for syllable counting. All language data and hyphenation patterns are embedded in the crate — no filesystem or network access is required at runtime.

This is a byte-accurate Rust port of the canonical PHP library and its Python port:

PHP (canonical): https://github.com/MADEVAL/ReadSight
Python port: https://github.com/MADEVAL/ReadSightPy

Output parity with the reference implementation is verified with golden vectors generated from the PHP library (see tests/golden).

See It in Action

Two texts of almost equal length — a plain sentence and a chunk of legal boilerplate:

let plain = "We made an app that reads your text. It tells you how easy it is to read. You get a score in one second.";
let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";

There is no "score everything" call — you loop over the formulas the language supports and call score() for each:

use readsight::ReadSight;
# let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";

let rs = ReadSight::new("en-us")?;

for formula in rs.supported_formulas() {
    let result = rs.score(&formula, legal)?;
    // result.score, result.grade_level, result.interpretation
    // ...
}
# Ok::<(), readsight::Error>(())

For both texts that produces (verbatim output of cargo run --example demo):

READABILITY FORMULA          | Plain text               | Legalese
------------------------------------------------------------------------------------
ari                          | -2.1  g0.0 Kindergarten  | 13.2  g13.2 College
coleman_liau                 | -0.4  g0.0 Kindergarten  | 16.5  g16.5 Graduate
dale_chall                   | 5.3  5th-6th grade       | 12.2  Graduate
flesch_kincaid_grade_level   | 0.3  g0.3 1st Grade      | 13.5  g13.5 College
flesch_reading_ease          | 107.1  Very Easy         | 23.4  Very Hard
gunning_fog                  | 3.2  g3.2 Very Easy      | 18.5  g18.5 Extremely Hard
lix                          | 8  Children's Books      | 49.71  Factual Information
smog                         | 3.1  g3.1 3rd Grade      | 15.2  g15.2 College
spache                       | 2.3  g2.3 2nd Grade      | 6.5  g5.0 Above 4th Grade

All 9 formulas for en-us agree the second text is far harder. The bundled example prints this grid plus syllable breakdowns and text statistics:

cargo run --example demo
cargo run --example multilingual

17 formulas, 86 languages, one consistent API. Five of the formulas are truly universal — Gunning Fog, SMOG, Coleman-Liau, ARI and LIX score text in every one of the 86 languages. The remaining 12 are language-aware, each carrying its own coefficients: Flesch Reading Ease and Flesch-Kincaid span 12 languages, the Wiener Sachtextformel speaks German, Gulpease speaks Italian, OSMAN speaks Arabic, and the Fernández-Huerta · Szigriszt-Pazos · Gutiérrez-Polini · Crawford family handles Spanish. supported_formulas() then hands each language exactly the slice that fits it — 9 formulas for en-us, 11 for es, 8 for de-1996, down to the 5 universal ones for a language like th — so an English-only metric never lands on a Thai sentence by mistake.

Installation

[dependencies]
readsight = "1.0"

Requirements:

Rust >= 1.74 (edition 2021)

The crate depends on regex, fancy-regex, serde, serde_json, include_dir, and indexmap. All language and hyphenation data is bundled in the crate, so there is no runtime filesystem or network access.

Quick Start

use readsight::ReadSight;

let engine = ReadSight::new("en-us")?;

// Syllable counting
assert_eq!(engine.syllable_count("banana"), 3);          // 3
assert_eq!(engine.split_syllables("hyphenation"),        // ["hyp", "hen", "ati", "on"] (heuristic split)
           vec!["hyp", "hen", "ati", "on"]);
assert_eq!(engine.split_word("hyphenation"),             // ["hy", "phen", "ation"] (TeX hyphenation points)
           vec!["hy", "phen", "ation"]);

// Text analysis
let stats = engine.analyze("The quick brown fox jumps over the lazy dog.")?;
println!("Words: {}, Syllables: {}", stats.word_count, stats.syllable_count);

// Readability formulas
let fre = engine.flesch_reading_ease("The quick brown fox jumps over the lazy dog.")?;
println!("Flesch Reading Ease: {} - {}", fre.score, fre.interpretation);

let fog = engine.gunning_fog("The quick brown fox jumps over the lazy dog.")?;
println!("Gunning Fog: {} (grade {:?})", fog.score, fog.grade_level);
# Ok::<(), readsight::Error>(())

All formula methods return Result<FormulaResult, Error>. analyze (and hence every formula) returns [Error::EmptyText] for empty input.

Syllable Counting Modes

ReadSight has three syllable counting modes, configured per language via syllableMode in data/languages/*.json:

Mode	How it works	`count` accuracy	`split` accuracy
`heuristic`	Vowel patterns + word list + prefix/suffix rules	exact	≈ approximate
`tex`	Frank M. Liang hyphenation algorithm (TeX `.tex` patterns)	≈ approximate	exact
`composite`	Heuristic first, TeX as fallback	exact	≈ approximate (uses heuristic split)

80 languages use tex, 2 use composite (en-us, en-gb), 4 use heuristic (ru, uk, be, bg). The default mode is tex.

Why tex count is approximate: TeX hyphenation patterns are optimised for line-breaking, not phonetic syllabification. They respect hyphenMins and avoid awkward break points, so the number of pieces can differ from the true syllable count. For scripts where one syllable = one vowel (e.g. Cyrillic Slavic languages), TeX under- or over-counts. Those languages use heuristic mode with a per-language vowel pattern and "vowelMode": "individual" so each vowel is counted as a syllable. split_word() keeps using the exact TeX hyphenator regardless of mode.

Vowel counting: `vowelMode`

The syllableHeuristics block accepts a vowelMode field:

`vowelMode`	Behaviour
`"cluster"` (default)	Each run of consecutive vowels = 1 syllable
`"individual"`	Each vowel letter = 1 syllable (Slavic Cyrillic)

use readsight::ReadSight;

// Russian uses heuristic + vowelMode "individual"
let ru = ReadSight::new("ru")?;
assert_eq!(ru.syllable_count("дыхание"), 4);
# Ok::<(), readsight::Error>(())

Example: "hyphenation" in each mode

use readsight::ReadSight;

let en = ReadSight::new("en-us")?;           // composite mode - heuristic wins
assert_eq!(en.syllable_count("hyphenation"), 4);
assert_eq!(en.split_syllables("hyphenation"), vec!["hyp", "hen", "ati", "on"]); // heuristic
assert_eq!(en.split_word("hyphenation"), vec!["hy", "phen", "ation"]);          // TeX

let de = ReadSight::new("de-1996")?;          // tex mode
assert_eq!(de.syllable_count("hyphenation"), 4);
assert_eq!(de.split_syllables("hyphenation"), vec!["hy", "phena", "ti", "on"]); // TeX
assert_eq!(de.split_word("hyphenation"), vec!["hy", "phena", "ti", "on"]);      // same
# Ok::<(), readsight::Error>(())

Tip: split_word() always uses the TeX hyphenator (exact). split_syllables() may use the heuristic split (approximate) in composite/heuristic modes. Syllable counts are exact in heuristic/composite mode; in tex mode they follow TeX break points and are approximate (see note above).

Note: add_hyphenations() adds overrides to the TeX hyphenator. These affect split_word() but NOT split_syllables() in composite/heuristic modes (the heuristic counter doesn't see them).

Examples

Run the bundled demo to see ReadSight in action:

cargo run --example demo

This scores the plain/legal texts side by side and outputs:

Readability grid with every applicable formula, its score, grade level and interpretation
Syllable breakdown with heuristic split and TeX hyphenation points
Text statistics — letters, words, sentences, syllables, and a syllable histogram

Compare a short sample across several languages:

cargo run --example multilingual

Supported Languages

86 languages across 19 writing systems: Latin, Cyrillic, Arabic, Hebrew, Devanagari, Bengali, Tamil, Thai, Greek, Armenian, Georgian, Gujarati, Gurmukhi, Kannada, Malayalam, Odia, Telugu, Ethiopic, Coptic.

use readsight::ReadSight;

let _ru = ReadSight::new("ru")?;       // Russian
let _de = ReadSight::new("de-1996")?;  // German (1996 reform)
let _es = ReadSight::new("es")?;       // Spanish
let _th = ReadSight::new("th")?;       // Thai

// List all supported languages (sorted)
let langs = ReadSight::supported_languages(None);
assert_eq!(langs.len(), 86);
# Ok::<(), readsight::Error>(())

ReadSight::supported_languages(None) returns all 86 codes, sorted, mirroring the JSON files under data/languages exactly:

af, ar, as, be, bg, bn, ca, cop, cs, cu, cy, da, de-1901, de-1996, de-ch-1901,
el-monoton, el-polyton, en-gb, en-us, eo, es, et, eu, fa, fi, fi-x-school, fr,
fur, ga, gl, grc, gu, he, hi, hr, hsb, hu, hy, ia, id, is, it, ka, kk, kmr, kn,
la, la-x-classic, la-x-liturgic, lt, lv, mk, ml, mn-cyrl, mn-cyrl-x-lmc, mr,
mul-ethi, nb, nl, nn, oc, or, pa, pi, pl, pms, pt, rm, ro, ru, sa, sh-cyrl,
sh-latn, sk, sl, sq, sr-cyrl, sv, ta, te, th, tk, tr, uk, vi, zh-latn-pinyin

Readability Formulas

Universal (all 86 languages)

Formula	`name` key	Method	Type
Gunning Fog	`gunning_fog`	`gunning_fog()`	Syllable-based
SMOG Index	`smog`	`smog_index()`	Syllable-based
Coleman-Liau	`coleman_liau`	`coleman_liau()`	Letter-based
ARI	`ari`	`automated_readability_index()`	Letter-based
LIX	`lix`	`lix()`	Letter-based

Language-Specific

Language(s)	Formulas
`en-us`, `en-gb`, `de-*`, `ru`, `es`, `it`, `fr`, `nl`, `pt`, `tr` (12 codes)	Flesch Reading Ease, Flesch-Kincaid Grade Level
English (`en-us`, `en-gb`)	Dale-Chall, Spache
German (`de-1996`, `de-1901`, `de-ch-1901`)	Wiener Sachtextformel (4 variants)
Spanish (`es`)	Fernández-Huerta, Szigriszt-Pazos, Gutiérrez-Polini, Crawford
Italian (`it`)	Gulpease
Polish (`pl`)	FOG-PL
Arabic (`ar`)	OSMAN

* Note: Dale-Chall and Spache use a syllable-based heuristic to estimate difficult words (1-syllable ≈ easy). This is a simplified estimation, not the original Dale/Spache word lists.

Generic dispatching by name:

use readsight::ReadSight;

let rs = ReadSight::new("de-1996")?;
let r = rs.score("gunning_fog", "Ein einfacher deutscher Satz. Und noch einer.")?;
assert_eq!(r.formula_name, "gunning_fog");

// Wiener Sachtextformel supports variants 1..=4
let w = rs.wiener_sachtextformel("Ein einfacher deutscher Satz. Und noch einer.", 1)?;
assert_eq!(w.formula_name, "wiener_sachtextformel_1");
# Ok::<(), readsight::Error>(())

FormulaResult

# use readsight::FormulaResult;
# fn _doc(result: FormulaResult) {
result.score;           // f64 - raw (rounded) formula score
result.grade_level;     // Option<f64> - normalized grade level (FKGL, GF, SMOG, CL, ARI, Spache)
result.interpretation;  // String - qualitative interpretation ("Easy", "Hard", ...)
result.formula_name;    // String - formula key
result.language_code;   // String - language code used
result.inputs;          // BTreeMap<String, f64> - intermediate values for debugging
# }

API Reference

ReadSight (aliased as Engine) is the entry point.

Text / syllable methods

engine.syllable_count(word: &str) -> i64
engine.split_word(word: &str) -> Vec<String>
engine.split_syllables(word: &str) -> Vec<String>
engine.word_count(text: &str) -> i64
engine.sentence_count(text: &str) -> i64
engine.letter_count(text: &str) -> i64
engine.total_syllables(text: &str) -> i64
engine.average_syllables_per_word(text: &str) -> f64
engine.average_words_per_sentence(text: &str) -> f64
engine.polysyllable_count(text: &str, count_proper_nouns: bool) -> i64
engine.words_with_more_than_n_syllables(text: &str, n: i64, count_proper_nouns: bool) -> i64
engine.histogram_syllables(text: &str) -> BTreeMap<i64, i64>
engine.analyze(text: &str) -> Result<TextStatistics>
engine.add_hyphenations(iter)   // (word, "hy-phen-a-ted") overrides

Formula methods

engine.score(name: &str, text: &str) -> Result<FormulaResult>
engine.flesch_reading_ease(text) / flesch_kincaid_grade_level(text)
engine.gunning_fog(text) / smog_index(text) / coleman_liau(text)
engine.automated_readability_index(text) / lix(text)
engine.gulpease(text) / fernandez_huerta(text) / szigriszt_pazos(text)
engine.gutierrez_polini(text) / crawford(text) / fog_pl(text)
engine.dale_chall(text) / spache(text) / osman(text)
engine.wiener_sachtextformel(text, variant: i32)   // variant 1..=4

Static

ReadSight::supported_languages(config: Option<&Config>) -> Vec<String>

Data Source

By default all data is embedded at compile time (via include_dir!): 86 language JSON files and 86 hyph-*.tex pattern files under data/. To load from the filesystem instead:

use readsight::{Config, ReadSight};

let config = Config::from_dirs("data/patterns", "data/languages");
let rs = ReadSight::with_config("en-us", config)?;
# Ok::<(), readsight::Error>(())

Custom hyphenation overrides affect split_word (not split_syllables in composite/heuristic modes):

use readsight::ReadSight;

let engine = ReadSight::new("en-us")?;
engine.add_hyphenations([("customword", "cus-tom-word")]);
assert_eq!(engine.split_word("customword"), vec!["cus", "tom", "word"]);
# Ok::<(), readsight::Error>(())

Architecture

ReadSight (facade, aliased as Engine)
  ├── TextAnalyzer (syllable counting, text metrics)
  │   ├── SyllableCounterKind (tex | heuristic | composite)
  │   │   ├── CompositeSyllableCounter (heuristic problem words → heuristic, rest → TeX)
  │   │   ├── HeuristicSyllableCounter (vowel patterns + word list, vowelMode)
  │   │   └── TexSyllableCounter → LiangHyphenator (TeX hyphenation)
  │   ├── LiangHyphenator
  │   │   ├── parse_tex / TexSource (parses .tex from hyph-utf8)
  │   │   ├── PatternsCollection (pattern data)
  │   │   └── HyphenationExceptionsCollection (word overrides)
  │   └── TextSplitter (word/sentence/letter counting)
  ├── Language (JSON config per language, syllableMode + formula configs)
  └── FormulaRegistry (17 formulas)
      ├── FleschReadingEase / FleschKincaidGradeLevel (lang-specific coefficients)
      ├── GunningFog, SmogIndex, ColemanLiau, ARI, LIX (universal)
      └── WienerSachtextformel, Gulpease, FernandezHuerta, ... (lang-specific)

Development

cargo test                                  # unit + golden parity + smoke tests
cargo clippy --all-targets -- -D warnings   # lints (clean)
cargo fmt --check                           # formatting (clean)

The suite currently runs 90 integration tests, plus the crate's doc tests (including the runnable Rust examples in this README, which are compiled and executed by cargo test):

Golden parity against the PHP reference: supported_formulas, analyze, and the applicable formulas over all 86 languages, plus per-word syllable vectors for a 20-language subset (tests/golden).
Ported unit tests from the PHP/Python suites (hyphenation, syllables, text splitting, formulas, grade-level interpretation).
Full-language smoke test that builds every language and runs every supported formula.

The library is #![forbid(unsafe_code)].

License

MIT — see LICENSE. Author of the original library: Yevhen Leonidov. Readability data and hyphenation patterns originate from the canonical PHP project and the hyph-utf8 package; the TeX pattern files are packaged under their original licenses (see individual file headers).

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
data		data
examples		examples
src		src
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

ReadSight (Rust) — Multilingual Readability Engine

See It in Action

Table of Contents

Installation

Quick Start

Syllable Counting Modes

Vowel counting: `vowelMode`

Example: "hyphenation" in each mode

Examples

Supported Languages

Readability Formulas

Universal (all 86 languages)

Language-Specific

FormulaResult

API Reference

Text / syllable methods

Formula methods

Static

Data Source

Architecture

Development

License

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

ReadSight (Rust) — Multilingual Readability Engine

See It in Action

Table of Contents

Installation

Quick Start

Syllable Counting Modes

Vowel counting: vowelMode

Example: "hyphenation" in each mode

Examples

Supported Languages

Readability Formulas

Universal (all 86 languages)

Language-Specific

FormulaResult

API Reference

Text / syllable methods

Formula methods

Static

Data Source

Architecture

Development

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Vowel counting: `vowelMode`

Packages