English | 简体中文
A high-performance Rust crate for multi-pattern string matching, validation, filtering, and replacement.
- Find all sensitive words:
find_all - First match with metadata (
Match):find_first_match - Validate text contains sensitive words:
validate - Remove sensitive words:
filter - Replace sensitive words with a character:
replace - Multi-algorithm engine: Aho-Corasick, Wu-Manber, Regex
- Noise removal via configurable regex
- Variant detection (拼音、形似字) — pinyin plus a 50+ group shape-confusable map
- Parallel search with optional
rayonsupport (parallelfeature, enabled by default) - LRU cache for hot queries
- Batch processing:
find_all_batch - Layered matching:
find_all_layered - Streaming processing:
find_all_streaming - Criterion benchmarks and runnable examples for release validation
The engine auto-selects based on vocabulary size:
| Patterns | Algorithm | Rationale |
|---|---|---|
| 0–100 | Wu-Manber | Small tables, quick scan |
| 101–10,000 | Aho-Corasick | O(n) automaton scan regardless of count |
| 10,000+ | Regex | Compilation overhead amortized |
Override with Filter::with_algorithm(...) or --algorithm on the CLI.
| Platform | Status | How |
|---|---|---|
| Linux / macOS / Windows | Full support | default features / --all-features |
| WASM (browser / Node.js) | Supported | wasm feature; no file/network I/O (use loadWords) |
Embedded (no_std) |
Experimental | --no-default-features; core exact matching only |
| Async (tokio) | Supported | async-io / net-async features |
[dependencies]
sensitive-rs = { version = "1.3", default-features = false, features = ["wasm"] }import init, { WasmFilter } from 'sensitive-rs';
await init();
const filter = new WasmFilter();
filter.addWord('赌博');
filter.findAll('含有赌博内容'); // ['赌博']
filter.loadWords('色情\n诈骗'); // bulk-load from in-memory text[dependencies]
sensitive-rs = { version = "1.3", default-features = false }Core find_all / find_in / replace / filter work without std. Pinyin/shape variant
detection, the LRU cache, and the file/network loaders require the std feature (on by default).
[dependencies]
sensitive-rs = { version = "1.3", features = ["async-io"] }#[tokio::main]
async fn main() -> std::io::Result<()> {
use sensitive_rs::Filter;
let mut filter = Filter::new();
filter.load_word_dict_async("dict/dict.txt").await?;
Ok(())
}Add to your Cargo.toml:
[dependencies]
sensitive-rs = "1.3.0"For environments that should avoid rayon (for example WASM or embedded targets), disable default features:
[dependencies]
sensitive-rs = { version = "1.3.0", default-features = false }use sensitive_rs::Filter;
fn main() {
let mut filter = Filter::new();
filter.add_words(&["rust", "filter", "敏感词"]);
let text = "hello rust, this is a filter demo 包含敏感词";
let found = filter.find_all(text);
println!("Found: {:?}", found);
let cleaned = filter.replace(text, '*');
println!("Cleaned: {}", cleaned);
}Batch processing:
let texts = vec!["text1", "text2"];
let results = filter.find_all_batch(&texts);Layered matching:
let layered = filter.find_all_layered("some long text");Streaming large files:
use std::fs::File;
use std::io::BufReader;
let reader = BufReader::new(File::open("large.txt")?);
let stream_results = filter.find_all_streaming(reader)?;Install with the cli feature:
[dependencies]
sensitive-rs = { version = "1.3.0", features = ["cli"] }Or install directly:
cargo install sensitive-rs --features cliBoth sensitive and sensitive-rs commands are available after installation.
# Find sensitive words
sensitive check "含有赌博和色情内容"
# Validate (exit 1 if sensitive words found)
sensitive validate "clean text"
# Replace sensitive words
sensitive replace '*' "含有赌博内容"
# Remove sensitive words
sensitive filter "含有赌博内容"
# Read from file
sensitive check --file input.txt
# Pipe from stdin
echo "text" | sensitive check--dict <path>— custom dictionary file--dict-all— use extended dictionary (27k words)--algorithm <algo>— force algorithm:aho-corasick,wumanber,regex--variant— enable pinyin and shape variant detection--noise-pattern <regex>— custom noise removal regex--json— JSON output format--color— force colored output
cargo run --example basic
cargo run --example batch
cargo run --example custom_dict
cargo run --example variant
cargo benchFor detailed documentation, please refer to Documentation.
Licensed under either of
- Apache License, Version 2.0, LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0
- MIT license LICENSE-MIT or http://opensource.org/licenses/MIT
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 or MIT license, shall be dual licensed as above, without any additional terms or conditions.
