Text

Right-to-left text

Right-to-left text done right: a text's direction, mixed Arabic or Hebrew text put in display order for printers, terminals and images, user text isolated so it cannot flip a page, and Trojan Source direction controls found and removed.

Version 1.0.0 · 5 functions · 2 crates

#unicode#languages#security#printing

Functions

What use bidi.qwer gives a program, with the types Quillo sees. To take only some, or under other names: use bidi_direction, bidi_visual from bidi (Quillo 0.6).

bidi_direction(text: str) → any

The direction of text: "rtl" or "ltr" by its first strong letter (as dir="auto" decides), "neutral" when it has none (digits and punctuation only); { direction, has_rtl, has_ltr, mixed }.

bidi_visual(text: str, base: str) → stror an error

text in display order, line by line (the Unicode Bidirectional Algorithm), for output that shows characters left to right as stored: thermal printers, terminals, text drawn on images. base is "auto" (each paragraph by its first strong letter), "ltr" or "rtl".

bidi_isolate(text: str) → str

text wrapped in a first-strong isolate (U+2068 … U+2069) with any controls inside removed: a user's name or a product title in Arabic or Hebrew put into a sentence without turning the words around it. (In HTML, <bdi> does the same.)

bidi_strip(text: str) → str

text without the invisible direction controls (embeddings, overrides, isolates and marks): for usernames, file names and code, where they only disguise.

bidi_check(text: str) → any

The direction controls in text, "Trojan Source" style: { safe, controls: [{ position, code, name }], unbalanced }; unbalanced when an embedding, override or isolate is never closed, which lets it flip what follows. safe when there are no overrides, embeddings or isolates (lone marks are allowed).

Get it

$ quillo install bidi               # bidi.qwer into this folder$ quillo install bidi --example     # and its example, bidi.qwe

Or with curl:

$ curl -O https://quillolang.org/plugins/files/bidi.qwer$ curl -O https://quillolang.org/plugins/files/bidi.qwe

Then, in your program, all of it or only what you call:

main.qwe
use bidi.qweruse bidi_direction, bidi_visual from bidi      // Quillo 0.6: only these

It uses these crates, which cargo fetches the first time the plugin is built:

[dependencies]
unicode-bidi = "=0.3.18"serde_json = { version = "1", features = ["preserve_order"] }

Example

bidi.qwe
// A marketplace with sellers writing in Arabic and Hebrew: the page direction of each listing, names isolated in English sentences, receipts for a printer with no bidi, and usernames checked for hidden direction tricks.////   quillo run bidi.qwe      then open http://localhost:8080/bidiuse "bidi.qwer"route GET /bidi {    {        arabic: bidi_direction("مرحبا بالعالم"),        english: bidi_direction("Hello, world").direction,        mixed: bidi_direction("שלום world"),        digits: bidi_direction("123-456").direction,        // "Order 15 for עלמה": in memory the Hebrew letters come first to last; a printer must get them reversed.        receipt: bidi_visual("Order 15 for עלמה", "ltr")?,        hebrew_line: bidi_visual("abc שלום", "rtl")?,        isolated: replace(replace(bidi_isolate("עלמה"), "\u{2068}", "[FSI]"), "\u{2069}", "[PDI]"),        // A file name that shows as "invoice‮fdp.exe" reads "invoiceexe.pdf" on screen.        trick: bidi_check("invoice\u{202E}fdp.exe"),        cleaned: bidi_strip("invoice\u{202E}fdp.exe"),        plain: bidi_check("Plain name").safe,    }}route GET /listing ? { title: str = "שולחן עץ" } {    let dir = bidi_direction(title).direction    <html><body>        <h1 dir={dir == "rtl" ? "rtl" : "ltr"}>{title}</h1>        <p>Sold by <bdi>{title}</bdi> in 3 days.</p>    </body></html>}
$ quillo run bidi.qwe$ curl 'localhost:8080/bidi'

Source

bidi.qwer
//! Right-to-left text done right: a text's direction (Arabic, Hebrew, Persian, Urdu...), mixed text put in display order for places with no bidi support (receipt printers, terminals, images, PDFs), user text isolated so it cannot flip a page's layout, and the invisible direction controls of "Trojan Source" attacks found and removed.//! version: 1.0.0//!//! [dependencies]//! unicode-bidi = "=0.3.18"//! serde_json = { version = "1", features = ["preserve_order"] }use serde_json::{json, Value};use unicode_bidi::{bidi_class, BidiClass, BidiInfo, Level};const CONTROLS: [(char, &str); 12] = [    ('\u{202A}', "LEFT-TO-RIGHT EMBEDDING"), ('\u{202B}', "RIGHT-TO-LEFT EMBEDDING"), ('\u{202C}', "POP DIRECTIONAL FORMATTING"),    ('\u{202D}', "LEFT-TO-RIGHT OVERRIDE"), ('\u{202E}', "RIGHT-TO-LEFT OVERRIDE"), ('\u{2066}', "LEFT-TO-RIGHT ISOLATE"),    ('\u{2067}', "RIGHT-TO-LEFT ISOLATE"), ('\u{2068}', "FIRST STRONG ISOLATE"), ('\u{2069}', "POP DIRECTIONAL ISOLATE"),    ('\u{200E}', "LEFT-TO-RIGHT MARK"), ('\u{200F}', "RIGHT-TO-LEFT MARK"), ('\u{061C}', "ARABIC LETTER MARK"),];fn is_rtl_class(c: BidiClass) -> bool { matches!(c, BidiClass::R | BidiClass::AL) }/// The direction of `text`: `"rtl"` or `"ltr"` by its first strong letter (as `dir="auto"` decides), `"neutral"` when it has none (digits and punctuation only); `{ direction, has_rtl, has_ltr, mixed }`.pub fn bidi_direction(text: &str) -> Value {    let mut first = None;    let (mut r, mut l) = (false, false);    for c in text.chars() {        let cls = bidi_class(c);        if is_rtl_class(cls) { r = true; if first.is_none() { first = Some("rtl") } }        else if cls == BidiClass::L { l = true; if first.is_none() { first = Some("ltr") } }    }    json!({ "direction": first.unwrap_or("neutral"), "has_rtl": r, "has_ltr": l, "mixed": r && l })}/// `text` in display order, line by line (the Unicode Bidirectional Algorithm), for output that shows characters left to right as stored: thermal printers, terminals, text drawn on images./// `base` is `"auto"` (each paragraph by its first strong letter), `"ltr"` or `"rtl"`.pub fn bidi_visual(text: &str, base: &str) -> Result<String, String> {    let level = match base.trim().to_lowercase().as_str() { "" | "auto" => None, "ltr" => Some(Level::ltr()), "rtl" => Some(Level::rtl()), b => return Err(format!("base is auto, ltr or rtl, not `{}`", b)) };    let mut out = Vec::new();    for line in text.split('\n') {        let info = BidiInfo::new(line, level);        let mut s = String::new();        for para in &info.paragraphs { s.push_str(&info.reorder_line(para, para.range.clone())) }        out.push(s);    }    Ok(out.join("\n"))}/// `text` wrapped in a first-strong isolate (U+2068 … U+2069) with any controls inside removed: a user's name or a product title in Arabic or Hebrew put into a sentence without turning the words around it. (In HTML, `<bdi>` does the same.)pub fn bidi_isolate(text: &str) -> String {    format!("\u{2068}{}\u{2069}", bidi_strip(text))}/// `text` without the invisible direction controls (embeddings, overrides, isolates and marks): for usernames, file names and code, where they only disguise.pub fn bidi_strip(text: &str) -> String {    text.chars().filter(|c| !CONTROLS.iter().any(|(x, _)| x == c)).collect()}/// The direction controls in `text`, "Trojan Source" style: `{ safe, controls: [{ position, code, name }], unbalanced }`; `unbalanced` when an embedding, override or isolate is never closed, which lets it flip what follows. `safe` when there are no overrides, embeddings or isolates (lone marks are allowed).pub fn bidi_check(text: &str) -> Value {    let mut found = Vec::new();    let (mut emb, mut iso) = (0i64, 0i64);    let mut dangerous = false;    for (i, c) in text.chars().enumerate() {        if let Some((_, name)) = CONTROLS.iter().find(|(x, _)| *x == c) {            found.push(json!({ "position": i, "code": format!("U+{:04X}", c as u32), "name": name }));            match c {                '\u{202A}'..='\u{202B}' | '\u{202D}'..='\u{202E}' => { emb += 1; dangerous = true }                '\u{202C}' => emb -= 1,                '\u{2066}'..='\u{2068}' => { iso += 1; dangerous = true }                '\u{2069}' => iso -= 1,                _ => {}            }        }        if c == '\n' { if emb != 0 || iso != 0 { dangerous = true } }    }    json!({ "safe": !dangerous, "controls": found, "unbalanced": emb != 0 || iso != 0 })}
Japanese kana kana.qwer · 1.0.0Japanese kana: romaji typed as hiragana or katakana, kana written in Hepburn romaji, hiragana and katakana each to the other, half-width katakana from bank and till systems made full-width, and which scripts a text uses.unicodelanguagesformstext6 views · 10 downloadsBraille braille.qwer · 1.0.0Unicode braille from English text: grade 1 letter by letter, and Unified English Braille grade 2 with its wordsigns, groupsigns, positional rules and shortforms; read back cell by cell where the cells allow it; dot numbers and the Braille ASCII of `.brf` files both ways.brailleaccessibilityunicodeembosser3 views · 6 downloadsChinese words chinese_segment.qwer · 1.0.0Chinese text split into words for search indexes and counts, with parts of speech, and written in pinyin with tones, tone numbers or initials for sorting and finding names.chinesesearchtextlanguages7 views · 7 downloadsEmoji emoji.qwer · 1.0.0Emoji for chat and comments: :shortcodes: turned into emoji and back, emoji found with their names, stripped from subjects, and searched by name for a picker.emojichattextunicode7 views · 8 downloadsHyphenation hyphenation.qwer · 1.0.0Words hyphenated by the rules of 80 languages and spellings, patterns built in: a word's syllables, soft hyphens put in text and HTML for narrow columns and e-mails, and text wrapped at a width with words broken where they may be.textlanguagesemailwriting6 views · 7 downloadsNumbers in words number_words.qwer · 1.0.0Numbers and amounts of money in words for cheques, invoices and contracts, in English, French and Ukrainian, with ordinals, years and Roman numerals both ways.invoicesmoneytextlanguages8 views · 7 downloads
Thread
57Useful? Like it, or add to its thread below.

Thread 0 posts

On Right-to-left text: questions, tips, corrections, and code that uses it.

No posts yet. The first one opens the thread.