Data formats

Canonical JSON

Canonical JSON (RFC 8785): one way to write a value, keys sorted and numbers as JavaScript writes them, so hashes and signatures of JSON hold however it was spaced or ordered.

Version 1.0.0 · 6 functions · 4 crates

#json#signing#checksums#webhooks

Functions

What use json_canonical.qwer gives a program, with the types Quillo sees. To take only some, or under other names: use jcs_canonical, jcs_canonical_text from json_canonical (Quillo 0.6).

jcs_canonical(value: any) → stror an error

value as canonical JSON text (RFC 8785): no spaces, map keys sorted by UTF-16 code units, numbers as JavaScript writes them (1.0 is 1, 1e30 is 1e+30), strings with only the escapes JSON needs. Integers past 2^53 are an error.

jcs_canonical_text(json: str) → stror an error

JSON text, as received, made canonical: the form to check a signature against. Numbers are read exactly as written (333333333.33333329 is 333333333.3333333, as in JavaScript).

jcs_sha256(value: any) → stror an error

The SHA-256 of the value's canonical form, in hex: the same for the same data however its JSON was written. For content ids, deduplication and detached signatures.

jcs_hmac(value: any, secret: str) → stror an error

An HMAC-SHA256 of the value's canonical form with secret, in hex: a signature that survives the JSON being re-spaced or its keys reordered on the way.

jcs_hmac_verify(value: any, secret: str, signature: str) → boolor an error

Whether signature (hex, from jcs_hmac) is the HMAC of the value's canonical form with secret, compared in constant time.

jcs_same(a: str, b: str) → boolor an error

Whether two JSON texts hold the same data, whatever their spacing, key order and number spelling (1.0 and 1, 1E2 and 100).

Get it

$ quillo install json_canonical               # json_canonical.qwer into this folder$ quillo install json_canonical --example     # and its example, json_canonical.qwe

Or with curl:

$ curl -O https://quillolang.org/plugins/files/json_canonical.qwer$ curl -O https://quillolang.org/plugins/files/json_canonical.qwe$ curl -O https://quillolang.org/plugins/files/json_canonical_sample.json

Then, in your program, all of it or only what you call:

main.qwe
use json_canonical.qweruse jcs_canonical, jcs_canonical_text from json_canonical      // Quillo 0.6: only these

It uses these crates, which cargo fetches the first time the plugin is built:

[dependencies]
serde_json = { version = "1", features = ["float_roundtrip"] }ryu-js = "1"sha2 = "0.10"hmac = "0.12"

Example

json_canonical.qwe
// Signed JSON between two services: the payload made canonical (RFC 8785) before it is hashed or signed, so// re-spacing or reordering on the way does not break the signature.////   quillo run json_canonical.qwe      then open http://localhost:8080/canonicaluse "json_canonical.qwer"let SECRET = env("PARTNER_SECRET") ?? "partner-shared-secret"// The RFC's own example, read as text and made canonical.route GET /canonical = respond(200, jcs_canonical_text(read_file("json_canonical_sample.json")?)?, { "content-type": "application/json" })// Keys ordered as JavaScript orders them (UTF-16), the RFC's sorting example.route GET /order = jcs_canonical_text("{\"\\u20ac\": \"Euro\", \"\\r\": \"CR\", \"\\ufb33\": \"Hebrew\", \"1\": \"One\", \"\\ud83d\\ude00\": \"Smiley\", \"\\u0080\": \"Control\", \"\\u00f6\": \"Latin\"}")?// A partner's webhook: the same order sent spaced and reordered still checks.route GET /signed {    let order = { id: "A-1042", total: 15.3, lines: [{ sku: "TEA-01", qty: 2 }], paid: true }    let sig = jcs_hmac(order, SECRET)?    let received = json.parse("{ \"paid\": true, \"lines\": [ { \"qty\": 2, \"sku\": \"TEA-01\" } ], \"total\": 15.30, \"id\": \"A-1042\" }")    {        canonical: jcs_canonical(order)?,        signature: sig,        received_checks: jcs_hmac_verify(received, SECRET, sig)?,        tampered_checks: jcs_hmac_verify(merge(received, { total: 1.53 }), SECRET, sig)?,        same: jcs_same("{\"a\": 1.0, \"b\": [1E2]}", "{\"b\":[100],\"a\":1}")?,        sha256: jcs_sha256(order)?,    }}
$ quillo run json_canonical.qwe$ curl 'localhost:8080/canonical'$ curl 'localhost:8080/order'$ curl 'localhost:8080/signed'

Source

json_canonical.qwer
//! Canonical JSON (RFC 8785, JCS): the one way to write a value, keys sorted and numbers as JavaScript writes them, so a signature or hash of JSON holds however it was spaced or ordered.//! version: 1.0.0//!//! [dependencies]//! serde_json = { version = "1", features = ["float_roundtrip"] }//! ryu-js = "1"//! sha2 = "0.10"//! hmac = "0.12"use hmac::{Hmac, Mac};use serde_json::Value;use sha2::{Digest, Sha256};/// Integers past 2^53 are not exact as JSON numbers in JavaScript, which JCS follows.const SAFE: i64 = 9_007_199_254_740_991;fn number(n: &serde_json::Number, out: &mut String) -> Result<(), String> {    if let Some(i) = n.as_i64() {        if i.abs() <= SAFE {            out.push_str(&i.to_string());            return Ok(());        }    }    if n.as_u64().is_some() || n.as_i64().is_some() {        return Err(format!("{} is past 2^53, where JSON numbers stop being exact: send it as text", n));    }    let f = n.as_f64().ok_or_else(|| format!("{} is not a number JSON can hold", n))?;    if !f.is_finite() {        return Err("NaN and infinity have no JSON form".into());    }    out.push_str(ryu_js::Buffer::new().format(f));    Ok(())}fn write(v: &Value, out: &mut String, depth: usize) -> Result<(), String> {    if depth > 512 {        return Err("nested more than 512 deep".into());    }    match v {        Value::Null => out.push_str("null"),        Value::Bool(b) => out.push_str(if *b { "true" } else { "false" }),        Value::Number(n) => number(n, out)?,        // serde_json escapes as JCS asks: \b \t \n \f \r \" \\, other controls as \u00xx, the rest as it is.        Value::String(s) => out.push_str(&serde_json::to_string(s).map_err(|e| e.to_string())?),        Value::Array(a) => {            out.push('[');            for (i, x) in a.iter().enumerate() {                if i > 0 {                    out.push(',');                }                write(x, out, depth + 1)?;            }            out.push(']');        }        Value::Object(m) => {            // Keys sorted by their UTF-16 code units, as JavaScript compares strings.            let mut keys: Vec<(&String, Vec<u16>)> = m.keys().map(|k| (k, k.encode_utf16().collect())).collect();            keys.sort_by(|a, b| a.1.cmp(&b.1));            out.push('{');            for (i, (k, _)) in keys.iter().enumerate() {                if i > 0 {                    out.push(',');                }                out.push_str(&serde_json::to_string(k).map_err(|e| e.to_string())?);                out.push(':');                write(&m[k.as_str()], out, depth + 1)?;            }            out.push('}');        }    }    Ok(())}fn canon(v: &Value) -> Result<String, String> {    let mut out = String::new();    write(v, &mut out, 0)?;    Ok(out)}fn parse(text: &str) -> Result<Value, String> {    serde_json::from_str(text.trim_start_matches('\u{feff}')).map_err(|e| format!("not JSON: {}", e))}fn hex(b: &[u8]) -> String {    b.iter().map(|x| format!("{:02x}", x)).collect()}fn mac(secret: &str, text: &str) -> Hmac<Sha256> {    let mut m = Hmac::<Sha256>::new_from_slice(secret.as_bytes()).expect("HMAC takes any key length");    m.update(text.as_bytes());    m}/// `value` as canonical JSON text (RFC 8785): no spaces, map keys sorted by UTF-16 code units, numbers as JavaScript/// writes them (1.0 is `1`, 1e30 is `1e+30`), strings with only the escapes JSON needs. Integers past 2^53 are an error.pub fn jcs_canonical(value: serde_json::Value) -> Result<String, String> {    canon(&value)}/// JSON text, as received, made canonical: the form to check a signature against. Numbers are read exactly as written/// (`333333333.33333329` is `333333333.3333333`, as in JavaScript).pub fn jcs_canonical_text(json: &str) -> Result<String, String> {    canon(&parse(json)?)}/// The SHA-256 of the value's canonical form, in hex: the same for the same data however its JSON was written. For/// content ids, deduplication and detached signatures.pub fn jcs_sha256(value: serde_json::Value) -> Result<String, String> {    Ok(hex(&Sha256::digest(canon(&value)?.as_bytes())))}/// An HMAC-SHA256 of the value's canonical form with `secret`, in hex: a signature that survives the JSON being/// re-spaced or its keys reordered on the way.pub fn jcs_hmac(value: serde_json::Value, secret: &str) -> Result<String, String> {    Ok(hex(&mac(secret, &canon(&value)?).finalize().into_bytes()))}/// Whether `signature` (hex, from `jcs_hmac`) is the HMAC of the value's canonical form with `secret`, compared in constant time.pub fn jcs_hmac_verify(value: serde_json::Value, secret: &str, signature: &str) -> Result<bool, String> {    let text = canon(&value)?;    let s = signature.trim();    if s.len() % 2 != 0 {        return Ok(false);    }    let bytes: Option<Vec<u8>> = (0..s.len()).step_by(2).map(|i| u8::from_str_radix(s.get(i..i + 2)?, 16).ok()).collect();    Ok(bytes.is_some_and(|b| mac(secret, &text).verify_slice(&b).is_ok()))}/// Whether two JSON texts hold the same data, whatever their spacing, key order and number spelling (`1.0` and `1`, `1E2` and `100`).pub fn jcs_same(a: &str, b: &str) -> Result<bool, String> {    Ok(canon(&parse(a)?)? == canon(&parse(b)?)?)}
Thread
75Useful? Like it, or add to its thread below.

Thread 0 posts

On Canonical JSON: questions, tips, corrections, and code that uses it.

No posts yet. The first one opens the thread.