Time and IDs

Snowflake ids

Snowflake ids: 64-bit numbers that sort by time, unique across 1,024 servers with no coordination, with their time, server and sequence read back and the first id of a time for range queries.

Version 1.0.0 · 3 functions · 0 crates

#ids#database#distributed#time

Functions

What use snowflake.qwer gives a program, with the types Quillo sees. To take only some, or under other names: use snowflake, snowflake_parse from snowflake (Quillo 0.6).

snowflake(worker: int) → intor an error

A new id from server worker (0 to 1023; give each server or process its own). Ids from one server only grow, even when its clock steps back (they then run ahead of it); 4,096 can be made each millisecond, then it waits for the next. Store them as 64-bit integers; send them to browsers as text, since JavaScript numbers lose digits past 2^53.

snowflake_parse(id: int) → anyor an error

What an id holds: { time_ms, time, worker, sequence }, time_ms in milliseconds since 1970 (as now() gives) and time the same as ISO 8601 UTC text.

snowflake_at(time_ms: int) → intor an error

The smallest id of a time in milliseconds since 1970: where id >= snowflake_at(t) finds the rows made since t, with no index on a date column.

Get it

$ quillo install snowflake               # snowflake.qwer into this folder$ quillo install snowflake --example     # and its example, snowflake.qwe

Or with curl:

$ curl -O https://quillolang.org/plugins/files/snowflake.qwer$ curl -O https://quillolang.org/plugins/files/snowflake.qwe

Then, in your program, all of it or only what you call:

main.qwe
use snowflake.qweruse snowflake, snowflake_parse from snowflake      // Quillo 0.6: only these

It uses these crates, which cargo fetches the first time the plugin is built:

[dependencies]

Example

snowflake.qwe
// Ids for messages made on several servers at once, sortable by time, and "since yesterday" without a date column.////   quillo run snowflake.qwe      then open http://localhost:8080/idsuse "snowflake.qwer"let WORKER = int(env("WORKER_ID") ?? "7")route GET /ids {    let ids = range(0, 3) | map((i) => snowflake(WORKER)?)    { ids: ids | map((i) => str(i)), sorted: sort(ids) == ids, first: snowflake_parse(ids[0])? }}route GET /since = { yesterday: str(snowflake_at(now() - 86400000)?), fixed: snowflake_parse(snowflake_at(1767225600000)?)? }
$ quillo run snowflake.qwe$ curl 'localhost:8080/ids'$ curl 'localhost:8080/since'

Source

snowflake.qwer
//! Snowflake ids: 64-bit numbers that sort by the time they were made, unique across up to 1,024 servers with no coordination, 4,096 a millisecond each; and the time, server and sequence read back.//! version: 1.0.0//!//! [dependencies]use serde_json::{json, Value};use std::sync::Mutex;/// Ids count milliseconds from 2020-01-01 UTC: 41 bits last until 2089.const EPOCH_MS: i64 = 1_577_836_800_000;const WORKER_BITS: i64 = 10;const SEQ_BITS: i64 = 12;static STATE: Mutex<(i64, i64)> = Mutex::new((0, 0));fn now_ms() -> i64 {    std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_millis() as i64)}/// A new id from server `worker` (0 to 1023; give each server or process its own). Ids from one server only grow, even when its clock steps back (they then run ahead of it); 4,096 can be made each millisecond, then it waits for the next. Store them as 64-bit integers; send them to browsers as text, since JavaScript numbers lose digits past 2^53.pub fn snowflake(worker: i64) -> Result<i64, String> {    if !(0..1024).contains(&worker) {        return Err("worker is from 0 to 1023".into());    }    let mut st = STATE.lock().unwrap_or_else(|e| e.into_inner());    let (last, seq) = *st;    let mut t = now_ms().max(last);    let mut s = if t == last { seq + 1 } else { 0 };    if s >= 1 << SEQ_BITS {        // This millisecond is used up: wait for the next one.        while now_ms() <= last {            std::thread::yield_now();        }        t = now_ms().max(last + 1);        s = 0;    }    *st = (t, s);    Ok(((t - EPOCH_MS) << (WORKER_BITS + SEQ_BITS)) | (worker << SEQ_BITS) | s)}/// What an id holds: `{ time_ms, time, worker, sequence }`, `time_ms` in milliseconds since 1970 (as `now()` gives) and `time` the same as ISO 8601 UTC text.pub fn snowflake_parse(id: i64) -> Result<Value, String> {    if id < 0 {        return Err("snowflake ids are not negative".into());    }    let ms = (id >> (WORKER_BITS + SEQ_BITS)) + EPOCH_MS;    let secs = ms.div_euclid(1000);    let (days, rem) = (secs.div_euclid(86400), secs.rem_euclid(86400));    // Days since 1970 to a civil date (Howard Hinnant's algorithm).    let z = days + 719468;    let era = z.div_euclid(146097);    let doe = z - era * 146097;    let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);    let mp = (5 * doy + 2) / 153;    let d = doy - (153 * mp + 2) / 5 + 1;    let m = if mp < 10 { mp + 3 } else { mp - 9 };    let y = yoe + era * 400 + if m <= 2 { 1 } else { 0 };    let time = format!("{:04}-{:02}-{:02}T{:02}:{:02}:{:02}.{:03}Z", y, m, d, rem / 3600, rem % 3600 / 60, rem % 60, ms.rem_euclid(1000));    Ok(json!({        "time_ms": ms,        "time": time,        "worker": (id >> SEQ_BITS) & ((1 << WORKER_BITS) - 1),        "sequence": id & ((1 << SEQ_BITS) - 1),    }))}/// The smallest id of a time in milliseconds since 1970: `where id >= snowflake_at(t)` finds the rows made since `t`, with no index on a date column.pub fn snowflake_at(time_ms: i64) -> Result<i64, String> {    let t = time_ms - EPOCH_MS;    if !(0..(1i64 << 41)).contains(&t) {        return Err("times from 2020-01-01 to 2089".into());    }    Ok(t << (WORKER_BITS + SEQ_BITS))}
Thread
77Useful? Like it, or add to its thread below.

Thread 0 posts

On Snowflake ids: questions, tips, corrections, and code that uses it.

No posts yet. The first one opens the thread.