Snowflake ids
Snowflake ids: 64-bit numbers that sort by time, unique across 1,024 servers with no coordination, with their time, server and sequence read back and the first id of a time for range queries.
Version 1.0.0 · 3 functions · 0 crates
Functions
What use snowflake.qwer gives a program, with the types Quillo sees. To take only some, or under other names: use snowflake, snowflake_parse from snowflake (Quillo 0.6).
snowflake(worker: int) → intor an errorA new id from server worker (0 to 1023; give each server or process its own). Ids from one server only grow, even when its clock steps back (they then run ahead of it); 4,096 can be made each millisecond, then it waits for the next. Store them as 64-bit integers; send them to browsers as text, since JavaScript numbers lose digits past 2^53.
snowflake_parse(id: int) → anyor an errorWhat an id holds: { time_ms, time, worker, sequence }, time_ms in milliseconds since 1970 (as now() gives) and time the same as ISO 8601 UTC text.
snowflake_at(time_ms: int) → intor an errorThe smallest id of a time in milliseconds since 1970: where id >= snowflake_at(t) finds the rows made since t, with no index on a date column.
Get it
$ quillo install snowflake # snowflake.qwer into this folder$ quillo install snowflake --example # and its example, snowflake.qweOr with curl:
$ curl -O https://quillolang.org/plugins/files/snowflake.qwer$ curl -O https://quillolang.org/plugins/files/snowflake.qweThen, in your program, all of it or only what you call:
use snowflake.qweruse snowflake, snowflake_parse from snowflake // Quillo 0.6: only theseIt uses these crates, which cargo fetches the first time the plugin is built:
Example
// Ids for messages made on several servers at once, sortable by time, and "since yesterday" without a date column.//// quillo run snowflake.qwe then open http://localhost:8080/idsuse "snowflake.qwer"let WORKER = int(env("WORKER_ID") ?? "7")route GET /ids { let ids = range(0, 3) | map((i) => snowflake(WORKER)?) { ids: ids | map((i) => str(i)), sorted: sort(ids) == ids, first: snowflake_parse(ids[0])? }}route GET /since = { yesterday: str(snowflake_at(now() - 86400000)?), fixed: snowflake_parse(snowflake_at(1767225600000)?)? }$ quillo run snowflake.qwe$ curl 'localhost:8080/ids'$ curl 'localhost:8080/since'Source
//! Snowflake ids: 64-bit numbers that sort by the time they were made, unique across up to 1,024 servers with no coordination, 4,096 a millisecond each; and the time, server and sequence read back.//! version: 1.0.0//!//! [dependencies]use serde_json::{json, Value};use std::sync::Mutex;/// Ids count milliseconds from 2020-01-01 UTC: 41 bits last until 2089.const EPOCH_MS: i64 = 1_577_836_800_000;const WORKER_BITS: i64 = 10;const SEQ_BITS: i64 = 12;static STATE: Mutex<(i64, i64)> = Mutex::new((0, 0));fn now_ms() -> i64 { std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_millis() as i64)}/// A new id from server `worker` (0 to 1023; give each server or process its own). Ids from one server only grow, even when its clock steps back (they then run ahead of it); 4,096 can be made each millisecond, then it waits for the next. Store them as 64-bit integers; send them to browsers as text, since JavaScript numbers lose digits past 2^53.pub fn snowflake(worker: i64) -> Result<i64, String> { if !(0..1024).contains(&worker) { return Err("worker is from 0 to 1023".into()); } let mut st = STATE.lock().unwrap_or_else(|e| e.into_inner()); let (last, seq) = *st; let mut t = now_ms().max(last); let mut s = if t == last { seq + 1 } else { 0 }; if s >= 1 << SEQ_BITS { // This millisecond is used up: wait for the next one. while now_ms() <= last { std::thread::yield_now(); } t = now_ms().max(last + 1); s = 0; } *st = (t, s); Ok(((t - EPOCH_MS) << (WORKER_BITS + SEQ_BITS)) | (worker << SEQ_BITS) | s)}/// What an id holds: `{ time_ms, time, worker, sequence }`, `time_ms` in milliseconds since 1970 (as `now()` gives) and `time` the same as ISO 8601 UTC text.pub fn snowflake_parse(id: i64) -> Result<Value, String> { if id < 0 { return Err("snowflake ids are not negative".into()); } let ms = (id >> (WORKER_BITS + SEQ_BITS)) + EPOCH_MS; let secs = ms.div_euclid(1000); let (days, rem) = (secs.div_euclid(86400), secs.rem_euclid(86400)); // Days since 1970 to a civil date (Howard Hinnant's algorithm). let z = days + 719468; let era = z.div_euclid(146097); let doe = z - era * 146097; let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365; let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); let mp = (5 * doy + 2) / 153; let d = doy - (153 * mp + 2) / 5 + 1; let m = if mp < 10 { mp + 3 } else { mp - 9 }; let y = yoe + era * 400 + if m <= 2 { 1 } else { 0 }; let time = format!("{:04}-{:02}-{:02}T{:02}:{:02}:{:02}.{:03}Z", y, m, d, rem / 3600, rem % 3600 / 60, rem % 60, ms.rem_euclid(1000)); Ok(json!({ "time_ms": ms, "time": time, "worker": (id >> SEQ_BITS) & ((1 << WORKER_BITS) - 1), "sequence": id & ((1 << SEQ_BITS) - 1), }))}/// The smallest id of a time in milliseconds since 1970: `where id >= snowflake_at(t)` finds the rows made since `t`, with no index on a date column.pub fn snowflake_at(time_ms: i64) -> Result<i64, String> { let t = time_ms - EPOCH_MS; if !(0..(1i64 << 41)).contains(&t) { return Err("times from 2020-01-01 to 2089".into()); } Ok(t << (WORKER_BITS + SEQ_BITS))}Related
ksuid.qwer · 1.0.0KSUIDs, Segment's K-Sortable Unique IDs: 27 base62 characters sorted by the second they were made, with 128 random bits; made, read back and checked.8 views · 7 downloadsCUID2 cuid2.qwer · 1.0.0CUID2 ids: secure, collision-resistant, starting with a letter, of the length you choose, that give away neither the time nor a count; and a check that a string is one.7 views · 7 downloadsShort ids short_ids.qwer · 1.0.0Short ids for URLs made from your database's numbers and read back (Sqids): your own alphabet, a minimum length, no rude words, and typos never reaching another row.6 views · 8 downloadsTypeID typeid.qwer · 1.0.0TypeIDs such as user_01h455vb4pex5vsknk084sn02q: a type prefix and a UUIDv7 in base32, made, read back to their UUID and time, and checked against the type a URL expects.6 views · 7 downloadsULIDs ulid.qwer · 1.0.0Unique ids that sort by when they were made: 26 characters, safe in URLs, good primary keys.8 views · 8 downloadsUUID tools uuid_tools.qwer · 1.0.0UUIDs beyond the random uuid(): version 7 sorted by time for database keys, versions 5 and 3 that give the same id for the same name, and any UUID read with its version and time.7 views · 6 downloadsThread 0 posts
On Snowflake ids: questions, tips, corrections, and code that uses it.
No posts yet. The first one opens the thread.