Disk cache
A disk cache for slow answers: text under a key with a lifetime, stored by content hash and checked when read.
Version 1.0.0 · 6 functions · 1 crate
Functions
What use cache_disk.qwer gives a program, with the types Quillo sees. To take only some, or under other names: use disk_cache_put, disk_cache_get from cache_disk (Quillo 0.6).
disk_cache_put(dir: str, key: str, value: str, ttl_seconds: int) → stror an errorKeeps value (text) under key in the cache folder dir (made if missing), for ttl_seconds (0: until removed). Gives the content's hash (sha256-...).
disk_cache_get(dir: str, key: str) → str?or an errorThe text under key, or null when it is missing or expired. The content is checked against its hash: a damaged file is an error, not wrong data.
disk_cache_info(dir: str, key: str) → anyor an errorAbout an entry, without reading it: { key, hash, size, written, expires } (times in Unix milliseconds, expires 0 for never), or null.
disk_cache_remove(dir: str, key: str) → nullor an errorForgets key (its content stays until disk_cache_clear, as other keys may share it).
disk_cache_clear(dir: str) → nullor an errorEmpties the whole cache folder.
disk_cache_keys(dir: str, limit: int) → listor an errorThe keys in the cache that are not expired, at most limit (1 to 10,000).
Get it
$ quillo install cache_disk # cache_disk.qwer into this folder$ quillo install cache_disk --example # and its example, cache_disk.qweOr with curl:
$ curl -O https://quillolang.org/plugins/files/cache_disk.qwer$ curl -O https://quillolang.org/plugins/files/cache_disk.qweThen, in your program, all of it or only what you call:
use cache_disk.qweruse disk_cache_put, disk_cache_get from cache_disk // Quillo 0.6: only theseIt uses this crate, which cargo fetches the first time the plugin is built:
cacache = { version = "13.1", default-features = false, features = ["tokio-runtime"] }Example
// A slow report cached on disk for a minute: the first request makes it, the next ones read it.//// quillo run cache_disk.qwe then open http://localhost:8080/reportuse "cache_disk.qwer"let CACHE = "data/cache"fn slow_report() = json.stringify({ total: sum(range(1, 100001)), made: str(now()) })route GET /report { let hit = disk_cache_get(CACHE, "report:daily")? if hit != null { return { from: "cache", report: json.parse(hit) } } disk_cache_put(CACHE, "report:daily", slow_report(), 60)? { from: "made", report: json.parse(disk_cache_get(CACHE, "report:daily")?) }}route GET /info = disk_cache_info(CACHE, "report:daily")?route GET /flush { disk_cache_put(CACHE, "short", "gone soon", 1)? disk_cache_remove(CACHE, "report:daily")? { keys: disk_cache_keys(CACHE, 100)?, report: disk_cache_get(CACHE, "report:daily")? }}$ quillo run cache_disk.qwe$ curl 'localhost:8080/report'$ curl 'localhost:8080/flush'Source
//! A disk cache for slow answers (API calls, rendered pages, thumbnails as base64): text kept under a key with a lifetime, stored by content hash so equal values are kept once and checked on read.//! version: 1.0.0//!//! [dependencies]//! cacache = { version = "13.1", default-features = false, features = ["tokio-runtime"] }use serde_json::{json, Value};const MAX_VALUE: usize = 64 << 20;fn now_ms() -> u128 { std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis()).unwrap_or(0)}/// The entry's metadata when it is there and its time is not up.fn fresh(dir: &str, key: &str) -> Result<Option<cacache::Metadata>, String> { let Some(m) = cacache::metadata_sync(dir, key).map_err(|e| e.to_string())? else { return Ok(None) }; let exp = m.metadata.get("expires").and_then(Value::as_u64).unwrap_or(0) as u128; if exp > 0 && exp <= now_ms() { return Ok(None); } Ok(Some(m))}/// Keeps `value` (text) under `key` in the cache folder `dir` (made if missing), for `ttl_seconds` (0: until removed). Gives the content's hash (`sha256-...`).pub fn disk_cache_put(dir: &str, key: &str, value: &str, ttl_seconds: i64) -> Result<String, String> { if value.len() > MAX_VALUE { return Err("a cached value is at most 64 MB".into()); } let expires = if ttl_seconds > 0 { now_ms() as u64 + ttl_seconds as u64 * 1000 } else { 0 }; let mut w = cacache::WriteOpts::new().metadata(json!({ "expires": expires })).open_sync(dir, key).map_err(|e| e.to_string())?; std::io::Write::write_all(&mut w, value.as_bytes()).map_err(|e| e.to_string())?; w.commit().map(|i| i.to_string()).map_err(|e| e.to_string())}/// The text under `key`, or null when it is missing or expired. The content is checked against its hash: a damaged file is an error, not wrong data.pub fn disk_cache_get(dir: &str, key: &str) -> Result<Option<String>, String> { if fresh(dir, key)?.is_none() { return Ok(None); } let bytes = cacache::read_sync(dir, key).map_err(|e| e.to_string())?; String::from_utf8(bytes).map(Some).map_err(|_| "the cached value is not text".to_string())}/// About an entry, without reading it: `{ key, hash, size, written, expires }` (times in Unix milliseconds, `expires` 0 for never), or null.pub fn disk_cache_info(dir: &str, key: &str) -> Result<Value, String> { Ok(match fresh(dir, key)? { Some(m) => json!({ "key": m.key, "hash": m.integrity.to_string(), "size": m.size, "written": m.time as u64, "expires": m.metadata.get("expires").cloned().unwrap_or(json!(0)) }), None => Value::Null, })}/// Forgets `key` (its content stays until `disk_cache_clear`, as other keys may share it).pub fn disk_cache_remove(dir: &str, key: &str) -> Result<(), String> { match cacache::remove_sync(dir, key) { Ok(()) => Ok(()), Err(e) => Err(e.to_string()), }}/// Empties the whole cache folder.pub fn disk_cache_clear(dir: &str) -> Result<(), String> { cacache::clear_sync(dir).map_err(|e| e.to_string())}/// The keys in the cache that are not expired, at most `limit` (1 to 10,000).pub fn disk_cache_keys(dir: &str, limit: i64) -> Result<Vec<String>, String> { if !std::path::Path::new(dir).exists() { return Ok(vec![]); } let t = now_ms(); let mut out = Vec::new(); for m in cacache::list_sync(dir) { let m = m.map_err(|e| e.to_string())?; let exp = m.metadata.get("expires").and_then(Value::as_u64).unwrap_or(0) as u128; if exp == 0 || exp > t { out.push(m.key); } if out.len() >= limit.clamp(1, 10_000) as usize { break; } } out.sort(); Ok(out)}Related
kv.qwer · 1.0.0A fast key-value store in one file: values with an optional lifetime, atomic counters, and keys listed by prefix, kept across restarts.7 views · 6 downloadsPoloDB polodb.qwer · 1.0.0An embedded document database in one file with a MongoDB-like API: insert maps, find them with $gt and $in filters, update with $set and $inc.7 views · 8 downloadsFont subsets font_subset.qwer · 1.0.0TrueType fonts cut down to the characters a page or a PDF uses, as .ttf, as .woff for the web, or as a data URL for CSS, with the characters the font lacks.6 views · 7 downloadsLZ4 lz4.qwer · 1.0.0LZ4 compression of files and text in the standard .lz4 frame format: the fastest to pack and unpack, for caches, logs and data passed between services.6 views · 7 downloadsHashes hashes.qwer · 1.0.0MD5, SHA-1, SHA-2, SHA-3, BLAKE2, xxHash and CRC-32 of text and files, and Subresource Integrity values for script tags.7 views · 6 downloadsBackblaze B2 backblaze_b2.qwer · 1.0.0Keep files on Backblaze B2 through its native API: uploads with their SHA-1, large files in parts, upload URLs replaced when a pod is busy, download links and checked downloads, paged listings, and an expired token renewed without the program noticing.4 views · 6 downloadsThread 0 posts
On Disk cache: questions, tips, corrections, and code that uses it.
No posts yet. The first one opens the thread.