laussywikilapse
laussywikilapse is a python script that makes a time-lapse of this wiki filling itself in. It watches the two navigation boxes that promise the most and deliver the least—{{Months}}, which lists all 366 days of the year, and {{Years}}, which lists 1900 to 2039—and draws one frame per month showing which of those links are still red and which have become pages. Played back, one watches a decade of writing arrive: a red grid slowly turning blue, a day at a time.
Written with the help of Claude Opus 5 on 16 August (2026). Version 1.0.0.
The idea
It began as the sibling of an earlier toy that photographs my Google Scholar profile every day, and it was meant to be the same thing pointed at the wiki: a cron job, a screenshot, a year of patience before there is anything to watch. The second thought is the one worth keeping. Google publishes no history, so a citation count not caught on the day is lost for good and a daemon really is the only way; a wiki, on the other hand, is its history. Every page carries the date it was created and the size it was at every revision since, which means the entire series—every frame, back to the beginning—can be computed at any moment from what the database already knows.
So there is no daemon, no timer, nothing running in the background. There is one command, and it takes about four minutes:
python3 wiki_shot.py && ./make_video.sh
That this is possible at all is not a small convenience. It means the layout can be changed and the whole past redrawn in the new design; it means a cadence of one frame per day, month or year is a command-line flag rather than a decision one is stuck with; and it means nothing was lost by not having thought of the idea five years ago.
The two templates
| template | what a cell is | entries today | written | series begins |
|---|---|---|---|---|
| {{Months}} | one day of the year | 368 | 28 (7.6%) | February 2016 |
| {{Years}} | one year | 140 | 21 (15.0%) | January 2014 |
The 368 cells of {{Months}} are the 366 days plus February 30 and February 31, which live in the template's Hypertime row; the two Wikipedia links beside them are shown greyed out and left out of the count, since an external link can never turn blue.
Anything built on {{Navbox}} works—the grid is read straight out of its groupN and listN parameters—so pointing it at another template is a line in targets.json. It only ever reads: nothing here writes to the wiki.
Reading a frame
| meaning | |
|---|---|
| pink cell | still a redlink |
| blue cell | the page exists; the deeper the blue, the more is written |
| gold ring | created since the previous frame—the footer names them |
| grey cell | an external link, which can never turn blue |
| row label | itself a page where the template links one, coloured the same way |
Blue saturates at 4 kB and the scale is fixed rather than relative to the biggest page of the day, so a colour means the same thing in every frame of the series: a cell that darkens is a page that grew, not a page that was overtaken.
The chart at the top right is the whole series to that point. It is scaled to the curve and not to the total, deliberately: against 140 possible entries a handful of pages is a flat line lying on the floor, and the shape of the growth is the one thing the big number to its left cannot show. Its top only steps on a 1–2–5 boundary, so it does not rescale from one frame to the next.
The grid grows too
The nicest thing the wiki's memory buys is that the template has a history as well, and it is read alongside the pages. Each frame is therefore drawn in the grid the navbox actually had that day. {{Years}} has been edited five times since 2018 and its first version ran only from 2005 to 2019, so the frames from 2014 are fifteen cells wide and read 1 / 15; the grid then widens under the growing blue as the template is extended, first backwards to 1900, then forwards to 2039. {{Months}} has four revisions and has been the full calendar throughout.
What it cannot see
Two things, and it is worth knowing which of them bites.
A page created and later deleted leaves no trace in this reconstruction, since the query that finds it asks only about pages that exist now. That is the real gap—so I checked it rather than warning about it: across 5399 entries of the delete and move logs, the only tracked titles ever deleted are 2097, 2098 and 2099, which are outside the range {{Years}} covers. For these two templates the reconstruction is therefore complete, with nothing missing.
The other is benign. For dates before the navbox itself existed the earliest revision is used as the grid, since pages predating the template are worth showing and there is no other grid to show them in—which is why the {{Years}} series can start in 2014, four years before the template was written.
Drawn, not screenshotted
The frames are not pictures of the rendered template. They are drawn here, from the wikitext parsed out of each revision, and that is what makes the historical grids possible at all: there is no way to photograph how {{Years}} looked in 2016. It also puts the geometry beyond the reach of the skin, the navbox CSS and the collapse state, none of which should be allowed to break the alignment of a series meant to run for years.
Alignment is the whole game, so the grid fits itself to however many rows and columns the template has and the canvas never changes size: a revision that adds a row makes the cells slightly smaller rather than making the video refuse to assemble. A row whose labels do not fit its column takes two or three columns where the row is short enough to spare them, which is what keeps the Hypertime row from reading Februa.
Two details, both learned the hard way on the Google Scholar sibling, keep successive renders byte-identical: every vertical position is an integer sum of integers—a half-pixel offset from align-items:center rounds differently between runs and jitters the frame by a pixel—and Chrome is given 200 px of slack below the frame which is then cropped off, because content ending near the bottom of the rasterised area comes out subtly different from one run to the next. Two full rebuilds of the 127 {{Months}} frames give 127 identical files. Nothing shimmers.
Usage
python3 wiki_shot.py # rebuild everything, one frame a month
python3 wiki_shot.py --only years # just one target
python3 wiki_shot.py --cadence daily # or yearly
python3 wiki_shot.py --since 2020-01-01 --until 2024-12-31
python3 wiki_shot.py --offline # re-render from the cache, no wiki
./make_video.sh # assemble, 6 frames a second
./make_video.sh 3 gif # slower, and write GIFs too
./make_video.sh 6 "" years # one target's video
--offline is the useful one while fiddling with the layout: wiki-data.json.gz holds every revision the series is built from, so changing the design and redrawing thirteen years of frames needs no wiki at all. To try a template that is not in the configuration: --page Template:X --slug x --title "X".
Everything lives in ~/scrap/timelapses/wiki/, the frames as targets/<slug>/frames/YYYY-MM-DD.png at 3120×2384 (drawn at 1560×1192 and rasterised at 2× for sharpness), the film beside them as timelapse.mp4, with totals.csv and entries.csv giving the numbers behind every frame. A frame costs some 330 kB, so both series together come to about 90 MB at a monthly cadence; a daily one over the same span would be 1.3 GB, which is why monthly is the default.
The Google Scholar sibling
The same machinery, in ~/scrap/timelapses/pub/, photographs six Google Scholar profiles once a day—mine, Elena's, Kavokin's, Baumberg's, Sanvitto's and Cirac's—drawing the top twenty publications and the citation histogram in the same fixed geometry. That one does need its daily timer, for the reason this one does not.
Source
#!/usr/bin/env python3
"""Timelapse of which entries of a wiki navbox have been written.
{{Months}} lists every day of the year and {{Years}} every year from 1900;
most of those links are still red. This draws one frame per month (or day, or
year) showing which ones have turned blue, from the first page ever written up
to today.
Nothing needs to run in the background: MediaWiki keeps the history, so the
whole series is rebuilt from scratch, correct to the day, whenever you ask.
./wiki_shot.py # rebuild every target, monthly frames
./wiki_shot.py --only years # just one
./wiki_shot.py --cadence daily # one frame per day instead
./wiki_shot.py --offline # re-render from the cache, no wiki
./wiki_shot.py --page Template:X --slug x --title "X" # ad-hoc target
Each frame is drawn against the template **as it stood that day**: the wiki's
own revisions of the navbox are read too, so a row added in 2025 does not
appear in a 2016 frame.
"""
import argparse
import datetime as dt
import glob
import gzip
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import requests
import frame as frame_tpl
__version__ = "1.0.0"
BASE = os.path.dirname(os.path.abspath(__file__))
TARGETS_JSON = os.path.join(BASE, "targets.json")
API = "http://localhost/laussywiki/api.php"
SCALE = 2 # device pixel ratio; 2 keeps text crisp
BATCH = 50 # titles per API query for anonymous access
TOTALS_COLS = ["date", "total", "defined", "missing", "external", "bytes"]
ENTRY_COLS = ["date", "title", "size"]
def log(msg):
print(f"[{dt.datetime.now():%Y-%m-%d %H:%M:%S}] {msg}", flush=True)
class Target:
"""One tracked template and where its files live."""
def __init__(self, slug, page, title=None, subtitle=""):
self.slug, self.page = slug, page
self.title = title or page
self.subtitle = subtitle
self.dir = os.path.join(BASE, "targets", slug)
self.frames = os.path.join(self.dir, "frames")
self.cache = os.path.join(self.dir, "wiki-data.json.gz")
self.totals = os.path.join(self.dir, "totals.csv")
self.entries = os.path.join(self.dir, "entries.csv")
def make_dirs(self):
os.makedirs(self.frames, exist_ok=True)
def frame_path(self, date_str):
return os.path.join(self.frames, f"{date_str}.png")
def load_targets():
with open(TARGETS_JSON, encoding="utf-8") as f:
return [Target(t["slug"], t["page"], t.get("title"),
t.get("subtitle", "")) for t in json.load(f)]
# ------------------------------------------------------------------------ api
def api(**params):
params.setdefault("action", "query")
params.setdefault("format", "json")
params.setdefault("formatversion", "2")
r = requests.get(API, params=params, timeout=60)
r.raise_for_status()
data = r.json()
if "error" in data:
raise RuntimeError(f"API error: {data['error'].get('info')}")
return data
def template_revisions(page):
"""[(date, wikitext), ...] for every revision of the navbox, oldest first.
These templates change a handful of times a decade, so fetching the lot
with content costs nothing and lets each frame use the grid of its day.
"""
out, cont = [], {}
while True:
data = api(prop="revisions", titles=page, rvslots="main",
rvprop="timestamp|content", rvlimit="max", rvdir="newer",
**cont)
pages = data["query"]["pages"]
if not pages or pages[0].get("missing"):
raise RuntimeError(f"{page} does not exist on the wiki")
for r in pages[0].get("revisions", []):
out.append((r["timestamp"][:10], r["slots"]["main"]["content"]))
if "continue" not in data:
return out
cont = data["continue"]
def existing(titles):
"""The subset of titles that exist right now."""
out = []
for i in range(0, len(titles), BATCH):
data = api(titles="|".join(titles[i:i + BATCH]), prop="info")
q = data["query"]
# The API normalises titles ("january 1" -> "January 1"); map back so
# the caller can look up by exactly the string the template used.
back = {n["to"]: n["from"] for n in q.get("normalized", [])}
for p in q["pages"]:
if not p.get("missing"):
out.append(back.get(p["title"], p["title"]))
return out
def revision_history(title):
"""[(date, size), ...] for every revision of a page, oldest first."""
revs, cont = [], {}
while True:
data = api(prop="revisions", titles=title, rvprop="timestamp|size",
rvlimit="max", rvdir="newer", **cont)
pages = data["query"]["pages"]
if not pages or pages[0].get("missing"):
return []
for r in pages[0].get("revisions", []):
revs.append((r["timestamp"][:10], r.get("size", 0)))
if "continue" not in data:
return revs
cont = data["continue"]
# ---------------------------------------------------------------------- parse
def split_params(body):
"""Split a template call's body on its own top-level pipes.
Pipes inside nested templates, links and tags belong to those, not to the
navbox: {{*}} separators and [[January 1|1]] labels would otherwise cut
the parameters to shreds.
"""
parts, depth, buf = [], 0, []
i = 0
while i < len(body):
two = body[i:i + 2]
if two in ("{{", "[["):
depth += 1
buf.append(two)
i += 2
elif two in ("}}", "]]"):
depth = max(0, depth - 1)
buf.append(two)
i += 2
elif body[i] == "|" and depth == 0:
parts.append("".join(buf))
buf = []
i += 1
else:
buf.append(body[i])
i += 1
parts.append("".join(buf))
return parts
def parse_navbox(text):
"""Rows of a Navbox as [(label_wikitext, list_wikitext), ...].
Rows come out in the template's own order; the numbering has gaps in it
({{Years}} jumps from list12 to list14) so they are sorted by number
rather than by position.
"""
text = re.sub(r"<noinclude>.*?</noinclude>", "", text, flags=re.S)
text = re.sub(r"<!--.*?-->", "", text, flags=re.S)
start = text.find("{{")
if start < 0:
raise RuntimeError("no template call found on the page")
end = text.rfind("}}")
params = split_params(text[start + 2:end])[1:] # [0] is the name
groups, lists = {}, {}
for p in params:
m = re.match(r"\s*(group|list)(\d+)\s*=(.*)", p, flags=re.S)
if m:
(groups if m.group(1) == "group" else lists)[int(m.group(2))] = \
m.group(3).strip()
if not lists:
raise RuntimeError("navbox has no listN parameters")
return [(groups.get(n, ""), lists[n]) for n in sorted(lists)]
LINK_RE = re.compile(r"\[\[\s*([^\]|#<>{}]+?)\s*(?:\|\s*(.*?)\s*)?\]\]"
r"|\[(?:https?:)?//?[^\s\]]+\s+([^\]]+)\]")
def parse_entries(wt):
"""Entries of one navbox row, in order.
Internal links become tracked entries; external links (the wiki has a
couple pointing at Wikipedia) are shown greyed out and left out of the
count, since they can never turn blue.
"""
out = []
for m in LINK_RE.finditer(wt):
target, label, ext = m.group(1), m.group(2), m.group(3)
if target:
if target.split(":")[0] in ("File", "Image", "Category"):
continue # decoration, not an entry
out.append({"title": target, "text": label or target,
"state": "missing"})
elif ext:
out.append({"title": "", "text": strip_markup(ext),
"state": "external"})
return out
def natural_key(title):
"""Sort 'January 9' before 'January 20', not after it."""
return [int(p) if p.isdigit() else p.lower()
for p in re.split(r"(\d+)", title)]
def strip_markup(s):
s = re.sub(r"<[^>]+>", "", s)
s = re.sub(r"'{2,}", "", s)
return re.sub(r"\s+", " ", s).strip()
def row_label(group_wt, entries):
"""Label for a row: its navbox group, or the span of what it holds."""
if group_wt:
m = LINK_RE.search(group_wt)
if m and m.group(1):
return m.group(1), m.group(1)
return strip_markup(group_wt), None
real = [e for e in entries if e["title"]]
if not real:
return "", None
if len(real) == 1:
return real[0]["text"], None
return f"{real[0]['text']}–{real[-1]['text']}", None
def build_rows(page_text):
"""The template's grid: rows of entries, with the titles to look up."""
rows = []
for group_wt, list_wt in parse_navbox(page_text):
entries = parse_entries(list_wt)
if not entries:
continue
label, label_title = row_label(group_wt, entries)
rows.append({"label": label, "label_title": label_title,
"state": "plain", "cells": entries})
return rows
def all_titles(rows):
seen, out = set(), []
for row in rows:
for t in [row["label_title"]] + [c["title"] for c in row["cells"]]:
if t and t not in seen:
seen.add(t)
out.append(t)
return out
def apply_existence(rows, sizes, new_titles=()):
"""Colour the grid in from {title: size}, marking the fresh arrivals."""
defined = missing = external = nbytes = 0
for row in rows:
lt = row["label_title"]
row["state"] = ("plain" if not lt
else "exists" if lt in sizes else "missing")
for cell in row["cells"]:
if cell["state"] == "external":
external += 1
continue
if cell["title"] in sizes:
cell["state"] = "exists"
cell["size"] = sizes[cell["title"]]
cell["new"] = cell["title"] in new_titles
defined += 1
nbytes += cell["size"]
else:
cell["state"] = "missing"
missing += 1
return {"total": defined + missing, "defined": defined,
"missing": missing, "external": external, "bytes": nbytes}
# --------------------------------------------------------------------- render
def chrome_binary():
for name in ("google-chrome", "chromium", "chromium-browser",
"google-chrome-stable"):
path = shutil.which(name)
if path:
return path
raise SystemExit("no Chrome/Chromium binary found")
def render(doc, out_png):
"""Rasterise the frame at exactly frame.W x frame.H (times SCALE).
Rendered taller than the frame and cropped back: content ending close to
the bottom of the rasterised area comes out a pixel different from run to
run, which would make the timelapse shimmer.
"""
from PIL import Image
SLACK = 200
tmp = tempfile.mkdtemp(prefix="wikishot-")
try:
page = os.path.join(tmp, "frame.html")
with open(page, "w", encoding="utf-8") as f:
f.write(doc)
raw = os.path.join(tmp, "raw.png")
res = subprocess.run([
chrome_binary(), "--headless=new", "--disable-gpu", "--no-sandbox",
"--hide-scrollbars", "--force-color-profile=srgb",
"--disable-extensions", "--disable-background-networking",
f"--user-data-dir={tmp}/profile",
f"--force-device-scale-factor={SCALE}",
"--virtual-time-budget=10000",
f"--window-size={frame_tpl.W},{frame_tpl.H + SLACK}",
f"--screenshot={raw}", "file://" + page,
], capture_output=True, text=True, timeout=180)
if not os.path.exists(raw):
raise RuntimeError("chrome produced no screenshot:\n"
+ res.stderr[-1500:])
img = Image.open(raw).convert("RGB")
want = (frame_tpl.W * SCALE, frame_tpl.H * SCALE)
canvas = Image.new("RGB", want, "#ffffff") # exact size, always
canvas.paste(img.crop((0, 0, min(img.width, want[0]),
min(img.height, want[1]))), (0, 0))
canvas.save(out_png, optimize=True)
finally:
shutil.rmtree(tmp, ignore_errors=True)
# ------------------------------------------------------------------ wiki data
def gather(target):
"""Everything about this target the wiki knows, cached to disk.
Every revision of the navbox, and every revision of every page it has
ever linked to. That is the whole series in one bundle: rebuilding the
frames afterwards needs no wiki at all.
"""
log(f"{target.slug}: reading {target.page} and its links")
revisions = template_revisions(target.page)
titles, seen = [], set() # the union over all revisions, so a
for _, text in revisions: # page dropped from the template is
for t in all_titles(build_rows(text)): # still known to have existed
if t not in seen:
seen.add(t)
titles.append(t)
pages = {}
for title in existing(titles):
revs = revision_history(title)
if revs:
pages[title] = revs
log(f" {len(revisions)} template revision(s), {len(titles)} entries, "
f"{len(pages)} written")
data = {"page": target.page, "fetched": dt.date.today().isoformat(),
"revisions": revisions, "pages": pages}
with gzip.open(target.cache, "wt", encoding="utf-8") as f:
json.dump(data, f)
return data
def load_cache(target):
if not os.path.exists(target.cache):
raise RuntimeError(f"no cache at {target.cache}; run without --offline")
with gzip.open(target.cache, "rt", encoding="utf-8") as f:
data = json.load(f)
data["revisions"] = [tuple(r) for r in data["revisions"]]
return data
def as_of(revisions, date_str):
"""The template's wikitext on a given day.
Before its first revision the earliest text is used: pages that predate
the navbox are worth showing, and there is no other grid to show them in.
"""
text = revisions[0][1]
for ts, body in revisions:
if ts <= date_str:
text = body
return text
def cadence_dates(first, last, cadence):
"""Frame dates from `first` to `last` inclusive, at the given cadence."""
d0 = dt.date.fromisoformat(first)
d1 = dt.date.fromisoformat(last)
if d1 < d0:
return [last]
if cadence == "daily":
return [(d0 + dt.timedelta(days=i)).isoformat()
for i in range((d1 - d0).days + 1)]
out, y, m = [], d0.year, d0.month
while True:
if cadence == "yearly":
d = dt.date(y, 12, 31)
else: # monthly: last day of month
d = dt.date(y + m // 12, m % 12 + 1, 1) - dt.timedelta(days=1)
if d >= d1:
break
if d >= d0:
out.append(d.isoformat())
if cadence == "yearly":
y += 1
else:
y, m = (y + 1, 1) if m == 12 else (y, m + 1)
out.append(d1.isoformat()) # always end on the last day
return out
# ----------------------------------------------------------------------- data
def write_csv(path, cols, rows):
with open(path, "w", newline="", encoding="utf-8") as f:
f.write(",".join(cols) + "\n")
for r in rows:
f.write(",".join(csv_field(r[c]) for c in cols) + "\n")
def csv_field(value):
s = str(value)
return f'"{s}"' if any(c in s for c in ',"\n') else s
# ----------------------------------------------------------------------- main
def rebuild(target, args):
target.make_dirs()
data = load_cache(target) if args.offline else gather(target)
revisions, pages = data["revisions"], data["pages"]
starts = [revs[0][0] for revs in pages.values()] or [revisions[0][0]]
first = args.since or min(starts)
dates = cadence_dates(first, args.until, args.cadence)
for path in glob.glob(os.path.join(target.frames, "*.png")):
os.remove(path)
totals, entries, prev_titles, prev_count = [], [], None, None
for date_str in dates:
sizes = {}
for title, revs in pages.items():
live = [s for ts, s in revs if ts <= date_str]
if live:
sizes[title] = live[-1]
new_titles = (set() if prev_titles is None
else {t for t in sizes if t not in prev_titles})
rows = build_rows(as_of(revisions, date_str))
stats = apply_existence(rows, sizes, new_titles)
totals.append(dict(date=date_str, **{k: stats[k]
for k in TOTALS_COLS[1:]}))
entries.extend({"date": date_str, "title": t, "size": s}
for t, s in sorted(sizes.items(), key=lambda kv:
natural_key(kv[0])))
render(frame_tpl.build(
date_str, target.title, target.subtitle, rows, stats,
prev_defined=prev_count,
history=[(t["date"], t["defined"]) for t in totals],
created=sorted(new_titles, key=natural_key),
source="laussywiki"), target.frame_path(date_str))
prev_titles, prev_count = set(sizes), stats["defined"]
write_csv(target.totals, TOTALS_COLS, totals)
write_csv(target.entries, ENTRY_COLS, entries)
log(f"{target.slug}: {len(dates)} frames, {dates[0]} to {dates[-1]}, "
f"{totals[-1]['defined']}/{totals[-1]['total']} defined")
def main():
ap = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--version", action="version",
version=f"wiki_shot {__version__}")
ap.add_argument("--only", metavar="SLUG", help="just this target")
ap.add_argument("--cadence", default="monthly",
choices=("daily", "monthly", "yearly"),
help="one frame per what (default monthly)")
ap.add_argument("--since", metavar="YYYY-MM-DD",
help="start here instead of at the first page written")
ap.add_argument("--until", metavar="YYYY-MM-DD",
default=dt.date.today().isoformat())
ap.add_argument("--offline", action="store_true",
help="re-render from the cached wiki data, no requests")
ap.add_argument("--page", help="ad-hoc template, not from targets.json")
ap.add_argument("--slug", help="directory name for --page")
ap.add_argument("--title", help="heading to print for --page")
ap.add_argument("--subtitle", default="")
args = ap.parse_args()
if args.page:
targets = [Target(args.slug or "adhoc", args.page, args.title,
args.subtitle)]
else:
targets = load_targets()
if args.only:
targets = [t for t in targets if t.slug == args.only]
if not targets:
raise SystemExit(f"no target with slug {args.only!r} "
f"in {TARGETS_JSON}")
failed = []
for target in targets:
try:
rebuild(target, args)
except Exception as e: # one bad target must not
log(f"{target.slug}: FAILED - {e}") # sink the others
failed.append(target.slug)
if failed:
log(f"finished with failures: {', '.join(failed)}")
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
"""Build the HTML for one wiki-coverage frame.
A frame is a grid: one cell per entry of the tracked template, red while the
entry is still a redlink and blue once the page exists, deepening as the page
grows. The canvas is a fixed size and every position in it is an integer, so a
frame from 2031 lines up pixel-for-pixel with a frame from 2026.
The grid fits itself to however many rows and columns the template has, so a
template that gains a row keeps producing frames of the same size (the cells
just get slightly smaller) rather than breaking the series.
"""
import datetime as dt
import html
import math
# ------------------------------------------------------------------ geometry
W = 1560
PAD = 40
HEADER_H = 128
STATS_H = 188 # must clear the chart: title + plot + caption + padding
GRID_H = 700
FOOTER_H = 96
H = PAD + HEADER_H + STATS_H + GRID_H + FOOTER_H + PAD
LBL_W = 172 # row-label column
CELL_MAX_W = 120 # cells stop growing so a 10-column grid stays legible
CELL_MAX_H = 54
GAP = 4 # gutter between cells, taken out of the cell box
# A page this big counts as fully written; the fill tops out there. Fixed
# rather than relative to the biggest page of the day, so a colour means the
# same thing in every frame of the series.
SIZE_REF = 4000
CHART_W = 902
CHART_H = 108
def esc(s):
return html.escape(str(s or ""), quote=True)
def nice_max(value):
"""Round a maximum up to a stable, round number.
Keeps the chart's vertical scale from rescaling every single day: it only
steps when the curve crosses a 1-2-5 boundary.
"""
if value <= 0:
return 10
step = 10 ** (len(str(int(value))) - 1)
for mult in (1, 1.5, 2, 2.5, 3, 4, 5, 6, 7, 8, 9, 10):
top = step * mult
if top >= value:
return int(top)
return int(step * 10)
def mix(a, b, t):
"""Blend two #rrggbb colours, t in [0, 1]."""
t = max(0.0, min(1.0, t))
pa = [int(a[i:i + 2], 16) for i in (1, 3, 5)]
pb = [int(b[i:i + 2], 16) for i in (1, 3, 5)]
return "#%02x%02x%02x" % tuple(round(x + (y - x) * t)
for x, y in zip(pa, pb))
def fill_for(cell):
"""Background colour of one cell, and whether its text must go light."""
if cell["state"] == "external":
return "#fafafa", False
if cell["state"] == "missing":
return "#fdf0f0", False
t = math.log1p(cell.get("size", 0)) / math.log1p(SIZE_REF)
return mix("#e8f0fe", "#4285f4", t), t > 0.62
def fmt_delta(d):
if not d:
return ""
return f"+{d}" if d > 0 else str(d)
def build(date_str, title, subtitle, rows, stats, prev_defined=None,
history=(), created=(), source=""):
"""Return the complete HTML document for one frame.
rows : [{label, state, cells: [{text, state, size, title, new}]}]
stats : {defined, total, pages, bytes}
prev_defined : how many entries existed in the previous frame
history : [(date, defined), ...] over the whole captured series
created : titles that appeared since the previous frame
"""
pretty = dt.datetime.strptime(date_str, "%Y-%m-%d").strftime("%d %B %Y")
total = max(stats.get("total", 1), 1)
defined = stats.get("defined", 0)
pct = 100.0 * defined / total
delta = None if prev_defined is None else defined - prev_defined
# ---- grid ---------------------------------------------------------
n_rows = max(len(rows), 1)
n_cols = max((len(r["cells"]) for r in rows), default=1) or 1
cw = min(CELL_MAX_W, (W - 2 * PAD - LBL_W) // n_cols)
ch = min(CELL_MAX_H, GRID_H // n_rows)
grid_w = LBL_W + n_cols * cw
left0 = (W - 2 * PAD - grid_w) // 2 # centre a narrow grid
lbl_fs = max(11, min(17, int(ch * 0.42)))
cells = []
for r, row in enumerate(rows):
y = r * ch
lcls = "rl " + row.get("state", "plain")
cells.append(
f'<div class="rlab {lcls}" style="left:{left0}px;top:{y}px;'
f'width:{LBL_W - 14}px;height:{ch}px;line-height:{ch - GAP}px;'
f'font-size:{lbl_fs}px">{esc(row["label"])}</div>')
# A row whose labels do not fit the column takes several columns, if
# it is short enough to have them spare: {{Months}}'s four-entry
# Hypertime row holds titles, not day numbers, and would otherwise be
# clipped to "Februa". Rows that fit are left alone, so the grid stays
# on one pitch wherever it can.
longest = max((len(c["text"]) for c in row["cells"]), default=1)
need = longest * 13 / 1.8 + 10 # 13px is as small as is kind
span = max(1, min(3, n_cols // max(len(row["cells"]), 1),
-int(-need // cw)))
w = span * cw - GAP
# Type size follows the cell and the longest label in the row, so the
# same template reads well as 31 columns of day numbers or 10 of years.
fs = max(9, min(17, int(ch * 0.42), int((w - 8) * 1.8 / longest)))
for c, cell in enumerate(row["cells"]):
bg, light = fill_for(cell)
x = left0 + LBL_W + c * span * cw
cls = "cell " + cell["state"] + (" new" if cell.get("new") else "")
style = (f'left:{x}px;top:{y}px;width:{w}px;'
f'height:{ch - GAP}px;line-height:{ch - GAP}px;'
f'font-size:{fs}px;background:{bg}')
if light:
style += ";color:#ffffff"
cells.append(f'<div class="{cls}" style="{style}">'
f'{esc(cell["text"])}</div>')
# ---- progress chart -----------------------------------------------
# The whole captured series, so the frame carries its own history: the
# line grows a little further to the right every day.
# Scaled to the curve rather than to the total: against 140 possible
# entries a handful of pages is a flat line at the floor, and the shape of
# the growth is the one thing the number above cannot show.
pts, chart_extra = [], ""
if len(history) >= 2:
axis = min(total, nice_max(max(v for _, v in history)))
n = len(history)
for i, (_, value) in enumerate(history):
px = i * (CHART_W - 2) / (n - 1) + 1
py = CHART_H - 1 - (CHART_H - 14) * value / max(axis, 1)
pts.append(f"{px:.1f},{py:.1f}")
area = (f'<polygon class="ca" points="1,{CHART_H - 1} '
f'{" ".join(pts)} {CHART_W - 1},{CHART_H - 1}"/>')
chart_extra = (
f'<line class="cx" x1="0" y1="{CHART_H - 1}" x2="{CHART_W}" '
f'y2="{CHART_H - 1}"/>{area}'
f'<polyline class="cl" points="{" ".join(pts)}"/>'
f'<circle class="cd" cx="{CHART_W - 1}" '
f'cy="{pts[-1].split(",")[1]}" r="4"/>'
f'<text class="cm" x="{CHART_W}" y="11" text-anchor="end">'
f'{axis:,}</text>')
span = (f'<div class="cfrom">{esc(history[0][0])}</div>'
f'<div class="cto">{len(history)} frames</div>')
else:
span = '<div class="cfrom">first frame — the line starts here</div>'
# ---- footer -------------------------------------------------------
if created:
news = ('<span class="nlab">New</span> '
+ ' <span class="dot">·</span> '.join(
esc(t) for t in created))
elif prev_defined is None:
news = '<span class="quiet">baseline — nothing to compare yet</span>'
else:
news = '<span class="quiet">no new entries since the previous frame</span>'
legend = "".join(
f'<span class="lg" style="background:{mix("#e8f0fe", "#4285f4", t)}">'
f'</span>' for t in (0.0, 0.25, 0.5, 0.75, 1.0))
return f"""<!doctype html>
<html><head><meta charset="utf-8"><style>
/* Every vertical position below is an integer sum of integers: no
align-items:center, no fractional line-heights. Half-pixel offsets round
unpredictably between runs and would jitter the frames by 1px. */
* {{ margin:0; padding:0; box-sizing:border-box; }}
body {{ width:{W}px; height:{H}px; background:#fff; overflow:hidden;
font-family:Roboto,Arial,'DejaVu Sans',sans-serif; color:#202124;
line-height:1; -webkit-font-smoothing:antialiased; }}
.page {{ padding:{PAD}px; }}
header {{ height:{HEADER_H}px; display:flex; align-items:flex-start;
border-bottom:1px solid #e8eaed; }}
.who {{ flex:1; min-width:0; padding-top:26px; }}
.name {{ font-size:34px; line-height:40px; letter-spacing:-.2px; }}
.sub {{ font-size:17px; line-height:22px; color:#5f6368; margin-top:8px;
white-space:nowrap; overflow:hidden; text-overflow:ellipsis; }}
.when {{ text-align:right; flex:none; padding:28px 0 0 24px; }}
.date {{ font-size:30px; line-height:40px; font-weight:500;
letter-spacing:-.3px; }}
.src {{ font-size:14px; line-height:16px; color:#80868b; margin-top:8px;
text-transform:uppercase; letter-spacing:1.2px; }}
.panel {{ height:{STATS_H}px; display:flex; align-items:flex-start;
padding-top:26px; }}
.stats {{ width:{W - 2 * PAD - CHART_W - 40}px; flex:none; }}
.sname {{ font-size:14px; line-height:18px; color:#5f6368;
text-transform:uppercase; letter-spacing:1px; }}
.sval {{ font-size:44px; line-height:52px; font-weight:500;
font-variant-numeric:tabular-nums; margin-top:4px; }}
.sof {{ font-size:22px; color:#9aa0a6; font-weight:400; }}
.tdelta {{ font-size:20px; color:#188038; font-weight:500; margin-left:10px;
vertical-align:2px; }}
.srec {{ font-size:15px; line-height:20px; color:#80868b; margin-top:6px; }}
.chart {{ width:{CHART_W}px; flex:none; margin-left:40px; padding-top:4px; }}
.ctitle {{ font-size:14px; line-height:18px; color:#5f6368;
text-transform:uppercase; letter-spacing:1px; }}
svg.plot {{ display:block; margin-top:8px; }}
.ca {{ fill:#e8f0fe; }}
.cl {{ fill:none; stroke:#1a73e8; stroke-width:2.5;
stroke-linejoin:round; stroke-linecap:round; }}
.cd {{ fill:#1a73e8; }}
.cx {{ stroke:#e8eaed; stroke-width:1; }}
.cm {{ fill:#9aa0a6; font-size:11px;
font-family:Roboto,Arial,'DejaVu Sans',sans-serif; }}
.cfoot {{ display:flex; font-size:12px; line-height:16px; color:#9aa0a6;
margin-top:4px; }}
.cfrom {{ flex:1; }}
.cto {{ text-align:right; }}
.grid {{ position:relative; height:{GRID_H}px; }}
.cell {{ position:absolute; text-align:center; border-radius:3px;
color:#1a0dab; font-variant-numeric:tabular-nums;
overflow:hidden; white-space:nowrap; }}
.cell.missing {{ color:#ba0000; box-shadow:inset 0 0 0 1px #f6dada; }}
.cell.external {{ color:#c8cbcf; }}
.cell.new {{ box-shadow:inset 0 0 0 2px #f9ab00; font-weight:700; }}
.rlab {{ position:absolute; text-align:right; white-space:nowrap;
overflow:hidden; text-overflow:ellipsis; }}
.rl.exists {{ color:#1a0dab; }}
.rl.missing {{ color:#ba0000; }}
.rl.plain {{ color:#5f6368; }}
footer {{ height:{FOOTER_H}px; display:flex; align-items:flex-start;
padding-top:22px; border-top:1px solid #e8eaed; }}
.news {{ flex:1; min-width:0; font-size:16px; line-height:24px;
color:#202124; white-space:nowrap; overflow:hidden;
text-overflow:ellipsis; padding-right:24px; }}
.nlab {{ font-size:12px; font-weight:700; color:#f9ab00;
letter-spacing:1px; margin-right:8px; }}
.dot {{ color:#bdc1c6; }}
.quiet {{ color:#9aa0a6; }}
.key {{ flex:none; text-align:right; font-size:12px; line-height:18px;
color:#80868b; }}
.lg {{ display:inline-block; width:22px; height:11px; border-radius:2px;
margin-left:2px; vertical-align:-1px; }}
.lgm {{ display:inline-block; width:22px; height:11px; border-radius:2px;
background:#fdf0f0; box-shadow:inset 0 0 0 1px #f6dada;
margin-right:6px; vertical-align:-1px; }}
/* Chrome jitters the vertical position of the very last element in flow by
a pixel between runs; a spacer keeps that jitter off the footer. */
.tail {{ height:24px; }}
</style></head>
<body><div class="page">
<header>
<div class="who">
<div class="name">{esc(title)}</div>
<div class="sub">{esc(subtitle)}</div>
</div>
<div class="when">
<div class="date">{pretty}</div>
<div class="src">{esc(source or 'laussywiki')}</div>
</div>
</header>
<div class="panel">
<div class="stats">
<div class="sname">Entries defined</div>
<div class="sval">{defined:,}<span class="sof"> / {total:,}</span><span
class="tdelta">{fmt_delta(delta)}</span></div>
<div class="srec">{pct:.1f}% written ·
{stats.get('bytes', 0):,} bytes across them</div>
</div>
<div class="chart">
<div class="ctitle">Coverage over the series</div>
<svg class="plot" width="{CHART_W}" height="{CHART_H}"
viewBox="0 0 {CHART_W} {CHART_H}">{chart_extra}</svg>
<div class="cfoot">{span}</div>
</div>
</div>
<div class="grid">{''.join(cells)}</div>
<footer>
<div class="news">{news}</div>
<div class="key"><span class="lgm"></span>redlink {legend}
written</div>
</footer>
<div class="tail"></div>
</div></body></html>"""
#!/usr/bin/env bash
# Assemble each target's frames into a timelapse.
#
# ./make_video.sh # every target, 6 frames/second
# ./make_video.sh 3 # slower: 3 frames/second
# ./make_video.sh 6 gif # also write timelapse.gif
# ./make_video.sh 6 "" years # just one target
#
# Each video is written next to its frames, as targets/<slug>/timelapse.mp4.
# Frames are half-scaled (they are rendered at 2x for sharpness).
set -euo pipefail
cd "$(dirname "$(readlink -f "$0")")"
FPS="${1:-6}"
ALSO_GIF="${2:-}"
ONLY="${3:-}"
for dir in targets/*/; do
slug=$(basename "$dir")
[ -n "$ONLY" ] && [ "$slug" != "$ONLY" ] && continue
[ -d "$dir/frames" ] || continue
count=$(find "$dir/frames" -name '*.png' | wc -l)
if [ "$count" -lt 2 ]; then
echo "$slug: only $count frame(s) - need at least 2 for a timelapse"
continue
fi
# Frames are named YYYY-MM-DD.png, so lexical order is chronological.
list="$dir/.frames.txt"
find "$dir/frames" -name '*.png' | sort | sed "s|^|file '$PWD/|;s|$|'|" > "$list"
ffmpeg -y -loglevel error -r "$FPS" -f concat -safe 0 -i "$list" \
-vf "scale=iw/2:ih/2:flags=lanczos,pad=ceil(iw/2)*2:ceil(ih/2)*2" \
-c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \
-movflags +faststart "$dir/timelapse.mp4"
echo "$slug: wrote timelapse.mp4 ($count frames at ${FPS}fps)"
if [ "$ALSO_GIF" = "gif" ]; then
ffmpeg -y -loglevel error -r "$FPS" -f concat -safe 0 -i "$list" \
-vf "scale=iw/2:ih/2:flags=lanczos,split[a][b];[a]palettegen[p];[b][p]paletteuse" \
"$dir/timelapse.gif"
echo "$slug: wrote timelapse.gif"
fi
rm -f "$list"
done