Sync𝕐
Syncy is a script, made with Claude, which synchronizes the 𝕐 and ℤ posts made online (typically in real time) to my local laussywiki (llw), so as to keep everything harmonious in one and the same branch of the multiverse.
It should be patched so that the cloudflare secret key is stored in the credential files, like llw2lw (maybe even using the same file as those are the same credentials).
Versions
- v°1.4 (10 August (2026)) — a file the gadget names in lowercase (its own lw_*.jpg uploads) was compared un-normalized against the API's ucfirst'd title, so it could never be found "present": every run downloaded it from online again and re-uploaded it into
fileexists-no-change, reported as FAILED for ever. Titles now compare throughcanon()(ucfirst, underscores) and—belt and braces, the llw2lw 2.5.0 lesson—fileexists-no-changecounts as "already here", never a failure. - v°1.5 (11 August (2026)) — ℤ joins 𝕐. Post ids are
[YZ]-and page selection covers both branches: a bare run syncs the current month of each, --all walks 𝕐 from May 2026 and ℤ (the Z/ pages) from August 2026. The merge machinery was branch-clean already, so this is mostly page selection. An explicit title of either branch is also what llw2lw 3.5.0 hands syncy, one page at a time, when its own guard finds online posts local lacks. - v°1.6 (11 August (2026)) — deletions travel. A post deleted ONLINE, which the gadget signs with an invisible
<!--deleted:ID-->tombstone, is dropped from the merge—deletion beats the local copy—instead of being faithfully resurrected as though it were a local post awaiting its push. Reported as "N deleted online". - v°1.7 (18 August (2026)) — no invented ids, and deletions travel both ways. A clash used to be settled by keeping both copies and giving the second a made-up id, Z-…-155034b; such a post is broken as written, because an id is a timestamp and {{thistime}} builds its stamp link from what the composer gave it, so the stamp lands on …-155034 and no longer matches its own blockquote, the gadget cannot lift it to the head of the post, and a {{continue}} hand-off from the other branch points at an anchor that is not on the page. Local's copy is kept, each declined copy is written to ~/.syncy-backups as …CLASH-<id>.txt, and the clash is reported on stdout—stderr being invisible inside an llw2lw run. A post deleted LOCALLY is now tombstoned too (gadget v1.46): the union with online used to hand it straight back and the next push republished it, so it could not be killed.
- v°1.8 (18 August (2026)) — nothing between the posts is thrown away. The rebuild regenerates the posts region from the posts it could parse and keeps only the preamble and the postamble, so any other text living between the first post and the last—a stray paragraph, or a post whose
<blockquote>wrapper a hand edit had broken—was destroyed in silence. The Würzburg post of 3 June (2026) went that way: a section edit on the 7th left its text outside any blockquote and the sync of the 9th swept it off the page. Such a page is now refused, with the first three hundred orphaned characters printed so one can see what is at stake. - v°1.9 (18 August (2026)) — post ids may be Anno Fabri tags. The composer (gadget v1.49) stopped naming a post by its timestamp spelled out and names it Y-006Nj instead: the same second in five base62 characters, unreadable without the Feistel of ~/bin/AF. Both spellings are recognised, and nothing here may go on slicing an id for its date—the day a post belongs to and its order inside that day are now decoded, an Anno Fabri tag neither showing its date at
[2:10]nor sorting by it. A month page holds both kinds at once and merges the same either way. - v°2.0 (24 August (2026)) — a clash is settled, not shrugged off: the two copies are judged against ~/.syncy-base, the texts as they stood at the last clean sync, and only a post that has moved on both wikis is a question—one that now stops the machinery instead of scrolling past. See below.
When the edit was made online
On 24 August (2026) I noticed that the sentences I had added at laussy.org to a post about a golden number in an enneadecaeteris were not there any more. They were not, and neither were eight further edits to the month pages, the oldest of them from 6 July (2026). Every one of those had been made ONLINE in the ordinary MediaWiki section editor rather than in the microblog composer, and two ordinary pages outside this tool's reach had gone the same way, which brings the edits the sync destroyed to eleven; three more that came back with them were nobody's doing but my own de-duplication of 15 August (2026).
That is the whole of the trap. Which of two copies of a post is the newer one is read off the gadget's invisible <!--lastedited:--> marker, and the ordinary editor does not touch that marker—it has never heard of it. An edit made online therefore leaves the two copies looking exactly the same age while saying different things, which is not a verdict but the absence of one. Until V2.0 syncy called it a CLASH and disposed of it by keeping local, because local comes first in the list it consults; from V1.7 the declined words went to ~/.syncy-backups as a CLASH file—before that both copies were kept, the second under an invented id—and the event was announced by a single plain line in the middle of a two-hundred-line llw2lw run. Nothing was lost on disk and everything was lost in practice. Worse, syncy was usually never called at all: llw2lw's month-page reconcile compared SETS of post ids, where a post local lacks is a phone post and a tombstoned one is a deletion, so a post present on both sides that merely said something different matched neither test and the push went straight over it—no warning, no backup, no line in the log. That half is fixed in llw2lw 3.9.0, which compares the posts' text as well as their ids.
So V2.0 asks a question the marker can answer. syncy now keeps, per page, ~/.syncy-base/August_(2026).json.gz—the local text, the online text and the moment they last agreed, spaces and slashes flattened out of the title. A clash then becomes an ordinary three-way merge: the side that has MOVED away from the base is the side somebody wrote on, and it wins, whichever wiki it is. Only when both sides have moved since their last common state is there a real question, and that one is now loud: a banner, the declined copy in a CLASH file, and the machine-readable line SYNCY-UNRESOLVED: <page> <id> <file>, with exit status 2. llw2lw 3.9.0 reads that and refuses the page with syncfirst, which lands it in the Failures and Pending-retry blocks at the end of the run, where a line cannot be missed the way one line mid-run could.
The base is deliberately NOT written while anything on the page is unresolved. Were it written, the next run would find neither side moved—both would match a base taken after the divergence—hand the post to local, and bury the question for good; leaving the base where it is, is what keeps the question being asked. The base also has to move at the other end: llw2lw advances it after a verified push, from the very bytes whose sha1 it has just checked online. Without that, the base's idea of "what online said last time" stays at the pre-push text, so the next perfectly ordinary local edit looks like both sides moving, and every month page deadlocks on a clash that is not one.
A page with no base cannot be judged and its clashes are unresolved by construction, which is what syncy --seed-base is for: it declares the two wikis, as they stand right now, to be the reference a later clash is measured against, and merges nothing. It is needed once, when the store is introduced, and again after a page has been repaired by hand—otherwise the run stops for ever on a difference that is in fact already settled.
The repair has a regression test, ~/bin/syncy-selftest, which touches neither wiki: it loads ~/bin/syncy and ~/bin/llw2lw as modules and exercises settle(), diverged_posts() and llw2lw's local_ever_held()—the sha1 test that tells a legacy sync stamp from an edit made online—on the real words of the lost post. Twenty-three checks in six groups: the 23 August loss replayed, its three mirror cases (local moved, both moved, neither moved) and the no-base case, the two-pushes-in-a-row sequence with a NEGATIVE CONTROL that shows the deadlock when the base is left un-moved, three cases where the guard must not cry wolf, six on the sha1 test that decides whether a restamp may overwrite what is online, and a post that quotes a blockquote inside itself. Run it after touching either tool.
laussy@azag:~$ syncy-selftest
The 23 August 2026 loss, replayed
✓ his online edit survives the merge
✓ nothing is left for him to sort out
✓ and the merge says why
✓ llw2lw would have reconciled the page first
The other three ways round
✓ local moved, online did not → local wins
✓ both moved → unresolved, and local kept meanwhile
✓ neither moved → no question asked
✓ no base yet → unresolved rather than silently guessed
Two pushes in a row must not deadlock
✓ local edit settles in local's favour
✓ NEGATIVE CONTROL: a base left un-moved deadlocks
✓ with the base moved after the push, no deadlock
✓ llw2lw computes the sha1 the wiki reports
The guard must not cry wolf
✓ identical copies
✓ local edited with the gadget (its marker is later)
✓ phone post online only: the id guards handle that one
Whose text is online? (the restamp guard)
✓ a sha1 on the first page of history is found
✓ a sha1 only on a LATER page is found too (continuation)
✓ and it really did have to page for it
✓ a sha1 the local wiki never had is NOT found — this is the refusal
✓ no sha1 at all → False; a guard that cannot see waves nothing through
✓ page gone locally → False
A post that quotes a blockquote is read whole
✓ llw2lw reaches the real end tag
✓ and reads it exactly as syncy does
All good.
Recovering the fourteen edits was possible at all only because llw2lw imports rather than overwrites, so every version a post ever had online is still in laussy.org's history—which is what syncloss reads: it walks the online history of the 𝕐 and ℤ month pages, flags every point where a post's body goes back to a state it already had (prose does not revert by itself, so the state in between is an edit somebody made and something undid), and cross-examines it against the local history and against both wikis as they stand today.
Code
#!/usr/bin/env python3
# ___
#/ __|_ _ _ _ __ _| _|
#\__ \ || | ' \/ _| _| _|
#|___/\_, |_||_\__| _|
# |__/ SyncY _|
# V2.0.1 Laussy&Claude _|
# Synchronize Y posts from lw to llw
#
# V2.0.1: the count above said thirteen edits from 13 June; the 13 June one
# (Y-20260613-011637) was superseded by a local rewrite the next day, not
# destroyed, so the oldest real loss is 6 July and there are eleven.
#
# V2.0: a clash is settled, not shrugged off. Two copies of one post with the
# same last-edit time and different words used to be "keep local, write
# the other out, print a line" — and the line goes by inside an llw2lw
# run. That is how the golden-number sentences of Y-01on2 died on
# 24 August 2026, along with twelve more edits going back to 13 June:
# every one of them made ONLINE in the ordinary MediaWiki editor, which
# does not touch the <!--lastedited:--> marker, so the two copies looked
# the same age and local won by being first in the list.
#
# The marker cannot tell them apart, so stop asking it to. syncy now
# remembers, per page, the two texts as they stood when it last ran
# (~/.syncy-base), and a clash becomes an ordinary three-way merge:
# whichever side has MOVED since that base is the side that wrote
# something, and it wins. Only when BOTH have moved is there a real
# question, and that one is now loud — a banner, a CLASH file, the line
# "SYNCY-UNRESOLVED: <page> <id> <file>" for llw2lw to read, and exit
# status 2. llw2lw 3.9.0 turns that into a refusal to push the page, so
# nothing can overwrite an answer nobody has given yet.
#
# The base is deliberately NOT written when anything is unresolved: were
# it written, the next run would see neither side move, hand it to local
# and bury the question for good.
#
# V1.9: post ids may be Anno Fabri tags. The composer (gadget v1.49) stopped
# naming a post by its timestamp spelled out and names it Y-006Nj
# instead — the same second, five base62 characters, unreadable without
# the Feistel of ~/bin/AF. Both spellings are recognised, and nothing
# else here may go on slicing an id for its date: the day a post belongs
# to and the order it takes inside that day are now DECODED (read_id),
# an Anno Fabri tag neither showing its date at [2:10] nor sorting by
# it. Old ids are untouched, so a month page holds both kinds at once
# and merges the same either way.
#
# V1.8: nothing between the posts is thrown away. The rebuild regenerates the
# posts region from the posts it could PARSE and keeps the preamble and
# postamble, so any other text living between the first post and the
# last — a stray paragraph, or a post whose <blockquote> wrapper a hand
# edit had broken — was silently destroyed. The 3 June 2026 Würzburg
# post went that way: a section edit on 7 June left its text outside any
# blockquote, and the sync of 9 June swept it off the page. syncy now
# refuses to write such a page and prints what is at stake.
#
# V1.7: no invented ids, and deletions travel BOTH ways.
#
# A CLASH — two copies of one post with the same last-edit time and
# different words — used to be settled by keeping both and giving the
# second a made-up id, Z-…-155034b. Such a post is broken as written: an
# id IS a timestamp, and {{thistime}} builds its stamp link from what
# the composer gave it, so the stamp lands on …-155034, never on the
# invented …-155034b. The
# stamp then does not match its own blockquote, so the gadget cannot
# lift it to the head of the post and it sits at the foot; and the
# {{continue}} hand-off from the other branch points at an anchor that
# is not on the page, so the link from 𝕐 to its ℤ half dies. Local's
# copy is kept, each declined copy is written to ~/.syncy-backups as
# …CLASH-<id>.txt, and the clash is reported on stdout — stderr being
# invisible inside an llw2lw run.
#
# Deletions: A post deleted LOCALLY used to be
# invisible to this merge — the union with online handed it straight
# back, the next push republished it, and it could not be killed. The
# gadget (v1.46) now signs a local delete with the same
# <!--deleted:ID--> tombstone it has always written online, and a
# tombstone in EITHER text buries the post here. The marker is kept
# only while the other side still carries the post, and dropped once it
# does not, so the pages do not accumulate them.
#
# V1.6: deletions travel. A post deleted ONLINE (gadget v1.36 leaves an
# invisible <!--deleted:ID--> tombstone there) is dropped from the
# merge — deletion beats the local copy — instead of being faithfully
# resurrected as if it were a local post awaiting its push. Reported
# as "N deleted online (…)". Local deletions never needed help: llw2lw
# pushes the whole local text.
#
# V1.5: ℤ joins 𝕐. Post ids are [YZ]-, and page selection covers both
# branches: the default run syncs the current month of each, --all
# walks 𝕐 from May 2026 and ℤ (the "Z/" pages) from August 2026. An
# explicit title of either branch always worked once the id regex
# allowed it — which is also what llw2lw 3.5.0 calls per page. The
# merge machinery was branch-clean already (ids were grouped by their
# [2:10] date slice — V1.9 decodes it instead — backups flatten "/").
#
# V1.4: a file referenced with a lowercase first letter (the gadget's own
# lw_*.jpg names) was compared un-normalized against the API's
# ucfirst'd title, so it could never be "present": every run
# re-downloaded it and re-uploaded it into fileexists-no-change,
# reported as FAILED for ever. Titles are now compared through canon()
# (ucfirst, underscores), and — belt and braces, the llw2lw 2.5.0
# lesson — fileexists-no-change counts as "already here", never a
# failure.
#
"""
syncy -> the CURRENT month of BOTH branches, 𝕐 and ℤ
(e.g. "June (2026)" and "Z/June (2026)")
syncy "May (2026)" -> a specific month page (either branch)
syncy --all -> every month from each branch's start (𝕐: May 2026,
ℤ: August 2026) to the current one
syncy --seed-base -> take the two wikis as they stand right now to be the
reference a later clash is judged against, and merge
nothing. Needed once, and again after a page has been
repaired by hand.
Exit status: 0 everything settled; 2 a post changed on BOTH wikis since their
last common state and only Fabrice can say which words to keep. llw2lw 3.9.0
reads that and refuses to push the page rather than overwrite the answer.
Pull a 𝕐 or ℤ month-page from the ONLINE wiki into the LOCAL wiki:
fetch both, merge by day/time (latest edit wins), and write the result back to
the LOCAL wiki only — local is the base of record. Online is never modified.
It writes only when online actually contributed something (a post local lacks,
or a newer edit of a post local has). If local already contains everything
online has, nothing is written. A backup of the local page is saved first.
Which copy of a post is the newer one is read off the gadget's invisible
<!--lastedited:--> marker — except that an edit made online in the ORDINARY
MediaWiki editor never touches it, so the two copies read as the same age with
different words. Since V2.0 that is settled against ~/.syncy-base, the two
texts as they stood at the end of the last clean sync of the page: the side
that has MOVED away from them is the side somebody wrote on, and it wins. Only
when both have moved is there a question, and a question stops the machinery.
Setup:
- Install on PATH: cp syncy ~/.local/bin/syncy && chmod +x ~/.local/bin/syncy
- Credentials: a MediaWiki bot password from the LOCAL wiki's
Special:BotPasswords. Grants needed: "Edit existing pages" AND "Upload,
replace, and move files" (uploads need their own grant). Provide via env
vars SYNCY_USER / SYNCY_PASS or a ~/.syncy file (chmod 600):
user = Fabrice@syncy
pass = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- Local must have $wgEnableUploads = true (it does, since the gadget uploads).
"""
import sys, os, re, html, json, gzip, datetime
import urllib.request, urllib.parse, http.cookiejar
# ---- config ---------------------------------------------------------------
LOCAL_BASE = 'http://azag/laussywiki/index.php/' # article path (read)
LOCAL_API = 'http://azag/laussywiki/api.php' # API (write)
ONLINE_BASE = 'https://laussy.org/wiki/' # article path (read)
USER_AGENT = 'Mozilla/5.0 (Y-sync)'
# Secret header to get the ONLINE read past Cloudflare (matches your WAF rule).
EXTRA_HEADERS = {
'X-Y-Sync': '5a09709a70313871f3c60dc58de8930c'
}
BACKUP_DIR = os.path.expanduser('~/.syncy-backups')
# The two texts as they stood at the end of the last clean sync of a page.
# A clash is decided by asking which side has moved away from them (V2.0).
BASE_DIR = os.path.expanduser('~/.syncy-base')
# Per-branch page prefix and first month — the inclusive lower bound used
# by --all. 𝕐 lives at the bare title, ℤ under the ASCII "Z/" prefix.
BRANCHES = [('', 2026, 5), # 𝕐: May 2026
('Z/', 2026, 8)] # ℤ: August 2026
# ---------------------------------------------------------------------------
MONTHS = ['January', 'February', 'March', 'April', 'May', 'June',
'July', 'August', 'September', 'October', 'November', 'December']
# ---- Anno Fabri -----------------------------------------------------------
# Since 18 August 2026 a post is named by its Anno Fabri tag — Y-006Nj — where
# it used to be named by its timestamp spelled out, Y-20260820-145603. The tag
# holds the same second in five base62 characters (two of era, three put
# through the 6-round Feistel of ~/bin/AF and Module:AF), plus a sixth letter
# carrying the UTC offset when the post was written away from home. Nothing is
# lost — the transform is a bijection — but a date no longer reads off the id
# by eye. Ids written before that day keep their old spelling and are still
# read: read_id() below takes either.
AF_A = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
AF_ZL = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
AF_OFF = [-39600, -36000, -34200, -32400, -28800, -25200, -21600, -18000,
-14400, -12600, -10800, -9000, -7200, -3600, 0, 3600, 7200, 10800,
12600, 14400, 16200, 18000, 19800, 20700, 21600, 23400, 25200,
28800, 31500, 32400, 34200, 36000, 37800, 39600, 43200, 45900,
46800, 49500, 50400]
AF_BASE, AF_ERA, AF_a, AF_b, AF_SALT = 1787004000, 238328, 62, 3844, 232959
AF_K = [0x2f4b6c1d, 0x7a19e3b5, 0x5c83d271, 0x1e6fa94b, 0x3d5e8a17, 0x64c2b90f]
# A post id in either spelling. Deliberately not used to FIND ids in free
# wikitext — five loose base62 characters would match half the prose — only to
# recognise one already taken from a blockquote or a tombstone.
ID_RE = r'[YZ]-(?:\d{8}-\d{6}|[0-9A-Za-z]{5,6})'
def _af_f(x, i, tw):
h = ((x ^ AF_K[i] ^ tw) * 0x9E3779B1) & 0xFFFFFFFF
h = ((h ^ (h >> 15)) * 0x85EBCA6B) & 0xFFFFFFFF
h = ((h ^ (h >> 13)) * 0xC2B2AE35) & 0xFFFFFFFF
return h ^ (h >> 16)
def _af_home(t):
"""Europe/Madrid's offset at t. CEST between the last Sundays of March and
October, each switching at 01:00 UTC: the whole EU rule."""
y = datetime.datetime.fromtimestamp(t, datetime.timezone.utc).year
def switch(mo):
d = datetime.date(y, mo, 31)
d -= datetime.timedelta(days=(d.weekday() + 1) % 7) # back to Sunday
return int(datetime.datetime(d.year, d.month, d.day,
tzinfo=datetime.timezone.utc)
.timestamp()) + 3600
return 7200 if switch(3) <= t < switch(10) else 3600
def af_read(tag):
"""A tag -> (the instant, the offset its wall clock is written in)."""
if not re.match(r'^[0-9A-Za-z]{5,6}$', tag):
return None
hi = lo = 0
for i in range(5):
k = AF_A.index(tag[i])
if i < 2:
hi = hi * 62 + k
else:
lo = lo * 62 + k
L, R, tw = lo // AF_b, lo % AF_b, hi * AF_SALT
for i in (4, 2, 0):
R = (R - _af_f(L, i + 1, tw)) % AF_b
L = (L - _af_f(R, i, tw)) % AF_a
t = AF_BASE + hi * AF_ERA + L * AF_b + R
if len(tag) == 6:
k = AF_ZL.find(tag[5])
if k < 0 or k >= len(AF_OFF):
return None
return t, AF_OFF[k]
return t, _af_home(t)
def read_id(pid):
"""A post id, in either spelling -> (the wall clock it was written at, as a
naive datetime, and the instant as a Unix time). None if it is not one."""
m = re.match(r'^[YZ]-(\d{4})(\d{2})(\d{2})-(\d{2})(\d{2})(\d{2})$', pid)
if m:
w = datetime.datetime(*map(int, m.groups()))
return w, w.astimezone().timestamp()
m = re.match(r'^[YZ]-([0-9A-Za-z]{5,6})$', pid)
r = af_read(m.group(1)) if m else None
if not r:
return None
t, off = r
return (datetime.datetime.fromtimestamp(t + off, datetime.timezone.utc)
.replace(tzinfo=None), t)
POST_RE = re.compile(r'<blockquote id="(%s)">(.*?)</blockquote>' % ID_RE, re.S)
POST_OPEN = re.compile(r'<blockquote id="(%s)">' % ID_RE)
# A post deleted ONLINE signs itself with this invisible marker (gadget
# v1.36): without it, a deletion there is indistinguishable from a local
# post not yet pushed, and the merge below would faithfully resurrect it.
TOMBSTONE = re.compile(r'<!--deleted:(%s)-->' % ID_RE)
BQ_OPEN = re.compile(r'<blockquote\b', re.I)
BQ_CLOSE = re.compile(r'</blockquote\s*>', re.I)
# The 𝕐 month pages read NEWEST FIRST (since 2026-07-27): the newest day at the
# top of the page, the newest post at the top of its day. Set this to False to
# write them back the old way round, oldest first.
NEWEST_FIRST = True
TEXT_RE = re.compile(r'<text\b[^>]*>(.*?)</text>', re.S)
EDIT_RE = re.compile(r'<!--lastedited:(\d{8}-\d{6})-->')
FILE_RE = re.compile(r'\[\[\s*(?:File|Image)\s*:\s*([^|\]]+?)\s*(?:\|[^\]]*)?\]\]', re.I)
# ---- read (Special:Export, no auth needed for public reads) ---------------
def fetch(base, title):
url = base + 'Special:Export/' + urllib.parse.quote(title.replace(' ', '_'))
headers = {'User-Agent': USER_AGENT}
headers.update(EXTRA_HEADERS)
req = urllib.request.Request(url, headers=headers)
with urllib.request.urlopen(req, timeout=30) as r:
xml = r.read().decode('utf-8', 'replace')
m = TEXT_RE.findall(xml)
return html.unescape(m[-1]) if m else '' # '' if the page doesn't exist
# ---- merge ----------------------------------------------------------------
def last_changed(pid, inner):
"""When this copy of the post was last touched, as 'YYYYMMDD-HHMMSS' — the
latest edit stamp in it, or, for a post never edited, the moment it was
written. That moment used to be pid[2:], the id being the timestamp; an
Anno Fabri id has to be decoded to give it, and must be, since the two
copies of a post are compared on this string."""
m = EDIT_RE.findall(inner)
if m:
return max(m)
r = read_id(pid)
return r[0].strftime('%Y%m%d-%H%M%S') if r else pid[2:]
def find_posts(text):
"""Yield (pid, inner, start, end) for every post.
A post may quote something in a <blockquote> of its own, so the first
</blockquote> after the opening tag is not necessarily the post's: count
the nesting. (POST_RE did take the first one, which silently truncated
such a post — and everything the truncated tail held, its {{thistime}}
included — every time a page was synced.)
"""
for m in POST_OPEN.finditer(text):
depth, pos = 1, m.end()
while depth:
nxt_c = BQ_CLOSE.search(text, pos)
if not nxt_c:
break # unbalanced: leave this one alone
nxt_o = BQ_OPEN.search(text, pos)
if nxt_o and nxt_o.start() < nxt_c.start():
depth, pos = depth + 1, nxt_o.end()
else:
depth, pos = depth - 1, nxt_c.end()
if depth == 0:
yield m.group(1), text[m.end():nxt_c.start()], m.start(), pos
def collect(texts):
kept, clashes = {}, {}
for t in texts:
for pid, inner, _start, _end in find_posts(t):
lc = last_changed(pid, inner)
if pid not in kept:
kept[pid] = (lc, inner)
else:
old_lc, old_inner = kept[pid]
if lc > old_lc:
kept[pid] = (lc, inner)
elif lc == old_lc and old_inner.strip() != inner.strip():
clashes.setdefault(pid, [old_inner]).append(inner)
return kept, clashes
def build(kept, clashes):
# No id is ever invented here (V1.7). A clash used to be resolved by
# keeping both copies and giving the second a made-up id — Z-…-155034b —
# and such a post is broken the moment it is written: a post's id IS its
# timestamp, and {{thistime}} builds its own stamp link from what the
# composer gave it: it lands on …-155034, never on …-155034b. The stamp
# therefore no longer matches its own blockquote, the gadget cannot find
# it to lift it to the top of the post (it stays at the foot, which is the
# visible symptom), and the {{continue}} hand-off written by the other
# branch points at an anchor that is not on the page — the link from 𝕐 to
# its ℤ half simply dies. The losing text is written out beside the backup
# instead, and the clash reported, for Fabrice to merge by hand.
posts = {pid: inner for pid, (lc, inner) in kept.items()}
# The day a post belongs to, and the instant that orders it inside that
# day, both come out of its id — read, not sliced: an Anno Fabri id is
# its date just as much as the old spelling was, but it neither shows it
# at pid[2:10] nor sorts by it (the low three characters are scrambled on
# purpose). An id that will not decode keeps its string order, which is
# the best that can be done with it and puts it nowhere surprising.
days, when = {}, {}
for pid in posts:
r = read_id(pid)
days.setdefault(r[0].strftime('%Y%m%d') if r else pid[2:10],
[]).append(pid)
when[pid] = r[1] if r else 0
out = []
for day in sorted(days, reverse=NEWEST_FIRST):
y, mo, d = int(day[0:4]), int(day[4:6]), int(day[6:8])
out.append('== {{thisday|%d|%s|%d}} ==' % (d, MONTHS[mo - 1], y))
first = True
for pid in sorted(days[day], key=lambda q: (when[q], q),
reverse=NEWEST_FIRST):
if not first:
out.append('-----')
first = False
out.append('<blockquote id="%s">%s</blockquote>' % (pid, posts[pid]))
out.append('')
return '\n'.join(out).rstrip() + '\n'
def normalize(kept):
"""Post-level view for change detection (ignores whitespace/formatting)."""
return {pid: inner.strip() for pid, (lc, inner) in kept.items()}
# ---- preserve non-post chrome (e.g. the {{MonthNav}} header) --------------
def split_chrome(text):
"""Return (preamble, postamble): the LOCAL page content before the first
post element and after the last one. This keeps non-post material — above
all the {{MonthNav}} header with its trip annotations — from being lost
when the posts section is regenerated. Local is authoritative for chrome;
online's header (if any) is ignored."""
heads = [m.start() for m in re.finditer(r'(?m)^==\s*\{\{[Tt]hisday\b', text)]
posts = list(find_posts(text))
starts = heads + [p[2] for p in posts]
if not starts:
return text.strip('\n'), '' # no posts: keep it all as preamble
first = min(starts)
last = max([p[3] for p in posts]) if posts else first
return text[:first].strip('\n'), text[last:].strip('\n')
def orphan_text(text):
"""Wikitext BETWEEN the posts that the rebuild would not put back.
build() regenerates the posts region from the posts it could parse, and
with_chrome() keeps the preamble and the postamble — so anything else
living between the first post and the last simply ceases to exist. A
stray paragraph, or a post whose <blockquote> wrapper got damaged by a
hand edit, is destroyed by the next sync without a word.
That is not a theory: the 3 June 2026 Würzburg post was lost exactly so.
A section edit on 7 June left its text on the page but outside any
blockquote; the sync of 9 June ("syncy: import online posts") swept it
away, and it was two months before anyone noticed.
Returns whatever would be dropped, stripped of the furniture the rebuild
does reproduce — day headings, separators, tombstones, blank space.
"""
posts = sorted(find_posts(text), key=lambda p: p[2])
if not posts:
return ''
heads = [m.start() for m in re.finditer(r'(?m)^==\s*\{\{[Tt]hisday\b', text)]
first = min([posts[0][2]] + heads)
last = posts[-1][3]
gaps, pos = [], first
for _pid, _inner, st, en in posts:
if st > pos:
gaps.append(text[pos:st])
pos = max(pos, en)
if last > pos:
gaps.append(text[pos:last])
rest = '\n'.join(gaps)
rest = re.sub(r'(?m)^==\s*\{\{[Tt]hisday\|[^}]*\}\}\s*==\s*$', '', rest)
rest = re.sub(r'(?m)^-{3,}\s*$', '', rest)
rest = TOMBSTONE.sub('', rest)
return rest.strip()
def with_chrome(preamble, posts, postamble):
chunks = [c for c in (preamble.strip('\n'), posts.strip('\n'),
postamble.strip('\n')) if c]
return '\n\n'.join(chunks) + '\n'
# ---- LOCAL API session (login once; reuse for query, upload, edit) --------
def load_creds():
u, p = os.environ.get('SYNCY_USER'), os.environ.get('SYNCY_PASS')
if u and p:
return u, p
path = os.path.expanduser('~/.syncy')
if os.path.exists(path):
d = {}
for line in open(path, encoding='utf-8'):
line = line.strip()
if line and not line.startswith('#') and '=' in line:
k, v = line.split('=', 1)
d[k.strip().lower()] = v.strip()
if d.get('user') and d.get('pass'):
return d['user'], d['pass']
sys.exit('No credentials: set SYNCY_USER/SYNCY_PASS or create ~/.syncy '
'(user=..., pass=...).')
def make_api():
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
def call(params, post=False):
params = dict(params); params['format'] = 'json'
if post:
req = urllib.request.Request(
LOCAL_API, data=urllib.parse.urlencode(params).encode('utf-8'),
headers={'User-Agent': USER_AGENT})
else:
req = urllib.request.Request(
LOCAL_API + '?' + urllib.parse.urlencode(params),
headers={'User-Agent': USER_AGENT})
with opener.open(req, timeout=30) as r:
return json.loads(r.read().decode('utf-8'))
return call, opener
def login(call):
user, pwd = load_creds()
lt = call({'action': 'query', 'meta': 'tokens', 'type': 'login'})
lt = lt['query']['tokens']['logintoken']
r = call({'action': 'login', 'lgname': user, 'lgpassword': pwd, 'lgtoken': lt}, post=True)
if r.get('login', {}).get('result') != 'Success':
sys.exit('LOGIN FAILED: ' + json.dumps(r.get('login', r)))
def csrf_token(call):
return call({'action': 'query', 'meta': 'tokens'})['query']['tokens']['csrftoken']
def edit_page(call, csrf, title, text, summary):
r = call({'action': 'edit', 'title': title, 'text': text,
'summary': summary, 'token': csrf, 'assert': 'user'}, post=True)
if 'error' in r:
sys.exit('EDIT FAILED: ' + json.dumps(r['error']))
return r['edit']
# ---- files: find references, see what's missing locally, transfer ---------
def canon(name):
"""A file name as MediaWiki's title normalization sees it: underscores,
and the first letter uppercased (titles are ucfirst, so the gadget's
lw_*.jpg answers locally to Lw_*.jpg)."""
n = name.strip().replace(' ', '_')
return (n[0].upper() + n[1:]) if n else n
def referenced_files(text):
names, seen = [], set()
for m in FILE_RE.finditer(text):
n = m.group(1).strip().replace(' ', '_')
if n and canon(n) not in seen:
names.append(n)
seen.add(canon(n))
return names
def local_missing(call, names):
"""Read-only (no login): which of these File: names lack a file locally."""
missing = []
for i in range(0, len(names), 50):
chunk = names[i:i + 50]
r = call({'action': 'query', 'prop': 'imageinfo', 'iiprop': 'timestamp',
'titles': '|'.join('File:' + n for n in chunk), 'formatversion': '2'})
present = set()
for pg in r.get('query', {}).get('pages', []):
if 'imageinfo' in pg: # a file actually exists for this title
present.add(canon(pg['title'].split(':', 1)[1]))
for n in chunk:
if canon(n) not in present:
missing.append(n)
return missing
def download_online_file(name):
"""Exact bytes of a file from the online wiki via Special:FilePath (which
302-redirects to the real image; urllib carries our headers across it)."""
url = ONLINE_BASE + 'Special:FilePath/' + urllib.parse.quote(name)
headers = {'User-Agent': USER_AGENT}
headers.update(EXTRA_HEADERS)
req = urllib.request.Request(url, headers=headers)
with urllib.request.urlopen(req, timeout=120) as r:
return r.read()
def upload_local(opener, csrf, name, data, comment):
boundary = '----syncy' + os.urandom(8).hex()
def field(n, v):
return ('--%s\r\nContent-Disposition: form-data; name="%s"\r\n\r\n%s\r\n'
% (boundary, n, v)).encode('utf-8')
head = b''.join([field('action', 'upload'), field('filename', name),
field('token', csrf), field('ignorewarnings', '1'),
field('comment', comment), field('format', 'json')])
filehdr = ('--%s\r\nContent-Disposition: form-data; name="file"; filename="%s"\r\n'
'Content-Type: application/octet-stream\r\n\r\n' % (boundary, name)).encode('utf-8')
body = head + filehdr + data + b'\r\n' + ('--%s--\r\n' % boundary).encode('utf-8')
req = urllib.request.Request(LOCAL_API, data=body, headers={
'User-Agent': USER_AGENT,
'Content-Type': 'multipart/form-data; boundary=%s' % boundary})
with opener.open(req, timeout=300) as r:
return json.loads(r.read().decode('utf-8'))
def transfer_files(opener, csrf, names):
done, already, failed = [], [], []
for n in names:
try:
data = download_online_file(n)
r = upload_local(opener, csrf, n, data, 'syncy: import file from online')
if r.get('upload', {}).get('result') in ('Success', 'Warning'):
done.append(n)
elif r.get('error', {}).get('code') == 'fileexists-no-change':
already.append(n) # byte-identical copy is here: a no-op, not a failure
else:
failed.append((n, json.dumps(r)[:200]))
except Exception as e:
failed.append((n, str(e)))
return done, already, failed
def backup(title, text):
os.makedirs(BACKUP_DIR, exist_ok=True)
from datetime import datetime
name = '%s.%s.txt' % (title.replace(' ', '_').replace('/', '_'),
datetime.now().strftime('%Y%m%d-%H%M%S'))
path = os.path.join(BACKUP_DIR, name)
with open(path, 'w', encoding='utf-8') as f:
f.write(text)
return path
def base_path(title):
return os.path.join(BASE_DIR,
title.replace(' ', '_').replace('/', '_') + '.json.gz')
def read_base(title):
"""The local and online texts as they stood when this page last synced
cleanly. {} when there is none yet — a first run, or a run left with a
question outstanding."""
try:
with gzip.open(base_path(title), 'rt', encoding='utf-8') as f:
return json.load(f)
except Exception:
return {}
def write_base(title, local_text, online_text):
os.makedirs(BASE_DIR, exist_ok=True)
tmp = base_path(title) + '.tmp'
with gzip.open(tmp, 'wt', encoding='utf-8') as f:
json.dump({'local': local_text, 'online': online_text,
'when': datetime.datetime.now().isoformat(timespec='seconds')},
f, ensure_ascii=False)
os.replace(tmp, base_path(title))
def bodies(text):
return {pid: inner for pid, inner, _s, _e in find_posts(text)}
def settle(title, clashes, both, local_text, online_text):
"""Three-way merge for the posts the last-edit marker cannot separate.
The marker is written by the gadget and by nothing else, so a post edited
at laussy.org in the ordinary editor carries the SAME marker as the local
copy it no longer matches. Comparing the two copies to each other can
therefore only ever say "same age, different words" — which is not an
answer. Comparing each of them to what it was at the last sync is: the
side that has moved is the side somebody wrote on.
Returns (resolved, unresolved); `both` is updated in place for the ones
that could be decided.
"""
base = read_base(title)
bloc, bonl = bodies(base.get('local', '')), bodies(base.get('online', ''))
loc, onl = bodies(local_text), bodies(online_text)
resolved, unresolved = {}, {}
for pid, variants in clashes.items():
l, o = loc.get(pid), onl.get(pid)
if l is None or o is None or pid not in bloc or pid not in bonl:
unresolved[pid] = variants # no base to judge against
continue
moved_local = l.strip() != bloc[pid].strip()
moved_online = o.strip() != bonl[pid].strip()
if moved_online and not moved_local:
both[pid] = (both[pid][0], o)
resolved[pid] = 'online moved since the last sync, local did not'
elif moved_local and not moved_online:
both[pid] = (both[pid][0], l)
resolved[pid] = 'local moved since the last sync, online did not'
elif not moved_local and not moved_online:
both[pid] = (both[pid][0], l)
resolved[pid] = 'neither side moved; kept local'
else:
unresolved[pid] = variants
return resolved, unresolved
def save_clash(title, pid, variants):
"""Write the copies of one clashing post side by side, so that a text
syncy declined to publish is never a text syncy lost."""
os.makedirs(BACKUP_DIR, exist_ok=True)
from datetime import datetime
name = '%s.%s.CLASH-%s.txt' % (title.replace(' ', '_').replace('/', '_'),
datetime.now().strftime('%Y%m%d-%H%M%S'), pid)
path = os.path.join(BACKUP_DIR, name)
with open(path, 'w', encoding='utf-8') as f:
f.write('%s — %d copies with the same last-edit time.\n'
'The FIRST is the one kept (local is the base of record); the\n'
'rest were declined. Merge by hand and edit the post normally.\n'
% (pid, len(variants)))
for i, v in enumerate(variants):
f.write('\n==== copy %d %s ====\n%s\n'
% (i + 1, '(kept)' if i == 0 else '(declined)', v.strip()))
return path
# ---- page selection -------------------------------------------------------
def page_for(year, month, prefix=''):
return '%s%s (%d)' % (prefix, MONTHS[month - 1], year)
def current_pages():
"""The current month of every branch (V1.5: 𝕐 and ℤ)."""
t = datetime.date.today()
return [page_for(t.year, t.month, pre) for pre, _y, _m in BRANCHES]
def all_pages():
"""Every month of every branch, from that branch's start to the current
month, inclusive."""
t = datetime.date.today()
pages = []
for pre, y, m in BRANCHES:
while (y, m) <= (t.year, t.month):
pages.append(page_for(y, m, pre))
m += 1
if m > 12:
m, y = 1, y + 1
return pages
# ---- one page -------------------------------------------------------------
class Api:
"""Holds the cookie session; logs in lazily and caches the CSRF token,
so a --all run authenticates once and reuses it for every month."""
def __init__(self):
self.call, self.opener = make_api()
self.csrf = None
def auth(self):
if self.csrf is None:
login(self.call)
self.csrf = csrf_token(self.call)
return self.csrf
def sync_page(api, title):
try:
local_text = fetch(LOCAL_BASE, title)
except Exception as e:
raise RuntimeError('reading local failed: %s' % e)
try:
online_text = fetch(ONLINE_BASE, title)
except Exception as e:
raise RuntimeError('reading online failed: %s '
'(Cloudflare? check EXTRA_HEADERS)' % e)
both, clashes = collect([local_text, online_text])
local_only, _ = collect([local_text])
# Deletion beats every copy, and it travels BOTH ways (V1.7). This merge
# is a UNION of the two texts, so a post deleted on one side is simply
# handed back by the other unless something says it was deleted on
# purpose. Online deletions have said so since V1.6; local ones said
# nothing, on the theory that the push propagates them — true only if the
# push happens first. Sync first instead and the post walked straight back
# in, was pushed back online, and came back for ever after. So a tombstone
# from EITHER text buries the post.
dead = (set(TOMBSTONE.findall(online_text))
| set(TOMBSTONE.findall(local_text)))
killed = sorted(pid for pid in dead if pid in both)
for pid in dead:
both.pop(pid, None)
clashes.pop(pid, None)
# Decide the clashes BEFORE the page is rebuilt, so a post the base can
# settle goes into the merge like any other (V2.0).
resolved, unresolved = settle(title, clashes, both, local_text, online_text)
for pid in resolved:
clashes.pop(pid, None)
preamble, postamble = split_chrome(local_text)
# A tombstone guards against exactly one thing: the other side still
# holding the post. Once online no longer has it there is nothing left to
# guard, and the marker goes — so a month page does not silt up with the
# record of every post ever deleted from it. Until then it stays, which is
# what stops the next sync (or the next push) from digging the post up.
online_ids = {pid for pid, _inner, _s, _e in find_posts(online_text)}
stones = sorted(dead & online_ids)
preamble = TOMBSTONE.sub('', preamble)
postamble = TOMBSTONE.sub('', postamble).strip('\n')
if stones:
postamble = (postamble + '\n' if postamble else '') + '\n'.join(
'<!--deleted:%s-->' % pid for pid in stones)
merged = with_chrome(preamble, build(both, clashes), postamble)
# A tombstone that has expired is a real change to the local page even
# when no post moved, or it would never actually be cleared away.
textchanged = (normalize(both) != normalize(local_only)
or set(TOMBSTONE.findall(local_text)) != set(stones))
new_ids = sorted(set(both) - set(local_only))
upd_ids = sorted(pid for pid in both if pid in local_only
and both[pid][1].strip() != local_only[pid][1].strip())
refs = referenced_files(merged)
missing = local_missing(api.call, refs) # read-only, no login required
# A clash is a question, not a change: two copies of one post carrying the
# same last-edit time and different words. The kept copy is local's, and
# every declined one is written out where it can be read, named and loudly
# reported — on stdout, since a warning on stderr inside an llw2lw run is
# a warning nobody sees.
# Nothing between the posts may be thrown away (V1.8). The rebuild can
# only put back what it could parse, so if the local page carries text
# there that is not a post, writing this page would destroy it — refuse,
# and say exactly what is at stake. Read from the LOCAL text, which is the
# one about to be overwritten. Fix the page (usually by giving the orphan
# text its <blockquote id="…"> back) and the sync goes through.
orphan = orphan_text(local_text)
if orphan:
preview = orphan if len(orphan) <= 300 else orphan[:300] + '…'
print('\u2716 %s: REFUSING to write — %d character(s) sit between the '
'posts without being in one, and the rebuild would drop them:'
% (title, len(orphan)))
for line in preview.split('\n'):
print(' | %s' % line)
print(' Put that text back inside a <blockquote id="…"> (or move it '
'above the first post / below the last), then sync again.')
return 'refused'
for pid in sorted(resolved):
print('\u2699 %s: %s settled — %s' % (title, pid, resolved[pid]))
# What the base could not settle is a question for Fabrice, and a question
# must stop the machinery rather than be printed past. The marker line is
# what llw2lw 3.9.0 reads to refuse the push of this page.
clash_files = [save_clash(title, pid, variants)
for pid, variants in sorted(unresolved.items())]
if unresolved:
print('\n\u2716 %s: %d POST(S) CHANGED ON BOTH WIKIS SINCE THE LAST '
'SYNC — nobody but you can say which words to keep.'
% (title, len(unresolved)))
for pid, path in zip(sorted(unresolved), clash_files):
print(' %s: kept the local copy for now, the other is in %s'
% (pid, path))
print('SYNCY-UNRESOLVED: %s %s %s' % (title, pid, path))
if unresolved:
print(' Merge each by hand, edit the post normally, and sync again. '
'Until then this page is not pushed and the base is not moved.\n')
if not textchanged and not missing:
print('\u00b7 %s: up to date' % title)
if not unresolved:
write_base(title, local_text, online_text)
return 'unresolved' if unresolved else 'ok'
csrf = api.auth() # uploads/edits need auth
done, already, failed = transfer_files(api.opener, csrf, missing) \
if missing else ([], [], [])
rev = bpath = None
if textchanged:
bpath = backup(title, local_text)
rev = edit_page(api.call, csrf, title, merged,
'syncy: import online posts').get('newrevid')
bits = ['%d new, %d updated' % (len(new_ids), len(upd_ids))] if textchanged \
else ['no text changes']
if killed:
bits.insert(1 if textchanged else 0,
'%d deleted online (%s)' % (len(killed), ', '.join(killed)))
bits.append('%d files transferred, %d present, %d failed'
% (len(done), len(refs) - len(missing) + len(already), len(failed)))
print('\u2713 %s: %s' % (title, '; '.join(bits)))
if textchanged:
print(' local rev %s; backup %s' % (rev, bpath))
for n, why in failed:
print(' FAILED %s: %s' % (n, why))
# The base moves only when the page is settled. Leaving it where it is
# while a question is open is what keeps the question being asked: move it
# now and the next run would find neither side changed, hand the post to
# local, and the declined words would be gone for good.
if not unresolved:
write_base(title, merged if textchanged else local_text, online_text)
return 'unresolved' if unresolved else 'ok'
# ---- main -----------------------------------------------------------------
def seed_base(title):
"""Declare the two wikis, as they stand right now, to be the reference a
later clash is judged against.
Needed once, when the base store is introduced or after a page has been
repaired by hand: without a base every clash is unresolvable, and the run
would stop on a difference that is in fact already settled.
"""
write_base(title, fetch(LOCAL_BASE, title), fetch(ONLINE_BASE, title))
print('\u2261 %s: base set to the two wikis as they stand now' % title)
def main():
do_all, seed, titles = False, False, []
for a in sys.argv[1:]:
if a in ('-a', '--all'):
do_all = True
elif a == '--seed-base':
seed = True
elif a in ('-h', '--help'):
sys.stdout.write(__doc__)
return
elif a.startswith('-'):
sys.exit('unknown option: %s' % a)
else:
titles.append(a)
if do_all:
pages = all_pages()
elif titles:
pages = titles
else:
pages = current_pages()
if seed:
for title in pages:
try:
seed_base(title)
except Exception as e:
sys.stderr.write(' ! %s: %s\n' % (title, e))
return
api = Api()
multi = do_all or len(pages) > 1
stuck = []
for title in pages:
try:
if sync_page(api, title) == 'unresolved':
stuck.append(title)
except Exception as e:
if multi:
sys.stderr.write(' ! %s: %s (skipped)\n' % (title, e))
else:
sys.exit('ERROR with "%s": %s' % (title, e))
# Exit 2 says "I could not settle everything". llw2lw reads it, and so
# does anyone running syncy from a script: a clash left open must not look
# like a clean run (V2.0).
if stuck:
print('Unresolved on: %s. Nothing on those pages should be pushed '
'until they are merged by hand.' % ', '.join(stuck))
sys.exit(2)
if __name__ == '__main__':
main()