llw2lw

llw2lw (local-laussywiki to laussywiki) is a python script to export to my online laussy.org mediawiki web, changes made on my local copy. I.e., this synchronizes the master copy with its public version. This is not to be confused with syncy which does the opposite but for public 𝕐-tweets (yeets) instead (and only), since by the nature of this snapshoting-of-my-mind tool, I sometime have to yeet online.

A log is kept in ~/comp/lw/llw2lw-log.txt.

Other comparisons can be made through the MediaWiki API in case one suspects this script has not been faithfully importing everything.

Versions

v°1.4.19 (20 July (2025)) — first version (with Grok3).
v°1.5.0 (16 August (2025)) — exclude the file namespace (6) as files are uploaded separately and the inclusion of their metadata in the xml causes problems.
v°1.6.0 (5 September (2025)) — store the time of last update as well as the date!
v°1.7.0 (22 March (2026)) — don't include deleted pages from local wiki.
v°2.0.0 (10 June (2026)) — rewrote extensively (with claude) to import automatically, both pages and images.
v°2.1.0 (19 July (2026)) — Blog namespace (100) draft policy, for BlogTitles v2: a published post (one carrying the blogpost_pubdate page-prop) travels like any other page, a draft is withheld, and a copy of a draft found online is deleted—which is how unpublishing propagates. The --blog-sweep option reconciles the whole Blog corpus at once, regardless of Recent Changes.
v°2.2.0 — online deletion now demands positive evidence: only a page whose props explicitly carry blogpost_draft is ever deleted. Withheld-by-absence—no props at all, e.g. because page_props has not been rebuilt after a raw import—is warned about by name and left alone, so a page cannot be unpublished by accident.
v°2.3.0 (27 July (2026)) — recovers from errors instead of losing what they hit; see below.
v°2.4.0 (30 July (2026)) — an edit refused with a 406 by the host's mod_security is retried as multipart/form-data, which got Gadget-monthnav.js through.
v°2.5.0 (1 August (2026)) — that was not enough: a page the host refuses in any encoding is now imported rather than edited; see below.
v°2.6.0 (6 August (2026)) — a title with nothing behind it on the local wiki is dropped instead of being retried for ever; see below.
v°2.7.0 (8 August (2026)) — pages corrected by silent-lwedit never reach Recent Changes; their manifest is folded into every run automatically; see below.
v°2.8.0 (8 August (2026)) — the online history is an exact mirror of the local one: every page travels as an import carrying its true date, author and summary; see below.
v°2.8.1 — no terminal, no crash: run from cron or a pipe, the start-datetime prompt takes its own default instead of dying on EOF.
v°2.8.2 (9 August (2026)) — LAST UPDATE records the moment Recent Changes was read, not the moment the run finished writing; see below.
v°2.8.3 — --throttle=SECONDS between items, a courtesy to the shared host when a run carries hundreds of pages.
v°2.8.4 — a silent edit on a blog draft no longer haunts the manifest: drafts are withheld by policy, so the title is accounted for and cleared.
v°2.9.0 (9 August (2026)) — the online top revisions are read in one batched query per 50 pages instead of one per page: a third of the remote round-trips gone. The read-back that verifies each import stays per-page (it must follow what it verifies), and so do the local reads, on purpose: a hand edit made while a long run is going out is picked up rather than overwritten.
v°3.0.0 — -v/-h at last, from a __version__ the script can actually print (the number used to live only in the ASCII-art banner). The milestone under the round number: the day's 303-page mirror ran clean, 0 content differences.
v°3.1.0 — an import that lands below the online top is retried once, restamped. MediaWiki breaks a timestamp tie by revision id, so when something online already holds that same second, an import carrying the equal stamp files into history without becoming current—and every later run repeated it under identical conditions, for ever. Template:Chang16a sat like that behind an M-O edit sharing its 2024 second.
v°3.2.0 (10 August (2026)) — the sub-blogs named in LOCAL_ONLY_BLOGS—for now just Blog:Blog/, my own private one—never travel, published or not, and a copy found online is deleted: publishing there means "visible on localhost". Membership is read off the title itself, so unlike the draft policy of 2.2.0 no page-props evidence is wanted, there being nothing a stale page_props could get wrong; keeping such a post home used to take a hand-typed --no-export on every run.
v°3.3.0 (11 August (2026)) — a silent edit no longer moves the online date on a legacy page. The restamp-to-now defence of 2.8.0 fired on any page whose online top still carried an old sync-time stamp, i.e. most of the wiki, so Synaesthesia's silent typo fix went online dated the day of the fix instead of staying where it had always been—the exact thing a silent edit exists to avoid. A manifest title is now restamped to the online top's own second, and its tie-lost retry to one second past whichever stamp is ahead, invisible at the footer's minute resolution; never to now. A real push keeps its one-time restamp, which is the intended transition.
v°3.4.0 — llw2lw can no longer eat phone posts: a month page whose online top is ahead of local is fetched once, only then, and read for post anchors the local page lacks. Any found refuse that page with syncfirst, the rest of the run being unaffected—without which the legacy restamp made the local text current online, and syncy, which reads only the online current revision, could never have got them back.
v°3.5.0 — that guard heals instead of refusing: syncy is run on that one page, right there in the run, and the push then carries the merged text, so there is no "run syncy, then rerun llw2lw" round trip through me. It stays out of every run that does not need it, and syncfirst remains the outcome when syncy fails or resolves nothing.
v°3.6.0 (11 August (2026)) — a deletion made online travels back: a phone delete signs itself with an invisible  tombstone (microblog gadget v1.36), syncy V1.6 drops such a post from its merge, and the push that follows sweeps post and tombstone off the online page together. The whole inspection now runs on every month-page push and not only when online's top is ahead—one local edit after the phone activity used to hide the timestamp signal and reopen the hole.
v°3.7.0 (18 August (2026)) — a deletion made locally stops coming back. A post buried here is missing locally and still present online, which is the exact shape of a phone post local lacks, so the reconcile pulled it in and this push republished it, for ever. The gadget (v1.46) now tombstones a local delete just as it has always tombstoned an online one, syncy V1.7 honours a tombstone from either side, and the push itself is what carries the deletion online.
v°3.7.1 — a month page changed only online is noticed at last. Recent Changes is the local wiki's, so an online-only edit—a phone post, and above all a phone delete, which leaves no local trace whatever—never made the page a candidate and the tombstone inspection never ran on it. The current month page of each branch is now checked against the online wiki on every run: two reads, no syncy.
v°3.8.0 (18 August (2026)) — a post id may be an Anno Fabri tag. The composer (gadget v1.49) has stopped naming a post by its timestamp spelled out, Y-20260818-205730, and names it Y-006Nj: the same second in five base62 characters. Nothing here ever reads a date out of an id—ids are only matched, to compare the two wikis' posts and to find tombstones—so both spellings are simply admitted, without which a page whose posts are all Anno Fabri would look to the reconcile like a page with no posts at all.
v°3.9.0 (24 August (2026)) — a post edited online is no longer invisible: the guards compare the two copies' text and not only their sets of ids, a post nobody has settled refuses its page with syncfirst, which lands in the end-of-run Failures and Pending-retry blocks where nothing goes by unread, the probe covers the previous month page as well as the current one, and a verified push moves syncy's base forward with it; see below.
v°3.10.0 (24 August (2026)) — before restamping a page whose online top is ahead of the local revision that must replace it, ask whose text that is. A page mirrored from here has online text this wiki once held, so its sha1 stands somewhere in the local revision history—an import carries the bytes through unchanged and they hash the same on both sides. A sha1 that stands nowhere in it was written online: the page is refused with onlineahead and the online words are saved to ~/.llw2lw-backups rather than walked over. Of the 1658 pages still wearing a pre-2.8.0 sync stamp, 200 sampled all pass; the two that would have failed were Poetry and Fabrice. The same question now guards the verify-failure retry, which used to restamp to now and so bulldoze whatever had landed online during a run, reporting it as a routine note.
v°3.10.1 (24 August (2026)) — a count, not code: the note above said thirteen losses from 13 June. The 13 June one was superseded by a local rewrite the next day rather than destroyed, so the oldest is 6 July and there are eleven, plus three that my own de-duplication of 15 August (2026) undid.
v°3.11.0 (24 August (2026)) — the same question, asked of every ordinary page rather than only of those whose online clock is ahead. Hung on the restamp branch it missed the case with my name on it: edit a page at laussy.org, then edit it here before the next push, and local is the newer one, the branch is skipped and the online words go without even a failed verify to show for it. Month pages stay exempt, their content being settled per post above.

Error recovery

Until 2.3.0, a failed item was simply announced at the end of the run and then lost: LAST UPDATE had moved past it, so Recent Changes never mentioned it again and the online wiki stayed silently out of date. That is what happened on 27 July (2026) to Working with Claude, whose upload died on an Invalid CSRF token. in the middle of an otherwise successful run.

The log is now a ledger: every item enumerated by a run is either pushed or listed in a == Pending retry == block at the end of its entry, and the next invocation attempts those again on top of whatever Recent Changes reports. Each pending line keeps its last error and an attempt count, and an item that has failed three times or more is flagged loudly rather than retried forever in silence. A clean run writes None there, which is how the queue empties.

Three kinds of trouble are dealt with on the spot, without waiting for the next run:

a stale session—the Invalid CSRF token. above, which a long run invites, since the login can expire under it—triggers one automatic re-login and retry;
a transient server mood—rate limit, maxlag, read-only, a network timeout—is retried once after a short pause;
an error on one title no longer aborts the run: the others are pushed, and Ctrl-C queues the untouched remainder for next time.

A file whose bytes are already online is not a failure, and no longer reported as one. MediaWiki answers such an upload with the error fileexists-no-change—an error, although nothing is wrong—which the script now reads for what it means: the run prints Identical and the log records the file under == Already identical online ==, apart from what was really sent.

When the host refuses the edit

The host runs mod_security in front of MediaWiki, and it answers some perfectly legitimate edits with a text/html Error 406 - Not Acceptable before the wiki ever sees them—which the API layer meets as "the server did not return JSON". It is my own gadgets that trip it: Gadget-monthnav.js on 30 July (2026), then Gadget-microblog.js, which was stuck for three runs on 1 August (2026).

Probing it read-only—by sending the payload as an ignored text= parameter on action=query, so nothing is written whatever the answer—says what it is and, more usefully, what it is not:

not a bad line. Every line of the file passes on its own, and so does each half of it. The rule is CRS-style anomaly scoring, a total accumulated over the argument, so there is nothing to reword;
not the size. 96 kB of aaaa… goes through untroubled, while the real 64 kB of JavaScript is already refused at 16 kB. What is scored is what the text looks like, not how much of it there is;
not the encoding either. 2.4.0 sent the same bytes as multipart/form-data and they passed, so I thought the matter closed; in truth multipart fields are scanned just the same, and monthnav.js had merely scored under the line. Microblog.js is refused both ways;
but a file part is not scored at all. That is why uploading images has never hit the wall: the bytes ride in the file part of the body, which the rule does not look at.

Hence 2.5.0: when an edit comes back 406, the page is taken from the local wiki as an XML export and handed to action=import as a file—the road Special:Import took before 2.0.0, and the reason the old way never met this wall. The content arrives byte-exact, the run says Imported instead of Pushed, and the revision keeps its local timestamp and author, so the online history says when the page was really written rather than when it managed to travel.

When the title has nothing behind it

Recent Changes records the title a page carried at the moment it was edited, not the one it carries when the push finally runs. Rename it in between—or delete it—and the run is left holding a title with nothing behind it. That is how Bibbig2itsybitsy, a misspelling of Bigbib2itsybitsy corrected within the hour, came to fail could not fetch from local wiki on 5 August (2026) in the very run that pushed the properly named page beside it.

Entering the queue was the small half of it; staying there was the rest. The pending list of 2.3.0 is replayed blindly, on the assumption that whatever failed can be attempted again—true of a stale token or of a server in a bad mood, false of a page that has ceased to exist. Nothing in the loop could conclude that a title was never coming back, so it returned at every run, failed identically, and reached eight attempts before the ← stuck, look into it flag was actually read.

Since 2.6.0 a single batched existence check runs before anything is listed, over what Recent Changes reports and what the queue revives alike, so the dry run promises only what --doit can really attempt. What it finds is printed apart and never enters the work list; and since every run rewrites the whole pending block from that list, the same step is what empties the queue—there is no separate cleanup to remember.

== Pages to push ==
August (2026)

== Files to push ==
None

== Gone from the local wiki — dropped, not retried ==
Bibbig2itsybitsy

Two details are less obvious than they look. A File: title is judged on its imageinfo rather than on its page, because a description page can outlive the file behind it and the upload would then fail exactly as though the page were missing. And the log is rewritten even when dropping was the only thing a run had to do, since otherwise the stale pending block would outlive the very run meant to forget it.

When the edit was silent

A page corrected by silent-lwedit keeps its timestamp—that is the whole point of that tool—so Recent Changes never hears of it and the scan alone would let it drift from the online copy for ever. Such edits leave their titles in ~/.silent-lwedit-manifest, and since 2.7.0 every run folds that file into its work list exactly as it folds in the pending block, tagging them (silent edit) in the listing. A title is cleared once it is pushed, found identical online or gone locally; a failed one stays for the next run. There is no option to remember: if the manifest has titles, they travel.

The online history is the local one

Until 2.8.0 a page travelled as action=edit, so laussy.org stamped it with the hour of the sync rather than the hour of the writing. The import fallback of 2.5.0 did carry the true date—but MediaWiki stacks a bookkeeping "1 revision imported" revision on top of every API import, core behaviour no hook reaches, and that marker is what the footer then showed. Since 2.8.0 every page travels as an import of its local export: timestamp, author and edit summary arrive intact, and a guard patched into core's ImportReporter.php (deployed with pushcode; re-apply after every MediaWiki upgrade, the tell of a reverted patch being those markers reappearing) keeps the marker from being minted at all, the import log surviving it. The online history is thus a mirror of the local one—and the 406 wall of 2.4.0/2.5.0 is gone for free, imports riding the unscored file part.

Two traps guard the road. An import older than the online top files into history without becoming the current text—MediaWiki elects the current revision by timestamp—so a page whose online stamp is a legacy sync time ahead of its true local date is restamped to the present once, (restamped) in the run, and rides its real dates from the next edit on. And no import is trusted on its word: the run only says Mirrored once the online top matches the local content, anything else joining the pending list as notcurrent. An import at a timestamp equal to the online top does become current, which is what lets a silent-lwedit correction stay silent online too.

The race the ledger could not catch

The pending list of 2.3.0 rescues what a run tried and failed to push. It cannot rescue what a run never saw—and until 2.8.2 there was a way for an edit to be never seen. LAST UPDATE was stamped when the run finished writing, while Recent Changes had been read when it began: an edit landing in between fell after the harvest and before the stamp, so every later run asked "anything since?" and honestly heard nothing.

Usually that gap is seconds. On 8 August (2026) a run carrying a thousand templates took ten minutes, and a picture resized on August (2026) at 21:58:08 lost to a 21:58:13 stamp by five seconds—then stayed lost, invisible to every subsequent run, because a page that never enters the work list can never enter the pending list either.

Since 2.8.2 the horizon written to the log is the instant Recent Changes was read. An edit made mid-run is simply picked up by the next one, and re-examining the overlap costs nothing: a page pushed by the same run answers sha1-identical and is skipped as Identical.

When the ids matched and the words did not

On 24 August (2026) the sentences I had written at laussy.org into a post of mine about a golden number in an enneadecaeteris had gone. Eight more edits to the month pages had gone the same way, the oldest from 6 July (2026): a "contant" put right to "constant", a "Brian" put right to "Bryan" Ferry—wiped twice, both times—and the cross-post cut of an entry, whose loss had left the same list standing on both 𝕐 and ℤ. Every one of them made in the ordinary MediaWiki section editor online, and every one destroyed by a push that said nothing whatever about it.

Two bugs, one behind the other. Every guard this script has ever had compares the two wikis by their sets of post ids: a post local lacks is a phone post to be pulled, a post carrying a tombstone is a deletion to be honoured. A post that exists on both sides and merely says something different matches neither test. Such a page therefore never went to syncy at all, and the push went straight over it—no warning, no backup, no line in the log.

And when syncy did run, it ranked the two copies of a post by the gadget's invisible  marker, which the ordinary section editor never touches. An edit made at laussy.org thus left the two copies looking exactly the same age with different words; syncy called that a clash, kept local because local comes first in a list, wrote the loser to ~/.syncy-backups/*.CLASH-*.txt (only from V1.7—before that it kept both copies, the second under an invented id), and said so in one indented line in the middle of a two-hundred-line run. The cyan line I do notice is this script announcing the reconcile; the clash itself went by uncoloured.

Nothing was lost for good, for a reason that is luck rather than foresight: since 2.8.0 a page travels as an import, so the online history keeps every version a post ever had there, buried under the later pushes but intact. Fourteen edits came back. Eleven of them the sync had destroyed—nine posts on the month pages, and two ordinary pages that no guard covered at all: Poetry, where I had added Mayakovsky on 3 August (2026), and Fabrice, where "a video animator" had become a link on 21 August (2026), both of them still standing online and waiting for the next push to take them. The other three were nobody's fault but mine, wording I had undone in my own de-duplication of 15 August (2026). silent-lwedit 1.5 grew an --at option to file the two pages back at the hour they were really written rather than at the hour I noticed.

The fix is in two tools. syncy V2.0 stops asking the marker a question it cannot answer and keeps a base instead—~/.syncy-base/August_(2026).json.gz, the local text, the online text and the moment they last agreed, spaces and slashes flattened out of the title. A clash becomes an ordinary three-way merge: the side that has moved away from the base is the side somebody wrote on, and it wins. Only when both have moved is there a real question, and that one is loud—a banner, a CLASH file, the machine-readable line SYNCY-UNRESOLVED: <page> <id> <file>, and exit status 2. The base is deliberately not written while anything is unresolved: writing it would make the next run see neither side move, hand the post to local, and bury the question for good. --seed-base declares the two wikis as they stand now to be the reference, which is wanted once, and again after repairing a page by hand.

On this side, 3.9.0 compares post text as well as ids, using the marker only to let the ordinary case through—online older than local is a normal push. Any post where online is not older makes the page reconcile with syncy first, exactly as a phone post does, and an unsettled one refuses the page with syncfirst, which lands in the end-of-run Failures and Pending-retry blocks, where a line cannot be missed the way one line mid-run could. Two smaller repairs ride with it: the online probe covers the previous month page too (on the 1st of a month, everything written the day before had quietly stopped being watched), and a verified push moves syncy's base forward from the very bytes whose sha1 it has just checked online—without which the base's idea of what online said last time goes stale, the next ordinary local edit looks like both sides moving, and every month page deadlocks on a clash that is not one. ~/bin/syncy-selftest replays that loss and its mirror cases, negative control for the deadlock included, and touches neither wiki: it is what to run after touching either tool.

That still left the reason Poetry and Fabrice could sit there for weeks: only the month pages were guarded at all, every other page being imported over on the assumption—stated as fact in this script's own header, where 2.9.0 reads the online tops in one batch because "the online wiki changes only through this tool"—that nothing else is ever written online. 3.10.0 closes it with the same test syncloss uses: a page whose online top holds text this wiki has never held is refused with onlineahead and its online words written to ~/.llw2lw-backups. 3.10.0 hung that question on the restamp branch, which fires only where online's stamp is ahead of local's—so edit a page at laussy.org, then edit the same page here before the next push, and local is the newer one, nothing is asked, and the words go. 3.11.0 asks it of every ordinary page instead, which is one localhost query on the only kind of page that reaches the import at all; 400 pages sampled, 200 with online ahead and 200 with local ahead, all pass. Month pages are exempt on purpose: their question is settled per post by the reconcile above, and after a syncy merge the local text is the merge, never byte-identical to what is online, so the sha1 test would refuse the very phone post it had just pulled. syncloss is now the backstop for what neither of them thought of, rather than for a hole either of them leaves open.

Todo

~~Blogs should be treated specially as not all pages get exported (for unknown reasons).~~ 2.1.0.
~~The update should take into account the time of day too, in addition to the calendar date itself.~~ 1.6.0.
~~The date of page creation and modification should be fixed.~~ 2.8.0—modification dates now mirror the local wiki; creation dates of pages that predate the mirror stay as frozen history, by choice.

Usage

Run llw2lw alone first: it is a dry run and lists what would travel. Then llw2lw --doit to actually push. Since 2.0.0 there is no manual XML file, no Special:Import and no Special:BatchUpload step; since 2.8.0 a page travels as an import of its local export—its true date, author and summary with it—and files are re-uploaded byte-exact with action=upload, straight into the online wiki. Credentials for both wikis live in ~/.llw2lw (chmod 600).

laussy@azag:~$ llw2lw --help
Usage: llw2lw [--doit] [--blog-sweep] [--no-export=<t1>,<t2>,...] [--throttle=SECONDS] [-h] [-v]
Push recent changes from the local MediaWiki directly to the online wiki.

Options:
  --doit                Actually write to the online wiki and update the log file.
                        Without it, a dry run shows what would be pushed.
  --blog-sweep          Reconcile the entire Blog namespace, ignoring Recent
                        Changes: push every locally-published post and delete
                        from the online wiki any local draft found there.

Local-only blogs (Blog): their posts never
sync, published or not, and any copy found online is deleted (a --blog-sweep
checks them all). Edit LOCAL_ONLY_BLOGS in this script to change the list.
  --no-export=<titles>  Comma-separated titles to exclude (spaces/underscores
                        interchangeable), e.g. Test4,File:Shot.png
  --throttle=SECONDS    Sleep this long between items, a courtesy to the
                        shared host on a run of hundreds (default 0).
  -h, --help            Show this help message and exit.
  -v, --version         Show the version and exit.

Titles listed in /home/laussy/.silent-lwedit-manifest (edited by silent-lwedit, so
invisible to Recent Changes) are folded into every run automatically
and cleared from that file once they are safely online.

Prompts for a start datetime (YYYY-MM-DD [HH:MM:SS]); defaults to the
LAST UPDATE in /home/laussy/comp/lw/llw2lw-log.txt, or yesterday if absent.
Credentials are read from /home/laussy/.llw2lw (see header of this script).

A run in write mode looks like this, the pending item of the previous one coming back with it:

Running in write mode
Retrying 1 item(s) left pending by the previous run.

== Pages to push ==
July (2026)
Working with Claude  (retry)

== Files to push ==
File:Screenshot 20260727 175840.png

Uploaded: File:Screenshot 20260727 175840.png
Mirrored: July (2026)
Mirrored: Working with Claude
Log file updated: /home/laussy/comp/lw/llw2lw-log.txt