<span class="mw-page-title-main">Llw2lw</span>
Fabrice P. Lauss𝕪's Web

llw2lw

llw2lw (local-laussywiki to laussywiki) is a python script to export to my online laussy.org mediawiki web, changes made on my local copy. I.e., this synchronizes the master copy with its public version. This is not to be confused with syncy which does the opposite but for public 𝕐-tweets (yeets) instead (and only), since by the nature of this snapshoting-of-my-mind tool, I sometime have to yeet online.

A log is kept in ~/comp/lw/llw2lw-log.txt.

Other comparisons can be made through the MediaWiki API in case one suspects this script has not been faithfully importing everything.

Versions

  • v°1.4.19 (20 July (2025)) — first version (with Grok3).
  • v°1.5.0 (16 August (2025)) — exclude the file namespace (6) as files are uploaded separately and the inclusion of their metadata in the xml causes problems.
  • v°1.6.0 (5 September (2025)) — store the time of last update as well as the date!
  • v°1.7.0 (22 March (2026)) — don't include deleted pages from local wiki.
  • v°2.0.0 (10 June (2026)) — rewrote extensively (with claude) to import automatically, both pages and images.
  • v°2.1.0 (19 July (2026)) — Blog namespace (100) draft policy, for BlogTitles v2: a published post (one carrying the blogpost_pubdate page-prop) travels like any other page, a draft is withheld, and a copy of a draft found online is deleted—which is how unpublishing propagates. The --blog-sweep option reconciles the whole Blog corpus at once, regardless of Recent Changes.
  • v°2.2.0 — online deletion now demands positive evidence: only a page whose props explicitly carry blogpost_draft is ever deleted. Withheld-by-absence—no props at all, e.g. because page_props has not been rebuilt after a raw import—is warned about by name and left alone, so a page cannot be unpublished by accident.
  • v°2.3.0 (27 July (2026)) — recovers from errors instead of losing what they hit; see below.
  • v°2.4.0 (30 July (2026)) — an edit refused with a 406 by the host's mod_security is retried as multipart/form-data, which got Gadget-monthnav.js through.
  • v°2.5.0 (1 August (2026)) — that was not enough: a page the host refuses in any encoding is now imported rather than edited; see below.
  • v°2.6.0 (6 August (2026)) — a title with nothing behind it on the local wiki is dropped instead of being retried for ever; see below.

Error recovery

Until 2.3.0, a failed item was simply announced at the end of the run and then lost: LAST UPDATE had moved past it, so Recent Changes never mentioned it again and the online wiki stayed silently out of date. That is what happened on 27 July (2026) to Working with Claude, whose upload died on an Invalid CSRF token. in the middle of an otherwise successful run.

The log is now a ledger: every item enumerated by a run is either pushed or listed in a == Pending retry == block at the end of its entry, and the next invocation attempts those again on top of whatever Recent Changes reports. Each pending line keeps its last error and an attempt count, and an item that has failed three times or more is flagged loudly rather than retried forever in silence. A clean run writes None there, which is how the queue empties.

Three kinds of trouble are dealt with on the spot, without waiting for the next run:

  • a stale session—the Invalid CSRF token. above, which a long run invites, since the login can expire under it—triggers one automatic re-login and retry;
  • a transient server mood—rate limit, maxlag, read-only, a network timeout—is retried once after a short pause;
  • an error on one title no longer aborts the run: the others are pushed, and Ctrl-C queues the untouched remainder for next time.

A file whose bytes are already online is not a failure, and no longer reported as one. MediaWiki answers such an upload with the error fileexists-no-change—an error, although nothing is wrong—which the script now reads for what it means: the run prints Identical and the log records the file under == Already identical online ==, apart from what was really sent.

When the host refuses the edit

The host runs mod_security in front of MediaWiki, and it answers some perfectly legitimate edits with a text/html Error 406 - Not Acceptable before the wiki ever sees them—which the API layer meets as "the server did not return JSON". It is my own gadgets that trip it: Gadget-monthnav.js on 30 July (2026), then Gadget-microblog.js, which was stuck for three runs on 1 August (2026).

Probing it read-only—by sending the payload as an ignored text= parameter on action=query, so nothing is written whatever the answer—says what it is and, more usefully, what it is not:

  • not a bad line. Every line of the file passes on its own, and so does each half of it. The rule is CRS-style anomaly scoring, a total accumulated over the argument, so there is nothing to reword;
  • not the size. 96 kB of aaaa… goes through untroubled, while the real 64 kB of JavaScript is already refused at 16 kB. What is scored is what the text looks like, not how much of it there is;
  • not the encoding either. 2.4.0 sent the same bytes as multipart/form-data and they passed, so I thought the matter closed; in truth multipart fields are scanned just the same, and monthnav.js had merely scored under the line. Microblog.js is refused both ways;
  • but a file part is not scored at all. That is why uploading images has never hit the wall: the bytes ride in the file part of the body, which the rule does not look at.

Hence 2.5.0: when an edit comes back 406, the page is taken from the local wiki as an XML export and handed to action=import as a file—the road Special:Import took before 2.0.0, and the reason the old way never met this wall. The content arrives byte-exact, the run says Imported instead of Pushed, and the revision keeps its local timestamp and author, so the online history says when the page was really written rather than when it managed to travel.

When the title has nothing behind it

Recent Changes records the title a page carried at the moment it was edited, not the one it carries when the push finally runs. Rename it in between—or delete it—and the run is left holding a title with nothing behind it. That is how Bibbig2itsybitsy, a misspelling of Bigbib2itsybitsy corrected within the hour, came to fail could not fetch from local wiki on 5 August (2026) in the very run that pushed the properly named page beside it.

Entering the queue was the small half of it; staying there was the rest. The pending list of 2.3.0 is replayed blindly, on the assumption that whatever failed can be attempted again—true of a stale token or of a server in a bad mood, false of a page that has ceased to exist. Nothing in the loop could conclude that a title was never coming back, so it returned at every run, failed identically, and reached eight attempts before the ← stuck, look into it flag was actually read.

Since 2.6.0 a single batched existence check runs before anything is listed, over what Recent Changes reports and what the queue revives alike, so the dry run promises only what --doit can really attempt. What it finds is printed apart and never enters the work list; and since every run rewrites the whole pending block from that list, the same step is what empties the queue—there is no separate cleanup to remember.

== Pages to push ==
August (2026)

== Files to push ==
None

== Gone from the local wiki — dropped, not retried ==
Bibbig2itsybitsy

Two details are less obvious than they look. A File: title is judged on its imageinfo rather than on its page, because a description page can outlive the file behind it and the upload would then fail exactly as though the page were missing. And the log is rewritten even when dropping was the only thing a run had to do, since otherwise the stale pending block would outlive the very run meant to forget it.

Todo

  • Blogs should be treated specially as not all pages get exported (for unknown reasons). 2.1.0.
  • The update should take into account the time of day too, in addition to the calendar date itself. 1.6.0.
  • The date of page creation and modification should be fixed.

Usage

Run llw2lw alone first: it is a dry run and lists what would travel. Then llw2lw --doit to actually push. Since 2.0.0 there is no XML file, no Special:Import and no Special:BatchUpload step—pages are written with action=edit and files re-uploaded byte-exact with action=upload, straight into the online wiki. Credentials for both wikis live in ~/.llw2lw (chmod 600).

laussy@azag:~$ llw2lw --help
Usage: llw2lw [--doit] [--blog-sweep] [--no-export=<title1>,<title2>,...] [--help]
Push recent changes from the local MediaWiki directly to the online wiki.

Options:
  --doit                Actually write to the online wiki and update the log file.
                        Without it, a dry run shows what would be pushed.
  --blog-sweep          Reconcile the entire Blog namespace, ignoring Recent
                        Changes: push every locally-published post and delete
                        from the online wiki any local draft found there.
  --no-export=<titles>  Comma-separated titles to exclude (spaces/underscores
                        interchangeable), e.g. Test4,File:Shot.png
  --help                Show this help message and exit.

Prompts for a start datetime (YYYY-MM-DD [HH:MM:SS]); defaults to the
LAST UPDATE in /home/laussy/comp/lw/llw2lw-log.txt, or yesterday if absent.
Credentials are read from /home/laussy/.llw2lw (see header of this script).

A run in write mode looks like this, the pending item of the previous one coming back with it:

Running in write mode
Retrying 1 item(s) left pending by the previous run.

== Pages to push ==
July (2026)
Working with Claude  (retry)

== Files to push ==
File:Screenshot 20260727 175840.png

Uploaded: File:Screenshot 20260727 175840.png
Pushed:   July (2026)
Pushed:   Working with Claude
Log file updated: /home/laussy/comp/lw/llw2lw-log.txt