<span class="mw-page-title-main">Lw2llw</span>
Fabrice P. Lauss𝕪’s ℤootechnic Web

lw2llw

lw2llw (laussywiki to local-laussywiki) is the mirror of llw2lw: a python script that brings into my local mediawiki the edits made at laussy.org, for every page but the 𝕐 month pages, which are syncy's. llw2lw runs it at the start of every run, so that a page changed online is here before anything goes out.

Versions

  • v°1.0.0 (7 November (2026)) — first version. Its first runs brought down an edit of September 4 made online two months before, two pages edited online in July 2025, a page created online, and 33 files uploaded at laussy.org that had never come down—six of them shown as broken images here.

The idea

Since llw2lw 2.8.0 every page goes online as an import of a revision of this wiki, byte for byte, so every pushed revision on laussy.org has the same SHA-1 as a local one. The last state the two wikis shared is therefore already written in their histories: it is the newest online revision whose SHA-1 stands in the local history of the same title, and everything above it was written online. No file of its own keeps that state, so nothing can go stale—which is what had happened to syncy's, the morning this was written.

What happens to a page

online, against here what lw2llw does
only the online side moved since the common revision its revisions are imported here, each with its own date, author and summary; the two wikis are then identical
both moved a three-way merge, paragraph by paragraph (one line is one paragraph on these pages). If it is clean, the merge becomes a revision stamped one second after the later of the two edits, so the footer shows when I last really wrote, the online revisions are filed beneath it, and llw2lw pushes the merge in the same run
the same paragraph changed on both nothing is written on either side; the three texts and the marked merge are saved, and the page is refused both ways at every run until I merge it by hand, or say lw2llw --resolved "Title"
created online imported, with its history
deleted or renamed here, arrived online by a move, its text already here under another title, or online revisions older than a deletion or rename here left alone, and listed
a file uploaded online brought here with its upload date, uploader and description

Two edits on neighbouring lines, such as two items of the same list, count as one paragraph: they are refused, and asked. Every account's edits come down. laussy.org is only ever read: lw2llw does not log in there.

The writes go through importDump, as silent-lwedit's do, so that a pulled revision keeps the moment it was written online; that needs the local MediaWiki install, so the script runs on azag wherever it is typed, as llw2lw does. A file comes with its page through importDump --uploads; MediaWiki then adds one bookkeeping revision on top of the description, same text, dated at the pull.

Usage

laussy@azag:~$ lw2llw -h
lw2llw — bring edits made at laussy.org into the local wiki (every page but
the 𝕐/ℤ month pages, which are syncy's).

  lw2llw                      dry run: what the last day or so would bring
  lw2llw --doit               do it
  lw2llw --all [--doit]       the whole online history (audit; ~2 minutes)
  lw2llw --since DATE         window from DATE (YYYY-MM-DD[ HH:MM[:SS]], or ISO Z)
  lw2llw --page "Title" ...   these pages only, whole history (repeatable)
  lw2llw --resolved "Title"   you merged a refused page by hand: file the online
                              revisions under your text so the push can go
  lw2llw -v | -h

The default window starts one day before llw2lw's LAST UPDATE. llw2lw (3.22.0)
runs it at the start of every run, with --from-llw2lw, which adds lines it
reads: LW2LLW-PULLED / LW2LLW-MERGED / LW2LLW-REFUSED.
Exit status: 0, 2 when a page is refused (changed on both wikis), 1 on error.

lw2llw-selftest walks every case above on two wikis kept in memory, with a writer that behaves like importDump (the newest stamp becomes current, an older one is filed beneath, a second already taken is skipped).

Code

#!/usr/bin/env python3
#  _      ___  _ _
# | |_ __|_  )| | |_ __ __
# | | V  V / / | | | V  V /
# |_|\_/\_/___||_|_|\_/\_/
# lw2llw - Version 1.0.0
# Claude & F.P. Laussy
#
# The mirror of llw2lw: brings the edits made at laussy.org into the LOCAL
# wiki, for every page, so that changing [[Mano Negra]] online is echoed here
# at the next llw2lw (study 0TP9X, audit 0TLak, 7 Nov 2026). The 𝕐/ℤ month
# pages and [[Photo repository]] stay syncy's.
#
# NEW in 1.0.0 — first version. The idea: since llw2lw 2.8.0 every page goes
#     online as an import of a local revision, byte for byte, so each pushed
#     online revision carries the sha1 of a local one. The common base of a
#     page is therefore already written in the two histories — the newest
#     online revision whose sha1 stands in the local history of that title —
#     and everything above it online was written online. No base file, nothing
#     kept per machine (the stale ~/.syncy-base of the same morning, 0T2If).
#
#     What happens to a page whose online text this wiki has never held:
#       fast-forward  this wiki has not moved since the base: the online
#                     revisions are imported, each with its own date, author
#                     and summary; both wikis are then identical.
#       merge         both moved: three-way merge by paragraph (git
#                     merge-file; one line is one paragraph on these pages).
#                     Clean: the merged text becomes a revision stamped one
#                     second after the later of the two edits (Fabrice's
#                     choice: the footer shows when he last really wrote),
#                     written FIRST, the online revisions filed beneath it,
#                     and the title goes to the silent manifest so llw2lw
#                     pushes the merge. Conflict: nothing is written on either
#                     side, the three texts and the marked merge go to
#                     ~/.llw2lw-backups/<title>.CLASH.txt, the page is refused
#                     both ways and asked again every run (his choice).
#       new           created online, never here: imported with its history.
#       left alone    deleted or renamed here, arrived online by an online
#                     move, its text already here under another title, or
#                     online revisions older than a deletion/rename here (a
#                     move onto a redirect deletes it): listed, never pulled.
#     Files uploaded online come down through importImages with their own
#     upload date, uploader and description. Every account's edits come down
#     (his choice). laussy.org is only ever READ: lw2llw never logs in there.
#     Writes go through importDump, as silent-lwedit's do, so they need the
#     local MediaWiki install: like llw2lw (3.21.0) it runs on azag, wherever
#     it is typed.
"""
lw2llw — bring edits made at laussy.org into the local wiki (every page but
the 𝕐/ℤ month pages, which are syncy's).

  lw2llw                      dry run: what the last day or so would bring
  lw2llw --doit               do it
  lw2llw --all [--doit]       the whole online history (audit; ~2 minutes)
  lw2llw --since DATE         window from DATE (YYYY-MM-DD[ HH:MM[:SS]], or ISO Z)
  lw2llw --page "Title" ...   these pages only, whole history (repeatable)
  lw2llw --resolved "Title"   you merged a refused page by hand: file the online
                              revisions under your text so the push can go
  lw2llw -v | -h

The default window starts one day before llw2lw's LAST UPDATE. llw2lw (3.22.0)
runs it at the start of every run, with --from-llw2lw, which adds lines it
reads: LW2LLW-PULLED / LW2LLW-MERGED / LW2LLW-REFUSED.
Exit status: 0, 2 when a page is refused (changed on both wikis), 1 on error.
"""
import datetime as dt
import hashlib
import importlib.machinery
import importlib.util
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import time
from xml.sax.saxutils import escape

__version__ = '1.0.0'

_l = importlib.machinery.SourceFileLoader('llw2lw_mod', os.path.expanduser('~/bin/llw2lw'))
_s = importlib.util.spec_from_loader('llw2lw_mod', _l)
LL = importlib.util.module_from_spec(_s)
_l.exec_module(LL)                       # definitions only: llw2lw has a __main__ guard

MW = '/var/www/html/laussywiki'
PHP = ['php', os.path.join(MW, 'maintenance', 'run.php')]
HOLD_FILE = os.path.expanduser('~/.lw2llw-hold')
LOG_FILE = os.path.expanduser('~/comp/lw/lw2llw-log.txt')
MANIFEST = LL.SILENT_MANIFEST
BACKUP_DIR = LL.ONLINE_BACKUP_DIR
MERGE_USER = 'Fabrice'
WINDOW_MARGIN = dt.timedelta(days=1)
UTC = dt.timezone.utc
MOVE_COMMENT = re.compile(r'moved page \[\[([^\]]+)\]\] to \[\[([^\]]+)\]\]')
ISO = '%Y-%m-%dT%H:%M:%SZ'


class Refusal(Exception):
    pass


def norm(t):
    return (t or '').replace('_', ' ').strip()


# --------------------------------------------------------------- the wikis ---
class Wiki:
    """Reads one wiki through its API (formatversion 2). A test passes another
    object with the same methods."""

    def __init__(self, name, session, api, pause=0.0):
        self.name, self.s, self.api, self.pause = name, session, api, pause

    def get(self, **p):
        p.update(format='json', formatversion='2')
        for attempt in range(4):
            try:
                r = self.s.get(self.api, params=p, timeout=LL.TIMEOUT)
                d = r.json()
                break
            except Exception as e:
                if attempt == 3:
                    raise LL.ApiFailure(f"{self.name} API unreachable: {e}")
                time.sleep(5 * (attempt + 1))
        if 'error' in d:
            raise LL.ApiFailure(f"{self.name} API error: {d['error']}")
        if self.pause:
            time.sleep(self.pause)
        return d

    def walk(self, key, **p):
        p.setdefault('action', 'query')     # without it MediaWiki answers with its HTML help page
        while True:
            d = self.get(**p)
            yield from d.get('query', {}).get(key, [])
            if 'continue' not in d:
                return
            p.update(d['continue'])

    @staticmethod
    def _rev(title, ns, rv):
        return {'title': title, 'ns': ns, 'revid': rv['revid'], 'parentid': rv.get('parentid'),
                'timestamp': rv['timestamp'], 'sha1': rv.get('sha1'), 'user': rv.get('user', ''),
                'comment': rv.get('comment', ''), 'size': rv.get('size')}

    def revisions_since(self, since):
        """Every revision stamped at or after `since` (ISO Z), all namespaces."""
        out = []
        for pg in self.walk('allrevisions', list='allrevisions', arvdir='older', arvend=since,
                            arvprop='ids|timestamp|sha1|user|comment|size', arvlimit='500'):
            out += [self._rev(pg['title'], pg['ns'], rv) for rv in pg['revisions']]
        return out

    def all_revisions(self):
        out = []
        for pg in self.walk('allrevisions', list='allrevisions', arvdir='newer',
                            arvprop='ids|timestamp|sha1|user|comment|size', arvlimit='500'):
            out += [self._rev(pg['title'], pg['ns'], rv) for rv in pg['revisions']]
        return out

    def history(self, title):
        """(ns or None, revisions oldest first); None ns when the page is absent."""
        revs, ns, p = [], None, dict(action='query', prop='revisions', titles=title, rvlimit='max',
                                     rvprop='ids|timestamp|sha1|user|comment|size')
        while True:
            d = self.get(**p)
            pg = d['query']['pages'][0]
            if pg.get('missing') or pg.get('invalid'):
                return None, []
            ns = pg['ns']
            revs += [self._rev(pg['title'], ns, rv) for rv in pg.get('revisions', [])]
            if 'continue' not in d:
                break
            p.update(d['continue'])
        revs.sort(key=lambda r: (r['timestamp'], r['revid']))
        return ns, revs

    def top(self, title):
        """The CURRENT revision (page_latest), or None. Not the newest of the
        sorted history: silent edits share their second with the revision before
        them, and only the wiki knows which one is current."""
        d = self.get(action='query', prop='revisions', titles=title,
                     rvprop='ids|timestamp|sha1|user|comment|size')
        pg = d['query']['pages'][0]
        if pg.get('missing') or pg.get('invalid') or not pg.get('revisions'):
            return None
        return self._rev(pg['title'], pg['ns'], pg['revisions'][0])

    def texts(self, revids):
        """{revid: (content, model, format)}"""
        out, revids = {}, sorted({r for r in revids if r})
        for i in range(0, len(revids), 50):
            d = self.get(action='query', prop='revisions', revids='|'.join(map(str, revids[i:i + 50])),
                         rvprop='ids|content|contentmodel', rvslots='main')
            for pg in d['query']['pages']:
                for rv in pg.get('revisions', []):
                    sl = rv['slots']['main']
                    out[rv['revid']] = (sl.get('content', ''), sl.get('contentmodel', 'wikitext'),
                                        sl.get('contentformat', 'text/x-wiki'))
        return out

    def logs_for(self, title):
        """Log entries whose log title is `title`: deletions here, moves away from here."""
        return [{'type': e['type'], 'action': e.get('action'), 'timestamp': e['timestamp'],
                 'target': norm((e.get('params') or {}).get('target_title'))}
                for e in self.walk('logevents', list='logevents', letitle=title, lelimit='500',
                                   leprop='type|timestamp|details')
                if e['type'] in ('delete', 'move')]

    def logs_since(self, letype, since):
        return [{'title': norm(e.get('title')), 'type': e['type'], 'action': e.get('action'),
                 'timestamp': e['timestamp'], 'user': e.get('user', ''), 'comment': e.get('comment', ''),
                 'target': norm((e.get('params') or {}).get('target_title'))}
                for e in self.walk('logevents', list='logevents', letype=letype, ledir='older',
                                   leend=since, lelimit='500', leprop='title|type|timestamp|user|comment|details')]

    def all_logs(self, letype):
        out = {}
        for e in self.walk('logevents', list='logevents', letype=letype, lelimit='500',
                           leprop='title|type|timestamp|details'):
            out.setdefault(norm(e.get('title')), []).append(
                {'type': e['type'], 'action': e.get('action'), 'timestamp': e['timestamp'],
                 'target': norm((e.get('params') or {}).get('target_title'))})
        return out

    def files(self, names):
        """{name: {...} or None}; name as in File:<name> (spaces or underscores)."""
        out = {}
        names = list(dict.fromkeys(names))
        for i in range(0, len(names), 50):
            d = self.get(action='query', prop='imageinfo', titles='|'.join('File:' + n for n in names[i:i + 50]),
                         iiprop='sha1|url|timestamp|user|comment|size', iilimit='max')
            for pg in d['query']['pages']:
                n = pg['title'].split(':', 1)[1]
                ii = pg.get('imageinfo')
                out[n] = None if not ii else {'sha1': ii[0].get('sha1'), 'url': ii[0].get('url'),
                                              'timestamp': ii[0].get('timestamp'), 'user': ii[0].get('user', ''),
                                              'comment': ii[0].get('comment', ''), 'size': ii[0].get('size'),
                                              'history': [v.get('sha1') for v in ii]}
        return out

    def all_files(self):
        return {norm(f['name']): f.get('sha1') for f in
                self.walk('allimages', list='allimages', aiprop='sha1', ailimit='500')}

    def download(self, url):
        r = self.s.get(url, timeout=LL.UPLOAD_TIMEOUT)
        if r.status_code != 200:
            raise LL.ApiFailure(f"download failed ({r.status_code}): {url}")
        return r.content

    def users_exist(self, names):
        names = sorted({n for n in names if n})
        if not names:
            return set()
        d = self.get(action='query', list='users', ususers='|'.join(names[:50]))
        return {u['name'] for u in d['query']['users'] if not u.get('missing') and not u.get('invalid')}


# ------------------------------------------------------------------ the plan ---
def plan_page(title, ns, online, local, llogs, global_sha=None):
    """What to do with one page. `online` and `local` are revision lists, oldest
    first (local empty when the page is absent here); `llogs` the local log
    entries of this title (deletions, moves away); `global_sha` the sha1 of
    every local revision of any title (only an --all run has it).
    Returns (kind, info)."""
    if LL.MONTH_PAGE.match(title) or title in LL.UNION_PAGES:
        return 'syncy', {}
    if ns in LL.LOCAL_ONLY_NAMESPACES or LL.is_local_only(title):
        return 'excluded', {}
    if not online:
        return 'in step', {}
    otop = online[-1]
    lsha = {r['sha1'] for r in local if r['sha1']}
    if otop['sha1'] in lsha:
        return 'in step', {}
    dels = [e for e in llogs if e['type'] == 'delete' and e['action'] in ('delete', 'delete_redir')]
    away = [e for e in llogs if e['type'] == 'move']
    cut = max([e['timestamp'] for e in dels + away], default=None)
    moves = [MOVE_COMMENT.search(r['comment'] or '') for r in online]
    arrived = any(m and norm(m.group(2)) == title for m in moves)
    left = [r for r, m in zip(online, moves) if m and norm(m.group(1)) == title]
    base_i = max((i for i, r in enumerate(online) if r['sha1'] in lsha), default=None)
    above = online[base_i + 1:] if base_i is not None else list(online)
    fresh = [r for r in above if cut is None or r['timestamp'] > cut]
    if not local:
        if dels:
            return 'deleted here', {'when': dels[-1]['timestamp']}
        if away:
            return 'renamed here', {'when': away[-1]['timestamp'], 'target': away[-1]['target']}
        if arrived:
            return 'renamed online', {}
        if global_sha is not None and otop['sha1'] in global_sha:
            return 'text here under another title', {}
        return 'new', {'revisions': list(online)}
    if not fresh:
        return 'stale', {'cut': cut}
    if left and left[-1]['timestamp'] >= fresh[-1]['timestamp']:
        return 'renamed online', {}
    if base_i is None:
        if ns == 6:
            return 'file description upload text', {}
        return 'no base', {'revisions': fresh}
    ltop = local[-1]
    if ltop['sha1'] == online[base_i]['sha1']:
        return 'fast-forward', {'revisions': fresh, 'local_top': ltop}
    return 'merge', {'base': online[base_i], 'local_top': ltop, 'online_top': otop, 'revisions': fresh}


def merge3(base, mine, theirs):
    """(clean?, merged text) — git merge-file, line (= paragraph) granularity."""
    with tempfile.TemporaryDirectory() as d:
        paths = []
        for n, s in (('local', mine), ('base', base), ('online', theirs)):
            p = os.path.join(d, n)
            with open(p, 'w', encoding='utf-8') as f:
                f.write(s.rstrip('\n') + '\n')
            paths.append(p)
        r = subprocess.run(['git', 'merge-file', '-p', '--diff3', '-L', 'here', '-L', 'base', '-L', 'online']
                           + paths, capture_output=True, text=True)
        if r.returncode < 0 or r.returncode > 127:
            raise RuntimeError('git merge-file failed: ' + r.stderr)
        return r.returncode == 0, r.stdout.rstrip('\n')


def sha1_hex(text):
    return hashlib.sha1(text.encode('utf-8')).hexdigest()


def after(ts, taken=()):
    """One second after ts, and further while that second is already taken."""
    t = LL.one_second_after(ts)
    while t in taken:
        t = LL.one_second_after(t)
    return t


# ----------------------------------------------------------------- writing ---
def rev_xml(r):
    return (f"    <revision>\n      <timestamp>{r['timestamp']}</timestamp>\n"
            f"      <contributor><username>{escape(r['user'] or MERGE_USER)}</username></contributor>\n"
            f"      <comment>{escape(r['comment'] or '')}</comment>\n"
            f"      <model>{escape(r['model'])}</model>\n      <format>{escape(r['format'])}</format>\n"
            f"      <text xml:space=\"preserve\">{escape(r['text'])}</text>\n    </revision>\n")


def import_dump(title, ns, revisions):
    """Write revisions (dicts with timestamp, user, comment, text, model, format)
    into this wiki with importDump, in the order given. Exact bytes, no
    recent-changes entry, no bookkeeping revision."""
    xml = ('<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" version="0.11" xml:lang="en">\n'
           f"  <page>\n    <title>{escape(title)}</title>\n    <ns>{ns}</ns>\n"
           + ''.join(rev_xml(r) for r in revisions) + '  </page>\n</mediawiki>\n')
    for r in revisions:
        if r['text'] and not xml.count(escape(r['text'])):
            raise RuntimeError('XML escaping lost a text')      # the silent-lwedit 5-byte lesson
    with tempfile.NamedTemporaryFile('w', suffix='.xml', delete=False, encoding='utf-8') as tf:
        tf.write(xml)
        tmp = tf.name
    try:
        r = subprocess.run(PHP + ['importDump', tmp], capture_output=True, text=True, cwd=MW)
        if r.returncode != 0 or 'Done!' not in r.stdout:
            raise RuntimeError(f'importDump failed: {r.stdout[-400:]} {r.stderr[-400:]}')
    finally:
        os.unlink(tmp)


def b36(digest):
    n, out = int.from_bytes(digest, 'big'), ''
    while n:
        n, r = divmod(n, 36)
        out = '0123456789abcdefghijklmnopqrstuvwxyz'[r] + out
    return out.rjust(31, '0')


def import_new_file(name, data, info, page_revisions):
    """A file absent here, brought with importDump --uploads: the bytes with their
    online date, uploader and summary, and the description page with its online
    revisions, text exact. (MediaWiki adds one bookkeeping 'uploaded' revision on
    top, same text, stamped now: only that page's footer shows the pull date.)"""
    import base64
    up = (f"    <upload>\n      <timestamp>{info['timestamp']}</timestamp>\n"
          f"      <contributor><username>{escape(info.get('user') or MERGE_USER)}</username></contributor>\n"
          f"      <comment>{escape(info.get('comment') or '')}</comment>\n"
          f"      <filename>{escape(name.replace(' ', '_'))}</filename>\n      <size>{len(data)}</size>\n"
          f"      <sha1base36>{b36(hashlib.sha1(data).digest())}</sha1base36>\n"
          f"      <contents encoding=\"base64\">{base64.b64encode(data).decode()}</contents>\n    </upload>\n")
    xml = ('<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" version="0.11" xml:lang="en">\n'
           f"  <page>\n    <title>{escape('File:' + name)}</title>\n    <ns>6</ns>\n"
           + ''.join(rev_xml(r) for r in page_revisions) + up + '  </page>\n</mediawiki>\n')
    with tempfile.NamedTemporaryFile('w', suffix='.xml', delete=False, encoding='utf-8') as tf:
        tf.write(xml)
        tmp = tf.name
    try:
        r = subprocess.run(PHP + ['importDump', '--uploads', tmp], capture_output=True, text=True, cwd=MW)
        if r.returncode != 0 or 'Done!' not in r.stdout:
            raise RuntimeError(f'importDump --uploads failed: {r.stdout[-400:]} {r.stderr[-400:]}')
    finally:
        os.unlink(tmp)


def import_file(name, data, info, description, users_here):
    """Upload one file here with importImages, keeping its online date and
    uploader (the uploader must exist here; else Maintenance script)."""
    with tempfile.TemporaryDirectory() as d, tempfile.TemporaryDirectory() as dd:
        path = os.path.join(d, name.replace(' ', '_'))     # importImages takes EVERY file of d
        with open(path, 'wb') as f:
            f.write(data)
        desc = os.path.join(dd, 'description.txt')         # so the description lives apart
        with open(desc, 'w', encoding='utf-8') as f:
            f.write(description or '')
        cmd = PHP + ['importImages', '--comment-file=' + desc,
                     '--summary=' + (info.get('comment') or 'lw2llw: file uploaded at laussy.org'),
                     '--timestamp=' + info['timestamp']]
        if info.get('user') in users_here:
            cmd.append('--user=' + info['user'])
        if info.get('overwrite'):
            cmd.append('--overwrite')
        r = subprocess.run(cmd + [d], capture_output=True, text=True, cwd=MW)
        if r.returncode != 0:
            raise RuntimeError(f'importImages failed: {r.stdout[-400:]} {r.stderr[-400:]}')


def manifest_add(titles):
    try:
        seen = set(open(MANIFEST, encoding='utf-8').read().splitlines())
    except OSError:
        seen = set()
    with open(MANIFEST, 'a', encoding='utf-8') as f:
        for t in titles:
            if t not in seen:
                f.write(t + '\n')
                seen.add(t)


def read_hold():
    try:
        return dict(l.rstrip('\n').split('\t', 1) for l in open(HOLD_FILE, encoding='utf-8') if '\t' in l)
    except OSError:
        return {}


def write_hold(hold):
    if not hold:
        if os.path.exists(HOLD_FILE):
            os.remove(HOLD_FILE)
        return
    with open(HOLD_FILE, 'w', encoding='utf-8') as f:
        for t, why in sorted(hold.items()):
            f.write(f'{t}\t{why}\n')


def save_clash(title, base, mine, theirs, marked, info):
    os.makedirs(BACKUP_DIR, exist_ok=True)
    path = os.path.join(BACKUP_DIR, title.replace(' ', '_').replace('/', '_') + '.CLASH.txt')
    with open(path, 'w', encoding='utf-8') as f:
        f.write(f"{title} — changed on BOTH wikis since their last common revision (lw2llw {__version__}).\n"
                f"Merge by hand here (edit the page normally, carrying the online words you want), then run\n"
                f"    lw2llw --resolved \"{title}\"\nor just llw2lw again: if your text already holds the online "
                f"changes, the merge comes out clean by itself.\n\n")
        f.write(f"==== MARKED MERGE (here / base / online) ====\n{marked}\n\n")
        f.write(f"==== ONLINE (rev {info['online_top']['revid']}, {info['online_top']['timestamp']}) ====\n{theirs}\n\n")
        f.write(f"==== HERE (rev {info['local_top']['revid']}, {info['local_top']['timestamp']}) ====\n{mine}\n\n")
        f.write(f"==== BASE (online rev {info['base']['revid']}, {info['base']['timestamp']}) ====\n{base}\n")
    return path


def log(lines):
    if not lines:
        return
    os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
    with open(LOG_FILE, 'a', encoding='utf-8') as f:
        f.write(f"== {dt.datetime.now().strftime('%Y-%m-%d %H:%M:%S')} (lw2llw {__version__}) ==\n")
        f.writelines(l + '\n' for l in lines)
        f.write('\n')


# ------------------------------------------------------------- the engine ---
class Engine:
    def __init__(self, online, local, doit, machine=False, writer=None, purge=None):
        self.on, self.lo, self.doit, self.machine = online, local, doit, machine
        self.writer = writer or import_dump
        self.purge = purge or (lambda titles: None)
        self.res = {}                       # kind -> [(title, note)]
        self.written, self.merged, self.refused, self.logl = [], [], {}, []
        self.hold = read_hold()

    def note(self, kind, title, msg=''):
        self.res.setdefault(kind, []).append((title, msg))

    def payload(self, revs, texts):
        out = []
        for r in revs:
            text, model, fmt = texts[r['revid']]
            out.append({'timestamp': r['timestamp'], 'user': r['user'], 'comment': r['comment'],
                        'text': text, 'model': model, 'format': fmt, 'sha1': r['sha1']})
        return out

    def do_page(self, title, ns, online, local, llogs, global_sha=None, again=True):
        if title in self.hold:
            return 'held'                       # refused once, at the start of the run
        kind, info = plan_page(title, ns, online, local, llogs, global_sha)
        if kind in ('in step', 'syncy', 'excluded'):
            return kind
        if kind in ('deleted here', 'renamed here', 'renamed online', 'text here under another title', 'stale',
                    'file description upload text'):
            self.note(kind, title, info.get('target') or info.get('when') or info.get('cut') or '')
            return kind
        if kind == 'no base':
            self.refuse(title, 'online and here share no revision — compare them by hand, then '
                               f'lw2llw --resolved "{title}"')
            return kind
        revs = info['revisions']
        if kind == 'merge':
            return self.do_merge(title, ns, info, local)
        texts = self.on.texts([r['revid'] for r in revs])
        pay = self.payload(revs, texts)
        if kind == 'fast-forward' and revs[-1]['timestamp'] <= info['local_top']['timestamp']:
            last = pay[-1]          # an online edit stamped no later than the local top: bring it on top
            pay.append(dict(last, timestamp=after(info['local_top']['timestamp'], {r['timestamp'] for r in local}),
                            user=MERGE_USER, comment='lw2llw: the online edit brought on top'))
        expect = pay[-1]['sha1'] or sha1_hex(pay[-1]['text'])
        desc = (f"{len(revs)} online edit(s), {revs[0]['timestamp'][:10]}"
                + (f"…{revs[-1]['timestamp'][:10]}" if len(revs) > 1 else '')
                + f", by {', '.join(sorted({r['user'] for r in revs}))}")
        if not self.doit:
            self.note('would ' + kind, title, desc)
            return kind
        if not self.unchanged_here(title, local):
            if again:
                ns2, local2 = self.lo.history(title)
                return self.do_page(title, ns, online, local2, self.lo.logs_for(title), global_sha, again=False)
            self.note('changed here meanwhile', title, 'left for the next run')
            return 'raced'
        self.writer(title, ns, pay)
        ok = self.verify(title, expect)
        if not ok:
            self.hold_page(title, f'lw2llw {kind} did not end on the online text; check the history '
                                  f'here, then lw2llw --resolved "{title}"')
            return 'failed'
        self.written.append(title)
        self.note(kind, title, desc)
        self.logl.append(f'{kind}: {title} — {desc}')
        self.emit('PULLED', title)
        return kind

    def do_merge(self, title, ns, info, local):
        revs = info['revisions']
        texts = self.on.texts([info['base']['revid'], info['online_top']['revid']] + [r['revid'] for r in revs])
        ltext = self.lo.texts([info['local_top']['revid']])[info['local_top']['revid']]
        base, theirs, mine = texts[info['base']['revid']][0], texts[info['online_top']['revid']][0], ltext[0]
        clean, merged = merge3(base, mine, theirs)
        when = ', '.join(sorted({r['timestamp'][:10] for r in revs}))
        if not clean:
            path = save_clash(title, base, mine, theirs, merged, info)
            self.refuse(title, f'the same paragraph changed on both wikis (online {when}); the three '
                               f'texts and the marked merge are in {path}')
            return 'conflict'
        newest = max(r['timestamp'] for r in revs)
        pay = self.payload(revs, texts)
        if merged == mine.rstrip('\n') and info['local_top']['timestamp'] > newest:
            kind, expect = 'already merged here', info['local_top']['sha1']
            write = pay                                   # filed beneath the local text, which stays current
        else:
            kind = 'merge'
            m = {'timestamp': after(max(info['local_top']['timestamp'], newest), {r['timestamp'] for r in local}),
                 'user': MERGE_USER, 'text': merged, 'model': ltext[1], 'format': ltext[2],
                 'comment': 'lw2llw: merged with the online edit(s) of ' + when
                            + ''.join(f' ({c})' for c in sorted({r['comment'] for r in revs if r['comment']})[:2])}
            expect = sha1_hex(merged)
            write = [m] + pay                             # the merge FIRST: current at once
        desc = f"{len(revs)} online edit(s) of {when} merged with the local text"
        if not self.doit:
            self.note('would ' + kind, title, desc)
            return kind
        if not self.unchanged_here(title, local):
            self.note('changed here meanwhile', title, 'left for the next run')
            return 'raced'
        self.writer(title, ns, write)
        if not self.verify(title, expect):
            self.hold_page(title, 'the merge did not end on the merged text; check the history here, '
                                  f'then lw2llw --resolved "{title}"')
            return 'failed'
        manifest_add([title])                             # so llw2lw pushes it
        self.written.append(title)
        self.merged.append(title)
        self.note(kind, title, desc)
        self.logl.append(f'{kind}: {title} — {desc}')
        self.emit('MERGED', title)
        return kind

    def resolve(self, title):
        """He merged by hand: file the online revisions under his text."""
        ns, online = self.on.history(title)
        ns_l, local = self.lo.history(title)
        if not online or not local:
            raise Refusal(f'{title}: needs the page on both wikis')
        lsha = {r['sha1'] for r in local}
        base_i = max((i for i, r in enumerate(online) if r['sha1'] in lsha), default=-1)
        revs = online[base_i + 1:]
        if not revs:
            self.hold.pop(title, None)
            self.note('nothing to resolve', title, 'online holds nothing new')
            return
        texts = self.on.texts([r['revid'] for r in revs])
        pay = self.payload(revs, texts)
        ltop = local[-1]
        ltext = self.lo.texts([ltop['revid']])[ltop['revid']]
        newest = max(r['timestamp'] for r in revs)
        write, expect = pay, ltop['sha1']
        if ltop['timestamp'] <= newest:                   # his text must stay on top
            write = [{'timestamp': after(newest, {r['timestamp'] for r in local}), 'user': MERGE_USER,
                      'comment': 'lw2llw --resolved: merged by hand', 'text': ltext[0],
                      'model': ltext[1], 'format': ltext[2]}] + pay
            expect = sha1_hex(ltext[0])
        if not self.doit:
            self.note('would resolve', title, f'{len(revs)} online revision(s) filed under the local text')
            return
        self.writer(title, ns_l, write)
        if not self.verify(title, expect):
            raise Refusal(f'{title}: the local text is not on top after filing — look at the history')
        self.hold.pop(title, None)
        manifest_add([title])
        self.written.append(title)
        self.note('resolved', title, f'{len(revs)} online revision(s) filed under the local text')
        self.logl.append(f'resolved: {title}')

    def unchanged_here(self, title, local):
        ns, now = self.lo.history(title)
        return [r['revid'] for r in now] == [r['revid'] for r in local]

    def verify(self, title, sha1):
        cur = self.lo.top(title)
        return bool(cur) and cur['sha1'] == sha1

    def refuse(self, title, why):
        self.refused[title] = why
        self.note('REFUSED', title, why)
        self.emit('REFUSED', title, why)

    def hold_page(self, title, why):
        self.hold[title] = why
        self.refuse(title, why)

    def emit(self, what, title, why=''):
        if self.machine:
            print(f'LW2LLW-{what}: {title}' + (f'\t{why}' if why else ''), flush=True)

    # ------------------------------------------------------------ files ---
    def do_files(self, names, deleted_here, moved_here):
        names = sorted({norm(n) for n in names})
        if not names:
            return set()
        here, there = self.lo.files(names), self.on.files(names)
        users_here = self.lo.users_exist([i['user'] for i in there.values() if i])
        handled = set()
        for n in names:
            o, l = there.get(n), here.get(n)
            if not o:
                continue
            t = 'File:' + n
            if l and l['sha1'] == o['sha1']:
                continue
            if not l and t in deleted_here:
                self.note('deleted here', t, deleted_here[t]); handled.add(t); continue
            if not l and t in moved_here:
                self.note('renamed here', t, moved_here[t]); handled.add(t); continue
            over = False
            if l:
                if l['sha1'] in (o['history'] or []):
                    over = True                          # online uploaded a newer version over ours
                else:
                    self.refuse(t, 'the file changed on both wikis — compare the two versions by hand')
                    handled.add(t); continue
            ns, ohist = self.on.history(t)
            otexts = self.on.texts([r['revid'] for r in ohist]) if ohist else {}
            desc = otexts.get(ohist[-1]['revid'], ('',))[0] if ohist else ''
            note = f"uploaded online {o['timestamp'][:10]} by {o['user']}" + (' (new version)' if over else '')
            if not self.doit:
                self.note('would bring the file', t, note); handled.add(t); continue
            try:
                if over:
                    import_file(n, self.on.download(o['url']), dict(o, overwrite=True), desc, users_here)
                else:
                    import_new_file(n, self.on.download(o['url']), o, self.payload(ohist, otexts))
            except Exception as e:
                self.note('FAILED', t, str(e)[:200]); handled.add(t); continue
            got = self.lo.files([n]).get(n)
            if not got or got['sha1'] != o['sha1']:
                self.note('FAILED', t, 'the bytes here do not match laussy.org after the import'); handled.add(t)
                continue
            self.written.append(t)
            self.note('file brought', t, note)
            self.logl.append(f'file: {t} — {note}')
            self.emit('PULLED', t)
            handled.add(t)
        return handled


# --------------------------------------------------------------------- main ---
def parse_since(s):
    if re.fullmatch(r'\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ', s):
        return dt.datetime.strptime(s, ISO).replace(tzinfo=UTC)
    return LL.parse_since(s).astimezone(UTC)


def main():
    a = sys.argv[1:]
    if '-h' in a or '--help' in a:
        print(__doc__.strip()); return 0
    if '-v' in a or '--version' in a:
        print(f'lw2llw {__version__}'); return 0
    LL.run_on_sync_host('lw2llw', 'LW2LLW_FORWARDED')
    doit, machine, do_all = '--doit' in a, '--from-llw2lw' in a, '--all' in a
    pages, resolved, since = [], [], None
    i = 0
    while i < len(a):
        x = a[i]
        if x in ('--page', '--resolved', '--since') and i + 1 < len(a):
            {'--page': pages, '--resolved': resolved}.get(x, []).append(a[i + 1])
            if x == '--since':
                since = parse_since(a[i + 1])
            i += 2; continue
        if x not in ('--doit', '--from-llw2lw', '--all'):
            sys.exit(f'lw2llw: unknown argument {x!r} (see -h)')
        i += 1
    cfg = LL.read_config()
    on = Wiki('online', LL.mw_session(cfg['headers']), cfg['online_api'], pause=0.2 if do_all else 0.0)
    lsession = LL.mw_session()
    if cfg.get('local_user') and cfg.get('local_pass'):
        try:
            LL.mw_login(lsession, cfg['local_api'], cfg['local_user'], cfg['local_pass'])
        except Exception as e:
            print(f'  (local login failed, purges skipped: {e})')
    lo = Wiki('local', lsession, cfg['local_api'])

    def purge(titles):
        for k in range(0, len(titles), 50):
            try:
                lsession.post(cfg['local_api'], data={'action': 'purge', 'forcelinkupdate': '1', 'format': 'json',
                                                      'titles': '|'.join(titles[k:k + 50])}, timeout=120)
            except Exception as e:
                print(f'  (purge failed: {e})')

    eng = Engine(on, lo, doit, machine, purge=purge)
    for t, why in sorted(eng.hold.items()):
        if t not in resolved:                 # a held page may look in step by sha1: refuse it regardless
            eng.refuse(t, why)
    print(f"lw2llw {__version__} — {'writing' if doit else 'dry run (--doit to write)'}"
          + (' — whole online history' if do_all else ''))

    if resolved:
        for t in resolved:
            try:
                eng.resolve(t)
            except Refusal as e:
                print(f'\033[1;31m{e}\033[0m')
        purge(eng.written)
        write_hold(eng.hold) if doit else None
        report(eng); log(eng.logl if doit else []); return 0

    # ---- the candidates ---------------------------------------------------
    global_sha, llog_all, ldel_files, lmov_files = None, None, {}, {}
    if do_all:
        print('reading both revision catalogues (a minute or two)…', flush=True)
        orows, lrows = on.all_revisions(), lo.all_revisions()
        global_sha = {r['sha1'] for r in lrows if r['sha1']}
        llog_all = {}
        for k in ('delete', 'move'):
            for t, es in lo.all_logs(k).items():
                llog_all.setdefault(t, []).extend(es)
        ofiles, lfiles = on.all_files(), lo.all_files()
        file_cands = [n for n, s in ofiles.items() if lfiles.get(n) != s]
    elif pages:
        orows, lrows = [], []
        file_cands = [norm(p.split(':', 1)[1]) for p in pages if p.startswith(('File:', 'Image:'))]
    else:
        given = since is not None
        if since is None:
            since = LL.get_last_update_date().astimezone(UTC) - WINDOW_MARGIN
        s_iso = since.strftime(ISO)
        print(f"online edits since {since.astimezone(LL.PARIS).strftime('%Y-%m-%d %H:%M')}"
              + ('' if given else " (a day before llw2lw's LAST UPDATE; --since for another)"))
        orows, lrows = on.revisions_since(s_iso), None
        file_cands = [e['title'].split(':', 1)[1] for e in on.logs_since('upload', s_iso)]
        for e in on.logs_since('move', s_iso):
            eng.note('renamed online (listed, not applied)', e['title'], '→ ' + e['target'])
        for e in on.logs_since('delete', s_iso):
            if e['action'] in ('delete', 'delete_redir'):
                eng.note('deleted online (listed, not applied)', e['title'], e['comment'][:80])

    by_title = {}
    for r in orows:
        by_title.setdefault(r['title'], []).append(r)
    for t in pages:
        by_title.setdefault(norm(t), [])
    lby = None
    if lrows is not None and do_all:
        lby = {}
        for r in lrows:
            lby.setdefault(r['title'], []).append(r)
    # files first: a file absent here comes with its description page
    if file_cands:
        if do_all:
            dh = {t: es[-1]['timestamp'] for t, es in llog_all.items() if t.startswith('File:')
                  and any(e['type'] == 'delete' for e in es)}
            mh = {t: es[-1]['target'] for t, es in llog_all.items() if t.startswith('File:')
                  and any(e['type'] == 'move' for e in es)}
        else:
            dh, mh = {}, {}
            for n in file_cands:
                es = lo.logs_for('File:' + norm(n))
                if any(e['type'] == 'delete' for e in es):
                    dh['File:' + norm(n)] = es[-1]['timestamp']
                if any(e['type'] == 'move' for e in es):
                    mh['File:' + norm(n)] = es[-1]['target']
        done_files = eng.do_files(file_cands, dh, mh)
    else:
        done_files = set()

    titles = sorted(by_title)
    print(f'{len(titles)} page(s) with online revisions to look at', flush=True)
    for t in titles:
        if t in done_files:
            continue
        rows = sorted(by_title[t], key=lambda r: (r['timestamp'], r['revid']))
        try:
            if lby is not None:
                local = sorted(lby.get(t, []), key=lambda r: (r['timestamp'], r['revid']))
                llogs = llog_all.get(t, [])
                online, ns = rows, rows[0]['ns'] if rows else 0
            else:
                ns_l, local = lo.history(t)
                if local:
                    cur = lo.top(t)
                    if cur and cur['revid'] != local[-1]['revid']:
                        local = [r for r in local if r['revid'] != cur['revid']] + [cur]
                lsha = {r['sha1'] for r in local}
                if rows and rows[-1]['sha1'] in lsha:
                    continue                              # a push of ours: in step, nothing to ask online
                ns, online = on.history(t)
                if ns is None:
                    continue
                if online and online[-1]['sha1'] in lsha:
                    continue
                llogs = lo.logs_for(t)
            if ns == 6 and not local and t in done_files:
                continue
            eng.do_page(t, ns, online, local, llogs, global_sha)
        except Exception as e:                  # one page's surprise must not stop the others
            eng.note('FAILED', t, f'{type(e).__name__}: {e}'[:200])
    if doit:
        purge([t for t in eng.written if not t.startswith('File:')] + [t for t in eng.written if t.startswith('File:')])
        write_hold(eng.hold)
        log(eng.logl)
    report(eng, full=do_all or bool(pages))
    return 2 if eng.refused else (1 if eng.res.get('FAILED') else 0)


ORDER = ['fast-forward', 'new', 'file brought', 'merge', 'already merged here', 'resolved',
         'would fast-forward', 'would new', 'would bring the file', 'would merge', 'would already merged here',
         'would resolve', 'REFUSED', 'FAILED', 'changed here meanwhile', 'deleted here', 'renamed here',
         'renamed online', 'renamed online (listed, not applied)', 'deleted online (listed, not applied)',
         'text here under another title', 'stale', 'file description upload text', 'nothing to resolve']
HEAD = {'fast-forward': 'Online edits brought here', 'new': 'Pages created online, brought here',
        'file brought': 'Files brought here', 'merge': 'Online edits merged with local ones (llw2lw pushes the merge)',
        'already merged here': 'Online edits already merged here (filed beneath the local text)',
        'resolved': 'Resolved by hand', 'REFUSED': 'CHANGED ON BOTH WIKIS — merge by hand',
        'FAILED': 'FAILED', 'deleted here': 'Left alone: deleted here', 'renamed here': 'Left alone: renamed here',
        'renamed online': 'Left alone: arrived online by a move',
        'text here under another title': 'Left alone: its text is here under another title',
        'stale': 'Left alone: online revisions older than a deletion/rename here',
        'file description upload text': 'File descriptions that differ only by their upload text'}
QUIET = {'file description upload text', 'stale', 'deleted here', 'renamed here'}


def report(eng, full=True):
    if not full and set(eng.res) <= {'file description upload text'}:
        n = len(eng.res.get('file description upload text', []))
        print('Nothing written online that this wiki lacks.'
              + (f' ({n} file description(s) differ only by their upload text: ignored.)' if n else ''))
        return
    if not eng.res:
        print('Nothing written online that this wiki lacks.')
        return
    for k in ORDER + sorted(set(eng.res) - set(ORDER)):
        items = eng.res.get(k)
        if not items:
            continue
        if not full and k == 'file description upload text':
            print(f'\n({len(items)} file description(s) differ only by their upload text: ignored.)')
            continue
        head = HEAD.get(k) or HEAD.get(k[6:], k[6:]) + ' (dry run)' if k.startswith('would ') else HEAD.get(k, k)
        col = '\033[1;31m' if k in ('REFUSED', 'FAILED') else '\033[1;36m' if not k.startswith(('would', 'Left')) else '\033[1m'
        print(f'\n{col}== {head} ({len(items)}) ==\033[0m')
        shown = items if k not in QUIET or len(items) <= 12 else items[:8]
        for t, msg in shown:
            print(f'{t}' + (f' — {msg}' if msg else ''))
        if len(shown) < len(items):
            print(f'… and {len(items) - len(shown)} more')


if __name__ == '__main__':
    try:
        sys.exit(main())
    except LL.ApiFailure as e:
        sys.exit(f'lw2llw: {e}')