MediaWiki API

An Application Programming Interface (API) is a connection between computers or between computer programs. I use the MediaWiki API mainly to check consistency between my online (lw) and local (llw) wikis.

My llw2lw script should keep things in sync unless I inadvertently forget or disrupt this myself (say, by updating something online directly without syncinc it locally).

In ~/comp/python/compare_llw_lw I have a series of scripts to make various types of comparisons between the two:

compare_wikis.py

This is the main (first) script.

→ Collects all page titles + revids from both wikis

→ Saves them in local_pages.json and online_pages.json

→ Tells you how many pages differ by revid

local_pages.json and online_pages.json

Dictionaries of { "Page title": "revid" } (or timestamp possible but timezones are different)

compare_real_content.py

A second script, which makes the actual comparison.

→ Takes the list of revid-differing pages

→ Downloads the actual wikitext from both wikis

→ Compares SHA256 hashes of the content

→ Tells you which pages really differ (ignores null edits, imports, revid-only changes)

different_pages

This generates different_pages.txt and different_pages_with_revid.txt as well as non_file_differences.txt and real_content_differences.txt

They are the output files from the previous one-liners and scripts

→ Lists of pages that differ (by revid or by real content)

Procedure

• Run the main revid comparison (fast, ~5–15 min depending on number of pages)

Run the main revid comparison (fast, ~5–15 min depending on number of pages)

→ requires password but this is put in the file for convenience.

→ Produces updated local_pages.json and online_pages.json.

→ Tells you the total number of pages that differ by revid (was 1457 on first run).

• Quickly see how many are real content differences (most important step)

python compare_real_content.py

→ Needs password again (in file too)

→ It automatically reads the latest JSON files

→ Only checks the pages that differ by revid

→ Prints the number of pages with actual wikitext differences

→ Saves the clean list to real_content_differences.txt. This is the number to really care about — usually much lower than 1457 (many are just import artifacts / null edits).

Exclude File:differences:

python3 -c '
import json
local = json.load(open("local_pages.json"))
online = json.load(open("online_pages.json"))
diff = sorted(t for t in set(local)|set(online) if local.get(t) != online.get(t) and not t.startswith("File:"))
print("Non-File differences:", len(diff))
print("\n".join(diff[:50]))
' > non_file_differences.txt

If I get rate-limited or timeouts in compare_real_content.py, then increase the time.sleep(0.7) to time.sleep(1.5).

This requires cloudflare to be disabled or to grant access (not under-attack mode).