wikigrep
wikigrep is a python script that greps the whole of laussywiki—every page, exactly, including regular expressions—by keeping a local copy of all the wikitext and searching that instead of asking the wiki.
Written with the help of Claude Opus 5 on 18 August (2026), after a search that lied to me. Version 1.0.1.
The idea
The wiki's own search works, but it answers a different question from the one you think you asked. list=search defaults to srwhat=title: it matches titles and reports totalhits=0 for words sitting in plain view in the body of a page. Ask it for srwhat=text and it behaves. Nothing warns you; you simply get an empty answer that looks like an answer.
That is how this script came to exist. While rolling out Anno Fabri 7.1.0 I searched for a handful of tags, got nothing, concluded they were nowhere on the wiki, and missed both a published 𝕐 post and an entire fourth implementation of the algorithm—the header clock. The lesson is not about MediaWiki. When a search returns nothing, suspect the search before believing the silence.
So the tool inverts the trust: it holds its own copy of every page and greps it with python's regex engine, where a negative result is a real negative and means what it says.
Usage
wikigrep PATTERN # regex over every page: "Title:line: text"
wikigrep -F 01RJr # a literal, no metacharacter surprises
wikigrep -l -i feistel # just the titles that match
wikigrep -c "\{\{af\|" # how many matches on each page
wikigrep -C 2 SALT # two lines of context around each hit
wikigrep -n 0,100 polariton # only these namespaces
wikigrep '0B[A-Za-z0-9]{3}' # a real regex: every era-0B tag
wikigrep --status # what the cache holds, and how fresh
Output is Title:line:text and the exit status is 1 when nothing matched, so it pipes and scripts exactly like grep.
The cache
The first run downloads everything: 20,121 pages, 12 MB, in about ten seconds. After that every run asks recentchanges what has changed since the last sync—normally one API call—and re-fetches only those pages, so a search costs about 0.4 s and is never stale. Deleted and renamed pages fall out of the cache on their own, because a title that no longer resolves is dropped.
The cache lives in ~/.cache/wikigrep/. --full re-downloads everything, --offline never touches the network, and $WIKIGREP_API points it at another wiki.
| namespace | pages |
|---|---|
| 0 (main) | 2615 |
| 6 (File) | 11,206 |
| 10 (Template) | 5754 |
| 100 (Blog) | 458 |
| 828 (Module) | 30 |
| others | 58 |
When to use which
Use the wiki's search for prose, where relevance ranking is the point—just remember srwhat=text. Use wikigrep when you want certainty or precision: exact short tokens (AF tags, constants, gadget names), anything containing punctuation or wiki markup, regular expressions, substrings, and stopwords. without sits on 325 pages of this wiki and the search engine scores it zero, because MySQL full-text drops it as too common. There is no stemming either, so microcavity will not find Microcavities.
CirrusSearch would fix the ranking and add insource:, but it wants an Elasticsearch daemon that the online host could never run, and for exact matching a grep beats a search engine anyway.
Limitations
It greps wikitext, not rendered HTML: text that only exists after a template expands is not there to be found. It knows nothing of page history—only current revisions. And it is as fresh as its last sync, which is to say very, unless --offline.
Version history
- 1.0.0 (18 August (2026)): first version.
- 1.0.1:
--helpprinted every line run together, and its header still repeated the wrong diagnosis of the wiki's search.
The file
~/bin/wikigrep
#!/usr/bin/env python3
# wikigrep — v1.0.1 — grep the WHOLE local wiki, exactly.
#
# The wiki's own search is fine, but you must ask it for text: list=search
# defaults to srwhat=title and reports nothing for words plainly on the page.
# Even asked properly it drops stopwords ("without" is on 325 pages and scores
# zero), and it has no substrings and no regex. CirrusSearch is not installed,
# so insource: is not an operator either. This keeps a local copy of every
# page's wikitext and greps that, so a negative here is a real negative.
#
# The cache refreshes itself incrementally from recentchanges on every run —
# normally one API call — so results are current without a full re-download.
#
# wikigrep PATTERN regex over every page, "Title:line: text"
# wikigrep -F 01RJr literal, no regex metacharacters
# wikigrep -l -i feistel just the titles that match
# wikigrep -n 0,100 foo only these namespaces (numbers or names)
# wikigrep -C 2 SALT two lines of context
# wikigrep --status what the cache holds, and how fresh
# wikigrep --full re-download everything, then search
# wikigrep --offline PATTERN do not touch the network
#
# Cache: ~/.cache/wikigrep/ API: $WIKIGREP_API or localhost/laussywiki
# F.P. Laussy
import argparse, json, os, re, sys, time, urllib.parse, urllib.request
API = os.environ.get("WIKIGREP_API", "http://localhost/laussywiki/api.php")
CACHE = os.path.expanduser("~/.cache/wikigrep")
PAGES = os.path.join(CACHE, "pages.jsonl")
META = os.path.join(CACHE, "meta.json")
UA = "wikigrep/1.0.1 (local; Fabrice)"
def api(**params):
params.setdefault("format", "json"); params.setdefault("formatversion", "2")
req = urllib.request.Request(API + "?" + urllib.parse.urlencode(params),
headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=60) as r:
return json.load(r)
def load():
if not os.path.exists(PAGES): return {}, {}
pages = {}
with open(PAGES, encoding="utf-8") as f:
for line in f:
if line.strip():
p = json.loads(line); pages[p["title"]] = p
meta = json.load(open(META)) if os.path.exists(META) else {}
return pages, meta
def save(pages, meta):
os.makedirs(CACHE, exist_ok=True)
tmp = PAGES + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
for t in sorted(pages):
f.write(json.dumps(pages[t], ensure_ascii=False) + "\n")
os.replace(tmp, PAGES)
json.dump(meta, open(META, "w"))
def now_iso():
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
def fetch_titles(titles, pages, verbose):
"""Fetch content for these titles; drop the ones that no longer exist."""
titles = list(titles); got = 0
for i in range(0, len(titles), 50):
batch = titles[i:i + 50]
d = api(action="query", prop="revisions", rvprop="content|timestamp",
rvslots="main", titles="|".join(batch))
for p in d.get("query", {}).get("pages", []):
if p.get("missing"):
pages.pop(p["title"], None); continue
rev = p["revisions"][0]
pages[p["title"]] = {"title": p["title"], "ns": p["ns"],
"ts": rev["timestamp"],
"text": rev["slots"]["main"].get("content", "")}
got += 1
if verbose and len(titles) > 200:
print(f"\r fetched {min(i+50,len(titles))}/{len(titles)}", end="", file=sys.stderr)
if verbose and len(titles) > 200: print(file=sys.stderr)
return got
def full_sync(verbose=True):
pages = {}
ns = [int(k) for k in api(action="query", meta="siteinfo",
siprop="namespaces")["query"]["namespaces"] if int(k) >= 0]
for n in ns:
cont = {}
while True:
d = api(action="query", generator="allpages", gapnamespace=n,
gaplimit="50", prop="revisions", rvprop="content|timestamp",
rvslots="main", **cont)
for p in d.get("query", {}).get("pages", []):
if p.get("missing") or "revisions" not in p: continue
rev = p["revisions"][0]
pages[p["title"]] = {"title": p["title"], "ns": p["ns"],
"ts": rev["timestamp"],
"text": rev["slots"]["main"].get("content", "")}
if "continue" in d: cont = d["continue"]
else: break
if verbose:
print(f"\r namespace {n:>3}: {len(pages)} pages so far", end="", file=sys.stderr)
if verbose: print(file=sys.stderr)
return pages, {"last_sync": now_iso()}
def incremental(pages, meta, verbose=True):
since = meta.get("last_sync")
if not since: return None # caller falls back to full
changed, cont = set(), {}
while True:
d = api(action="query", list="recentchanges", rcdir="newer",
rcstart=since, rcend=now_iso(), rclimit="500",
rcnamespace="*", rcprop="title|timestamp", **cont)
if "error" in d: return None
for r in d["query"]["recentchanges"]: changed.add(r["title"])
if "continue" in d: cont = d["continue"]
else: break
if changed:
if verbose: print(f" {len(changed)} page(s) changed since {since}", file=sys.stderr)
fetch_titles(changed, pages, verbose)
meta["last_sync"] = now_iso()
return len(changed)
def main():
ap = argparse.ArgumentParser(add_help=False)
ap.add_argument("pattern", nargs="?")
ap.add_argument("-i", "--ignore-case", action="store_true")
ap.add_argument("-F", "--fixed", action="store_true")
ap.add_argument("-l", "--files-with-matches", action="store_true")
ap.add_argument("-c", "--count", action="store_true")
ap.add_argument("-C", "--context", type=int, default=0)
ap.add_argument("-n", "--namespace", default=None)
ap.add_argument("--full", action="store_true")
ap.add_argument("--offline", action="store_true")
ap.add_argument("--status", action="store_true")
ap.add_argument("-h", "--help", action="store_true")
ap.add_argument("-v", "--version", action="store_true")
a = ap.parse_args()
if a.help or (not a.pattern and not a.status and not a.full and not a.version):
for l in open(__file__).read().split("\n")[1:]:
if not l.startswith("#"): break
print(l[2:] if l.startswith("# ") else l[1:])
return 0
if a.version:
print(open(__file__).read().split("\n")[1][2:]); return 0
pages, meta = load()
if not a.offline:
try:
if a.full or not pages:
pages, meta = full_sync()
else:
if incremental(pages, meta) is None:
pages, meta = full_sync()
save(pages, meta)
except Exception as e:
print(f"wikigrep: cannot reach the wiki ({e}); using the cache as it stands",
file=sys.stderr)
if a.status:
by = {}
for p in pages.values(): by[p["ns"]] = by.get(p["ns"], 0) + 1
total = sum(len(p["text"]) for p in pages.values())
print(f" cache : {PAGES}")
print(f" pages : {len(pages)} text: {total/1048576:.1f} MB")
print(f" synced : {meta.get('last_sync','never')}")
print(" by namespace: " + ", ".join(f"{k}:{v}" for k, v in sorted(by.items())))
if not a.pattern: return 0
wanted = None
if a.namespace:
wanted = set()
for tok in a.namespace.split(","):
tok = tok.strip()
if tok.lstrip("-").isdigit(): wanted.add(int(tok))
else:
for k, v in api(action="query", meta="siteinfo",
siprop="namespaces")["query"]["namespaces"].items():
if (v.get("name") or "(main)").lower() == tok.lower(): wanted.add(int(k))
pat = re.escape(a.pattern) if a.fixed else a.pattern
rx = re.compile(pat, re.IGNORECASE if a.ignore_case else 0)
hits = nmatch = 0
for title in sorted(pages):
p = pages[title]
if wanted is not None and p["ns"] not in wanted: continue
lines = p["text"].split("\n")
found = [(i, l) for i, l in enumerate(lines, 1) if rx.search(l)]
if not found: continue
hits += 1; nmatch += len(found)
if a.files_with_matches: print(title); continue
if a.count: print(f"{title}:{len(found)}"); continue
for i, l in found:
if a.context:
for j in range(max(1, i - a.context), min(len(lines), i + a.context) + 1):
sep = ":" if j == i else "-"
print(f"{title}:{j}{sep}{lines[j-1]}")
print("--")
else:
print(f"{title}:{i}:{l}")
if not a.files_with_matches and not a.count:
print(f"\n {nmatch} match(es) on {hits} page(s) of {len(pages)}", file=sys.stderr)
return 0 if hits else 1
if __name__ == "__main__":
sys.exit(main())