html2md
html2md is a python script that turns a selection copied out of a browser—typically a piece of a Claude answer read in its own window—into wikitext, with the maths back in $\mathrm{\TeX}$ form. The name is older than the behaviour: until 2.0.0 it produced Markdown, which is still there under -m, but this is a wiki-writing machine and wikitext is what I paste nine times out of ten.
Reading an answer on the web, I see the rendered page: the Markdown is gone, and every formula is a heap of KaTeX spans. Selecting a paragraph with the mouse and pasting it into an edit box gives ugly prose without any of its structure, and formulae reduced to loose glyphs—usually repeated, because the rendering and the MathML behind it are both in the selection. This script gives back the source. It sits on the clipboard and does it by itself: there is nothing to press.
Written with the help of Claude Opus 5 on 1 August (2026). Version 2.5.0.
The idea
There is nothing to reverse-engineer: the $\mathrm{\TeX}$ is already on the clipboard. A browser puts a copied selection there twice, once as text/plain—what the eye sees—and once as text/html, the real markup. And KaTeX, which is what renders maths in a chat window, keeps the $\mathrm{\TeX}$ it was handed inside the MathML annotation of every formula it builds:
<span class="katex"><span class="katex-mathml"><math><semantics>
... <annotation encoding="application/x-tex">2g\sqrt{n}</annotation>
</semantics></math></span><span class="katex-html" aria-hidden="true">...</span></span>
That annotation travels with the text/html flavour. Only the plain-text flavour throws it away—and the plain-text flavour is precisely the one every text form takes when I hit Ctrl+V. So the whole trick is to read the other target of the same clipboard:
xclip -selection clipboard -t TARGETS -o # see the flavours on offer
xclip -selection clipboard -t text/html -o # the one that has everything
No userscript, no extension, no export, nothing to install in the browser. This is the same observation mdtable2wiki makes for tables; html2md generalises it to a whole selection.
It runs by itself
html2md --watch sits on the clipboard and converts every copy as it arrives, so the gesture is just copy, paste—no keystroke in between, and it works into Emacs, a mail, the wiki edit box, anything. It is started at login from ~/.config/autostart/net.local.html2md-watch.desktop.
Nothing is lost by letting it rewrite every copy, because it only adds: the Markdown goes on as text/plain—the flavour Emacs and every text field read—while the original text/html is put back untouched, so pasting into a mail or a word processor still gets the formatting. Every other flavour of the copy (images, the source URL) is carried over too.
Three details make it work rather than nearly work. First, the watcher must ignore its own write or it converts its output forever—but the test for that is who owns the clipboard, never whether the HTML looks familiar. Comparing content was the first attempt, and it fails the moment I copy the same passage twice: the browser hands over byte-identical markup, the watcher takes the second copy for its own echo, and from then on nothing happens however hard I press Ctrl+C. It looks exactly like a crash, and it is not one. Ownership tells the two apart with no ambiguity: after my write I own the selection, after a copy the browser does.
Second, the notification must be fired with Popen, never run: notify-send lives as long as the toast is on screen, so waiting for it freezes the event loop and the next copy is missed entirely.
Third, an exception escaping a slot is fatal in PyQt5—one malformed copy would take the watcher down and every paste after it—so the whole thing runs inside a try, and what it decided goes to ~/.cache/html2md/watch.log. A one-second timer calls the same code as the signal, so even if dataChanged ever stops arriving, nothing is lost for more than a second. That last sentence was wrong for a year and it took a day of silence to find out why: see #When it stops working.
Which paste, exactly
Two things nearly undid all of this at the last step, both of them about where an application looks when it pastes.
A terminal pastes with the middle button, and that is the PRIMARY selection, a different thing from the clipboard: Emacs took the converted Markdown from the clipboard while Konsole, one window away, was still handing back the browser's own glyphs—K from the MathML, K from the annotation and K from the rendering, the formula three times over. So a conversion now writes the Markdown to PRIMARY as well—and, since the middle button pastes what was merely selected, a selection that was never copied is converted too, provided it carries a formula. That last part is the intrusive one: owning PRIMARY may cost the highlight in the window I am reading, so --no-select gives it back, and --no-primary leaves PRIMARY alone altogether.
And Qt offers text/plain, UTF8_STRING, STRING and TEXT but not text/plain;charset=utf-8, which the browser does offer and which GTK and VTE applications ask for by name. They were getting an empty answer where a paste should have been. It is one line to add, and worth checking with xclip -t target by target rather than trusting that a clipboard is a clipboard.
Which language comes out
Wikitext is the default: bold, italics, == headings ==, * and # lists nested by lengthening the marker, [url label], {| class="wikitable" tables one cell to a line, <syntaxhighlight lang="…"> for code, <blockquote> for a quotation, and $…$ left as it is, since SimpleMathJax reads the dollars. Two details are worth the words: a pipe inside a cell becomes |, because MediaWiki cuts cells on it before anything else runs, and an ordered list that resumes at 3 is written <ol start="3">, since # restarts at 1 whatever one does.
-m gives the Markdown instead, and there is one place where it is required: inside the a= of {{AI}}. Module:AI converts Markdown itself, and it reads a leading ** as bold—so a wikitext nested list handed to it comes back as a stray asterisk and half a list. Measured on a specimen: six list items became four. An answer quoted in an {{AI}} box therefore goes in as Markdown, which is also right in principle, the box being a facsimile of what the machine wrote.
By hand
html2md # clipboard in, Markdown to stdout (a look at it)
html2md -c # clipboard in, Markdown back on the clipboard
html2md -l # convert the last clipboard again, from the cache
html2md -m # Markdown instead, for the a= of Template:AI
html2md -x -c -m -a # ... and {{AI|a=}}-proof: every | becomes {{!}}
html2md page.html # a file in
... | html2md # stdin in
Meta+F9 runs html2md -x -c -n for the same thing on demand, if the watcher is not running. Not a letter: a global shortcut is a grab on a keycode, and with three layouts loaded (fr,es,ru) a letter moves from one physical key to another—M is not where it was between French and Spanish, and under the Russian layout it has no keycode at all. The key then falls through to the application, which types a character instead. Function keys sit still in every layout.
# ~/.local/share/applications/net.local.html2md.desktop
[Desktop Entry]
Exec=/home/laussy/bin/html2md -x -c -n
Name=HTML to Markdown
NoDisplay=true
Type=Application
X-KDE-GlobalAccel-CommandShortcut=true
# ~/.config/kglobalshortcutsrc
[services][net.local.html2md.desktop]
_launch=Meta+F9
Ctrl+Alt+A runs html2md -x -c --notify: the copy comes back as wikitext, ready for a page. Ctrl+Alt+ShiftRight+A runs it with --md --ai instead—Markdown, every pipe written {{!}}—which is the form {{AI}} wants, the same ShiftRight convention as mdtable2wiki's two keys. That one is a letter, and legitimately so: it is a keymapper rule, and keymapper matches the scan code at the evdev level, which no layout can move (it reads {Q} in the file, that being the US position of my A). The rule against letters applies to KDE global shortcuts only.
kglobalacceld reads that file once, at startup, so it must be restarted for a new entry to take. That it took is checked without pressing anything—the number that comes back is the key combination, Meta (0x10000000) plus F9 (0x01000038) = 285212728:
busctl --user call org.kde.kglobalaccel /component/net_local_html2md_desktop \
org.kde.kglobalaccel.Component allShortcutInfos
When the page kept no $\mathrm{\TeX}$
KaTeX stores the source; MathJax does not. It ships the visual rendering plus an assistive MathML copy of the same formula, and no LaTeX anywhere—which is why a naive paste of a MathJax page comes out doubled, in Unicode italics: 𝑤𝑘 and then w k again.
MathML is nonetheless a faithful tree of the formula, so it is translated back. msub/msup become _ and ^, mfrac becomes \frac, munderover on a ∑ becomes the limits of a \sum but on a letter becomes \hat or \vec, mtable becomes a matrix, mathvariant becomes \mathcal, \mathbb, \mathfrak. Unicode goes home too: ⊗ to \otimes, α to \alpha, and the mathematical alphanumerics—𝑤, ℋ, ℝ—back to plain letters wearing the right font command, which unicodedata gives for free since their names say what they are (MATHEMATICAL ITALIC SMALL W).
Sym2: spanned by wj⊗wk+wk⊗wjw_j\otimes w_k+w_k... wj⊗wk <- pasted raw
$\mathrm{Sym}^2$: spanned by $w_j\otimes w_k+w_k\otimes w_j$ <- html2md
It is not the author's keystrokes—\frac comes back where \dfrac was written—but it is real LaTeX giving the same formula. A formula with neither annotation nor MathML cannot be recovered by any amount of cleverness: its glyphs are kept, once rather than twice, and counted in the report so I know what needs retyping.
A selection begun inside a formula
Selecting from the middle of an equation—which is what one does when quoting half a sentence—produces a copy the browser truncates inside the KaTeX span. The rendered half comes over; the katex-mathml sibling holding the annotation does not, because it sits before the point where the selection started. There is then no $\mathrm{\TeX}$ to recover, however well the reader is written:
grep -c 'class="katex"' last.html # 2
grep -c 'katex-mathml' last.html # 1 <- the first formula lost its source
What survives is the glyphs, and those are worth keeping as maths: sym() already turns Ψ into \Psi and ≠ into \neq, so the fallback is delimited like any other formula rather than spilled into the prose. Leaving it bare was a real bug—\Psi(n)=0 for all $n\neq N$, one formula rendered and the other not, in the same sentence.
\Psi(n)=0 for all $n\neq N$ <- 1.4.0, the truncated formula left bare
$\Psi(n)=0$ for all $n\neq N$ <- 1.5.0
KaTeX also pads its layout with zero-width spaces, invisible in the browser and litter in a text file; they are stripped on the way out.
What comes back
Headings, paragraphs, nested lists, blockquotes, rules, links, images, bold, italics, code, strikeout, tables and fenced code blocks—with the language taken from the class="language-xxx" the highlighter left behind. A paragraph comes out as one line, which is what this wiki wants anyway.
Code blocks need one precaution. In a chat window the <pre> holds more than the code: the little language label and the Copy button live inside it too, and a naive text extraction pastes python and Copy into the listing. So when a <pre> contains a <code>, only the <code> is kept, and <button>, <svg> and <script> are dropped everywhere.
Maths comes out as $...$ inline and $$...$$ on a single line for display, which are the delimiters this wiki understands, from LocalSettings.php:
$wgSmjExtraInlineMath = [ [ "$", "$" ], [ "\\(", "\\)" ] ];
$wgSmjDisplayMath = [ ['$$','$$'] ];
Inside a table cell, and only there, a | living in the maths becomes \vert: a pipe would end the cell, in Markdown as in MediaWiki. It renders identically.
Into the wiki
The output is Markdown, not wikitext, and that is deliberate: it goes straight into an {{AI}} call, whose Module:AI converts Markdown itself. See Working with Claude for the whole route.
{{AI|AI=Claude|v=Opus 5|q=my prompt, verbatim|a=
the markdown that was on the clipboard when I pasted
}}
A literal | anywhere in there breaks the a= parameter, which is a real obstacle and not a rare one: any table has a pipe in every row. -a is the answer—every pipe is written {{!}}, which the preprocessor expands back to | before Module:AI reads the string, so the module still sees an ordinary Markdown table and a code block still shows its pipe. Without -a the script counts the pipes and says so, and the older cure—a subpage and src=—is still there for a payload one would rather not touch at all.
A table that should become a real wikitable, rather than a Markdown one rendered inside the box, is a different job and belongs to mdtable2wiki—Ctrl+Alt+W, or one pipe:
html2md | mdtable2wiki
When it stops working
It has stopped twice now, and both times it was not what it looked like. The failure to expect is not a crash: it is a watcher that is perfectly healthy and completely deaf—alive, its event loop turning, its one-second tick firing, and not one line in the log for a whole day of copying. Nothing about it looks wrong from the outside, which is exactly the problem, so this is the order to go in.
First: is it running at all?
It is a systemd user unit, generated from the autostart file, so ask systemd rather than ps—and restart it that way too, because a copy relaunched by hand escapes the unit and the next login starts a second one.
systemctl --user status 'app-net.local.html2md\[email protected]'
systemctl --user restart 'app-net.local.html2md\[email protected]'
Mind the \x2d: systemd escapes the hyphen in the unit name, and the quotes are what stop the shell from eating the backslash.
Second: what does the log say?
ls -la ~/.cache/html2md/
tail -20 ~/.cache/html2md/watch.log
A frozen mtime on watch.log while the clipboard demonstrably keeps changing is the whole diagnosis in one line. If instead the log is running and merely says no html on the clipboard, left alone, the copy carried no HTML and the fault is upstream, in what was selected.
One trap in reading that log: a torn write once left NUL bytes in it, after which grep decides the file is binary and prints nothing at all—which made it look as though the restart markers were missing when they had been there the whole time. Filter it: tr -d '\000' < watch.log | grep watching.
Third: deaf, or genuinely hung?
These look identical from a distance and want opposite cures, and the answer is in the process's own counters rather than in anything it says.
grep -E '^(rchar|syscr)' /proc/$(pgrep -f 'html2md --watch')/io # twice, 10 s apart
Numbers that do not move at all mean a true hang, and the cure is the restart above. Numbers creeping up by two or three syscalls and some fifteen bytes a second mean the deafness described below: that is one tiny get_selection_owner round trip a second and nothing else—no TARGETS, no data transfer—so the loop is alive and asking, and simply not being told anything.
What deafness actually is
Qt does not poll the clipboard. It learns that the selection changed from an XFixes subscription, and everything it hands out is cached until that notification arrives. Lose the subscription—a plasmashell restart is enough—and two things break at once: dataChanged never fires again, and mimeData() goes on answering out of a cache that will never be refreshed.
Three things then conspired to make that silent and permanent. The one-second fallback tick, written precisely for the case where dataChanged stops arriving, is no defence at all: it reads through the very cache that broke, so it agrees with itself once a second forever. The fingerprint that stops the tick from saying no html sixty times a minute swallowed the log line too, so not even a complaint got written—last.txt was left at zero bytes, the process having latched onto the hash of the empty string. And nothing inside the process can ever notice, because every road out of it goes through the same cache. It has to be contradicted from outside.
That is also how to confirm it by hand: xclip -o and a freshly started PyQt5 client both read the clipboard perfectly while the old process cannot, and that is what proves the wedge is internal state rather than a broken X clipboard. Do not expect help from the usual places—gdb cannot attach (ptrace_scope), and the unit's stderr is gone as soon as the journal rotates.
What 2.1.0 does about it
A second timer, the sentry, runs every ten seconds and asks somebody else: it compares xclip -o against what Qt believes is on the clipboard. Agreement is the normal case and costs one xclip in ten seconds. Two consecutive disagreements—one is not enough, since a copy landing between the two reads would look like one—mean the cache is stale.
There is no way to make Qt re-subscribe from inside a running process, so the cure is to start over: the watcher execves itself. That keeps the pid, which is the point—systemd's MainPID stays valid and the unit never counts a restart. A counter in the environment caps it at five re-execs in an hour, after which it says so and stays deaf rather than looping; an hour of health wipes the slate. The comparison is skipped whenever the watcher owns the clipboard, that being its own write and nothing to check.
So the expected shape of a cured failure, in the log, is this—and if it ever appears, nothing needs doing:
watchdog: Qt 94ch/1a2b3c4d, X 94ch/9f8e7d6c - disagreement 1 of 2
watchdog: Qt 94ch/1a2b3c4d, X 94ch/9f8e7d6c - disagreement 2 of 2
watchdog: the clipboard cache is stale, starting over
watchdog: re-exec 1
watching (pid 3283397)
Lengths and a short digest, never the text: this log outlives the session, and the clipboard is where passwords travel.
And the older false alarm
Worth keeping in mind because it presents the same way—nothing happens however hard I press Ctrl+C—but has nothing to do with Qt. Before 1.3.1 the watcher recognised its own write by comparing the HTML, and copying the same passage twice hands over byte-identical markup, so the second copy was taken for an echo. The ownership test replaced it. The lesson the two share is that this script fails silently by default, and every version since has had to be argued into complaining.
Limitations
Text is not Markdown-escaped: a stray * or _ in the prose stays a stray * or _. Escaping them would litter every answer with backslashes to protect against something that hardly ever happens, and the output would be unpleasant to re-edit.
Converting the clipboard destroys the HTML it came from, which is the one thing needed to find out why an output is wrong. So whatever is read is kept in ~/.cache/html2md/last.html (or last.txt when the copy carried no HTML at all, which is itself the diagnosis), and html2md -l runs it again.
What cannot be recovered is what the page never showed. A collapsed thinking block, a truncated listing, an artifact behind a tab: only what is actually selected is in the clipboard.
Version history
- 2.5.0 (6 October (2026)) — a copy out of any other web page is left alone. 2.4.0 cured the wiki and nothing else: a DOI selected on a journal's page pasted as
[https://doi.org/10.1021/… 10.1021/…]. The pages worth converting are the few—the chat windows—so a copy is now converted only when Chromium'schromium/x-source-urlflavour names one of them, or when it carries maths. Any other page's copy is not even taken over, and a paste gives what it would give with no html2md running. A copy that names no source is converted as before, andMeta+F9still converts anything. - 2.4.0 (24 September (2026)) — a copy out of a wiki is handed over as plain text, the page's furniture dropped. A wiki page was written by hand, in wikitext, so there is no source to rebuild, and converting its rendering only put links and an
[edit]into a paste meant to carry words. The page is recognised by MediaWiki's own class names or by a link into one of the wiki hosts; the[edit]beside a heading is dropped on every path. - 2.3.0 (12 September (2026)) — a copied picture is left alone. Copy image puts the bitmap on the clipboard with a one-line
<img>beside it; the watcher converted that html and rebuilt the clipboard flavour by flavour, still advertising the forty-odd image formats Qt can encode while holding the bytes of none, so every pasted image was a broken icon. A clipboard carrying a bitmap is now skipped outright, and on the ordinary path no flavour is offered on unless its bytes really arrived. - 2.2.0 (27 August (2026)) — deaf again, on the 26th, and the sentry with it: after a conversion Qt went on saying the clipboard was html2md's for sixteen hours, so every copy was dismissed as an echo and the alarm excused itself on the same grounds. X is now asked before Qt's word on ownership is taken, and a heartbeat file is touched on every sentry tick, so an event loop that stops turning shows up in an
lsof the cache directory. - 2.0.1 (25 August (2026)) — a formula is found by
role="math"as well as by class. Claude's window strips the class names on the way to the clipboard, so the formula was walked into instead of converted and both halves came out,$M$M; the glyph half is thearia-hiddenone, which tells them apart. Display maths is taken from<math display="block">too. - 1.5.0 (2 August (2026)) — a formula truncated by the selection keeps its
$; zero-width spaces stripped; a selection never copied is converted too (--no-selectto stop it). - 1.4.0 (2 August (2026)) — the middle-button PRIMARY selection gets the Markdown too, and
text/plain;charset=utf-8is served rather than left empty. - 1.3.1 (2 August (2026)) — identical HTML is no longer a reason to skip: only ownership is. Copying the same passage twice had looked like the watcher dying.
- 1.3.0 (2 August (2026)) — ownership test instead of the content test, every slot guarded, a one-second fallback tick, and a log of what was decided.
- 1.2.1 (1 August (2026)) — the notification is fired with
Popen: waiting fornotify-sendfroze the watcher and it missed the copy that followed. - 1.2.0 (1 August (2026)) —
--watch, so there is no keystroke at all; the shortcut moved off a letter and ontoMeta+F9; the clipboard read is cached for-l. - 1.1.0 (1 August (2026)) — MathML translated back to LaTeX for pages that store no annotation (MathJax);
mathvariant, Unicode alphanumerics,<sup>/<sub>; the glyph fallback no longer doubles. - 1.0.0 (1 August (2026)) — first version. KaTeX annotations, block and inline structure, tables, fenced code with the chat chrome removed, clipboard in and out.
- 1.6.0 (16 August (2026)) —
-a/--ai, and theCtrl+Alt+Akeymapper binding that carries it: the whole copy comes back with every literal pipe as{{!}}, so an answer containing a table can be pasted into thea=of {{AI}} as it stands. The pipe warning now counts every pipe: it used to skip the lines starting with one, i.e. exactly the table rows that break the parameter. - 2.1.0 (25 August (2026)) — the sentry. The watcher had gone deaf on the 24th and stayed alive doing it, Qt's clipboard cache having been orphaned by a lost XFixes subscription; the one-second fallback tick was useless against it, reading through the same cache. A second timer now compares Qt against
xclipevery ten seconds and, on two disagreements,execves the process in place—same pid, so the systemd unit is none the wiser—capped at five an hour. Also: thenotify-sendtoasts are reaped instead of being left<defunct>one per formula. See #When it stops working. - 2.0.0 (16 August (2026)) — wikitext by default. Every emitter grew a second form and the flavour rides on the rendering context: marks, links, images, headings, rules, lists (by marker, not by indentation), tables, code blocks, quotations.
-mkeeps the Markdown, which stays the right thing for thea=of {{AI}}—wikitext there is actively wrong, Module:AI reading**as bold and losing the nested items. The keys became two:Ctrl+Alt+Afor a page,Ctrl+Alt+ShiftRight+Afor the box. The watcher, being a running process, has to be restarted for a new version to take.
The file
~/bin/html2md
#!/usr/bin/env python3
# _ _ _ ___ _
# | |_| |_ _ __ __| |_ )_ __ __| |
# | ' \ _| ' \/ _` |/ /| ' \/ _` |
# |_||_\__|_|_|_\__,_/___|_|_|_\__,_|
#
# html2md - Version 2.5.0
# Claude & F.P. Laussy
# https://laussy.org/wiki/html2md
#
# Select anything in a browser with the mouse, copy, run this: out comes
# the WIKITEXT it was rendered from, with the maths back in LaTeX form.
#
# Wikitext is the default since 2.0.0, this being a wiki-writing machine:
# '''bold''', ''italics'', == headings ==, * lists, [url label],
# {| class="wikitable" tables, <syntaxhighlight> code. -m/--md gives the
# Markdown instead, which is what Template:AI wants - Module:AI converts
# the Markdown itself, and an answer quoted in an {{AI}} box should read
# as the machine wrote it.
#
# Usage:
# html2md clipboard in, wikitext to stdout (preview)
# html2md -c clipboard in, wikitext back on the clipboard
# html2md page.html a file in
# ... | html2md stdin in
# html2md -l convert the last clipboard again
# html2md -x -c -m -a Markdown, {{AI|a=}}-proof: | -> {{!}}
#
# Why it works
# ------------
# A selection copied from a browser lands on the clipboard twice: as
# text/plain (what the eye sees - maths reduced to loose glyphs, often
# doubled) and as text/html (the real markup). KaTeX, which is what
# renders maths in Claude's window, keeps the LaTeX it was given inside
# the MathML annotation of every formula:
#
# <span class="katex"><span class="katex-mathml"><math><semantics>
# ... <annotation encoding="application/x-tex">E=mc^2</annotation>
#
# That annotation travels with the text/html flavour. So the original
# LaTeX is already in the clipboard - only the plain-text flavour throws
# it away. This tool reads the html one and puts everything back:
#
# 2.0.1: the wrapper is found by role="math" as well as by class. Claude's
# window strips the class names on the way to the clipboard - what arrives
# is <span role="math"> holding the MathML and, beside it, an
# aria-hidden="true" span of the rendered glyphs - so nothing matched
# "katex", the formula was walked into instead of converted, and out came
# both halves: $M$M, and $M=\lfloor N/2\rfloor$ with M=⌊N/2⌋ after it.
# The glyph half is aria-hidden by construction - it is the half a
# screen reader must not read - which is what tells the two apart
# once the classes are gone.
# Display maths is now taken from <math display="block"> too, KaTeX's own
# mark for it, since class="katex-display" goes the same way.
#
# * MATHS WITH NO ANNOTATION. MathJax stores no LaTeX at all, only the
# rendering plus an assistive MathML copy - which is why a naive
# paste of it comes out doubled and in Unicode italics. MathML is a
# faithful tree of the formula, so it is translated back: fractions,
# roots, sub- and superscripts, sums with limits, matrices, accents,
# Greek, operators, and \mathcal/\mathbb/\mathbf from mathvariant.
# Not the author's keystrokes, but LaTeX that gives the same formula;
# * $...$ inline, $$...$$ on one line for display maths (both are
# delimiters on this wiki: $wgSmjExtraInlineMath / $wgSmjDisplayMath);
# * headings, paragraphs, nested lists, blockquotes, rules, links,
# images, **bold**, *italics*, `code`, ~~strikeout~~, tables;
# * fenced code blocks, language taken from class="language-xxx", and
# the chat window's own chrome (the language label, the Copy button)
# dropped - only the <code> inside a <pre> is kept;
# * one paragraph = one line, no hard wrapping, as the wiki wants.
#
# Output is Markdown, not wikitext: that is what Module:AI wants inside
# {{AI|AI=Claude|...|a= }}, and it converts it itself. For a table that
# has to become a real wikitable, pipe it on: html2md | mdtable2wiki.
#
# A copied PICTURE is left alone (2.3.0)
# --------------------------------------
# "Copy image" in the browser puts the bitmap on the clipboard and, beside
# it, a one-line <img src="...">. That is html, so this watcher used to
# convert it to a link and hand back a clipboard it had rebuilt flavour by
# flavour - and the rebuilt one still ADVERTISED image/png, image/jpeg and
# the forty-odd formats Qt says it can make out of a picture while having
# the bytes for none of them. Every paste of a copied image was therefore a
# broken icon, curable only by re-copying the bitmap by hand:
#
# xclip -selection clipboard -o -t image/png > /tmp/img.png &&
# xclip -selection clipboard -t image/png -i /tmp/img.png
#
# which worked because it threw the html away, leaving nothing here to
# convert. A clipboard carrying a bitmap is now skipped outright: the
# browser keeps it and a picture pastes as a picture. The html is still
# stashed, so html2md -l gives the wikitext for it. And on the ordinary
# path no flavour is ever offered on unless its bytes really arrived.
#
# A copy out of a WIKI is handed over as TEXT (2.4.0)
# ---------------------------------------------------
# A chat window is markup that was never written by hand, and rebuilding
# its source is the whole point of this tool. A wiki page is the opposite:
# it WAS written by hand, in wikitext, and converting its rendering back
# puts markup into a paste that was only ever meant to carry words -
#
# [http://localhost/laussywiki/index.php/ICMM ICMM] - [http://...]
# === Meetings[[http://azag/...&action=edit§ion=3 edit]] ===
#
# for "ICMM - Campus de Cantoblanco" and "Meetings". The [edit] is worse
# than noise: it is the one link on the page that means nothing off it.
# So the WATCHER now recognises a copy out of a MediaWiki page - by
# MediaWiki's own class names, or by a link into one of WIKI_HOSTS - and
# hands it over as plain text, the page's furniture dropped.
#
# Nothing else changes. The hotkey and the command line convert whatever
# they are given, so Meta+F9 on the very same copy still gives the
# wikitext; and a selection carrying MATHS is converted as ever, wherever
# it came from, MathJax keeping no LaTeX of its own. The [edit] beside a
# heading is dropped on every path now (CHROME), being a page's furniture
# rather than anything it says.
#
# A copy out of any other WEB PAGE is left alone (2.5.0)
# ------------------------------------------------------
# 2.4.0 cured the wiki and nothing else: select a DOI on a journal's page,
# copy, paste, and out came
#
# [https://doi.org/10.1021/acs.nanolett.5c03585 10.1021/acs.nanolett.5c03585]
#
# where 10.1021/acs.nanolett.5c03585 was wanted. Recognising the pages NOT
# to convert one by one was the wrong way round: the pages worth converting
# are the few - the chat windows - and every other page is the many.
# Chromium says where each copy was made, in a chromium/x-source-url
# flavour beside the html, so the watcher now converts a copy only when it
# comes out of one of CHAT_HOSTS, or carries maths, from wherever. Any
# other page's copy is left exactly as the browser made it: the clipboard
# is not even taken over, so a paste gives what it would give with no
# html2md running. A copy that names no source (not out of a browser) is
# converted as before, and Meta+F9 still converts anything.
#
# Caveats
# -------
# * a formula with neither annotation nor MathML (an image-based
# renderer, a screenshot) cannot be recovered: its visible glyphs are
# kept, once rather than twice, and the count is reported;
# * what was read is kept in ~/.cache/html2md/last.html, because the
# conversion overwrites the clipboard it came from and that is the
# one thing needed to find out why an output is wrong: html2md -l;
# * text is not markdown-escaped, so a stray * or _ in the prose stays
# a stray * or _ - it reads better and re-edits better that way;
# * a literal | breaks an {{AI|a=...}} parameter (use src= instead);
# html2md warns when the output has one outside a table row.
import argparse
import os
import re
import shutil
import subprocess
import sys
import time
import unicodedata
from html.parser import HTMLParser
VERSION = '2.5.0'
CACHE = os.path.expanduser('~/.cache/html2md/last.html')
ALIVE = os.path.expanduser('~/.cache/html2md/alive')
VOID = {'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'link',
'meta', 'param', 'source', 'track', 'wbr'}
# never contributes text
SKIP = {'script', 'style', 'svg', 'noscript', 'button', 'select', 'option',
'textarea', 'template', 'iframe', 'head', 'title', 'input'}
# Nor do these, which are a PAGE's own furniture rather than anything it
# says: MediaWiki's [edit] beside every heading (2.4.0 - it was coming out
# as == Meetings[[http://.../index.php?title=X&action=edit§ion=3 edit]] ==,
# the one link on the page that means nothing anywhere else), its
# jump-to-content links and the TOC's show/hide word. Matched on the class,
# like the chat window's Copy button is matched on the tag.
CHROME = {'mw-editsection', 'mw-editsection-bracket', 'mw-jump-link',
'toctogglespan', 'mw-indicators'}
BLOCK = {'p', 'div', 'section', 'article', 'main', 'header', 'footer',
'aside', 'nav', 'figure', 'figcaption', 'ul', 'ol', 'li', 'dl',
'dt', 'dd', 'pre', 'blockquote', 'table', 'hr', 'form', 'address',
'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}
HEADING = {'h%d' % i: '#' * i for i in range(1, 7)}
# ------------------------------------------------------------------- tree
# HTMLParser is a stream parser; a small tree makes the rendering rules
# (lists inside lists, code inside a cell) plain recursion instead of a
# pile of depth counters.
class Node:
__slots__ = ('tag', 'attrs', 'kids')
def __init__(self, tag, attrs=None):
self.tag = tag
self.attrs = attrs or {}
self.kids = []
class Tree(HTMLParser):
def __init__(self):
HTMLParser.__init__(self, convert_charrefs=True)
self.root = Node('#root')
self.stack = [self.root]
def handle_starttag(self, tag, attrs):
n = Node(tag, dict(attrs))
self.stack[-1].kids.append(n)
if tag not in VOID:
self.stack.append(n)
def handle_startendtag(self, tag, attrs):
self.stack[-1].kids.append(Node(tag, dict(attrs)))
def handle_endtag(self, tag):
for i in range(len(self.stack) - 1, 0, -1):
if self.stack[i].tag == tag:
del self.stack[i:]
return # stray </div>: ignored
def handle_data(self, data):
self.stack[-1].kids.append(data)
def parse(html):
p = Tree()
p.feed(html)
p.close()
root = p.root
body = find(root, lambda n: n.tag == 'body')
return body or root
# ---------------------------------------------------------------- walking
def find(node, pred):
for k in node.kids:
if isinstance(k, str):
continue
if pred(k):
return k
got = find(k, pred)
if got is not None:
return got
return None
def has_class(node, name):
return name in (node.attrs.get('class') or '').split()
def is_chrome(node):
"""Furniture of the page it was copied from, not part of the text."""
return any(c in CHROME
for c in (node.attrs.get('class') or '').split())
def raw_text(node):
"""All text in the subtree, verbatim (newlines and runs kept)."""
if isinstance(node, str):
return node
if node.tag in SKIP or is_chrome(node):
return ''
return ''.join(raw_text(k) for k in node.kids)
def flat_text(node):
return re.sub(r'\s+', ' ', raw_text(node).replace('\xa0', ' ')).strip()
# ------------------------------------------------------- mathml -> latex
# When a page kept no x-tex annotation - MathJax is the usual case - the
# LaTeX is not stored anywhere, but the MathML is a faithful tree of the
# formula and translates back mechanically. It is not the author's exact
# source (\frac may come back where \dfrac was written) but it is real
# LaTeX that compiles to the same formula, which is the point.
GREEK = ('alpha beta gamma delta epsilon zeta eta theta iota kappa lambda '
'mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega').split()
SYMBOL = {
'′': "'", '″': "''", '±': r'\pm', '∓': r'\mp',
'×': r'\times', '÷': r'\div', '⋅': r'\cdot',
'∗': '*', '∘': r'\circ', '⊕': r'\oplus',
'⊗': r'\otimes', '⊙': r'\odot', '∑': r'\sum',
'∏': r'\prod', '∫': r'\int', '∬': r'\iint',
'∮': r'\oint', '√': r'\sqrt', '∞': r'\infty',
'∂': r'\partial', '∇': r'\nabla', '∀': r'\forall',
'∃': r'\exists', '∄': r'\nexists', '∅': r'\emptyset',
'∈': r'\in', '∉': r'\notin', '∋': r'\ni',
'⊂': r'\subset', '⊃': r'\supset', '⊆': r'\subseteq',
'⊇': r'\supseteq', '∪': r'\cup', '∩': r'\cap',
'≤': r'\leq', '≥': r'\geq', '≪': r'\ll', '≫': r'\gg',
'≠': r'\neq', '≡': r'\equiv', '≈': r'\approx',
'≃': r'\simeq', '∼': r'\sim', '∝': r'\propto',
'≅': r'\cong', '→': r'\to', '←': r'\leftarrow',
'↔': r'\leftrightarrow', '⇒': r'\Rightarrow',
'⇐': r'\Leftarrow', '⇔': r'\Leftrightarrow',
'↦': r'\mapsto', '∥': r'\parallel', '⊥': r'\perp',
'∠': r'\angle', 'ℵ': r'\aleph', 'ℏ': r'\hbar',
'ℓ': r'\ell', '℘': r'\wp', 'ℑ': r'\Im', 'ℜ': r'\Re',
'ϕ': r'\phi', 'φ': r'\varphi', 'ϑ': r'\vartheta',
'ϵ': r'\epsilon', 'ε': r'\varepsilon', 'ϱ': r'\varrho',
'ϖ': r'\varpi', 'ϰ': r'\varkappa', 'ς': r'\varsigma',
'¬': r'\neg', '∧': r'\wedge', '∨': r'\vee',
'∴': r'\therefore', '∵': r'\because', '⋯': r'\cdots',
'…': r'\ldots', '⋮': r'\vdots', '⋱': r'\ddots',
'⌈': r'\lceil', '⌉': r'\rceil', '⌊': r'\lfloor',
'⌋': r'\rfloor', '⟨': r'\langle', '⟩': r'\rangle',
'‖': r'\Vert', '†': r'\dagger', '‡': r'\ddagger',
'⊕︀': r'\oplus', '°': r'^\circ', '⊕ ': r'\oplus',
'{': r'\{', '}': r'\}', ' ': ' ', '': '', '': '',
'': '', '': '',
}
ACCENT = {'^': r'\hat', 'ˆ': r'\hat', '̂': r'\hat',
'¯': r'\bar', '̄': r'\bar', '‾': r'\bar',
'˜': r'\tilde', '̃': r'\tilde', '→': r'\vec',
'⃗': r'\vec', '˙': r'\dot', '̇': r'\dot',
'¨': r'\ddot', '̈': r'\ddot', '⏞': r'\overbrace',
'⏟': r'\underbrace'}
FONT = {'BOLD': r'\mathbf', 'ITALIC': '', 'BOLD ITALIC': r'\boldsymbol',
'SCRIPT': r'\mathcal', 'BOLD SCRIPT': r'\mathcal',
'FRAKTUR': r'\mathfrak', 'BLACK-LETTER': r'\mathfrak',
'BOLD FRAKTUR': r'\mathfrak', 'DOUBLE-STRUCK': r'\mathbb',
'SANS-SERIF': r'\mathsf', 'SANS-SERIF BOLD': r'\mathsf',
'SANS-SERIF ITALIC': r'\mathsf', 'MONOSPACE': r'\mathtt'}
STYLED = re.compile(r'^(?:MATHEMATICAL )?(%s)(?: (?:CAPITAL|SMALL))?'
r'(?: LETTER| DIGIT)? (\S+)$' % '|'.join(FONT))
BIG = {r'\sum', r'\prod', r'\int', r'\iint', r'\oint', r'\bigcup', r'\bigcap',
r'\lim', r'\max', r'\min', r'\sup', r'\inf'}
def unstyle(ch):
"""A Unicode maths letter back to ASCII + the font it was wearing.
'MATHEMATICAL ITALIC SMALL W' -> ('', 'w'); italic is what maths mode
does anyway, so it costs no markup."""
try:
name = unicodedata.name(ch)
except ValueError:
return None
m = STYLED.match(name)
if not m:
return None
font, base = FONT[m.group(1)], m.group(2)
if len(base) == 1:
letter = base
elif base.lower() in GREEK:
letter = '\\' + (base.capitalize() if 'CAPITAL' in name
else base.lower())
else:
digits = 'ZERO ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE'.split()
letter = str(digits.index(base)) if base in digits else None
if letter is None:
return None
return (font + '{' + letter + '}') if font else letter
def join(parts):
r"""Concatenate LaTeX bits, keeping a space where one command would
otherwise swallow the next letter: \alpha + b is \alpha b, never
\alphab. Doing it here, at the seam, is the only place where the two
tokens are still distinguishable."""
out = ''
for p in parts:
if out and re.search(r'\\[A-Za-z]+$', out) and re.match(r'[A-Za-z]', p):
out += ' '
out += p
return out
VARIANT = {'script': r'\mathcal', 'bold': r'\mathbf', 'fraktur': r'\mathfrak',
'double-struck': r'\mathbb', 'sans-serif': r'\mathsf',
'monospace': r'\mathtt', 'bold-italic': r'\boldsymbol',
'normal': r'\mathrm'}
def sym(text):
"""A run of characters as LaTeX: Greek stays Greek, operators become
their commands, styled letters lose the styling."""
out = []
for ch in text:
if ch in SYMBOL:
out.append(SYMBOL[ch])
continue
name = unicodedata.name(ch, '')
if name.startswith('GREEK ') and 'LETTER' in name:
base = name.split('LETTER ')[-1].lower()
if base in GREEK:
out.append('\\' + (base.capitalize() if 'CAPITAL' in name
else base))
continue
got = unstyle(ch)
out.append(got if got is not None else ch)
return join(out)
def brace(s):
"""Braces only where TeX needs them: x_{ab} but x_1."""
s = s.strip()
if len(s) == 1 or re.fullmatch(r'\\[A-Za-z]+', s):
return s
return '{' + s + '}'
def mrow(kids):
return join([mathml(k) for k in kids])
def mkids(node):
return [k for k in node.kids
if not (isinstance(k, str) and not k.strip())]
def mathml(node):
"""One MathML node as LaTeX."""
if isinstance(node, str):
return sym(node.strip())
t = node.tag.split(':')[-1]
kids = mkids(node)
if t in ('math', 'semantics', 'mrow', 'mstyle', 'mpadded', 'mtd',
'menclose', 'msrow', 'mscarries'):
return mrow(kids)
if t in ('annotation', 'annotation-xml', 'mphantom'):
return ''
if t in ('mi', 'mn', 'mtext', 'ms', 'mo'):
text = raw_text(node).strip()
if t == 'mtext':
return r'\text{%s}' % text if text else r'\ '
var = VARIANT.get((node.attrs.get('mathvariant') or '').lower())
if var and text:
return r'%s{%s}' % (var, text)
got = sym(text)
if t == 'mi' and len(text) > 1 and re.fullmatch(r'[A-Za-z]+', text):
return r'\%s' % text if text in (
'sin cos tan log ln exp lim max min sup inf det dim ker '
'deg gcd arg').split() else r'\mathrm{%s}' % text
return got
if t == 'mspace':
return r'\,'
if t in ('msub', 'msup', 'msubsup', 'munder', 'mover', 'munderover'):
return script(t, kids)
if t == 'mfrac':
if len(kids) == 2:
a, b = mathml(kids[0]), mathml(kids[1])
if node.attrs.get('linethickness') in ('0', '0pt'):
return r'\binom{%s}{%s}' % (a, b)
return r'\frac{%s}{%s}' % (a, b)
if t == 'msqrt':
return r'\sqrt{%s}' % mrow(kids)
if t == 'mroot' and len(kids) == 2:
return r'\sqrt[%s]{%s}' % (mathml(kids[1]), mathml(kids[0]))
if t == 'mfenced':
o = node.attrs.get('open', '(')
c = node.attrs.get('close', ')')
return r'\left%s %s \right%s' % (sym(o), mrow(kids), sym(c))
if t == 'mtable':
rows = [' & '.join(mathml(c) for c in mkids(r))
for r in kids if r.tag.split(':')[-1] in ('mtr', 'mlabeledtr')]
return r'\begin{matrix} %s \end{matrix}' % r' \\ '.join(rows)
if t == 'mtr':
return ' & '.join(mathml(c) for c in mkids(node))
return mrow(kids)
def script(t, kids):
"""Sub- and superscripts, and the same thing written under and over:
a limit under a sum is _ and ^ again, an accent over a letter is not."""
if not kids:
return ''
base = mathml(kids[0])
rest = [mathml(k) for k in kids[1:]]
if t in ('munder', 'mover', 'munderover') and base not in BIG:
if t == 'mover' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\overset{%s}{%s}' % (rest[0], base)
if t == 'munder' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\underset{%s}{%s}' % (rest[0], base)
if t in ('msub', 'munder'):
return '%s_%s' % (base, brace(rest[0]))
if t in ('msup', 'mover'):
return '%s^%s' % (base, brace(rest[0]))
if len(rest) == 2: # msubsup, munderover
return '%s_%s^%s' % (base, brace(rest[0]), brace(rest[1]))
return base + ''.join(rest)
def mathml_to_tex(node):
return re.sub(r'\s{2,}', ' ', mathml(node)).strip()
# ------------------------------------------------------------------- math
class Ctx:
"""Rendering state: counters for the report, plus the table flag,
which is the one place where a | inside maths has to be neutralised."""
def __init__(self, wiki=True):
self.math = 0
self.rebuilt = 0
self.lost = 0
self.in_table = False
self.wiki = wiki # wikitext out; -m/--md for Markdown
def is_annotation(n):
return (n.tag == 'annotation'
and n.attrs.get('encoding') == 'application/x-tex')
def latex_of(node):
"""The LaTeX a formula was built from, if the page kept it."""
ann = node if is_annotation(node) else find(node, is_annotation)
if ann is not None:
return raw_text(ann).strip()
for key in ('data-latex', 'data-tex', 'alt'): # other renderers
for probe in (node, find(node, lambda n: key in n.attrs)):
if probe is not None and probe.attrs.get(key):
return probe.attrs[key].strip()
tex = find(node, lambda n: n.tag == 'script'
and 'math/tex' in (n.attrs.get('type') or ''))
if tex is not None:
return raw_text(tex).strip()
return None
def visible_math(node):
"""Last resort: the glyphs, and only one copy of them. A formula is
on the page twice - the rendering and the MathML behind it - and
taking the subtree wholesale is what doubles it.
This is what a selection begun in the middle of a formula leaves
behind: the browser copies the rendered half and drops the MathML
sibling that held the annotation, so there is no LaTeX to be had.
The glyphs still carry the formula, and sym() puts the symbols back
in command form, so the result is worth delimiting as maths."""
for probe in ('katex-mathml', 'katex-html'):
got = find(node, lambda n, p=probe: has_class(n, p))
if got is not None:
return sym(zapzero(flat_text(got)))
vis = find(node, lambda n: n.tag in ('mjx-math', 'mjx-container'))
if vis is None:
# Classes stripped (2.0.1): the rendering is the aria-hidden half,
# and taking it alone is what keeps the MathML beside it from being
# counted a second time.
vis = find(node, lambda n: n.attrs.get('aria-hidden') == 'true')
return sym(zapzero(flat_text(vis if vis is not None else node)))
def zapzero(text):
"""KaTeX pads its layout with zero-width spaces; they are invisible
in the browser and litter in a text file."""
return re.sub('[\xad]', '', text)
def math_md(node, ctx, display=False):
tex = latex_of(node)
ctx.math += 1
if not display:
# KaTeX marks display maths on the MathML as well as with
# class="katex-display", and the attribute survives a copy that
# loses the classes (2.0.1). MathJax's assistive copy says it the
# same way.
m = node if node.tag == 'math' else find(node,
lambda n: n.tag == 'math')
if m is not None and m.attrs.get('display') == 'block':
display = True
if tex is None:
mml = node if node.tag == 'math' else find(node,
lambda n: n.tag == 'math')
if mml is not None:
tex = mathml_to_tex(mml)
if tex:
ctx.rebuilt += 1
if not tex:
ctx.lost += 1
tex = visible_math(node)
if not tex:
return ''
tex = re.sub(r'\s+', ' ', tex.replace('\xa0', ' ')).strip()
if ctx.in_table:
tex = tex.replace('|', r'\vert ') # a | would end the cell
d = '$$' if display else '$'
pad = ' ' if display else ''
return d + pad + tex + pad + d
def is_math(node):
# katex-mathml and a bare annotation are here for the same reason:
# a selection started inside a formula arrives without its outer
# wrapper, and what is left must still be read as maths rather than
# spilled into the prose as loose text.
return (has_class(node, 'katex') or has_class(node, 'katex-display')
or has_class(node, 'katex-mathml') or is_annotation(node)
or node.tag in ('math', 'mjx-container', 'mjx-assistive-mml')
or has_class(node, 'MathJax') or has_class(node, 'MathJax_Preview')
# ...and by ARIA, for a copy that arrives with its classes
# stripped (2.0.1): the wrapper still says what it is, and
# taking it whole is what keeps the rendered glyphs beside the
# MathML from being read as prose.
or node.attrs.get('role') == 'math')
# ----------------------------------------------------------------- inline
MARKS = {'b': '**', 'strong': '**', 'i': '*', 'em': '*',
's': '~~', 'del': '~~', 'strike': '~~'}
# The same set in wikitext. Apostrophe markup is quantity-sensitive:
# ''' is bold, '' italic, and five in a row is both - which is why the
# two are never nested by hand here, only ever wrapped one at a time.
WIKI_MARKS = {'b': "'''", 'strong': "'''", 'i': "''", 'em': "''",
's': None, 'del': None, 'strike': None}
def marked(inner, mark):
"""Put the markers against the text, not against its spaces:
' word ' -> ' **word** ', and nothing at all for empty runs.
mark is None for something wikitext has no markup for (struck-out
text): the HTML tag is kept, which MediaWiki renders happily."""
m = re.match(r'^(\s*)(.*?)(\s*)$', inner, re.S)
lead, core, trail = m.groups()
if not core:
return inner
if mark is None:
return lead + '<s>' + core + '</s>' + trail
return lead + mark + core + mark + trail
def code_span(inner):
inner = inner.replace('\n', ' ')
n = max((len(r) for r in re.findall(r'`+', inner)), default=0)
fence = '`' * (n + 1)
pad = ' ' if inner[:1] == '`' or inner[-1:] == '`' else ''
return fence + pad + inner + pad + fence
def inline(node, ctx):
if isinstance(node, str):
return re.sub(r'[ \t\r\n]+', ' ', node.replace('\xa0', ' '))
tag = node.tag
if tag in SKIP or is_chrome(node):
return ''
if is_math(node):
return math_md(node, ctx, display=has_class(node, 'katex-display'))
if tag == 'br':
return '\n'
if tag == 'img':
alt = node.attrs.get('alt', '').strip()
src = node.attrs.get('src', '')
if src[:4] != 'http':
return '![%s]' % alt if alt and not ctx.wiki else ''
# a remote image cannot be shown by MediaWiki, so it becomes a
# link rather than a red File: that will never resolve
return '[%s %s]' % (src, alt or 'image') if ctx.wiki else \
'' % (alt, src)
if tag == 'wbr':
return ''
inner = ''.join(inline(k, ctx) for k in node.kids)
if tag in MARKS:
return marked(inner, (WIKI_MARKS if ctx.wiki else MARKS)[tag])
if tag in ('sup', 'sub'):
# Markdown has no notation for these; the tag itself is understood
# both by MediaWiki and by every Markdown renderer worth the name
return '<%s>%s</%s>' % (tag, inner, tag) if inner.strip() else inner
if tag in ('code', 'tt', 'kbd', 'samp'):
if not inner.strip():
return inner
return ('<code>%s</code>' % inner.replace('\n', ' ') if ctx.wiki
else code_span(inner))
if tag == 'a':
href = node.attrs.get('href', '')
text = inner.strip()
if not href or href[:1] == '#' or href[:11] == 'javascript:':
return inner
if ctx.wiki:
# one bracket pair, space between target and label - and a
# bare url is left bare, MediaWiki links it by itself
if text in (href, href.rstrip('/')):
return href
return '[%s %s]' % (href, text) if text else ''
if text in (href, href.rstrip('/')):
return '<%s>' % href
return '[%s](%s)' % (text, href) if text else ''
return inner
# ------------------------------------------------------------------ block
def indent(text, pad, first=None):
lines = text.split('\n')
head = (first if first is not None else pad) + lines[0]
return '\n'.join([head] + [(pad + l if l.strip() else '') for l in lines[1:]])
def join_item(li, ctx):
"""The blocks of one <li>. A sub-list hangs directly under its item
(no blank line): that is what keeps it one list rather than two."""
body = ''
for part in blocks(li.kids, ctx):
if not body:
body = part
elif re.match(r'(?:[-*+]|\d+[.)]) ', part):
body += '\n' + part
else:
body += '\n\n' + part
return body
def list_wiki(node, ctx, prefix=''):
"""Wikitext nests a list by lengthening its marker - *, **, *# -
with no indentation and no blank line anywhere inside. A list item
is therefore one line, and a block that cannot be one line (a code
block, a table) is emitted after the item: MediaWiki closes the list
there, which is ugly but keeps the content, and there is no way
round it short of a template."""
mark = prefix + ('#' if node.tag == 'ol' else '*')
try:
start = int(node.attrs.get('start', 1))
except ValueError:
start = 1
out = []
for li in node.kids:
if isinstance(li, str) or li.tag != 'li':
continue
line, after = [], []
for k in li.kids:
if not isinstance(k, str) and k.tag in ('ul', 'ol'):
after.append(('list', k))
elif isinstance(k, str) or k.tag not in BLOCK or is_math(k):
line.append(inline(k, ctx))
else:
got = block(k, ctx)
if not got.strip():
continue
if '\n' in got.strip():
after.append(('block', got))
else:
line.append(got)
body = re.sub(r'\s+', ' ', ''.join(line)).strip()
if body:
out.append(mark + ' ' + body)
for kind, thing in after:
out.append(list_wiki(thing, ctx, mark) if kind == 'list' else thing)
body = '\n'.join(x for x in out if x.strip())
if node.tag == 'ol' and start > 1 and not prefix and body:
# # restarts at 1, always: an answer resuming at 3 has to say so
# in HTML - the same fix Module:AI carries for the same reason
items = ''.join('<li>%s</li>' % l[len(mark) + 1:]
for l in body.split('\n') if l.startswith(mark + ' '))
if items:
return '<ol start="%d">%s</ol>' % (start, items)
return body
def list_md(node, ctx):
ordered = node.tag == 'ol'
try:
n = int(node.attrs.get('start', 1))
except ValueError:
n = 1
out = []
for li in node.kids:
if isinstance(li, str) or li.tag != 'li':
continue
body = join_item(li, ctx)
if not body.strip():
continue
mark = ('%d. ' % n) if ordered else '- '
n += 1
out.append(indent(body, ' ' * len(mark), first=mark))
return '\n'.join(out)
def table_md(node, ctx):
was, ctx.in_table = ctx.in_table, True
rows = []
aligns = {}
for tr in [n for n in walk(node) if not isinstance(n, str)
and n.tag == 'tr']:
cells = []
for c in tr.kids:
if isinstance(c, str) or c.tag not in ('td', 'th'):
continue
style = c.attrs.get('style', '') or ''
m = re.search(r'text-align\s*:\s*(left|right|center)', style, re.I)
a = (m.group(1) if m else c.attrs.get('align', '')).lower()
if a:
aligns.setdefault(len(cells), a)
txt = re.sub(r'\s+', ' ', inline(c, ctx)).strip()
cells.append(txt.replace('\n', ' '))
if cells:
rows.append(cells)
ctx.in_table = was
if not rows:
return ''
ncol = max(len(r) for r in rows)
sep = ['|'.join({'center': ':---:', 'right': '---:', 'left': ':---'}
.get(aligns.get(i), '---') for i in range(ncol))]
def line(cells):
v = [c.replace('|', r'\|') for c in cells] # math | is \vert by now
return '| ' + ' | '.join(v + [''] * (ncol - len(v))) + ' |'
return '\n'.join([line(rows[0]), '|' + sep[0] + '|']
+ [line(r) for r in rows[1:]])
def table_wiki(node, ctx):
"""The house wikitable: full width, one cell per line, header row
with !. A | in the content becomes | - MediaWiki cuts a cell on
it before anything else runs - which is [[mdtable2wiki]]'s whole
subject; the same rule, applied here at the source."""
was, ctx.in_table = ctx.in_table, True
rows, heads = [], []
for tr in [n for n in walk(node) if not isinstance(n, str)
and n.tag == 'tr']:
cells, head = [], []
for c in tr.kids:
if isinstance(c, str) or c.tag not in ('td', 'th'):
continue
txt = re.sub(r'\s+', ' ', inline(c, ctx)).strip()
cells.append(txt.replace('|', '|'))
head.append(c.tag == 'th')
if cells:
rows.append(cells)
heads.append(all(head))
ctx.in_table = was
if not rows:
return ''
out = ['{| class="wikitable" style="width:100%;"']
for i, cells in enumerate(rows):
if i:
out.append('|-')
lead = '!' if heads[i] else '|'
out += ['%s %s' % (lead, c) for c in cells]
out.append('|}')
return '\n'.join(out)
def walk(node):
for k in node.kids:
yield k
if not isinstance(k, str):
for g in walk(k):
yield g
def pre_wiki(node):
src, lang = pre_parts(node)
if not lang:
return '<pre>\n%s\n</pre>' % src
return '<syntaxhighlight lang="%s">\n%s\n</syntaxhighlight>' % (lang, src)
def pre_md(node):
"""A code block. In a chat window the <pre> also holds the language
label and the Copy button; the <code> holds only the code."""
src, lang = pre_parts(node)
n = max((len(r) for r in re.findall(r'^\s*(`{3,})', src, re.M)), default=2)
fence = '`' * max(3, n + 1)
return '%s%s\n%s\n%s' % (fence, lang, src, fence)
def pre_parts(node):
code = find(node, lambda n: n.tag == 'code')
src = raw_text(code if code is not None else node)
src = src.replace('\xa0', ' ').replace('\r\n', '\n').strip('\n')
lang = ''
for probe in (code, node):
if probe is None:
continue
for c in (probe.attrs.get('class') or '').split():
m = re.match(r'(?:language|lang|highlight|hljs)[-_](\w+)$', c)
if m:
lang = m.group(1)
break
lang = lang or probe.attrs.get('data-language', '')
if lang:
break
return src, lang
def blocks(kids, ctx):
"""Children of a container, as a list of block-level chunks. Text and
inline elements between two blocks make a paragraph of their own."""
out = []
run = []
def flush():
if run:
para = re.sub(r'[ \t]*\n[ \t]*', '\n', ''.join(run)).strip()
if para:
out.append(para)
run.clear()
for k in kids:
if isinstance(k, str) or k.tag not in BLOCK or is_math(k):
run.append(inline(k, ctx))
continue
flush()
got = block(k, ctx)
if got.strip():
out.append(got)
flush()
return out
def block(node, ctx):
tag = node.tag
if tag in SKIP or is_chrome(node):
return ''
if tag == 'hr':
return '----' if ctx.wiki else '---'
if tag in HEADING:
text = re.sub(r'\s+', ' ', inline(node, ctx)).strip()
if not text:
return ''
if ctx.wiki:
bars = '=' * (len(HEADING[tag]) + 1) # h1 -> ==, the page
return '%s %s %s' % (bars, text, bars) # title being the h1
return '%s %s' % (HEADING[tag], text)
if tag == 'pre':
return pre_wiki(node) if ctx.wiki else pre_md(node)
if tag == 'table':
return table_wiki(node, ctx) if ctx.wiki else table_md(node, ctx)
if tag in ('ul', 'ol'):
return list_wiki(node, ctx) if ctx.wiki else list_md(node, ctx)
if tag == 'blockquote':
body = '\n\n'.join(blocks(node.kids, ctx))
if ctx.wiki:
return '<blockquote>\n%s\n</blockquote>' % body
return '\n'.join('>' + (' ' + l if l.strip() else '')
for l in body.split('\n'))
if tag in ('dt',):
return ("'''%s'''" if ctx.wiki else '**%s**') % flat_text(node)
# p, div, li, section, ... : just their contents
return '\n\n'.join(blocks(node.kids, ctx))
# --------------------------------------------------------- the text alone
# A copy out of a WIKI page is a different job from a copy out of a chat
# window (2.4.0). There is nothing to reconstruct: the page was written in
# wikitext to begin with, and turning its rendering back into wikitext puts
# markup into a paste that was meant to carry words -
#
# [http://localhost/laussywiki/index.php/ICMM ICMM] - [http://...]
# === Meetings[[http://azag/...&action=edit§ion=3 edit]] ===
#
# where what was wanted was "ICMM - Campus de Cantoblanco" and "Meetings".
# So the watcher hands such a copy over as TEXT, the page's furniture
# dropped. Nothing else changes: the hotkey and the command line still
# convert whatever they are given (Meta+F9 on the same copy gives the
# wikitext), and a selection carrying MATHS is converted as ever, that
# being the one thing on a wiki page a paste cannot rebuild for itself -
# MathJax keeps no LaTeX, so its rendering is the only source there is.
# The wikis of this machine, for recognising a link into one of them.
# Local is localhost/azag, online is laussy.org; add a host here and a copy
# out of it is treated the same way.
WIKI_HOSTS = ('localhost', '127.0.0.1', 'azag', 'laussy.org')
WIKI_PAGE = re.compile(
# MediaWiki's own class names, which no chat window has
r"""class=["'][^"']*\bmw-(?:headline|heading|editsection|parser-output"""
r"""|content-l|content-r|file-description|redirect)\b"""
# or a link into one of our wikis
r"""|href=["'][^"']*//(?:[^"'/]*\.)?(?:%s)[:/]"""
% '|'.join(h.replace('.', r'\.') for h in WIKI_HOSTS), re.I)
def from_wiki(html):
"""Was this copied out of a MediaWiki page?
Its own class names say so outright - a heading carries mw-headline and
the [edit] beside it mw-editsection - but a selection of plain prose
from the middle of a paragraph carries none of them, and then the only
tell is a link pointing back into the wiki. Hence the two halves.
The class names are deliberately enough on their own, so that a copy
out of ANY MediaWiki (Wikipedia included) is treated as the page it is;
the URL half is deliberately NOT, since a Claude answer citing a
wikipedia.org/wiki/... page would otherwise be taken for a wiki copy
and handed over unconverted - which is the one thing this tool is for."""
return WIKI_PAGE.search(html) is not None
def has_math(html):
return 'katex' in html or '<math' in html.lower() or 'mjx-' in html
# Where a copy was made (2.5.0). Chromium tags every copy with the page it
# came from, as a flavour of its own; Firefox does the same under its name.
SOURCE_FLAVOURS = ('chromium/x-source-url', 'text/x-moz-url-priv')
# The chat windows - the only pages whose rendering is worth turning back
# into markup, a copy out of any other page being left as the browser made
# it. Add a host here and a copy out of it is converted again.
CHAT_HOSTS = ('claude.ai',)
def source_host(data):
"""The host of the page a copy was made on, or '' when the copy does
not say - one out of an application that is not a browser, say."""
from urllib.parse import urlsplit
formats = data.formats()
for fmt in SOURCE_FLAVOURS:
if fmt not in formats:
continue
blob = bytes(data.data(fmt) or b'')
url = blob.decode('utf-16' if b'\0' in blob else 'utf-8', 'replace')
host = urlsplit(url.strip('\0').split('\n')[0].strip()).hostname
if host:
return host.lower()
return ''
def from_chat(host):
return any(host == h or host.endswith('.' + h) for h in CHAT_HOSTS)
def text_only(node, pre=False):
"""The visible text of a rendering: what the eye reads there, with the
page's furniture gone and its blocks kept apart. Not a conversion -
nothing is marked up, a link is its label and a heading is its words."""
if isinstance(node, str):
text = node.replace('\xa0', ' ')
return text if pre else re.sub(r'[ \t\r\n]+', ' ', text)
if node.tag in SKIP or is_chrome(node):
return ''
if node.tag == 'br':
return '\n'
if node.tag == 'img':
return node.attrs.get('alt', '')
if node.tag == 'pre':
body = ''.join(text_only(k, True) for k in node.kids)
return '\n\n%s\n\n' % body.strip('\n')
inner = ''.join(text_only(k, pre) for k in node.kids)
if node.tag in ('td', 'th'):
return inner.strip() + '\t'
if node.tag in ('tr', 'li', 'dt', 'dd'):
return inner.strip() + '\n'
if node.tag in BLOCK:
return '\n\n%s\n\n' % inner.strip()
return inner
def to_text(html):
"""html -> the words in it, blank lines between the blocks."""
text = text_only(parse(html))
text = re.sub(r'[ \t]*\n[ \t]*', '\n', text)
text = re.sub(r'\n{3,}', '\n\n', text)
text = re.sub(r'[ \t]+$', '', text, flags=re.M)
return text.strip()
def to_markdown(html, ctx):
"""HTML in, wikitext out - or Markdown, if the ctx says so."""
md = '\n\n'.join(blocks(parse(html).kids, ctx))
md = re.sub(r'[ \t]+$', '', md, flags=re.M)
md = re.sub(r'\n{3,}', '\n\n', md)
return md.strip() + '\n'
# -------------------------------------------------------------- clipboard
def _run(cmd):
if not shutil.which(cmd[0]):
return None
try:
return subprocess.run(cmd, capture_output=True, check=True
).stdout.decode('utf-8', 'replace')
except subprocess.CalledProcessError:
return None
def ai_escape(md):
"""Every literal pipe as {{!}}, so the Markdown can sit in {{AI|a=}}.
A template parameter is cut on |, so a table row - or a single | in
prose or in a code block - loses the rest of the answer. {{!}} is
expanded back to | by the preprocessor BEFORE Module:AI reads the
string, so the module still sees an ordinary Markdown table and the
code block still shows a pipe. Same trick as mdtable2wiki --ai.
"""
return md.replace('|', '{{!}}')
def clip_read():
"""The html flavour if the copy had one - that is the whole point."""
for cmd in (['wl-paste', '-t', 'text/html'],
['xclip', '-selection', 'clipboard', '-t', 'text/html', '-o']):
got = _run(cmd)
if got and got.strip():
return got, True
for cmd in (['wl-paste', '-n'], ['xclip', '-selection', 'clipboard', '-o'],
['xsel', '-ob']):
got = _run(cmd)
if got is not None:
return got, False
sys.exit('html2md: no clipboard tool (wl-clipboard / xclip / xsel)')
def clip_write(text):
for cmd in (['wl-copy'], ['xclip', '-selection', 'clipboard'],
['xsel', '-ib']):
if shutil.which(cmd[0]):
try:
subprocess.run(cmd, input=text.encode(), check=True)
return cmd[0]
except subprocess.CalledProcessError:
pass
return None
# ------------------------------------------------------------------ watch
# The hotkey works, but a hotkey is a thing to remember. Watching the
# clipboard removes the step altogether: copy in the browser, paste in
# Emacs, and what arrives is already Markdown.
#
# Nothing is lost by doing it: the converted Markdown goes on as
# text/plain - the flavour Emacs, a textarea and the wiki edit box all
# read - while the original text/html is put back untouched, so a rich
# paste into a mail or a word processor still gets its formatting.
# ------------------------------------------------------- the deaf watcher
# It went deaf on 24 August 2026 and looked perfectly healthy doing it:
# alive, the event loop turning, the one-second tick firing, and not one
# line in the log for a whole day of copying.
#
# Qt learns that the clipboard changed from an XFixes subscription, and
# everything it hands out is cached until that notification arrives. Lose
# the subscription - a compositor restart is enough - and dataChanged
# never fires again AND mimeData() keeps answering out of the stale
# cache, so the fallback tick, which reads through the very cache that
# broke, agrees with it once a second forever. The fingerprint on the
# unchanged text then swallows the log line too, and the failure is
# perfectly silent: the process must be contradicted from outside to
# find out it is wrong.
#
# So a second client reads the same selection, and disagreement is the
# alarm. There is no way to make Qt re-subscribe from inside, so the cure
# is to start over - os.execv, which keeps the pid and leaves the systemd
# unit none the wiser.
#
# 2.2.0: it went deaf again on 26 August 2026, at 18:06:49, on the line
# after a conversion - and this time the watchdog above went with it.
# Both look() and watchdog() begin by asking Qt whether the clipboard is
# ours, because a watcher must not convert its own output a second time;
# and OWNERSHIP is Qt's word too. Having written the converted text, Qt
# went on saying the clipboard was ours for the next sixteen hours, so
# every copy after it was dismissed as an echo AND the alarm excused
# itself on the same grounds. Not one line in the log for a whole day of
# copying - the same perfect silence as before, one layer further in.
# The second reader is therefore consulted BEFORE the ownership
# question: if X hands out something we never wrote, the clipboard is
# not ours, whatever Qt says. A heartbeat file is touched on every sentry
# tick as well, so that an event loop which stops turning altogether -
# the one failure no process can diagnose from inside - shows up in an
# ls of the cache directory.
REEXEC_ENV = 'HTML2MD_REEXEC'
REEXEC_AT = 'HTML2MD_REEXEC_AT'
def fingerprint(text):
"""Enough to tell two clipboards apart in the log, not enough to read
one back out of it."""
import hashlib
return hashlib.sha1(text.encode('utf-8', 'replace')).hexdigest()[:8]
def x_clipboard_text():
"""The clipboard as another X client sees it, or None if it cannot be
read. Deliberately not Qt: the whole point is a second opinion."""
if not shutil.which('xclip'):
return None
try:
r = subprocess.run(['xclip', '-selection', 'clipboard', '-o'],
capture_output=True, timeout=2)
except (subprocess.TimeoutExpired, OSError):
return None # a hung owner is not Qt's fault
if r.returncode != 0:
return '' # empty clipboard: xclip says so loudly
return r.stdout.decode('utf-8', 'replace')
def same_text(a, b):
"""Line endings and trailing space differ between the two readers for
reasons that have nothing to do with staleness."""
return a.replace('\r\n', '\n').strip() == b.replace('\r\n', '\n').strip()
def reexec():
"""Start again in place. execv keeps the pid, so systemd still sees
its MainPID and the unit never counts a restart."""
n = int(os.environ.get(REEXEC_ENV, '0') or '0')
last = float(os.environ.get(REEXEC_AT, '0') or '0')
if time.time() - last > 3600:
n = 0 # an hour of health wipes the slate
if n >= 5:
log('watchdog: %d re-execs in the hour, staying deaf rather than '
'looping' % n)
return False
env = dict(os.environ)
env[REEXEC_ENV] = str(n + 1)
env[REEXEC_AT] = '%d' % time.time()
log('watchdog: re-exec %d' % (n + 1))
try:
os.execve(sys.executable,
[sys.executable, os.path.abspath(sys.argv[0])]
+ sys.argv[1:], env)
except OSError as exc:
log('watchdog: re-exec failed: %r' % (exc,))
return False
def watch(o):
from PyQt5 import QtCore, QtGui, QtWidgets
app = QtWidgets.QApplication(sys.argv[:1])
app.setQuitOnLastWindowClosed(False)
board = app.clipboard()
seen = [None, None, 0.0, None, None, None] # plain, html, when,
# selection html, our own last write,
# the image or web copy left alone
deaf = [0] # consecutive watchdog disagreements
def look():
# Is this our own write coming back round? Ask who owns the
# clipboard, never whether the HTML looks familiar: copying the
# same passage twice hands us byte-identical markup, and a
# content test then mistakes a fresh copy for its own echo and
# sits there doing nothing - which is exactly what it did.
if board.ownsClipboard():
return
data = board.mimeData()
if data is None:
return
html = data.html() if data.hasHtml() else ''
# A PICTURE IS NOT A PAGE (2.3.0). Chromium's "Copy image" puts the
# bitmap on the clipboard and, beside it, a one-line
# <img src="..."> - which is html, so this watcher used to convert
# it to a link and re-publish a clipboard rebuilt format by format
# out of data.data() answers the browser had not given. The
# re-published copy still ADVERTISED image/png, image/jpeg and the
# forty-odd formats Qt says it can make out of a picture, and
# handed out nothing for every one of them: an empty answer where
# the bitmap should have been. That is the broken icon that came
# out of every paste of a copied image, and the reason for the
# xclip re-copy by hand. Anything carrying a bitmap is now left
# strictly alone - the clipboard keeps the browser's own owner and
# pastes as a picture, which is what copying a picture means. The
# html is still stashed, so html2md -l gives the wikitext for it.
if any(f.startswith('image/') for f in data.formats()):
fingerprint = hash((html, data.text() or ''))
if fingerprint != seen[5]:
seen[5] = fingerprint
if html:
keep(html)
log('an image is on the clipboard, left alone '
'(html2md -l for the wikitext)')
return
if not html:
# Nothing to do, and we take no ownership - so without a
# memory of it the tick would say so once a second forever.
fingerprint = hash(data.text() or '')
if fingerprint != seen[0]:
seen[0] = fingerprint
keep(data.text() or '', plain=True)
log('no html on the clipboard, left alone')
return
# ONLY A CHAT WINDOW IS CONVERTED (2.5.0). A copy out of any other
# web page - a DOI selected on a journal's site came out as
# [https://doi.org/10.1021/... 10.1021/...] - is left exactly as the
# browser made it, clipboard and all, so it pastes as its words.
# Maths still converts wherever it comes from, a wiki copy still
# goes the text way below, and a copy naming no source at all is
# converted as before. Meta+F9 converts this one after all.
host = source_host(data)
if host and not from_chat(host) and not has_math(html) \
and not from_wiki(html):
fingerprint = hash((html, data.text() or ''))
if fingerprint != seen[5]:
seen[5] = fingerprint
keep(html)
log('a copy out of %s, left as the browser made it '
'(Meta+F9 for wikitext)' % host)
return
# Identical HTML is NOT a reason to skip: copying the same
# passage twice is a perfectly ordinary thing to do. Only a
# repeat within a moment is suppressed, and only so that a failed
# hand-over cannot turn into a conversion loop.
if html == seen[1] and time.monotonic() - seen[2] < 2:
return
seen[1], seen[2] = html, time.monotonic()
keep(html)
# A WIKI PAGE IS NOT A CHAT WINDOW (2.4.0). Its rendering came from
# wikitext in the first place, so writing wikitext back into the
# clipboard is markup where words were wanted - the [edit] beside
# every heading included, which belongs to nothing but the page it
# was read on. Such a copy is handed over as plain text instead,
# the furniture dropped. Maths is the exception and stays a
# conversion: MathJax keeps no LaTeX, so its rendering is the only
# place the formula still exists. Meta+F9 (html2md -x -c) converts
# this copy after all, for the times when wikitext IS what is
# wanted.
if from_wiki(html) and not has_math(html):
text = to_text(html)
if not text:
log('a wiki page, but nothing in the selection')
return
hand_over(data, text)
log('a wiki page, handed over as text (Meta+F9 for wikitext)')
if not o.quiet and not o.notify:
sys.stderr.write('html2md: a wiki page, text left plain\n')
return
ctx = Ctx(wiki=not o.md)
md = to_markdown(html, ctx)
if not md.strip():
log('html converted to nothing, left alone')
return
hand_over(data, md)
note = '%d formula%s%s' % (ctx.math, 's' if ctx.math != 1 else '',
', %d rebuilt from MathML' % ctx.rebuilt
if ctx.rebuilt else '')
log('converted, %s' % note)
if o.quiet:
return
if o.notify and ctx.math:
report(o, 'converted', note)
elif not o.notify:
sys.stderr.write('html2md: converted, %s\n' % note)
def hand_over(data, out):
"""Publish `out` as the text of the clipboard, keeping every other
flavour the source offered - and offering on none it cannot
actually hand over."""
fresh = QtCore.QMimeData()
for fmt in data.formats(): # keep source URLs, anchors, everything
if fmt.startswith('text/plain') or fmt in ('UTF8_STRING',
'STRING', 'TEXT'):
continue
# Qt answers formats() with every picture format it could
# ENCODE, not the ones the owner actually has, so asking for
# all of them is forty fetches of one bitmap from a browser
# that serves them one at a time. An image copy never reaches
# this far now, and no picture format is ever re-served blind.
if fmt.startswith('image/') or fmt in ('application/x-qt-image',
'BITMAP', 'PIXMAP'):
continue
# TARGETS, MULTIPLE, TIMESTAMP and SAVE_TARGETS are the X
# protocol's own questions about a selection, not flavours of
# what is in it: Qt answers them itself, and asking for their
# contents is meaningless.
if fmt in ('TARGETS', 'MULTIPLE', 'TIMESTAMP', 'SAVE_TARGETS'):
continue
blob = data.data(fmt)
# An empty answer must not become an advertised format: a
# clipboard that offers what it cannot hand over is exactly
# how a paste turns into a broken icon.
if blob is None or len(blob) == 0:
log('%s came back empty, not offered on' % fmt)
continue
fresh.setData(fmt, blob)
fresh.setText(out)
# Qt offers text/plain, UTF8_STRING, STRING and TEXT but not
# text/plain;charset=utf-8, which the browser does offer and
# which GTK and VTE applications ask for by name: without it
# they get an empty answer where a paste should have been.
fresh.setData('text/plain;charset=utf-8', out.encode())
board.setMimeData(fresh)
seen[4] = out # what the watchdog expects to find out there
# A terminal pastes with the middle button, and that is the
# PRIMARY selection, not the clipboard - it would otherwise hand
# back the browser's own glyphs, the formula three times over.
if o.primary:
board.setText(out, QtGui.QClipboard.Selection)
def look_selection():
# The middle button pastes what was merely *selected*, with no
# copy at all. Converting that means owning PRIMARY, and the
# window I am reading may drop its highlight when it loses it -
# hence --no-select for anyone who finds that worse than the
# tripled formula it cures.
if board.ownsSelection():
return
data = board.mimeData(QtGui.QClipboard.Selection)
if data is None or not data.hasHtml():
return
html = data.html()
if html == seen[3] or 'katex' not in html and '<math' not in html \
and 'mjx-' not in html:
return # only formulae are worth the theft
seen[3] = html
ctx = Ctx(wiki=not o.md)
md = to_markdown(html, ctx)
if md.strip() and ctx.math:
board.setText(md, QtGui.QClipboard.Selection)
log('selection converted, %d formulas' % ctx.math)
def guarded_selection():
try:
look_selection()
except Exception as exc:
log('selection error: %r' % (exc,))
def guarded():
# An exception escaping a slot is fatal in PyQt5; one malformed
# copy would take the watcher down and every paste after it.
try:
look()
except Exception as exc:
log('error: %r' % (exc,))
def watchdog():
# Two readers, one selection: if they disagree Qt is answering out
# of a cache no notification will ever refresh again, and only a
# fresh process gets a fresh subscription.
#
# The second reader speaks FIRST (2.2.0). Asking Qt whether the
# clipboard is ours before asking X what is on it puts the whole
# alarm behind the very answer that goes stale: a watcher that
# wrongly believes it still owns the clipboard excuses itself
# from checking forever, which is precisely how 26 August was
# spent.
theirs = x_clipboard_text()
if theirs is None:
return # no xclip, or an owner that would not answer
if board.ownsClipboard():
# We may really own it - a conversion has just been handed
# over - and then X gives back what we wrote. Anything else
# out there is somebody else's copy, so the ownership was a
# memory, not a fact.
mine = seen[4]
if mine is None or same_text(mine, theirs):
deaf[0] = 0
return
else:
data = board.mimeData()
mine = ('' if data is None else (data.text() or ''))
if same_text(mine, theirs):
deaf[0] = 0
return
deaf[0] += 1
# Lengths and a digest, never the text: this log outlives the
# session and the clipboard is where passwords travel.
log('watchdog: Qt %dch/%s, X %dch/%s - disagreement %d of 2'
% (len(mine), fingerprint(mine), len(theirs),
fingerprint(theirs), deaf[0]))
if deaf[0] >= 2:
deaf[0] = 0
log('watchdog: the clipboard cache is stale, starting over')
reexec()
def guarded_watchdog():
beat()
try:
watchdog()
except Exception as exc:
log('watchdog error: %r' % (exc,))
reap()
board.dataChanged.connect(guarded)
if o.select:
board.selectionChanged.connect(guarded_selection)
tick = QtCore.QTimer() # in case dataChanged ever stops
tick.timeout.connect(guarded) # arriving: one round trip a second
tick.start(1000)
# The tick above cannot catch a deaf Qt - it reads the same cache.
# This one asks somebody else. Twice ten seconds apart before believing
# it, so that a copy landing between the two reads is not an alarm.
sentry = QtCore.QTimer()
sentry.timeout.connect(guarded_watchdog)
sentry.start(10000)
log('watching (pid %d)' % os.getpid())
if not o.quiet:
sys.stderr.write('html2md: watching the clipboard (Ctrl-C to stop)\n')
QtCore.QTimer.singleShot(0, guarded) # whatever is there now
sys.exit(app.exec_())
def log(line):
"""A record that outlives the session: when the watcher looks stuck,
this is what says whether it saw the copy at all."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = os.path.join(os.path.dirname(CACHE), 'watch.log')
if os.path.exists(path) and os.path.getsize(path) > 200000:
os.replace(path, path + '.1')
with open(path, 'a', encoding='utf-8') as f:
f.write('%s %s\n' % (time.strftime('%Y-%m-%d %H:%M:%S'), line))
except OSError:
pass
def keep(src, plain=False):
"""Stash what was read. Converting the clipboard destroys the html it
came from, and when the output is wrong that is exactly what one
needs to look at: html2md -l runs it again, html2md ~/.cache/... too."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = CACHE[:-5] + '.txt' if plain else CACHE
with open(path, 'w', encoding='utf-8') as f:
f.write(src)
except OSError:
pass
def beat():
"""A file whose mtime says the event loop is still turning. The log
only speaks when the clipboard changes, so silence there means either
a quiet morning or a dead watcher, and the two look identical. This
tells them apart from outside - which is the only place a stopped
event loop can be seen from."""
try:
os.makedirs(os.path.dirname(ALIVE), exist_ok=True)
with open(ALIVE, 'w') as f:
f.write('%d\n' % time.time())
except OSError:
pass
TOASTS = []
def reap():
"""notify-send is spawned and never waited for - waiting would hold the
event loop for as long as the toast is on screen and lose the next
copy - so somebody has to collect the corpses. A watcher that runs for
weeks accumulates them one per formula otherwise."""
TOASTS[:] = [t for t in TOASTS if t.poll() is None]
def report(o, title, body):
"""On a hotkey there is no terminal to write to, so say it in a toast."""
if o.quiet or not body:
return
if o.notify and shutil.which('notify-send'):
# Popen, not run: notify-send lives as long as the toast is on
# screen, and waiting for it would freeze the watcher's event
# loop - it would sit out the next copy entirely.
TOASTS.append(subprocess.Popen(
['notify-send', '-a', 'html2md', '-t', '4000',
'html2md: ' + title, body], start_new_session=True))
reap()
else:
sys.stderr.write('html2md: ' + body + '\n')
def looks_like_html(text):
return re.search(r'<(p|div|span|table|li|h[1-6]|pre|br|body)[\s/>]',
text, re.I) is not None
# --------------------------------------------------------------------- cli
def main():
p = argparse.ArgumentParser(
prog='html2md',
description='Rendered HTML on the clipboard -> wikitext '
'(-m: Markdown), maths back in LaTeX.')
p.add_argument('file', nargs='?', help='HTML file (default: clipboard)')
p.add_argument('-w', '--watch', action='store_true',
help='sit on the clipboard and convert every copy')
p.add_argument('--no-primary', dest='primary', action='store_false',
help='watch: do not put the conversion on PRIMARY on a copy')
p.add_argument('--no-select', dest='select', action='store_false',
help='watch: do not convert a selection that was never copied')
p.add_argument('-l', '--last', action='store_true',
help='convert the last clipboard again (%s)' % CACHE)
p.add_argument('-x', '--clipboard', action='store_true',
help='read the clipboard even if stdin is a pipe')
p.add_argument('-c', '--copy', action='store_true',
help='put the conversion back on the clipboard')
p.add_argument('-m', '--md', action='store_true',
help='Markdown out instead of wikitext (what {{AI}} '
'wants: Module:AI converts the Markdown itself)')
p.add_argument('--wiki', action='store_true',
help='wikitext out - the default, named for symmetry')
p.add_argument('-a', '--ai', action='store_true',
help='write every literal | as {{!}}, so the whole '
'answer - tables and code included - drops into '
'the a= of Template:AI')
p.add_argument('-q', '--quiet', action='store_true',
help='no report on stderr')
p.add_argument('-n', '--notify', action='store_true',
help='report in a desktop notification (for a hotkey)')
p.add_argument('-V', '--version', action='version',
version='html2md ' + VERSION)
o = p.parse_args()
if o.watch:
watch(o)
from_clip = False
if o.last:
o.file = CACHE
if o.file:
with open(o.file, encoding='utf-8', errors='replace') as f:
src = f.read()
elif o.clipboard or sys.stdin.isatty():
src, rich = clip_read()
from_clip = True
keep(src) # the conversion overwrites the clipboard
if not rich and not looks_like_html(src):
# plain text was copied: nothing to convert, pass it through
if o.copy:
clip_write(src)
sys.stdout.write(src if src.endswith('\n') else src + '\n')
report(o, 'nothing to do', 'the clipboard has no text/html '
'(plain text left as it was)')
return
else:
src = sys.stdin.read()
ctx = Ctx(wiki=not o.md)
md = to_markdown(src, ctx)
pipes = md.count('|')
if o.ai:
md = ai_escape(md)
if o.copy:
if clip_write(md) is None:
sys.exit('html2md: no clipboard tool to write with')
if not (o.copy and from_clip and sys.stdout.isatty()):
sys.stdout.write(md)
say = ['wikitext' if ctx.wiki else 'Markdown']
if ctx.math:
say.append('%d formula%s' % (ctx.math, 's' if ctx.math > 1 else ''))
if ctx.rebuilt:
say.append('%d rebuilt from MathML (no annotation)' % ctx.rebuilt)
if ctx.lost:
say.append('%d with no LaTeX kept by the page (glyphs only)'
% ctx.lost)
if o.ai:
if pipes:
say.append('%d pipe%s written {{!}}' % (pipes, 's' * (pipes > 1)))
elif pipes and ctx.wiki:
# on a page those pipes are the table itself; only a template
# parameter minds them
say.append('%d pipe%s - fine on a page, -a to put this in {{AI|a=}}'
% (pipes, 's' * (pipes > 1)))
elif pipes:
say.append('%d literal | - breaks {{AI|a=...}}, so -a or src='
% pipes)
if o.copy:
say.append('on the clipboard')
report(o, '%d line%s' % (md.count('\n'),
'' if md.count('\n') == 1 else 's'),
'; '.join(say))
if __name__ == '__main__':
main()