html2md is a python script that turns a selection copied out of a browser—typically a piece of a Claude answer read in its own window—into the Markdown it was rendered from, with the maths back in $\mathrm{\TeX}$ form.
Reading an answer on the web, I see the rendered page: the Markdown is gone, and every formula is a heap of KaTeX spans. Selecting a paragraph with the mouse and pasting it into an edit box gives ugly prose without any of its structure, and formulae reduced to loose glyphs—usually repeated, because the rendering and the MathML behind it are both in the selection. This script gives back the source. It sits on the clipboard and does it by itself: there is nothing to press.
Written with the help of Claude Opus 5 on 1 August (2026). Version 1.5.0.
There is nothing to reverse-engineer: the $\mathrm{\TeX}$ is already on the clipboard. A browser puts a copied selection there twice, once as text/plain—what the eye sees—and once as text/html, the real markup. And KaTeX, which is what renders maths in a chat window, keeps the $\mathrm{\TeX}$ it was handed inside the MathML annotation of every formula it builds:
<span class="katex"><span class="katex-mathml"><math><semantics>
... <annotation encoding="application/x-tex">2g\sqrt{n}</annotation>
</semantics></math></span><span class="katex-html" aria-hidden="true">...</span></span>
That annotation travels with the text/html flavour. Only the plain-text flavour throws it away—and the plain-text flavour is precisely the one every text form takes when I hit Ctrl+V. So the whole trick is to read the other target of the same clipboard:
xclip -selection clipboard -t TARGETS -o # see the flavours on offer
xclip -selection clipboard -t text/html -o # the one that has everything
No userscript, no extension, no export, nothing to install in the browser. This is the same observation mdtable2wiki makes for tables; html2md generalises it to a whole selection.
html2md --watch sits on the clipboard and converts every copy as it arrives, so the gesture is just copy, paste—no keystroke in between, and it works into Emacs, a mail, the wiki edit box, anything. It is started at login from ~/.config/autostart/net.local.html2md-watch.desktop.
Nothing is lost by letting it rewrite every copy, because it only adds: the Markdown goes on as text/plain—the flavour Emacs and every text field read—while the original text/html is put back untouched, so pasting into a mail or a word processor still gets the formatting. Every other flavour of the copy (images, the source URL) is carried over too.
Three details make it work rather than nearly work. First, the watcher must ignore its own write or it converts its output forever—but the test for that is who owns the clipboard, never whether the HTML looks familiar. Comparing content was the first attempt, and it fails the moment I copy the same passage twice: the browser hands over byte-identical markup, the watcher takes the second copy for its own echo, and from then on nothing happens however hard I press Ctrl+C. It looks exactly like a crash, and it is not one. Ownership tells the two apart with no ambiguity: after my write I own the selection, after a copy the browser does.
Second, the notification must be fired with Popen, never run: notify-send lives as long as the toast is on screen, so waiting for it freezes the event loop and the next copy is missed entirely.
Third, an exception escaping a slot is fatal in PyQt5—one malformed copy would take the watcher down and every paste after it—so the whole thing runs inside a try, and what it decided goes to ~/.cache/html2md/watch.log. A one-second timer calls the same code as the signal, so even if dataChanged ever stops arriving, nothing is lost for more than a second.
Two things nearly undid all of this at the last step, both of them about where an application looks when it pastes.
A terminal pastes with the middle button, and that is the PRIMARY selection, a different thing from the clipboard: Emacs took the converted Markdown from the clipboard while Konsole, one window away, was still handing back the browser's own glyphs—K from the MathML, K from the annotation and K from the rendering, the formula three times over. So a conversion now writes the Markdown to PRIMARY as well—and, since the middle button pastes what was merely selected, a selection that was never copied is converted too, provided it carries a formula. That last part is the intrusive one: owning PRIMARY may cost the highlight in the window I am reading, so --no-select gives it back, and --no-primary leaves PRIMARY alone altogether.
And Qt offers text/plain, UTF8_STRING, STRING and TEXT but not text/plain;charset=utf-8, which the browser does offer and which GTK and VTE applications ask for by name. They were getting an empty answer where a paste should have been. It is one line to add, and worth checking with xclip -t target by target rather than trusting that a clipboard is a clipboard.
html2md # clipboard in, Markdown to stdout (a look at it)
html2md -c # clipboard in, Markdown back on the clipboard
html2md -l # convert the last clipboard again, from the cache
html2md page.html # a file in
... | html2md # stdin in
Meta+F9 runs html2md -x -c -n for the same thing on demand, if the watcher is not running. Not a letter: a global shortcut is a grab on a keycode, and with three layouts loaded (fr,es,ru) a letter moves from one physical key to another—M is not where it was between French and Spanish, and under the Russian layout it has no keycode at all. The key then falls through to the application, which types a character instead. Function keys sit still in every layout.
# ~/.local/share/applications/net.local.html2md.desktop
[Desktop Entry]
Exec=/home/laussy/bin/html2md -x -c -n
Name=HTML to Markdown
NoDisplay=true
Type=Application
X-KDE-GlobalAccel-CommandShortcut=true
# ~/.config/kglobalshortcutsrc
[services][net.local.html2md.desktop]
_launch=Meta+F9
kglobalacceld reads that file once, at startup, so it must be restarted for a new entry to take. That it took is checked without pressing anything—the number that comes back is the key combination, Meta (0x10000000) plus F9 (0x01000038) = 285212728:
busctl --user call org.kde.kglobalaccel /component/net_local_html2md_desktop \
org.kde.kglobalaccel.Component allShortcutInfos
KaTeX stores the source; MathJax does not. It ships the visual rendering plus an assistive MathML copy of the same formula, and no LaTeX anywhere—which is why a naive paste of a MathJax page comes out doubled, in Unicode italics: 𝑤𝑘 and then w k again.
MathML is nonetheless a faithful tree of the formula, so it is translated back. msub/msup become _ and ^, mfrac becomes \frac, munderover on a ∑ becomes the limits of a \sum but on a letter becomes \hat or \vec, mtable becomes a matrix, mathvariant becomes \mathcal, \mathbb, \mathfrak. Unicode goes home too: ⊗ to \otimes, α to \alpha, and the mathematical alphanumerics—𝑤, ℋ, ℝ—back to plain letters wearing the right font command, which unicodedata gives for free since their names say what they are (MATHEMATICAL ITALIC SMALL W).
Sym2: spanned by wj⊗wk+wk⊗wjw_j\otimes w_k+w_k... wj⊗wk <- pasted raw
$\mathrm{Sym}^2$: spanned by $w_j\otimes w_k+w_k\otimes w_j$ <- html2md
It is not the author's keystrokes—\frac comes back where \dfrac was written—but it is real LaTeX giving the same formula. A formula with neither annotation nor MathML cannot be recovered by any amount of cleverness: its glyphs are kept, once rather than twice, and counted in the report so I know what needs retyping.
Selecting from the middle of an equation—which is what one does when quoting half a sentence—produces a copy the browser truncates inside the KaTeX span. The rendered half comes over; the katex-mathml sibling holding the annotation does not, because it sits before the point where the selection started. There is then no $\mathrm{\TeX}$ to recover, however well the reader is written:
grep -c 'class="katex"' last.html # 2
grep -c 'katex-mathml' last.html # 1 <- the first formula lost its source
What survives is the glyphs, and those are worth keeping as maths: sym() already turns Ψ into \Psi and ≠ into \neq, so the fallback is delimited like any other formula rather than spilled into the prose. Leaving it bare was a real bug—\Psi(n)=0 for all $n\neq N$, one formula rendered and the other not, in the same sentence.
\Psi(n)=0 for all $n\neq N$ <- 1.4.0, the truncated formula left bare
$\Psi(n)=0$ for all $n\neq N$ <- 1.5.0
KaTeX also pads its layout with zero-width spaces, invisible in the browser and litter in a text file; they are stripped on the way out.
Headings, paragraphs, nested lists, blockquotes, rules, links, images, bold, italics, code, strikeout, tables and fenced code blocks—with the language taken from the class="language-xxx" the highlighter left behind. A paragraph comes out as one line, which is what this wiki wants anyway.
Code blocks need one precaution. In a chat window the <pre> holds more than the code: the little language label and the Copy button live inside it too, and a naive text extraction pastes python and Copy into the listing. So when a <pre> contains a <code>, only the <code> is kept, and <button>, <svg> and <script> are dropped everywhere.
Maths comes out as $...$ inline and $$...$$ on a single line for display, which are the delimiters this wiki understands, from LocalSettings.php:
$wgSmjExtraInlineMath = [ [ "$", "$" ], [ "\\(", "\\)" ] ];
$wgSmjDisplayMath = [ ['$$','$$'] ];
Inside a table cell, and only there, a | living in the maths becomes \vert: a pipe would end the cell, in Markdown as in MediaWiki. It renders identically.
The output is Markdown, not wikitext, and that is deliberate: it goes straight into an {{AI}} call, whose Module:AI converts Markdown itself. See Working with Claude for the whole route.
{{AI|AI=Claude|v=Opus 5|q=my prompt, verbatim|a=
the markdown that was on the clipboard when I pasted
}}
A literal | anywhere in there breaks the a= parameter, so the script counts the pipes it leaves outside table rows and warns; the cure is a subpage and src=. And a table that should become a real wikitable is one pipe away:
html2md | mdtable2wiki
Text is not Markdown-escaped: a stray * or _ in the prose stays a stray * or _. Escaping them would litter every answer with backslashes to protect against something that hardly ever happens, and the output would be unpleasant to re-edit.
Converting the clipboard destroys the HTML it came from, which is the one thing needed to find out why an output is wrong. So whatever is read is kept in ~/.cache/html2md/last.html (or last.txt when the copy carried no HTML at all, which is itself the diagnosis), and html2md -l runs it again.
What cannot be recovered is what the page never showed. A collapsed thinking block, a truncated listing, an artifact behind a tab: only what is actually selected is in the clipboard.
$; zero-width spaces stripped; a selection never copied is converted too (--no-select to stop it).text/plain;charset=utf-8 is served rather than left empty.Popen: waiting for notify-send froze the watcher and it missed the copy that followed.--watch, so there is no keystroke at all; the shortcut moved off a letter and onto Meta+F9; the clipboard read is cached for -l.mathvariant, Unicode alphanumerics, <sup>/<sub>; the glyph fallback no longer doubles.~/bin/html2md#!/usr/bin/env python3
# _ _ _ ___ _
# | |_| |_ _ __ __| |_ )_ __ __| |
# | ' \ _| ' \/ _` |/ /| ' \/ _` |
# |_||_\__|_|_|_\__,_/___|_|_|_\__,_|
#
# html2md - Version 1.5.0
# Claude & F.P. Laussy
# https://laussy.org/wiki/html2md
#
# Select anything in a browser with the mouse, copy, run this: out comes
# the Markdown, with the maths back in LaTeX form.
#
# Usage:
# html2md clipboard in, Markdown to stdout (preview)
# html2md -c clipboard in, Markdown back on the clipboard
# html2md page.html a file in
# ... | html2md stdin in
# html2md -l convert the last clipboard again
#
# Why it works
# ------------
# A selection copied from a browser lands on the clipboard twice: as
# text/plain (what the eye sees - maths reduced to loose glyphs, often
# doubled) and as text/html (the real markup). KaTeX, which is what
# renders maths in Claude's window, keeps the LaTeX it was given inside
# the MathML annotation of every formula:
#
# <span class="katex"><span class="katex-mathml"><math><semantics>
# ... <annotation encoding="application/x-tex">E=mc^2</annotation>
#
# That annotation travels with the text/html flavour. So the original
# LaTeX is already in the clipboard - only the plain-text flavour throws
# it away. This tool reads the html one and puts everything back:
#
# * MATHS WITH NO ANNOTATION. MathJax stores no LaTeX at all, only the
# rendering plus an assistive MathML copy - which is why a naive
# paste of it comes out doubled and in Unicode italics. MathML is a
# faithful tree of the formula, so it is translated back: fractions,
# roots, sub- and superscripts, sums with limits, matrices, accents,
# Greek, operators, and \mathcal/\mathbb/\mathbf from mathvariant.
# Not the author's keystrokes, but LaTeX that gives the same formula;
# * $...$ inline, $$...$$ on one line for display maths (both are
# delimiters on this wiki: $wgSmjExtraInlineMath / $wgSmjDisplayMath);
# * headings, paragraphs, nested lists, blockquotes, rules, links,
# images, **bold**, *italics*, `code`, ~~strikeout~~, tables;
# * fenced code blocks, language taken from class="language-xxx", and
# the chat window's own chrome (the language label, the Copy button)
# dropped - only the <code> inside a <pre> is kept;
# * one paragraph = one line, no hard wrapping, as the wiki wants.
#
# Output is Markdown, not wikitext: that is what Module:AI wants inside
# {{AI|AI=Claude|...|a= }}, and it converts it itself. For a table that
# has to become a real wikitable, pipe it on: html2md | mdtable2wiki.
#
# Caveats
# -------
# * a formula with neither annotation nor MathML (an image-based
# renderer, a screenshot) cannot be recovered: its visible glyphs are
# kept, once rather than twice, and the count is reported;
# * what was read is kept in ~/.cache/html2md/last.html, because the
# conversion overwrites the clipboard it came from and that is the
# one thing needed to find out why an output is wrong: html2md -l;
# * text is not markdown-escaped, so a stray * or _ in the prose stays
# a stray * or _ - it reads better and re-edits better that way;
# * a literal | breaks an {{AI|a=...}} parameter (use src= instead);
# html2md warns when the output has one outside a table row.
import argparse
import os
import re
import shutil
import subprocess
import sys
import time
import unicodedata
from html.parser import HTMLParser
VERSION = '1.5.0'
CACHE = os.path.expanduser('~/.cache/html2md/last.html')
VOID = {'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'link',
'meta', 'param', 'source', 'track', 'wbr'}
# never contributes text
SKIP = {'script', 'style', 'svg', 'noscript', 'button', 'select', 'option',
'textarea', 'template', 'iframe', 'head', 'title', 'input'}
BLOCK = {'p', 'div', 'section', 'article', 'main', 'header', 'footer',
'aside', 'nav', 'figure', 'figcaption', 'ul', 'ol', 'li', 'dl',
'dt', 'dd', 'pre', 'blockquote', 'table', 'hr', 'form', 'address',
'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}
HEADING = {'h%d' % i: '#' * i for i in range(1, 7)}
# ------------------------------------------------------------------- tree
# HTMLParser is a stream parser; a small tree makes the rendering rules
# (lists inside lists, code inside a cell) plain recursion instead of a
# pile of depth counters.
class Node:
__slots__ = ('tag', 'attrs', 'kids')
def __init__(self, tag, attrs=None):
self.tag = tag
self.attrs = attrs or {}
self.kids = []
class Tree(HTMLParser):
def __init__(self):
HTMLParser.__init__(self, convert_charrefs=True)
self.root = Node('#root')
self.stack = [self.root]
def handle_starttag(self, tag, attrs):
n = Node(tag, dict(attrs))
self.stack[-1].kids.append(n)
if tag not in VOID:
self.stack.append(n)
def handle_startendtag(self, tag, attrs):
self.stack[-1].kids.append(Node(tag, dict(attrs)))
def handle_endtag(self, tag):
for i in range(len(self.stack) - 1, 0, -1):
if self.stack[i].tag == tag:
del self.stack[i:]
return # stray </div>: ignored
def handle_data(self, data):
self.stack[-1].kids.append(data)
def parse(html):
p = Tree()
p.feed(html)
p.close()
root = p.root
body = find(root, lambda n: n.tag == 'body')
return body or root
# ---------------------------------------------------------------- walking
def find(node, pred):
for k in node.kids:
if isinstance(k, str):
continue
if pred(k):
return k
got = find(k, pred)
if got is not None:
return got
return None
def has_class(node, name):
return name in (node.attrs.get('class') or '').split()
def raw_text(node):
"""All text in the subtree, verbatim (newlines and runs kept)."""
if isinstance(node, str):
return node
if node.tag in SKIP:
return ''
return ''.join(raw_text(k) for k in node.kids)
def flat_text(node):
return re.sub(r'\s+', ' ', raw_text(node).replace('\xa0', ' ')).strip()
# ------------------------------------------------------- mathml -> latex
# When a page kept no x-tex annotation - MathJax is the usual case - the
# LaTeX is not stored anywhere, but the MathML is a faithful tree of the
# formula and translates back mechanically. It is not the author's exact
# source (\frac may come back where \dfrac was written) but it is real
# LaTeX that compiles to the same formula, which is the point.
GREEK = ('alpha beta gamma delta epsilon zeta eta theta iota kappa lambda '
'mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega').split()
SYMBOL = {
'′': "'", '″': "''", '±': r'\pm', '∓': r'\mp',
'×': r'\times', '÷': r'\div', '⋅': r'\cdot',
'∗': '*', '∘': r'\circ', '⊕': r'\oplus',
'⊗': r'\otimes', '⊙': r'\odot', '∑': r'\sum',
'∏': r'\prod', '∫': r'\int', '∬': r'\iint',
'∮': r'\oint', '√': r'\sqrt', '∞': r'\infty',
'∂': r'\partial', '∇': r'\nabla', '∀': r'\forall',
'∃': r'\exists', '∄': r'\nexists', '∅': r'\emptyset',
'∈': r'\in', '∉': r'\notin', '∋': r'\ni',
'⊂': r'\subset', '⊃': r'\supset', '⊆': r'\subseteq',
'⊇': r'\supseteq', '∪': r'\cup', '∩': r'\cap',
'≤': r'\leq', '≥': r'\geq', '≪': r'\ll', '≫': r'\gg',
'≠': r'\neq', '≡': r'\equiv', '≈': r'\approx',
'≃': r'\simeq', '∼': r'\sim', '∝': r'\propto',
'≅': r'\cong', '→': r'\to', '←': r'\leftarrow',
'↔': r'\leftrightarrow', '⇒': r'\Rightarrow',
'⇐': r'\Leftarrow', '⇔': r'\Leftrightarrow',
'↦': r'\mapsto', '∥': r'\parallel', '⊥': r'\perp',
'∠': r'\angle', 'ℵ': r'\aleph', 'ℏ': r'\hbar',
'ℓ': r'\ell', '℘': r'\wp', 'ℑ': r'\Im', 'ℜ': r'\Re',
'ϕ': r'\phi', 'φ': r'\varphi', 'ϑ': r'\vartheta',
'ϵ': r'\epsilon', 'ε': r'\varepsilon', 'ϱ': r'\varrho',
'ϖ': r'\varpi', 'ϰ': r'\varkappa', 'ς': r'\varsigma',
'¬': r'\neg', '∧': r'\wedge', '∨': r'\vee',
'∴': r'\therefore', '∵': r'\because', '⋯': r'\cdots',
'…': r'\ldots', '⋮': r'\vdots', '⋱': r'\ddots',
'⌈': r'\lceil', '⌉': r'\rceil', '⌊': r'\lfloor',
'⌋': r'\rfloor', '⟨': r'\langle', '⟩': r'\rangle',
'‖': r'\Vert', '†': r'\dagger', '‡': r'\ddagger',
'⊕︀': r'\oplus', '°': r'^\circ', '⊕ ': r'\oplus',
'{': r'\{', '}': r'\}', ' ': ' ', '': '', '': '',
'': '', '': '',
}
ACCENT = {'^': r'\hat', 'ˆ': r'\hat', '̂': r'\hat',
'¯': r'\bar', '̄': r'\bar', '‾': r'\bar',
'˜': r'\tilde', '̃': r'\tilde', '→': r'\vec',
'⃗': r'\vec', '˙': r'\dot', '̇': r'\dot',
'¨': r'\ddot', '̈': r'\ddot', '⏞': r'\overbrace',
'⏟': r'\underbrace'}
FONT = {'BOLD': r'\mathbf', 'ITALIC': '', 'BOLD ITALIC': r'\boldsymbol',
'SCRIPT': r'\mathcal', 'BOLD SCRIPT': r'\mathcal',
'FRAKTUR': r'\mathfrak', 'BLACK-LETTER': r'\mathfrak',
'BOLD FRAKTUR': r'\mathfrak', 'DOUBLE-STRUCK': r'\mathbb',
'SANS-SERIF': r'\mathsf', 'SANS-SERIF BOLD': r'\mathsf',
'SANS-SERIF ITALIC': r'\mathsf', 'MONOSPACE': r'\mathtt'}
STYLED = re.compile(r'^(?:MATHEMATICAL )?(%s)(?: (?:CAPITAL|SMALL))?'
r'(?: LETTER| DIGIT)? (\S+)$' % '|'.join(FONT))
BIG = {r'\sum', r'\prod', r'\int', r'\iint', r'\oint', r'\bigcup', r'\bigcap',
r'\lim', r'\max', r'\min', r'\sup', r'\inf'}
def unstyle(ch):
"""A Unicode maths letter back to ASCII + the font it was wearing.
'MATHEMATICAL ITALIC SMALL W' -> ('', 'w'); italic is what maths mode
does anyway, so it costs no markup."""
try:
name = unicodedata.name(ch)
except ValueError:
return None
m = STYLED.match(name)
if not m:
return None
font, base = FONT[m.group(1)], m.group(2)
if len(base) == 1:
letter = base
elif base.lower() in GREEK:
letter = '\\' + (base.capitalize() if 'CAPITAL' in name
else base.lower())
else:
digits = 'ZERO ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE'.split()
letter = str(digits.index(base)) if base in digits else None
if letter is None:
return None
return (font + '{' + letter + '}') if font else letter
def join(parts):
r"""Concatenate LaTeX bits, keeping a space where one command would
otherwise swallow the next letter: \alpha + b is \alpha b, never
\alphab. Doing it here, at the seam, is the only place where the two
tokens are still distinguishable."""
out = ''
for p in parts:
if out and re.search(r'\\[A-Za-z]+$', out) and re.match(r'[A-Za-z]', p):
out += ' '
out += p
return out
VARIANT = {'script': r'\mathcal', 'bold': r'\mathbf', 'fraktur': r'\mathfrak',
'double-struck': r'\mathbb', 'sans-serif': r'\mathsf',
'monospace': r'\mathtt', 'bold-italic': r'\boldsymbol',
'normal': r'\mathrm'}
def sym(text):
"""A run of characters as LaTeX: Greek stays Greek, operators become
their commands, styled letters lose the styling."""
out = []
for ch in text:
if ch in SYMBOL:
out.append(SYMBOL[ch])
continue
name = unicodedata.name(ch, '')
if name.startswith('GREEK ') and 'LETTER' in name:
base = name.split('LETTER ')[-1].lower()
if base in GREEK:
out.append('\\' + (base.capitalize() if 'CAPITAL' in name
else base))
continue
got = unstyle(ch)
out.append(got if got is not None else ch)
return join(out)
def brace(s):
"""Braces only where TeX needs them: x_{ab} but x_1."""
s = s.strip()
if len(s) == 1 or re.fullmatch(r'\\[A-Za-z]+', s):
return s
return '{' + s + '}'
def mrow(kids):
return join([mathml(k) for k in kids])
def mkids(node):
return [k for k in node.kids
if not (isinstance(k, str) and not k.strip())]
def mathml(node):
"""One MathML node as LaTeX."""
if isinstance(node, str):
return sym(node.strip())
t = node.tag.split(':')[-1]
kids = mkids(node)
if t in ('math', 'semantics', 'mrow', 'mstyle', 'mpadded', 'mtd',
'menclose', 'msrow', 'mscarries'):
return mrow(kids)
if t in ('annotation', 'annotation-xml', 'mphantom'):
return ''
if t in ('mi', 'mn', 'mtext', 'ms', 'mo'):
text = raw_text(node).strip()
if t == 'mtext':
return r'\text{%s}' % text if text else r'\ '
var = VARIANT.get((node.attrs.get('mathvariant') or '').lower())
if var and text:
return r'%s{%s}' % (var, text)
got = sym(text)
if t == 'mi' and len(text) > 1 and re.fullmatch(r'[A-Za-z]+', text):
return r'\%s' % text if text in (
'sin cos tan log ln exp lim max min sup inf det dim ker '
'deg gcd arg').split() else r'\mathrm{%s}' % text
return got
if t == 'mspace':
return r'\,'
if t in ('msub', 'msup', 'msubsup', 'munder', 'mover', 'munderover'):
return script(t, kids)
if t == 'mfrac':
if len(kids) == 2:
a, b = mathml(kids[0]), mathml(kids[1])
if node.attrs.get('linethickness') in ('0', '0pt'):
return r'\binom{%s}{%s}' % (a, b)
return r'\frac{%s}{%s}' % (a, b)
if t == 'msqrt':
return r'\sqrt{%s}' % mrow(kids)
if t == 'mroot' and len(kids) == 2:
return r'\sqrt[%s]{%s}' % (mathml(kids[1]), mathml(kids[0]))
if t == 'mfenced':
o = node.attrs.get('open', '(')
c = node.attrs.get('close', ')')
return r'\left%s %s \right%s' % (sym(o), mrow(kids), sym(c))
if t == 'mtable':
rows = [' & '.join(mathml(c) for c in mkids(r))
for r in kids if r.tag.split(':')[-1] in ('mtr', 'mlabeledtr')]
return r'\begin{matrix} %s \end{matrix}' % r' \\ '.join(rows)
if t == 'mtr':
return ' & '.join(mathml(c) for c in mkids(node))
return mrow(kids)
def script(t, kids):
"""Sub- and superscripts, and the same thing written under and over:
a limit under a sum is _ and ^ again, an accent over a letter is not."""
if not kids:
return ''
base = mathml(kids[0])
rest = [mathml(k) for k in kids[1:]]
if t in ('munder', 'mover', 'munderover') and base not in BIG:
if t == 'mover' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\overset{%s}{%s}' % (rest[0], base)
if t == 'munder' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\underset{%s}{%s}' % (rest[0], base)
if t in ('msub', 'munder'):
return '%s_%s' % (base, brace(rest[0]))
if t in ('msup', 'mover'):
return '%s^%s' % (base, brace(rest[0]))
if len(rest) == 2: # msubsup, munderover
return '%s_%s^%s' % (base, brace(rest[0]), brace(rest[1]))
return base + ''.join(rest)
def mathml_to_tex(node):
return re.sub(r'\s{2,}', ' ', mathml(node)).strip()
# ------------------------------------------------------------------- math
class Ctx:
"""Rendering state: counters for the report, plus the table flag,
which is the one place where a | inside maths has to be neutralised."""
def __init__(self):
self.math = 0
self.rebuilt = 0
self.lost = 0
self.in_table = False
def is_annotation(n):
return (n.tag == 'annotation'
and n.attrs.get('encoding') == 'application/x-tex')
def latex_of(node):
"""The LaTeX a formula was built from, if the page kept it."""
ann = node if is_annotation(node) else find(node, is_annotation)
if ann is not None:
return raw_text(ann).strip()
for key in ('data-latex', 'data-tex', 'alt'): # other renderers
for probe in (node, find(node, lambda n: key in n.attrs)):
if probe is not None and probe.attrs.get(key):
return probe.attrs[key].strip()
tex = find(node, lambda n: n.tag == 'script'
and 'math/tex' in (n.attrs.get('type') or ''))
if tex is not None:
return raw_text(tex).strip()
return None
def visible_math(node):
"""Last resort: the glyphs, and only one copy of them. A formula is
on the page twice - the rendering and the MathML behind it - and
taking the subtree wholesale is what doubles it.
This is what a selection begun in the middle of a formula leaves
behind: the browser copies the rendered half and drops the MathML
sibling that held the annotation, so there is no LaTeX to be had.
The glyphs still carry the formula, and sym() puts the symbols back
in command form, so the result is worth delimiting as maths."""
for probe in ('katex-mathml', 'katex-html'):
got = find(node, lambda n, p=probe: has_class(n, p))
if got is not None:
return sym(zapzero(flat_text(got)))
vis = find(node, lambda n: n.tag in ('mjx-math', 'mjx-container'))
return sym(zapzero(flat_text(vis if vis is not None else node)))
def zapzero(text):
"""KaTeX pads its layout with zero-width spaces; they are invisible
in the browser and litter in a text file."""
return re.sub('[\xad]', '', text)
def math_md(node, ctx, display=False):
tex = latex_of(node)
ctx.math += 1
if tex is None:
mml = node if node.tag == 'math' else find(node,
lambda n: n.tag == 'math')
if mml is not None:
tex = mathml_to_tex(mml)
if tex:
ctx.rebuilt += 1
if not tex:
ctx.lost += 1
tex = visible_math(node)
if not tex:
return ''
tex = re.sub(r'\s+', ' ', tex.replace('\xa0', ' ')).strip()
if ctx.in_table:
tex = tex.replace('|', r'\vert ') # a | would end the cell
d = '$$' if display else '$'
pad = ' ' if display else ''
return d + pad + tex + pad + d
def is_math(node):
# katex-mathml and a bare annotation are here for the same reason:
# a selection started inside a formula arrives without its outer
# wrapper, and what is left must still be read as maths rather than
# spilled into the prose as loose text.
return (has_class(node, 'katex') or has_class(node, 'katex-display')
or has_class(node, 'katex-mathml') or is_annotation(node)
or node.tag in ('math', 'mjx-container', 'mjx-assistive-mml')
or has_class(node, 'MathJax') or has_class(node, 'MathJax_Preview'))
# ----------------------------------------------------------------- inline
MARKS = {'b': '**', 'strong': '**', 'i': '*', 'em': '*',
's': '~~', 'del': '~~', 'strike': '~~'}
def marked(inner, mark):
"""Put the markers against the text, not against its spaces:
' word ' -> ' **word** ', and nothing at all for empty runs."""
m = re.match(r'^(\s*)(.*?)(\s*)$', inner, re.S)
lead, core, trail = m.groups()
if not core:
return inner
return lead + mark + core + mark + trail
def code_span(inner):
inner = inner.replace('\n', ' ')
n = max((len(r) for r in re.findall(r'`+', inner)), default=0)
fence = '`' * (n + 1)
pad = ' ' if inner[:1] == '`' or inner[-1:] == '`' else ''
return fence + pad + inner + pad + fence
def inline(node, ctx):
if isinstance(node, str):
return re.sub(r'[ \t\r\n]+', ' ', node.replace('\xa0', ' '))
tag = node.tag
if tag in SKIP:
return ''
if is_math(node):
return math_md(node, ctx, display=has_class(node, 'katex-display'))
if tag == 'br':
return '\n'
if tag == 'img':
alt = node.attrs.get('alt', '').strip()
src = node.attrs.get('src', '')
return '' % (alt, src) if src[:4] == 'http' else (
'![%s]' % alt if alt else '')
if tag == 'wbr':
return ''
inner = ''.join(inline(k, ctx) for k in node.kids)
if tag in MARKS:
return marked(inner, MARKS[tag])
if tag in ('sup', 'sub'):
# Markdown has no notation for these; the tag itself is understood
# both by MediaWiki and by every Markdown renderer worth the name
return '<%s>%s</%s>' % (tag, inner, tag) if inner.strip() else inner
if tag in ('code', 'tt', 'kbd', 'samp'):
return code_span(inner) if inner.strip() else inner
if tag == 'a':
href = node.attrs.get('href', '')
text = inner.strip()
if not href or href[:1] == '#' or href[:11] == 'javascript:':
return inner
if text in (href, href.rstrip('/')):
return '<%s>' % href
return '[%s](%s)' % (text, href) if text else ''
return inner
# ------------------------------------------------------------------ block
def indent(text, pad, first=None):
lines = text.split('\n')
head = (first if first is not None else pad) + lines[0]
return '\n'.join([head] + [(pad + l if l.strip() else '') for l in lines[1:]])
def join_item(li, ctx):
"""The blocks of one <li>. A sub-list hangs directly under its item
(no blank line): that is what keeps it one list rather than two."""
body = ''
for part in blocks(li.kids, ctx):
if not body:
body = part
elif re.match(r'(?:[-*+]|\d+[.)]) ', part):
body += '\n' + part
else:
body += '\n\n' + part
return body
def list_md(node, ctx):
ordered = node.tag == 'ol'
try:
n = int(node.attrs.get('start', 1))
except ValueError:
n = 1
out = []
for li in node.kids:
if isinstance(li, str) or li.tag != 'li':
continue
body = join_item(li, ctx)
if not body.strip():
continue
mark = ('%d. ' % n) if ordered else '- '
n += 1
out.append(indent(body, ' ' * len(mark), first=mark))
return '\n'.join(out)
def table_md(node, ctx):
was, ctx.in_table = ctx.in_table, True
rows = []
aligns = {}
for tr in [n for n in walk(node) if not isinstance(n, str)
and n.tag == 'tr']:
cells = []
for c in tr.kids:
if isinstance(c, str) or c.tag not in ('td', 'th'):
continue
style = c.attrs.get('style', '') or ''
m = re.search(r'text-align\s*:\s*(left|right|center)', style, re.I)
a = (m.group(1) if m else c.attrs.get('align', '')).lower()
if a:
aligns.setdefault(len(cells), a)
txt = re.sub(r'\s+', ' ', inline(c, ctx)).strip()
cells.append(txt.replace('\n', ' '))
if cells:
rows.append(cells)
ctx.in_table = was
if not rows:
return ''
ncol = max(len(r) for r in rows)
sep = ['|'.join({'center': ':---:', 'right': '---:', 'left': ':---'}
.get(aligns.get(i), '---') for i in range(ncol))]
def line(cells):
v = [c.replace('|', r'\|') for c in cells] # math | is \vert by now
return '| ' + ' | '.join(v + [''] * (ncol - len(v))) + ' |'
return '\n'.join([line(rows[0]), '|' + sep[0] + '|']
+ [line(r) for r in rows[1:]])
def walk(node):
for k in node.kids:
yield k
if not isinstance(k, str):
for g in walk(k):
yield g
def pre_md(node):
"""A code block. In a chat window the <pre> also holds the language
label and the Copy button; the <code> holds only the code."""
code = find(node, lambda n: n.tag == 'code')
src = raw_text(code if code is not None else node)
src = src.replace('\xa0', ' ').replace('\r\n', '\n').strip('\n')
lang = ''
for probe in (code, node):
if probe is None:
continue
for c in (probe.attrs.get('class') or '').split():
m = re.match(r'(?:language|lang|highlight|hljs)[-_](\w+)$', c)
if m:
lang = m.group(1)
break
lang = lang or probe.attrs.get('data-language', '')
if lang:
break
n = max((len(r) for r in re.findall(r'^\s*(`{3,})', src, re.M)), default=2)
fence = '`' * max(3, n + 1)
return '%s%s\n%s\n%s' % (fence, lang, src, fence)
def blocks(kids, ctx):
"""Children of a container, as a list of block-level chunks. Text and
inline elements between two blocks make a paragraph of their own."""
out = []
run = []
def flush():
if run:
para = re.sub(r'[ \t]*\n[ \t]*', '\n', ''.join(run)).strip()
if para:
out.append(para)
run.clear()
for k in kids:
if isinstance(k, str) or k.tag not in BLOCK or is_math(k):
run.append(inline(k, ctx))
continue
flush()
got = block(k, ctx)
if got.strip():
out.append(got)
flush()
return out
def block(node, ctx):
tag = node.tag
if tag in SKIP:
return ''
if tag == 'hr':
return '---'
if tag in HEADING:
text = re.sub(r'\s+', ' ', inline(node, ctx)).strip()
return '%s %s' % (HEADING[tag], text) if text else ''
if tag == 'pre':
return pre_md(node)
if tag == 'table':
return table_md(node, ctx)
if tag in ('ul', 'ol'):
return list_md(node, ctx)
if tag == 'blockquote':
body = '\n\n'.join(blocks(node.kids, ctx))
return '\n'.join('>' + (' ' + l if l.strip() else '')
for l in body.split('\n'))
if tag in ('dt',):
return '**%s**' % flat_text(node)
# p, div, li, section, ... : just their contents
return '\n\n'.join(blocks(node.kids, ctx))
def to_markdown(html, ctx):
md = '\n\n'.join(blocks(parse(html).kids, ctx))
md = re.sub(r'[ \t]+$', '', md, flags=re.M)
md = re.sub(r'\n{3,}', '\n\n', md)
return md.strip() + '\n'
# -------------------------------------------------------------- clipboard
def _run(cmd):
if not shutil.which(cmd[0]):
return None
try:
return subprocess.run(cmd, capture_output=True, check=True
).stdout.decode('utf-8', 'replace')
except subprocess.CalledProcessError:
return None
def clip_read():
"""The html flavour if the copy had one - that is the whole point."""
for cmd in (['wl-paste', '-t', 'text/html'],
['xclip', '-selection', 'clipboard', '-t', 'text/html', '-o']):
got = _run(cmd)
if got and got.strip():
return got, True
for cmd in (['wl-paste', '-n'], ['xclip', '-selection', 'clipboard', '-o'],
['xsel', '-ob']):
got = _run(cmd)
if got is not None:
return got, False
sys.exit('html2md: no clipboard tool (wl-clipboard / xclip / xsel)')
def clip_write(text):
for cmd in (['wl-copy'], ['xclip', '-selection', 'clipboard'],
['xsel', '-ib']):
if shutil.which(cmd[0]):
try:
subprocess.run(cmd, input=text.encode(), check=True)
return cmd[0]
except subprocess.CalledProcessError:
pass
return None
# ------------------------------------------------------------------ watch
# The hotkey works, but a hotkey is a thing to remember. Watching the
# clipboard removes the step altogether: copy in the browser, paste in
# Emacs, and what arrives is already Markdown.
#
# Nothing is lost by doing it: the converted Markdown goes on as
# text/plain - the flavour Emacs, a textarea and the wiki edit box all
# read - while the original text/html is put back untouched, so a rich
# paste into a mail or a word processor still gets its formatting.
def watch(o):
from PyQt5 import QtCore, QtGui, QtWidgets
app = QtWidgets.QApplication(sys.argv[:1])
app.setQuitOnLastWindowClosed(False)
board = app.clipboard()
seen = [None, None, 0.0, None] # plain, html, when, selection html
def look():
# Is this our own write coming back round? Ask who owns the
# clipboard, never whether the HTML looks familiar: copying the
# same passage twice hands us byte-identical markup, and a
# content test then mistakes a fresh copy for its own echo and
# sits there doing nothing - which is exactly what it did.
if board.ownsClipboard():
return
data = board.mimeData()
if data is None:
return
html = data.html() if data.hasHtml() else ''
if not html:
# Nothing to do, and we take no ownership - so without a
# memory of it the tick would say so once a second forever.
fingerprint = hash(data.text() or '')
if fingerprint != seen[0]:
seen[0] = fingerprint
keep(data.text() or '', plain=True)
log('no html on the clipboard, left alone')
return
# Identical HTML is NOT a reason to skip: copying the same
# passage twice is a perfectly ordinary thing to do. Only a
# repeat within a moment is suppressed, and only so that a failed
# hand-over cannot turn into a conversion loop.
if html == seen[1] and time.monotonic() - seen[2] < 2:
return
seen[1], seen[2] = html, time.monotonic()
keep(html)
ctx = Ctx()
md = to_markdown(html, ctx)
if not md.strip():
log('html converted to nothing, left alone')
return
fresh = QtCore.QMimeData()
for fmt in data.formats(): # keep images, source URLs, everything
if fmt.startswith('text/plain') or fmt in ('UTF8_STRING',
'STRING', 'TEXT'):
continue
fresh.setData(fmt, data.data(fmt))
fresh.setText(md)
# Qt offers text/plain, UTF8_STRING, STRING and TEXT but not
# text/plain;charset=utf-8, which the browser does offer and
# which GTK and VTE applications ask for by name: without it
# they get an empty answer where a paste should have been.
fresh.setData('text/plain;charset=utf-8', md.encode())
board.setMimeData(fresh)
# A terminal pastes with the middle button, and that is the
# PRIMARY selection, not the clipboard - it would otherwise hand
# back the browser's own glyphs, the formula three times over.
if o.primary:
board.setText(md, QtGui.QClipboard.Selection)
note = '%d formula%s%s' % (ctx.math, 's' if ctx.math != 1 else '',
', %d rebuilt from MathML' % ctx.rebuilt
if ctx.rebuilt else '')
log('converted, %s' % note)
if o.quiet:
return
if o.notify and ctx.math:
report(o, 'converted', note)
elif not o.notify:
sys.stderr.write('html2md: converted, %s\n' % note)
def look_selection():
# The middle button pastes what was merely *selected*, with no
# copy at all. Converting that means owning PRIMARY, and the
# window I am reading may drop its highlight when it loses it -
# hence --no-select for anyone who finds that worse than the
# tripled formula it cures.
if board.ownsSelection():
return
data = board.mimeData(QtGui.QClipboard.Selection)
if data is None or not data.hasHtml():
return
html = data.html()
if html == seen[3] or 'katex' not in html and '<math' not in html \
and 'mjx-' not in html:
return # only formulae are worth the theft
seen[3] = html
ctx = Ctx()
md = to_markdown(html, ctx)
if md.strip() and ctx.math:
board.setText(md, QtGui.QClipboard.Selection)
log('selection converted, %d formulas' % ctx.math)
def guarded_selection():
try:
look_selection()
except Exception as exc:
log('selection error: %r' % (exc,))
def guarded():
# An exception escaping a slot is fatal in PyQt5; one malformed
# copy would take the watcher down and every paste after it.
try:
look()
except Exception as exc:
log('error: %r' % (exc,))
board.dataChanged.connect(guarded)
if o.select:
board.selectionChanged.connect(guarded_selection)
tick = QtCore.QTimer() # in case dataChanged ever stops
tick.timeout.connect(guarded) # arriving: one round trip a second
tick.start(1000)
log('watching (pid %d)' % os.getpid())
if not o.quiet:
sys.stderr.write('html2md: watching the clipboard (Ctrl-C to stop)\n')
QtCore.QTimer.singleShot(0, guarded) # whatever is there now
sys.exit(app.exec_())
def log(line):
"""A record that outlives the session: when the watcher looks stuck,
this is what says whether it saw the copy at all."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = os.path.join(os.path.dirname(CACHE), 'watch.log')
if os.path.exists(path) and os.path.getsize(path) > 200000:
os.replace(path, path + '.1')
with open(path, 'a', encoding='utf-8') as f:
f.write('%s %s\n' % (time.strftime('%Y-%m-%d %H:%M:%S'), line))
except OSError:
pass
def keep(src, plain=False):
"""Stash what was read. Converting the clipboard destroys the html it
came from, and when the output is wrong that is exactly what one
needs to look at: html2md -l runs it again, html2md ~/.cache/... too."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = CACHE[:-5] + '.txt' if plain else CACHE
with open(path, 'w', encoding='utf-8') as f:
f.write(src)
except OSError:
pass
def report(o, title, body):
"""On a hotkey there is no terminal to write to, so say it in a toast."""
if o.quiet or not body:
return
if o.notify and shutil.which('notify-send'):
# Popen, not run: notify-send lives as long as the toast is on
# screen, and waiting for it would freeze the watcher's event
# loop - it would sit out the next copy entirely.
subprocess.Popen(['notify-send', '-a', 'html2md', '-t', '4000',
'html2md: ' + title, body],
start_new_session=True)
else:
sys.stderr.write('html2md: ' + body + '\n')
def looks_like_html(text):
return re.search(r'<(p|div|span|table|li|h[1-6]|pre|br|body)[\s/>]',
text, re.I) is not None
# --------------------------------------------------------------------- cli
def main():
p = argparse.ArgumentParser(
prog='html2md',
description='Rendered HTML on the clipboard -> Markdown, '
'maths back in LaTeX.')
p.add_argument('file', nargs='?', help='HTML file (default: clipboard)')
p.add_argument('-w', '--watch', action='store_true',
help='sit on the clipboard and convert every copy')
p.add_argument('--no-primary', dest='primary', action='store_false',
help='watch: do not put the Markdown on PRIMARY on a copy')
p.add_argument('--no-select', dest='select', action='store_false',
help='watch: do not convert a selection that was never copied')
p.add_argument('-l', '--last', action='store_true',
help='convert the last clipboard again (%s)' % CACHE)
p.add_argument('-x', '--clipboard', action='store_true',
help='read the clipboard even if stdin is a pipe')
p.add_argument('-c', '--copy', action='store_true',
help='put the Markdown back on the clipboard')
p.add_argument('-q', '--quiet', action='store_true',
help='no report on stderr')
p.add_argument('-n', '--notify', action='store_true',
help='report in a desktop notification (for a hotkey)')
p.add_argument('-V', '--version', action='version',
version='html2md ' + VERSION)
o = p.parse_args()
if o.watch:
watch(o)
from_clip = False
if o.last:
o.file = CACHE
if o.file:
with open(o.file, encoding='utf-8', errors='replace') as f:
src = f.read()
elif o.clipboard or sys.stdin.isatty():
src, rich = clip_read()
from_clip = True
keep(src) # the conversion overwrites the clipboard
if not rich and not looks_like_html(src):
# plain text was copied: nothing to convert, pass it through
if o.copy:
clip_write(src)
sys.stdout.write(src if src.endswith('\n') else src + '\n')
report(o, 'nothing to do', 'the clipboard has no text/html '
'(plain text left as it was)')
return
else:
src = sys.stdin.read()
ctx = Ctx()
md = to_markdown(src, ctx)
if o.copy:
if clip_write(md) is None:
sys.exit('html2md: no clipboard tool to write with')
if not (o.copy and from_clip and sys.stdout.isatty()):
sys.stdout.write(md)
say = []
if ctx.math:
say.append('%d formula%s' % (ctx.math, 's' if ctx.math > 1 else ''))
if ctx.rebuilt:
say.append('%d rebuilt from MathML (no annotation)' % ctx.rebuilt)
if ctx.lost:
say.append('%d with no LaTeX kept by the page (glyphs only)'
% ctx.lost)
loose = [l for l in md.split('\n')
if '|' in l and not re.match(r'\s*\|', l)]
if loose:
say.append('%d line%s with a literal | - breaks {{AI|a=...}}, '
'use src=' % (len(loose), 's' if len(loose) > 1 else ''))
if o.copy:
say.append('on the clipboard')
report(o, '%d line%s of Markdown'
% (md.count('\n'), '' if md.count('\n') == 1 else 's'),
'; '.join(say))
if __name__ == '__main__':
main()