html2md
html2md is a python script that turns a selection copied out of a browser—typically a piece of a Claude answer read in its own window—into wikitext, with the maths back in $\mathrm{\TeX}$ form. The name is older than the behaviour: until 2.0.0 it produced Markdown, which is still there under -m, but this is a wiki-writing machine and wikitext is what I paste nine times out of ten.
Reading an answer on the web, I see the rendered page: the Markdown is gone, and every formula is a heap of KaTeX spans. Selecting a paragraph with the mouse and pasting it into an edit box gives ugly prose without any of its structure, and formulae reduced to loose glyphs—usually repeated, because the rendering and the MathML behind it are both in the selection. This script gives back the source. It sits on the clipboard and does it by itself: there is nothing to press.
Written with the help of Claude Opus 5 on 1 August (2026). Version 2.0.0.
The idea
There is nothing to reverse-engineer: the $\mathrm{\TeX}$ is already on the clipboard. A browser puts a copied selection there twice, once as text/plain—what the eye sees—and once as text/html, the real markup. And KaTeX, which is what renders maths in a chat window, keeps the $\mathrm{\TeX}$ it was handed inside the MathML annotation of every formula it builds:
<span class="katex"><span class="katex-mathml"><math><semantics>
... <annotation encoding="application/x-tex">2g\sqrt{n}</annotation>
</semantics></math></span><span class="katex-html" aria-hidden="true">...</span></span>
That annotation travels with the text/html flavour. Only the plain-text flavour throws it away—and the plain-text flavour is precisely the one every text form takes when I hit Ctrl+V. So the whole trick is to read the other target of the same clipboard:
xclip -selection clipboard -t TARGETS -o # see the flavours on offer
xclip -selection clipboard -t text/html -o # the one that has everything
No userscript, no extension, no export, nothing to install in the browser. This is the same observation mdtable2wiki makes for tables; html2md generalises it to a whole selection.
It runs by itself
html2md --watch sits on the clipboard and converts every copy as it arrives, so the gesture is just copy, paste—no keystroke in between, and it works into Emacs, a mail, the wiki edit box, anything. It is started at login from ~/.config/autostart/net.local.html2md-watch.desktop.
Nothing is lost by letting it rewrite every copy, because it only adds: the Markdown goes on as text/plain—the flavour Emacs and every text field read—while the original text/html is put back untouched, so pasting into a mail or a word processor still gets the formatting. Every other flavour of the copy (images, the source URL) is carried over too.
Three details make it work rather than nearly work. First, the watcher must ignore its own write or it converts its output forever—but the test for that is who owns the clipboard, never whether the HTML looks familiar. Comparing content was the first attempt, and it fails the moment I copy the same passage twice: the browser hands over byte-identical markup, the watcher takes the second copy for its own echo, and from then on nothing happens however hard I press Ctrl+C. It looks exactly like a crash, and it is not one. Ownership tells the two apart with no ambiguity: after my write I own the selection, after a copy the browser does.
Second, the notification must be fired with Popen, never run: notify-send lives as long as the toast is on screen, so waiting for it freezes the event loop and the next copy is missed entirely.
Third, an exception escaping a slot is fatal in PyQt5—one malformed copy would take the watcher down and every paste after it—so the whole thing runs inside a try, and what it decided goes to ~/.cache/html2md/watch.log. A one-second timer calls the same code as the signal, so even if dataChanged ever stops arriving, nothing is lost for more than a second.
Which paste, exactly
Two things nearly undid all of this at the last step, both of them about where an application looks when it pastes.
A terminal pastes with the middle button, and that is the PRIMARY selection, a different thing from the clipboard: Emacs took the converted Markdown from the clipboard while Konsole, one window away, was still handing back the browser's own glyphs—K from the MathML, K from the annotation and K from the rendering, the formula three times over. So a conversion now writes the Markdown to PRIMARY as well—and, since the middle button pastes what was merely selected, a selection that was never copied is converted too, provided it carries a formula. That last part is the intrusive one: owning PRIMARY may cost the highlight in the window I am reading, so --no-select gives it back, and --no-primary leaves PRIMARY alone altogether.
And Qt offers text/plain, UTF8_STRING, STRING and TEXT but not text/plain;charset=utf-8, which the browser does offer and which GTK and VTE applications ask for by name. They were getting an empty answer where a paste should have been. It is one line to add, and worth checking with xclip -t target by target rather than trusting that a clipboard is a clipboard.
Which language comes out
Wikitext is the default: bold, italics, == headings ==, * and # lists nested by lengthening the marker, [url label], {| class="wikitable" tables one cell to a line, <syntaxhighlight lang="…"> for code, <blockquote> for a quotation, and $…$ left as it is, since SimpleMathJax reads the dollars. Two details are worth the words: a pipe inside a cell becomes |, because MediaWiki cuts cells on it before anything else runs, and an ordered list that resumes at 3 is written <ol start="3">, since # restarts at 1 whatever one does.
-m gives the Markdown instead, and there is one place where it is required: inside the a= of {{AI}}. Module:AI converts Markdown itself, and it reads a leading ** as bold—so a wikitext nested list handed to it comes back as a stray asterisk and half a list. Measured on a specimen: six list items became four. An answer quoted in an {{AI}} box therefore goes in as Markdown, which is also right in principle, the box being a facsimile of what the machine wrote.
By hand
html2md # clipboard in, Markdown to stdout (a look at it)
html2md -c # clipboard in, Markdown back on the clipboard
html2md -l # convert the last clipboard again, from the cache
html2md -m # Markdown instead, for the a= of Template:AI
html2md -x -c -m -a # ... and {{AI|a=}}-proof: every | becomes {{!}}
html2md page.html # a file in
... | html2md # stdin in
Meta+F9 runs html2md -x -c -n for the same thing on demand, if the watcher is not running. Not a letter: a global shortcut is a grab on a keycode, and with three layouts loaded (fr,es,ru) a letter moves from one physical key to another—M is not where it was between French and Spanish, and under the Russian layout it has no keycode at all. The key then falls through to the application, which types a character instead. Function keys sit still in every layout.
# ~/.local/share/applications/net.local.html2md.desktop
[Desktop Entry]
Exec=/home/laussy/bin/html2md -x -c -n
Name=HTML to Markdown
NoDisplay=true
Type=Application
X-KDE-GlobalAccel-CommandShortcut=true
# ~/.config/kglobalshortcutsrc
[services][net.local.html2md.desktop]
_launch=Meta+F9
Ctrl+Alt+A runs html2md -x -c --notify: the copy comes back as wikitext, ready for a page. Ctrl+Alt+ShiftRight+A runs it with --md --ai instead—Markdown, every pipe written {{!}}—which is the form {{AI}} wants, the same ShiftRight convention as mdtable2wiki's two keys. That one is a letter, and legitimately so: it is a keymapper rule, and keymapper matches the scan code at the evdev level, which no layout can move (it reads {Q} in the file, that being the US position of my A). The rule against letters applies to KDE global shortcuts only.
kglobalacceld reads that file once, at startup, so it must be restarted for a new entry to take. That it took is checked without pressing anything—the number that comes back is the key combination, Meta (0x10000000) plus F9 (0x01000038) = 285212728:
busctl --user call org.kde.kglobalaccel /component/net_local_html2md_desktop \
org.kde.kglobalaccel.Component allShortcutInfos
When the page kept no $\mathrm{\TeX}$
KaTeX stores the source; MathJax does not. It ships the visual rendering plus an assistive MathML copy of the same formula, and no LaTeX anywhere—which is why a naive paste of a MathJax page comes out doubled, in Unicode italics: 𝑤𝑘 and then w k again.
MathML is nonetheless a faithful tree of the formula, so it is translated back. msub/msup become _ and ^, mfrac becomes \frac, munderover on a ∑ becomes the limits of a \sum but on a letter becomes \hat or \vec, mtable becomes a matrix, mathvariant becomes \mathcal, \mathbb, \mathfrak. Unicode goes home too: ⊗ to \otimes, α to \alpha, and the mathematical alphanumerics—𝑤, ℋ, ℝ—back to plain letters wearing the right font command, which unicodedata gives for free since their names say what they are (MATHEMATICAL ITALIC SMALL W).
Sym2: spanned by wj⊗wk+wk⊗wjw_j\otimes w_k+w_k... wj⊗wk <- pasted raw
$\mathrm{Sym}^2$: spanned by $w_j\otimes w_k+w_k\otimes w_j$ <- html2md
It is not the author's keystrokes—\frac comes back where \dfrac was written—but it is real LaTeX giving the same formula. A formula with neither annotation nor MathML cannot be recovered by any amount of cleverness: its glyphs are kept, once rather than twice, and counted in the report so I know what needs retyping.
A selection begun inside a formula
Selecting from the middle of an equation—which is what one does when quoting half a sentence—produces a copy the browser truncates inside the KaTeX span. The rendered half comes over; the katex-mathml sibling holding the annotation does not, because it sits before the point where the selection started. There is then no $\mathrm{\TeX}$ to recover, however well the reader is written:
grep -c 'class="katex"' last.html # 2
grep -c 'katex-mathml' last.html # 1 <- the first formula lost its source
What survives is the glyphs, and those are worth keeping as maths: sym() already turns Ψ into \Psi and ≠ into \neq, so the fallback is delimited like any other formula rather than spilled into the prose. Leaving it bare was a real bug—\Psi(n)=0 for all $n\neq N$, one formula rendered and the other not, in the same sentence.
\Psi(n)=0 for all $n\neq N$ <- 1.4.0, the truncated formula left bare
$\Psi(n)=0$ for all $n\neq N$ <- 1.5.0
KaTeX also pads its layout with zero-width spaces, invisible in the browser and litter in a text file; they are stripped on the way out.
What comes back
Headings, paragraphs, nested lists, blockquotes, rules, links, images, bold, italics, code, strikeout, tables and fenced code blocks—with the language taken from the class="language-xxx" the highlighter left behind. A paragraph comes out as one line, which is what this wiki wants anyway.
Code blocks need one precaution. In a chat window the <pre> holds more than the code: the little language label and the Copy button live inside it too, and a naive text extraction pastes python and Copy into the listing. So when a <pre> contains a <code>, only the <code> is kept, and <button>, <svg> and <script> are dropped everywhere.
Maths comes out as $...$ inline and $$...$$ on a single line for display, which are the delimiters this wiki understands, from LocalSettings.php:
$wgSmjExtraInlineMath = [ [ "$", "$" ], [ "\\(", "\\)" ] ];
$wgSmjDisplayMath = [ ['$$','$$'] ];
Inside a table cell, and only there, a | living in the maths becomes \vert: a pipe would end the cell, in Markdown as in MediaWiki. It renders identically.
Into the wiki
The output is Markdown, not wikitext, and that is deliberate: it goes straight into an {{AI}} call, whose Module:AI converts Markdown itself. See Working with Claude for the whole route.
{{AI|AI=Claude|v=Opus 5|q=my prompt, verbatim|a=
the markdown that was on the clipboard when I pasted
}}
A literal | anywhere in there breaks the a= parameter, which is a real obstacle and not a rare one: any table has a pipe in every row. -a is the answer—every pipe is written {{!}}, which the preprocessor expands back to | before Module:AI reads the string, so the module still sees an ordinary Markdown table and a code block still shows its pipe. Without -a the script counts the pipes and says so, and the older cure—a subpage and src=—is still there for a payload one would rather not touch at all.
A table that should become a real wikitable, rather than a Markdown one rendered inside the box, is a different job and belongs to mdtable2wiki—Ctrl+Alt+W, or one pipe:
html2md | mdtable2wiki
Limitations
Text is not Markdown-escaped: a stray * or _ in the prose stays a stray * or _. Escaping them would litter every answer with backslashes to protect against something that hardly ever happens, and the output would be unpleasant to re-edit.
Converting the clipboard destroys the HTML it came from, which is the one thing needed to find out why an output is wrong. So whatever is read is kept in ~/.cache/html2md/last.html (or last.txt when the copy carried no HTML at all, which is itself the diagnosis), and html2md -l runs it again.
What cannot be recovered is what the page never showed. A collapsed thinking block, a truncated listing, an artifact behind a tab: only what is actually selected is in the clipboard.
Version history
- 1.5.0 (2 August (2026)) — a formula truncated by the selection keeps its
$; zero-width spaces stripped; a selection never copied is converted too (--no-selectto stop it). - 1.4.0 (2 August (2026)) — the middle-button PRIMARY selection gets the Markdown too, and
text/plain;charset=utf-8is served rather than left empty. - 1.3.1 (2 August (2026)) — identical HTML is no longer a reason to skip: only ownership is. Copying the same passage twice had looked like the watcher dying.
- 1.3.0 (2 August (2026)) — ownership test instead of the content test, every slot guarded, a one-second fallback tick, and a log of what was decided.
- 1.2.1 (1 August (2026)) — the notification is fired with
Popen: waiting fornotify-sendfroze the watcher and it missed the copy that followed. - 1.2.0 (1 August (2026)) —
--watch, so there is no keystroke at all; the shortcut moved off a letter and ontoMeta+F9; the clipboard read is cached for-l. - 1.1.0 (1 August (2026)) — MathML translated back to LaTeX for pages that store no annotation (MathJax);
mathvariant, Unicode alphanumerics,<sup>/<sub>; the glyph fallback no longer doubles. - 1.0.0 (1 August (2026)) — first version. KaTeX annotations, block and inline structure, tables, fenced code with the chat chrome removed, clipboard in and out.
- 1.6.0 (16 August (2026)) —
-a/--ai, and theCtrl+Alt+Akeymapper binding that carries it: the whole copy comes back with every literal pipe as{{!}}, so an answer containing a table can be pasted into thea=of {{AI}} as it stands. The pipe warning now counts every pipe: it used to skip the lines starting with one, i.e. exactly the table rows that break the parameter. - 2.0.0 (16 August (2026)) — wikitext by default. Every emitter grew a second form and the flavour rides on the rendering context: marks, links, images, headings, rules, lists (by marker, not by indentation), tables, code blocks, quotations.
-mkeeps the Markdown, which stays the right thing for thea=of {{AI}}—wikitext there is actively wrong, Module:AI reading**as bold and losing the nested items. The keys became two:Ctrl+Alt+Afor a page,Ctrl+Alt+ShiftRight+Afor the box. The watcher, being a running process, has to be restarted for a new version to take.
The file
~/bin/html2md
#!/usr/bin/env python3
# _ _ _ ___ _
# | |_| |_ _ __ __| |_ )_ __ __| |
# | ' \ _| ' \/ _` |/ /| ' \/ _` |
# |_||_\__|_|_|_\__,_/___|_|_|_\__,_|
#
# html2md - Version 2.0.0
# Claude & F.P. Laussy
# https://laussy.org/wiki/html2md
#
# Select anything in a browser with the mouse, copy, run this: out comes
# the WIKITEXT it was rendered from, with the maths back in LaTeX form.
#
# Wikitext is the default since 2.0.0, this being a wiki-writing machine:
# '''bold''', ''italics'', == headings ==, * lists, [url label],
# {| class="wikitable" tables, <syntaxhighlight> code. -m/--md gives the
# Markdown instead, which is what Template:AI wants - Module:AI converts
# the Markdown itself, and an answer quoted in an {{AI}} box should read
# as the machine wrote it.
#
# Usage:
# html2md clipboard in, wikitext to stdout (preview)
# html2md -c clipboard in, wikitext back on the clipboard
# html2md page.html a file in
# ... | html2md stdin in
# html2md -l convert the last clipboard again
# html2md -x -c -m -a Markdown, {{AI|a=}}-proof: | -> {{!}}
#
# Why it works
# ------------
# A selection copied from a browser lands on the clipboard twice: as
# text/plain (what the eye sees - maths reduced to loose glyphs, often
# doubled) and as text/html (the real markup). KaTeX, which is what
# renders maths in Claude's window, keeps the LaTeX it was given inside
# the MathML annotation of every formula:
#
# <span class="katex"><span class="katex-mathml"><math><semantics>
# ... <annotation encoding="application/x-tex">E=mc^2</annotation>
#
# That annotation travels with the text/html flavour. So the original
# LaTeX is already in the clipboard - only the plain-text flavour throws
# it away. This tool reads the html one and puts everything back:
#
# * MATHS WITH NO ANNOTATION. MathJax stores no LaTeX at all, only the
# rendering plus an assistive MathML copy - which is why a naive
# paste of it comes out doubled and in Unicode italics. MathML is a
# faithful tree of the formula, so it is translated back: fractions,
# roots, sub- and superscripts, sums with limits, matrices, accents,
# Greek, operators, and \mathcal/\mathbb/\mathbf from mathvariant.
# Not the author's keystrokes, but LaTeX that gives the same formula;
# * $...$ inline, $$...$$ on one line for display maths (both are
# delimiters on this wiki: $wgSmjExtraInlineMath / $wgSmjDisplayMath);
# * headings, paragraphs, nested lists, blockquotes, rules, links,
# images, **bold**, *italics*, `code`, ~~strikeout~~, tables;
# * fenced code blocks, language taken from class="language-xxx", and
# the chat window's own chrome (the language label, the Copy button)
# dropped - only the <code> inside a <pre> is kept;
# * one paragraph = one line, no hard wrapping, as the wiki wants.
#
# Output is Markdown, not wikitext: that is what Module:AI wants inside
# {{AI|AI=Claude|...|a= }}, and it converts it itself. For a table that
# has to become a real wikitable, pipe it on: html2md | mdtable2wiki.
#
# Caveats
# -------
# * a formula with neither annotation nor MathML (an image-based
# renderer, a screenshot) cannot be recovered: its visible glyphs are
# kept, once rather than twice, and the count is reported;
# * what was read is kept in ~/.cache/html2md/last.html, because the
# conversion overwrites the clipboard it came from and that is the
# one thing needed to find out why an output is wrong: html2md -l;
# * text is not markdown-escaped, so a stray * or _ in the prose stays
# a stray * or _ - it reads better and re-edits better that way;
# * a literal | breaks an {{AI|a=...}} parameter (use src= instead);
# html2md warns when the output has one outside a table row.
import argparse
import os
import re
import shutil
import subprocess
import sys
import time
import unicodedata
from html.parser import HTMLParser
VERSION = '2.0.0'
CACHE = os.path.expanduser('~/.cache/html2md/last.html')
VOID = {'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'link',
'meta', 'param', 'source', 'track', 'wbr'}
# never contributes text
SKIP = {'script', 'style', 'svg', 'noscript', 'button', 'select', 'option',
'textarea', 'template', 'iframe', 'head', 'title', 'input'}
BLOCK = {'p', 'div', 'section', 'article', 'main', 'header', 'footer',
'aside', 'nav', 'figure', 'figcaption', 'ul', 'ol', 'li', 'dl',
'dt', 'dd', 'pre', 'blockquote', 'table', 'hr', 'form', 'address',
'h1', 'h2', 'h3', 'h4', 'h5', 'h6'}
HEADING = {'h%d' % i: '#' * i for i in range(1, 7)}
# ------------------------------------------------------------------- tree
# HTMLParser is a stream parser; a small tree makes the rendering rules
# (lists inside lists, code inside a cell) plain recursion instead of a
# pile of depth counters.
class Node:
__slots__ = ('tag', 'attrs', 'kids')
def __init__(self, tag, attrs=None):
self.tag = tag
self.attrs = attrs or {}
self.kids = []
class Tree(HTMLParser):
def __init__(self):
HTMLParser.__init__(self, convert_charrefs=True)
self.root = Node('#root')
self.stack = [self.root]
def handle_starttag(self, tag, attrs):
n = Node(tag, dict(attrs))
self.stack[-1].kids.append(n)
if tag not in VOID:
self.stack.append(n)
def handle_startendtag(self, tag, attrs):
self.stack[-1].kids.append(Node(tag, dict(attrs)))
def handle_endtag(self, tag):
for i in range(len(self.stack) - 1, 0, -1):
if self.stack[i].tag == tag:
del self.stack[i:]
return # stray </div>: ignored
def handle_data(self, data):
self.stack[-1].kids.append(data)
def parse(html):
p = Tree()
p.feed(html)
p.close()
root = p.root
body = find(root, lambda n: n.tag == 'body')
return body or root
# ---------------------------------------------------------------- walking
def find(node, pred):
for k in node.kids:
if isinstance(k, str):
continue
if pred(k):
return k
got = find(k, pred)
if got is not None:
return got
return None
def has_class(node, name):
return name in (node.attrs.get('class') or '').split()
def raw_text(node):
"""All text in the subtree, verbatim (newlines and runs kept)."""
if isinstance(node, str):
return node
if node.tag in SKIP:
return ''
return ''.join(raw_text(k) for k in node.kids)
def flat_text(node):
return re.sub(r'\s+', ' ', raw_text(node).replace('\xa0', ' ')).strip()
# ------------------------------------------------------- mathml -> latex
# When a page kept no x-tex annotation - MathJax is the usual case - the
# LaTeX is not stored anywhere, but the MathML is a faithful tree of the
# formula and translates back mechanically. It is not the author's exact
# source (\frac may come back where \dfrac was written) but it is real
# LaTeX that compiles to the same formula, which is the point.
GREEK = ('alpha beta gamma delta epsilon zeta eta theta iota kappa lambda '
'mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega').split()
SYMBOL = {
'′': "'", '″': "''", '±': r'\pm', '∓': r'\mp',
'×': r'\times', '÷': r'\div', '⋅': r'\cdot',
'∗': '*', '∘': r'\circ', '⊕': r'\oplus',
'⊗': r'\otimes', '⊙': r'\odot', '∑': r'\sum',
'∏': r'\prod', '∫': r'\int', '∬': r'\iint',
'∮': r'\oint', '√': r'\sqrt', '∞': r'\infty',
'∂': r'\partial', '∇': r'\nabla', '∀': r'\forall',
'∃': r'\exists', '∄': r'\nexists', '∅': r'\emptyset',
'∈': r'\in', '∉': r'\notin', '∋': r'\ni',
'⊂': r'\subset', '⊃': r'\supset', '⊆': r'\subseteq',
'⊇': r'\supseteq', '∪': r'\cup', '∩': r'\cap',
'≤': r'\leq', '≥': r'\geq', '≪': r'\ll', '≫': r'\gg',
'≠': r'\neq', '≡': r'\equiv', '≈': r'\approx',
'≃': r'\simeq', '∼': r'\sim', '∝': r'\propto',
'≅': r'\cong', '→': r'\to', '←': r'\leftarrow',
'↔': r'\leftrightarrow', '⇒': r'\Rightarrow',
'⇐': r'\Leftarrow', '⇔': r'\Leftrightarrow',
'↦': r'\mapsto', '∥': r'\parallel', '⊥': r'\perp',
'∠': r'\angle', 'ℵ': r'\aleph', 'ℏ': r'\hbar',
'ℓ': r'\ell', '℘': r'\wp', 'ℑ': r'\Im', 'ℜ': r'\Re',
'ϕ': r'\phi', 'φ': r'\varphi', 'ϑ': r'\vartheta',
'ϵ': r'\epsilon', 'ε': r'\varepsilon', 'ϱ': r'\varrho',
'ϖ': r'\varpi', 'ϰ': r'\varkappa', 'ς': r'\varsigma',
'¬': r'\neg', '∧': r'\wedge', '∨': r'\vee',
'∴': r'\therefore', '∵': r'\because', '⋯': r'\cdots',
'…': r'\ldots', '⋮': r'\vdots', '⋱': r'\ddots',
'⌈': r'\lceil', '⌉': r'\rceil', '⌊': r'\lfloor',
'⌋': r'\rfloor', '⟨': r'\langle', '⟩': r'\rangle',
'‖': r'\Vert', '†': r'\dagger', '‡': r'\ddagger',
'⊕︀': r'\oplus', '°': r'^\circ', '⊕ ': r'\oplus',
'{': r'\{', '}': r'\}', ' ': ' ', '': '', '': '',
'': '', '': '',
}
ACCENT = {'^': r'\hat', 'ˆ': r'\hat', '̂': r'\hat',
'¯': r'\bar', '̄': r'\bar', '‾': r'\bar',
'˜': r'\tilde', '̃': r'\tilde', '→': r'\vec',
'⃗': r'\vec', '˙': r'\dot', '̇': r'\dot',
'¨': r'\ddot', '̈': r'\ddot', '⏞': r'\overbrace',
'⏟': r'\underbrace'}
FONT = {'BOLD': r'\mathbf', 'ITALIC': '', 'BOLD ITALIC': r'\boldsymbol',
'SCRIPT': r'\mathcal', 'BOLD SCRIPT': r'\mathcal',
'FRAKTUR': r'\mathfrak', 'BLACK-LETTER': r'\mathfrak',
'BOLD FRAKTUR': r'\mathfrak', 'DOUBLE-STRUCK': r'\mathbb',
'SANS-SERIF': r'\mathsf', 'SANS-SERIF BOLD': r'\mathsf',
'SANS-SERIF ITALIC': r'\mathsf', 'MONOSPACE': r'\mathtt'}
STYLED = re.compile(r'^(?:MATHEMATICAL )?(%s)(?: (?:CAPITAL|SMALL))?'
r'(?: LETTER| DIGIT)? (\S+)$' % '|'.join(FONT))
BIG = {r'\sum', r'\prod', r'\int', r'\iint', r'\oint', r'\bigcup', r'\bigcap',
r'\lim', r'\max', r'\min', r'\sup', r'\inf'}
def unstyle(ch):
"""A Unicode maths letter back to ASCII + the font it was wearing.
'MATHEMATICAL ITALIC SMALL W' -> ('', 'w'); italic is what maths mode
does anyway, so it costs no markup."""
try:
name = unicodedata.name(ch)
except ValueError:
return None
m = STYLED.match(name)
if not m:
return None
font, base = FONT[m.group(1)], m.group(2)
if len(base) == 1:
letter = base
elif base.lower() in GREEK:
letter = '\\' + (base.capitalize() if 'CAPITAL' in name
else base.lower())
else:
digits = 'ZERO ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE'.split()
letter = str(digits.index(base)) if base in digits else None
if letter is None:
return None
return (font + '{' + letter + '}') if font else letter
def join(parts):
r"""Concatenate LaTeX bits, keeping a space where one command would
otherwise swallow the next letter: \alpha + b is \alpha b, never
\alphab. Doing it here, at the seam, is the only place where the two
tokens are still distinguishable."""
out = ''
for p in parts:
if out and re.search(r'\\[A-Za-z]+$', out) and re.match(r'[A-Za-z]', p):
out += ' '
out += p
return out
VARIANT = {'script': r'\mathcal', 'bold': r'\mathbf', 'fraktur': r'\mathfrak',
'double-struck': r'\mathbb', 'sans-serif': r'\mathsf',
'monospace': r'\mathtt', 'bold-italic': r'\boldsymbol',
'normal': r'\mathrm'}
def sym(text):
"""A run of characters as LaTeX: Greek stays Greek, operators become
their commands, styled letters lose the styling."""
out = []
for ch in text:
if ch in SYMBOL:
out.append(SYMBOL[ch])
continue
name = unicodedata.name(ch, '')
if name.startswith('GREEK ') and 'LETTER' in name:
base = name.split('LETTER ')[-1].lower()
if base in GREEK:
out.append('\\' + (base.capitalize() if 'CAPITAL' in name
else base))
continue
got = unstyle(ch)
out.append(got if got is not None else ch)
return join(out)
def brace(s):
"""Braces only where TeX needs them: x_{ab} but x_1."""
s = s.strip()
if len(s) == 1 or re.fullmatch(r'\\[A-Za-z]+', s):
return s
return '{' + s + '}'
def mrow(kids):
return join([mathml(k) for k in kids])
def mkids(node):
return [k for k in node.kids
if not (isinstance(k, str) and not k.strip())]
def mathml(node):
"""One MathML node as LaTeX."""
if isinstance(node, str):
return sym(node.strip())
t = node.tag.split(':')[-1]
kids = mkids(node)
if t in ('math', 'semantics', 'mrow', 'mstyle', 'mpadded', 'mtd',
'menclose', 'msrow', 'mscarries'):
return mrow(kids)
if t in ('annotation', 'annotation-xml', 'mphantom'):
return ''
if t in ('mi', 'mn', 'mtext', 'ms', 'mo'):
text = raw_text(node).strip()
if t == 'mtext':
return r'\text{%s}' % text if text else r'\ '
var = VARIANT.get((node.attrs.get('mathvariant') or '').lower())
if var and text:
return r'%s{%s}' % (var, text)
got = sym(text)
if t == 'mi' and len(text) > 1 and re.fullmatch(r'[A-Za-z]+', text):
return r'\%s' % text if text in (
'sin cos tan log ln exp lim max min sup inf det dim ker '
'deg gcd arg').split() else r'\mathrm{%s}' % text
return got
if t == 'mspace':
return r'\,'
if t in ('msub', 'msup', 'msubsup', 'munder', 'mover', 'munderover'):
return script(t, kids)
if t == 'mfrac':
if len(kids) == 2:
a, b = mathml(kids[0]), mathml(kids[1])
if node.attrs.get('linethickness') in ('0', '0pt'):
return r'\binom{%s}{%s}' % (a, b)
return r'\frac{%s}{%s}' % (a, b)
if t == 'msqrt':
return r'\sqrt{%s}' % mrow(kids)
if t == 'mroot' and len(kids) == 2:
return r'\sqrt[%s]{%s}' % (mathml(kids[1]), mathml(kids[0]))
if t == 'mfenced':
o = node.attrs.get('open', '(')
c = node.attrs.get('close', ')')
return r'\left%s %s \right%s' % (sym(o), mrow(kids), sym(c))
if t == 'mtable':
rows = [' & '.join(mathml(c) for c in mkids(r))
for r in kids if r.tag.split(':')[-1] in ('mtr', 'mlabeledtr')]
return r'\begin{matrix} %s \end{matrix}' % r' \\ '.join(rows)
if t == 'mtr':
return ' & '.join(mathml(c) for c in mkids(node))
return mrow(kids)
def script(t, kids):
"""Sub- and superscripts, and the same thing written under and over:
a limit under a sum is _ and ^ again, an accent over a letter is not."""
if not kids:
return ''
base = mathml(kids[0])
rest = [mathml(k) for k in kids[1:]]
if t in ('munder', 'mover', 'munderover') and base not in BIG:
if t == 'mover' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\overset{%s}{%s}' % (rest[0], base)
if t == 'munder' and len(rest) == 1:
acc = ACCENT.get(raw_text(kids[1]).strip())
if acc:
return r'%s{%s}' % (acc, base)
return r'\underset{%s}{%s}' % (rest[0], base)
if t in ('msub', 'munder'):
return '%s_%s' % (base, brace(rest[0]))
if t in ('msup', 'mover'):
return '%s^%s' % (base, brace(rest[0]))
if len(rest) == 2: # msubsup, munderover
return '%s_%s^%s' % (base, brace(rest[0]), brace(rest[1]))
return base + ''.join(rest)
def mathml_to_tex(node):
return re.sub(r'\s{2,}', ' ', mathml(node)).strip()
# ------------------------------------------------------------------- math
class Ctx:
"""Rendering state: counters for the report, plus the table flag,
which is the one place where a | inside maths has to be neutralised."""
def __init__(self, wiki=True):
self.math = 0
self.rebuilt = 0
self.lost = 0
self.in_table = False
self.wiki = wiki # wikitext out; -m/--md for Markdown
def is_annotation(n):
return (n.tag == 'annotation'
and n.attrs.get('encoding') == 'application/x-tex')
def latex_of(node):
"""The LaTeX a formula was built from, if the page kept it."""
ann = node if is_annotation(node) else find(node, is_annotation)
if ann is not None:
return raw_text(ann).strip()
for key in ('data-latex', 'data-tex', 'alt'): # other renderers
for probe in (node, find(node, lambda n: key in n.attrs)):
if probe is not None and probe.attrs.get(key):
return probe.attrs[key].strip()
tex = find(node, lambda n: n.tag == 'script'
and 'math/tex' in (n.attrs.get('type') or ''))
if tex is not None:
return raw_text(tex).strip()
return None
def visible_math(node):
"""Last resort: the glyphs, and only one copy of them. A formula is
on the page twice - the rendering and the MathML behind it - and
taking the subtree wholesale is what doubles it.
This is what a selection begun in the middle of a formula leaves
behind: the browser copies the rendered half and drops the MathML
sibling that held the annotation, so there is no LaTeX to be had.
The glyphs still carry the formula, and sym() puts the symbols back
in command form, so the result is worth delimiting as maths."""
for probe in ('katex-mathml', 'katex-html'):
got = find(node, lambda n, p=probe: has_class(n, p))
if got is not None:
return sym(zapzero(flat_text(got)))
vis = find(node, lambda n: n.tag in ('mjx-math', 'mjx-container'))
return sym(zapzero(flat_text(vis if vis is not None else node)))
def zapzero(text):
"""KaTeX pads its layout with zero-width spaces; they are invisible
in the browser and litter in a text file."""
return re.sub('[\xad]', '', text)
def math_md(node, ctx, display=False):
tex = latex_of(node)
ctx.math += 1
if tex is None:
mml = node if node.tag == 'math' else find(node,
lambda n: n.tag == 'math')
if mml is not None:
tex = mathml_to_tex(mml)
if tex:
ctx.rebuilt += 1
if not tex:
ctx.lost += 1
tex = visible_math(node)
if not tex:
return ''
tex = re.sub(r'\s+', ' ', tex.replace('\xa0', ' ')).strip()
if ctx.in_table:
tex = tex.replace('|', r'\vert ') # a | would end the cell
d = '$$' if display else '$'
pad = ' ' if display else ''
return d + pad + tex + pad + d
def is_math(node):
# katex-mathml and a bare annotation are here for the same reason:
# a selection started inside a formula arrives without its outer
# wrapper, and what is left must still be read as maths rather than
# spilled into the prose as loose text.
return (has_class(node, 'katex') or has_class(node, 'katex-display')
or has_class(node, 'katex-mathml') or is_annotation(node)
or node.tag in ('math', 'mjx-container', 'mjx-assistive-mml')
or has_class(node, 'MathJax') or has_class(node, 'MathJax_Preview'))
# ----------------------------------------------------------------- inline
MARKS = {'b': '**', 'strong': '**', 'i': '*', 'em': '*',
's': '~~', 'del': '~~', 'strike': '~~'}
# The same set in wikitext. Apostrophe markup is quantity-sensitive:
# ''' is bold, '' italic, and five in a row is both - which is why the
# two are never nested by hand here, only ever wrapped one at a time.
WIKI_MARKS = {'b': "'''", 'strong': "'''", 'i': "''", 'em': "''",
's': None, 'del': None, 'strike': None}
def marked(inner, mark):
"""Put the markers against the text, not against its spaces:
' word ' -> ' **word** ', and nothing at all for empty runs.
mark is None for something wikitext has no markup for (struck-out
text): the HTML tag is kept, which MediaWiki renders happily."""
m = re.match(r'^(\s*)(.*?)(\s*)$', inner, re.S)
lead, core, trail = m.groups()
if not core:
return inner
if mark is None:
return lead + '<s>' + core + '</s>' + trail
return lead + mark + core + mark + trail
def code_span(inner):
inner = inner.replace('\n', ' ')
n = max((len(r) for r in re.findall(r'`+', inner)), default=0)
fence = '`' * (n + 1)
pad = ' ' if inner[:1] == '`' or inner[-1:] == '`' else ''
return fence + pad + inner + pad + fence
def inline(node, ctx):
if isinstance(node, str):
return re.sub(r'[ \t\r\n]+', ' ', node.replace('\xa0', ' '))
tag = node.tag
if tag in SKIP:
return ''
if is_math(node):
return math_md(node, ctx, display=has_class(node, 'katex-display'))
if tag == 'br':
return '\n'
if tag == 'img':
alt = node.attrs.get('alt', '').strip()
src = node.attrs.get('src', '')
if src[:4] != 'http':
return '![%s]' % alt if alt and not ctx.wiki else ''
# a remote image cannot be shown by MediaWiki, so it becomes a
# link rather than a red File: that will never resolve
return '[%s %s]' % (src, alt or 'image') if ctx.wiki else \
'' % (alt, src)
if tag == 'wbr':
return ''
inner = ''.join(inline(k, ctx) for k in node.kids)
if tag in MARKS:
return marked(inner, (WIKI_MARKS if ctx.wiki else MARKS)[tag])
if tag in ('sup', 'sub'):
# Markdown has no notation for these; the tag itself is understood
# both by MediaWiki and by every Markdown renderer worth the name
return '<%s>%s</%s>' % (tag, inner, tag) if inner.strip() else inner
if tag in ('code', 'tt', 'kbd', 'samp'):
if not inner.strip():
return inner
return ('<code>%s</code>' % inner.replace('\n', ' ') if ctx.wiki
else code_span(inner))
if tag == 'a':
href = node.attrs.get('href', '')
text = inner.strip()
if not href or href[:1] == '#' or href[:11] == 'javascript:':
return inner
if ctx.wiki:
# one bracket pair, space between target and label - and a
# bare url is left bare, MediaWiki links it by itself
if text in (href, href.rstrip('/')):
return href
return '[%s %s]' % (href, text) if text else ''
if text in (href, href.rstrip('/')):
return '<%s>' % href
return '[%s](%s)' % (text, href) if text else ''
return inner
# ------------------------------------------------------------------ block
def indent(text, pad, first=None):
lines = text.split('\n')
head = (first if first is not None else pad) + lines[0]
return '\n'.join([head] + [(pad + l if l.strip() else '') for l in lines[1:]])
def join_item(li, ctx):
"""The blocks of one <li>. A sub-list hangs directly under its item
(no blank line): that is what keeps it one list rather than two."""
body = ''
for part in blocks(li.kids, ctx):
if not body:
body = part
elif re.match(r'(?:[-*+]|\d+[.)]) ', part):
body += '\n' + part
else:
body += '\n\n' + part
return body
def list_wiki(node, ctx, prefix=''):
"""Wikitext nests a list by lengthening its marker - *, **, *# -
with no indentation and no blank line anywhere inside. A list item
is therefore one line, and a block that cannot be one line (a code
block, a table) is emitted after the item: MediaWiki closes the list
there, which is ugly but keeps the content, and there is no way
round it short of a template."""
mark = prefix + ('#' if node.tag == 'ol' else '*')
try:
start = int(node.attrs.get('start', 1))
except ValueError:
start = 1
out = []
for li in node.kids:
if isinstance(li, str) or li.tag != 'li':
continue
line, after = [], []
for k in li.kids:
if not isinstance(k, str) and k.tag in ('ul', 'ol'):
after.append(('list', k))
elif isinstance(k, str) or k.tag not in BLOCK or is_math(k):
line.append(inline(k, ctx))
else:
got = block(k, ctx)
if not got.strip():
continue
if '\n' in got.strip():
after.append(('block', got))
else:
line.append(got)
body = re.sub(r'\s+', ' ', ''.join(line)).strip()
if body:
out.append(mark + ' ' + body)
for kind, thing in after:
out.append(list_wiki(thing, ctx, mark) if kind == 'list' else thing)
body = '\n'.join(x for x in out if x.strip())
if node.tag == 'ol' and start > 1 and not prefix and body:
# # restarts at 1, always: an answer resuming at 3 has to say so
# in HTML - the same fix Module:AI carries for the same reason
items = ''.join('<li>%s</li>' % l[len(mark) + 1:]
for l in body.split('\n') if l.startswith(mark + ' '))
if items:
return '<ol start="%d">%s</ol>' % (start, items)
return body
def list_md(node, ctx):
ordered = node.tag == 'ol'
try:
n = int(node.attrs.get('start', 1))
except ValueError:
n = 1
out = []
for li in node.kids:
if isinstance(li, str) or li.tag != 'li':
continue
body = join_item(li, ctx)
if not body.strip():
continue
mark = ('%d. ' % n) if ordered else '- '
n += 1
out.append(indent(body, ' ' * len(mark), first=mark))
return '\n'.join(out)
def table_md(node, ctx):
was, ctx.in_table = ctx.in_table, True
rows = []
aligns = {}
for tr in [n for n in walk(node) if not isinstance(n, str)
and n.tag == 'tr']:
cells = []
for c in tr.kids:
if isinstance(c, str) or c.tag not in ('td', 'th'):
continue
style = c.attrs.get('style', '') or ''
m = re.search(r'text-align\s*:\s*(left|right|center)', style, re.I)
a = (m.group(1) if m else c.attrs.get('align', '')).lower()
if a:
aligns.setdefault(len(cells), a)
txt = re.sub(r'\s+', ' ', inline(c, ctx)).strip()
cells.append(txt.replace('\n', ' '))
if cells:
rows.append(cells)
ctx.in_table = was
if not rows:
return ''
ncol = max(len(r) for r in rows)
sep = ['|'.join({'center': ':---:', 'right': '---:', 'left': ':---'}
.get(aligns.get(i), '---') for i in range(ncol))]
def line(cells):
v = [c.replace('|', r'\|') for c in cells] # math | is \vert by now
return '| ' + ' | '.join(v + [''] * (ncol - len(v))) + ' |'
return '\n'.join([line(rows[0]), '|' + sep[0] + '|']
+ [line(r) for r in rows[1:]])
def table_wiki(node, ctx):
"""The house wikitable: full width, one cell per line, header row
with !. A | in the content becomes | - MediaWiki cuts a cell on
it before anything else runs - which is [[mdtable2wiki]]'s whole
subject; the same rule, applied here at the source."""
was, ctx.in_table = ctx.in_table, True
rows, heads = [], []
for tr in [n for n in walk(node) if not isinstance(n, str)
and n.tag == 'tr']:
cells, head = [], []
for c in tr.kids:
if isinstance(c, str) or c.tag not in ('td', 'th'):
continue
txt = re.sub(r'\s+', ' ', inline(c, ctx)).strip()
cells.append(txt.replace('|', '|'))
head.append(c.tag == 'th')
if cells:
rows.append(cells)
heads.append(all(head))
ctx.in_table = was
if not rows:
return ''
out = ['{| class="wikitable" style="width:100%;"']
for i, cells in enumerate(rows):
if i:
out.append('|-')
lead = '!' if heads[i] else '|'
out += ['%s %s' % (lead, c) for c in cells]
out.append('|}')
return '\n'.join(out)
def walk(node):
for k in node.kids:
yield k
if not isinstance(k, str):
for g in walk(k):
yield g
def pre_wiki(node):
src, lang = pre_parts(node)
if not lang:
return '<pre>\n%s\n</pre>' % src
return '<syntaxhighlight lang="%s">\n%s\n
def pre_md(node):
also holds the language
label and the Copy button; the holds only the code."""
src, lang = pre_parts(node)
n = max((len(r) for r in re.findall(r'^\s*(`{3,})', src, re.M)), default=2)
fence = '`' * max(3, n + 1)
return '%s%s\n%s\n%s' % (fence, lang, src, fence)
def pre_parts(node):
code = find(node, lambda n: n.tag == 'code')
src = raw_text(code if code is not None else node)
src = src.replace('\xa0', ' ').replace('\r\n', '\n').strip('\n')
lang =
for probe in (code, node):
if probe is None:
continue
for c in (probe.attrs.get('class') or ).split():
m = re.match(r'(?:language|lang|highlight|hljs)[-_](\w+)$', c)
if m:
lang = m.group(1)
break
lang = lang or probe.attrs.get('data-language', )
if lang:
break
return src, lang
def blocks(kids, ctx):
"""Children of a container, as a list of block-level chunks. Text and
inline elements between two blocks make a paragraph of their own."""
out = []
run = []
def flush():
if run:
para = re.sub(r'[ \t]*\n[ \t]*', '\n', .join(run)).strip()
if para:
out.append(para)
run.clear()
for k in kids:
if isinstance(k, str) or k.tag not in BLOCK or is_math(k):
run.append(inline(k, ctx))
continue
flush()
got = block(k, ctx)
if got.strip():
out.append(got)
flush()
return out
def block(node, ctx):
tag = node.tag
if tag in SKIP:
return
if tag == 'hr':
return '----' if ctx.wiki else '---'
if tag in HEADING:
text = re.sub(r'\s+', ' ', inline(node, ctx)).strip()
if not text:
return
if ctx.wiki:
bars = '=' * (len(HEADING[tag]) + 1) # h1 -> ==, the page
return '%s %s %s' % (bars, text, bars) # title being the h1
return '%s %s' % (HEADING[tag], text)
if tag == 'pre':
return pre_wiki(node) if ctx.wiki else pre_md(node)
if tag == 'table':
return table_wiki(node, ctx) if ctx.wiki else table_md(node, ctx)
if tag in ('ul', 'ol'):
return list_wiki(node, ctx) if ctx.wiki else list_md(node, ctx)
if tag == 'blockquote':
body = '\n\n'.join(blocks(node.kids, ctx))
if ctx.wiki:
return '\n%s\n
' % body
return '\n'.join('>' + (' ' + l if l.strip() else )
for l in body.split('\n'))
if tag in ('dt',):
return ("%s" if ctx.wiki else '**%s**') % flat_text(node)
# p, div, li, section, ... : just their contents
return '\n\n'.join(blocks(node.kids, ctx))
def to_markdown(html, ctx):
"""HTML in, wikitext out - or Markdown, if the ctx says so."""
md = '\n\n'.join(blocks(parse(html).kids, ctx))
md = re.sub(r'[ \t]+$', , md, flags=re.M)
md = re.sub(r'\n{3,}', '\n\n', md)
return md.strip() + '\n'
# -------------------------------------------------------------- clipboard
def _run(cmd):
if not shutil.which(cmd[0]):
return None
try:
return subprocess.run(cmd, capture_output=True, check=True
).stdout.decode('utf-8', 'replace')
except subprocess.CalledProcessError:
return None
def ai_escape(md):
"""Every literal pipe as |, so the Markdown can sit in .
A template parameter is cut on |, so a table row - or a single | in
prose or in a code block - loses the rest of the answer. | is
expanded back to | by the preprocessor BEFORE Module:AI reads the
string, so the module still sees an ordinary Markdown table and the
code block still shows a pipe. Same trick as mdtable2wiki --ai.
"""
return md.replace('|', '|')
def clip_read():
"""The html flavour if the copy had one - that is the whole point."""
for cmd in (['wl-paste', '-t', 'text/html'],
['xclip', '-selection', 'clipboard', '-t', 'text/html', '-o']):
got = _run(cmd)
if got and got.strip():
return got, True
for cmd in (['wl-paste', '-n'], ['xclip', '-selection', 'clipboard', '-o'],
['xsel', '-ob']):
got = _run(cmd)
if got is not None:
return got, False
sys.exit('html2md: no clipboard tool (wl-clipboard / xclip / xsel)')
def clip_write(text):
for cmd in (['wl-copy'], ['xclip', '-selection', 'clipboard'],
['xsel', '-ib']):
if shutil.which(cmd[0]):
try:
subprocess.run(cmd, input=text.encode(), check=True)
return cmd[0]
except subprocess.CalledProcessError:
pass
return None
# ------------------------------------------------------------------ watch
# The hotkey works, but a hotkey is a thing to remember. Watching the
# clipboard removes the step altogether: copy in the browser, paste in
# Emacs, and what arrives is already Markdown.
#
# Nothing is lost by doing it: the converted Markdown goes on as
# text/plain - the flavour Emacs, a textarea and the wiki edit box all
# read - while the original text/html is put back untouched, so a rich
# paste into a mail or a word processor still gets its formatting.
def watch(o):
from PyQt5 import QtCore, QtGui, QtWidgets
app = QtWidgets.QApplication(sys.argv[:1])
app.setQuitOnLastWindowClosed(False)
board = app.clipboard()
seen = [None, None, 0.0, None] # plain, html, when, selection html
def look():
# Is this our own write coming back round? Ask who owns the
# clipboard, never whether the HTML looks familiar: copying the
# same passage twice hands us byte-identical markup, and a
# content test then mistakes a fresh copy for its own echo and
# sits there doing nothing - which is exactly what it did.
if board.ownsClipboard():
return
data = board.mimeData()
if data is None:
return
html = data.html() if data.hasHtml() else
if not html:
# Nothing to do, and we take no ownership - so without a
# memory of it the tick would say so once a second forever.
fingerprint = hash(data.text() or )
if fingerprint != seen[0]:
seen[0] = fingerprint
keep(data.text() or , plain=True)
log('no html on the clipboard, left alone')
return
# Identical HTML is NOT a reason to skip: copying the same
# passage twice is a perfectly ordinary thing to do. Only a
# repeat within a moment is suppressed, and only so that a failed
# hand-over cannot turn into a conversion loop.
if html == seen[1] and time.monotonic() - seen[2] < 2:
return
seen[1], seen[2] = html, time.monotonic()
keep(html)
ctx = Ctx(wiki=not o.md)
md = to_markdown(html, ctx)
if not md.strip():
log('html converted to nothing, left alone')
return
fresh = QtCore.QMimeData()
for fmt in data.formats(): # keep images, source URLs, everything
if fmt.startswith('text/plain') or fmt in ('UTF8_STRING',
'STRING', 'TEXT'):
continue
fresh.setData(fmt, data.data(fmt))
fresh.setText(md)
# Qt offers text/plain, UTF8_STRING, STRING and TEXT but not
# text/plain;charset=utf-8, which the browser does offer and
# which GTK and VTE applications ask for by name: without it
# they get an empty answer where a paste should have been.
fresh.setData('text/plain;charset=utf-8', md.encode())
board.setMimeData(fresh)
# A terminal pastes with the middle button, and that is the
# PRIMARY selection, not the clipboard - it would otherwise hand
# back the browser's own glyphs, the formula three times over.
if o.primary:
board.setText(md, QtGui.QClipboard.Selection)
note = '%d formula%s%s' % (ctx.math, 's' if ctx.math != 1 else ,
', %d rebuilt from MathML' % ctx.rebuilt
if ctx.rebuilt else )
log('converted, %s' % note)
if o.quiet:
return
if o.notify and ctx.math:
report(o, 'converted', note)
elif not o.notify:
sys.stderr.write('html2md: converted, %s\n' % note)
def look_selection():
# The middle button pastes what was merely *selected*, with no
# copy at all. Converting that means owning PRIMARY, and the
# window I am reading may drop its highlight when it loses it -
# hence --no-select for anyone who finds that worse than the
# tripled formula it cures.
if board.ownsSelection():
return
data = board.mimeData(QtGui.QClipboard.Selection)
if data is None or not data.hasHtml():
return
html = data.html()
if html == seen[3] or 'katex' not in html and '<math' not in html \
and 'mjx-' not in html:
return # only formulae are worth the theft
seen[3] = html
ctx = Ctx(wiki=not o.md)
md = to_markdown(html, ctx)
if md.strip() and ctx.math:
board.setText(md, QtGui.QClipboard.Selection)
log('selection converted, %d formulas' % ctx.math)
def guarded_selection():
try:
look_selection()
except Exception as exc:
log('selection error: %r' % (exc,))
def guarded():
# An exception escaping a slot is fatal in PyQt5; one malformed
# copy would take the watcher down and every paste after it.
try:
look()
except Exception as exc:
log('error: %r' % (exc,))
board.dataChanged.connect(guarded)
if o.select:
board.selectionChanged.connect(guarded_selection)
tick = QtCore.QTimer() # in case dataChanged ever stops
tick.timeout.connect(guarded) # arriving: one round trip a second
tick.start(1000)
log('watching (pid %d)' % os.getpid())
if not o.quiet:
sys.stderr.write('html2md: watching the clipboard (Ctrl-C to stop)\n')
QtCore.QTimer.singleShot(0, guarded) # whatever is there now
sys.exit(app.exec_())
def log(line):
"""A record that outlives the session: when the watcher looks stuck,
this is what says whether it saw the copy at all."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = os.path.join(os.path.dirname(CACHE), 'watch.log')
if os.path.exists(path) and os.path.getsize(path) > 200000:
os.replace(path, path + '.1')
with open(path, 'a', encoding='utf-8') as f:
f.write('%s %s\n' % (time.strftime('%Y-%m-%d %H:%M:%S'), line))
except OSError:
pass
def keep(src, plain=False):
"""Stash what was read. Converting the clipboard destroys the html it
came from, and when the output is wrong that is exactly what one
needs to look at: html2md -l runs it again, html2md ~/.cache/... too."""
try:
os.makedirs(os.path.dirname(CACHE), exist_ok=True)
path = CACHE[:-5] + '.txt' if plain else CACHE
with open(path, 'w', encoding='utf-8') as f:
f.write(src)
except OSError:
pass
def report(o, title, body):
"""On a hotkey there is no terminal to write to, so say it in a toast."""
if o.quiet or not body:
return
if o.notify and shutil.which('notify-send'):
# Popen, not run: notify-send lives as long as the toast is on
# screen, and waiting for it would freeze the watcher's event
# loop - it would sit out the next copy entirely.
subprocess.Popen(['notify-send', '-a', 'html2md', '-t', '4000',
'html2md: ' + title, body],
start_new_session=True)
else:
sys.stderr.write('html2md: ' + body + '\n')
def looks_like_html(text):
return re.search(r'<(p|div|span|table|li|h[1-6]|pre|br|body)[\s/>]',
text, re.I) is not None
# --------------------------------------------------------------------- cli
def main():
p = argparse.ArgumentParser(
prog='html2md',
description='Rendered HTML on the clipboard -> Markdown, '
'maths back in LaTeX.')
p.add_argument('file', nargs='?', help='HTML file (default: clipboard)')
p.add_argument('-w', '--watch', action='store_true',
help='sit on the clipboard and convert every copy')
p.add_argument('--no-primary', dest='primary', action='store_false',
help='watch: do not put the Markdown on PRIMARY on a copy')
p.add_argument('--no-select', dest='select', action='store_false',
help='watch: do not convert a selection that was never copied')
p.add_argument('-l', '--last', action='store_true',
help='convert the last clipboard again (%s)' % CACHE)
p.add_argument('-x', '--clipboard', action='store_true',
help='read the clipboard even if stdin is a pipe')
p.add_argument('-c', '--copy', action='store_true',
help='put the Markdown back on the clipboard')
p.add_argument('-m', '--md', action='store_true',
help='Markdown out instead of wikitext (what '
'wants: Module:AI converts the Markdown itself)')
p.add_argument('--wiki', action='store_true',
help='wikitext out - the default, named for symmetry')
p.add_argument('-a', '--ai', action='store_true',
help='write every literal | as |, so the whole '
'answer - tables and code included - drops into '
'the a= of Template:AI')
p.add_argument('-q', '--quiet', action='store_true',
help='no report on stderr')
p.add_argument('-n', '--notify', action='store_true',
help='report in a desktop notification (for a hotkey)')
p.add_argument('-V', '--version', action='version',
version='html2md ' + VERSION)
o = p.parse_args()
if o.watch:
watch(o)
from_clip = False
if o.last:
o.file = CACHE
if o.file:
with open(o.file, encoding='utf-8', errors='replace') as f:
src = f.read()
elif o.clipboard or sys.stdin.isatty():
src, rich = clip_read()
from_clip = True
keep(src) # the conversion overwrites the clipboard
if not rich and not looks_like_html(src):
# plain text was copied: nothing to convert, pass it through
if o.copy:
clip_write(src)
sys.stdout.write(src if src.endswith('\n') else src + '\n')
report(o, 'nothing to do', 'the clipboard has no text/html '
'(plain text left as it was)')
return
else:
src = sys.stdin.read()
ctx = Ctx(wiki=not o.md)
md = to_markdown(src, ctx)
pipes = md.count('|')
if o.ai:
md = ai_escape(md)
if o.copy:
if clip_write(md) is None:
sys.exit('html2md: no clipboard tool to write with')
if not (o.copy and from_clip and sys.stdout.isatty()):
sys.stdout.write(md)
say = ['wikitext' if ctx.wiki else 'Markdown']
if ctx.math:
say.append('%d formula%s' % (ctx.math, 's' if ctx.math > 1 else ))
if ctx.rebuilt:
say.append('%d rebuilt from MathML (no annotation)' % ctx.rebuilt)
if ctx.lost:
say.append('%d with no LaTeX kept by the page (glyphs only)'
% ctx.lost)
if o.ai:
if pipes:
say.append('%d pipe%s written |' % (pipes, 's' * (pipes > 1)))
elif pipes and ctx.wiki:
# on a page those pipes are the table itself; only a template
# parameter minds them
say.append('%d pipe%s - fine on a page, -a to put this in '
% (pipes, 's' * (pipes > 1)))
elif pipes:
say.append('%d literal | - breaks , so -a or src='
% pipes)
if o.copy:
say.append('on the clipboard')
report(o, '%d line%s' % (md.count('\n'),
if md.count('\n') == 1 else 's'),
'; '.join(say))
if __name__ == '__main__':
main()
</syntaxhighlight>
