VIP
VIP are Very Important Papers, refering to the list of papers cited in a given work, or one branch in the tree of citations.
They are easily extracted from a $\mathrm{\TeX}$ with the VIPfrom script. The other way round, the tree of the works that cite a paper is grown by vip.
VIPfrom
VIPfrom is a python script that lists the BibTeX keys cited by a TeX file, as they are found in my library: sci.bib, Books.bib and arXiv.bib, all three read by default. Where its sibling bibpapersfrom cuts the database by author, this one cuts it by paper: hand it a manuscript and it tells you which of my references that manuscript actually calls, in the form my wiki documents.
Written with the help of Claude Opus 5 on 29 July (2026). Version 1.1.
Usage
laussy@azag:~/publi/20/26/3--ghost$ VIPfrom -cs ghost.tex
* {{pittman95a}}
* {{strekalov95a}}
* {{bennink02a}}
* {{gatti04a}}
That is the whole point of the thing: -cs gives the list ready to be dropped into a wiki page, one bulleted transclusion per line. -c gives the curly brackets without the star, and the bare invocation gives the naked keys, which is what one pipes further:
laussy@azag:~/publi/20/26/3--ghost$ VIPfrom ghost.tex
pittman95a
strekalov95a
bennink02a
gatti04a
laussy@azag:~/publi/20/26/3--ghost$ VIPfrom ghost.tex | keys2wiki
The last line writes the wiki page of every paper cited by the manuscript, which is the reason the script was written in the first place. With no argument at all it takes every *.tex of the current directory, so VIPfrom -cs in a paper's folder is usually all that is needed. It reads stdin when piped, and .aux files work too, since LaTeX writes \citation{...} there.
Order
The keys come out chronologically by default—by year and month as given in the bib, undated entries last—because that is the order in which a bibliography of a subject wants to be read. -o tex keeps instead the order of first citation in the manuscript, i.e. the order in which the paper introduces its references, and -o alpha sorts by key. Repeated citations are collapsed to one entry, across all the files read.
The three bases
Papers are in sci.bib, but books are in Books.bib and preprints in arXiv.bib, and a manuscript of mine cites all three sorts. All three are therefore read by default, in that order, and a key found in several bases is taken from the first that has it. So the whole library answers, and nothing has to be spelled out:
laussy@azag:~/publi/20/25/1--polariton-BEC$ VIPfrom --count poco.tex > /dev/null
261 key(s) listed, 261 cited, 0 not in the bib
-b overrides the set, to restrict to one base or to bring in a paper's local copy. Restricting that same manuscript to sci.bib alone shows what the other two carry—three books and a preprint of ours:
laussy@azag:~/publi/20/25/1--polariton-BEC$ VIPfrom --count -b ~/bib/sci.bib poco.tex > /dev/null
VIPfrom: 4 cited key(s) not in the bib: kittel_book04a, weisbuch_book95a, arXiv_casalengua24a, meystre_book01a
257 key(s) listed, 261 cited, 4 not in the bib
Keys that no base knows are never silently dropped: they are reported on stderr, so that the list on stdout stays clean when it is piped. A base named explicitly with -b that cannot be read is a fatal error, whereas one of the three default ones merely warns and is skipped, so the script still works on a machine that has only sci.bib.
-m lists those orphans instead of the good keys—handy to find what still has to be entered in the base—and -a lists everything cited, in the bib or not. A key that differs from a real one only by its case is pointed at explicitly (did you mean), that being the usual typo.
What counts as a citation
Everything that ends in cite: \cite, \citep, \citet, \citeauthor, \nocite, and the biblatex family \parencite, \textcite, \autocite, including the multi-key forms \cites{a}{b}. Optional arguments are stepped over, so \citep[see][p.~3]{gatti04a} is read correctly, and \nocite{*} pulls in the whole base. TeX comments are ignored, which matters: the commented-out citations of a draft would otherwise pollute the list. \input, \include and \subfile are followed—-n if one does not want that—so a manuscript split in pieces needs only its master file.
Two hazards were worth the care. A group of braces following a citation is not read as keys, so \cite{gatti04a} {\it and then some} does not invent a reference; and a command such as \excitation{...}, which contains no cite at all but looks as if it might, is left alone.
Options
laussy@azag:~/$ VIPfrom -h
usage: VIPfrom [-h] [-b BIB] [-c] [-cs] [-o {chrono,tex,alpha}] [-m | -a] [-n]
[-q] [--count] [-V]
[TEXFILE ...]
List the BibTeX keys cited by a TeX file.
positional arguments:
TEXFILE TeX (or .aux) file(s); default: *.tex here
options:
-h, --help show this help message and exit
-b BIB, --bib BIB bib file(s), comma-separated (default:
~/bib/sci.bib,~/bib/Books.bib,~/bib/arXiv.bib)
-c, --curly surround each key with {{ }}
-cs, --curly-star like --curly, but also prefix each key with *
-o {chrono,tex,alpha}, --order {chrono,tex,alpha}
output order (default: chrono)
-m, --missing list instead the cited keys absent from the bib
-a, --all also list cited keys absent from the bib
-n, --no-recurse do not follow \input, \include and \subfile
-q, --quiet do not warn about keys absent from the bib
--count print a total count to stderr
-V, --version show program's version number and exit
The BibTeX parser is the one of bibpapersfrom, brace-matching and dependency-free, so @string, @comment and nested braces do not confuse it, and the two scripts always agree on what a key is.
Version
- v1°0: on 29 July (2026), cited keys from TeX (or aux) files, chronological by default, recursing through \input.
- v1°1: the same day, the whole library—sci.bib, Books.bib, arXiv.bib—read by default instead of sci.bib alone; an absent default base warns rather than aborts.
vip
vip is a python script that goes the other way. Where VIPfrom reads the references of a manuscript—what it stands on—vip follows a paper forward in time: it grows the tree of the most important works citing it, then of the most important works citing those, and so on, a few generations down. It is the branch in the tree of citations of the definition above, cut down to what matters.
Written with the help of Claude Opus 5.5 on 15 October (2026). Version 1.1.0. It is public, on GitHub, under the MIT license, and is built on citing, whose functions it uses.
Usage
laussy@azag:~$ vip weisbuch92a -n 4,2 --topic polariton
[n] year cites /yr
[0] 1992 2561 75.6 weisbuch92a Weisbuch Observation of the coupled exciton-photon mode splitting in a semicon…
[1] 2019 1462 187.9 R ├─ friskkockum19a Frisk Kockum Ultrastrong coupling between light and matter
[2] 2023 270 84.7 R │ ├─ 10.1021/acs.chemrev.2c00855 Mandal Theoretical Advances in Polariton Chemistry and M…
[3] 2025 82 66.9 │ └─ 10.1021/acsphotonics.5c00353 Abajo Roadmap for Photonics with 2D Materials
[4] 2006 3245 161.3 ├─ kasprzak06a Kasprzak Bose–Einstein condensation of exciton polaritons
[5] 2017 944 104.0 │ ├─ 10.1038/s41566-017-0006-2 St-Jean Lasing in topological edge states of a one-dimensi…
[6] 2014 1189 100.7 │ ├─ 10.1038/nphoton.2014.304 Liu Strong light–matter coupling in two-dimensional atomic …
│ ├─ see [7] carusotto13a
│ └─ see [10] torma15a
[7] 2013 2072 151.8 ├─ carusotto13a Carusotto Quantum fluids of light
[8] 2018 1462 181.4 │ ├─ 10.1103/physrevx.8.031079 Gong Topological Phases of Non-Hermitian Systems
[9] 2018 634 72.2 │ ├─ 10.1039/c8sc01043a Ribeiro Polariton chemistry: controlling molecular dynamics with …
│ └─ see [5] 10.1038/s41566-017-0006-2
[10] 2014 1644 139.2 └─ torma15a Törmä Strong coupling between surface plasmon polaritons and emitters: a review
[11] 2019 444 57.0 R ├─ 10.1039/c8cs00193f Hertzog Strong light–matter interactions: a new direction within …
[12] 2020 307 51.8 ├─ basov20a Basov Polariton panorama
└─ see [9] 10.1039/c8sc01043a
From the paper that opened the field, Weisbuch et al. 1992, this keeps the four works on polaritons that cite it with the most citations per year, and two for each of them. A work of my bib files is named by its key, the others by their DOI; R marks a review, P a preprint, * a work sharing an author with the one above it. A work is shown once: when it turns up again among the best citers of another, a see [n] line points to it. Here Carusotto and Ciuti's Quantum fluids of light [7] descends from Weisbuch directly, and is also among the most read works citing Kasprzak et al. [4].
The same tree drawn by Graphviz, with -o tree.svg: one row per year, the tree in solid lines and the other citations between its works dashed, reviews in a double frame, the works of my bib files shaded. In the SVG, each box links to its DOI.
Truncation
For each work, vip asks OpenAlex for the works citing it—the 200 most cited, and the 200 most cited of the last five years, so that a recent work much read is not lost behind older ones with more citations—counts a preprint and its published version once, ranks them and keeps the best.
- -n 5 keeps five under each work; -n 10,5,3 keeps ten, then five under each of those, then three, and the depth is then the length of the list (otherwise -d, two by default).
- -r rate, the default, ranks by citations per year since publication, so that a work of 2020 cited 300 times stands with one of 2005 cited 1000 times; the age counts at least one year, so that last month's paper does not come first on two citations. -r recent asks what is read now (citations this year and the two before), -r cites what has been read most.
- --topic, --type, --since, --until and --min-cites filter every generation, --no-self leaves out the works that share an author with the one they cite, and --max bounds the whole tree (300 works).
Without --topic, a tree soon leaves its field through its most cited descendants. From Weisbuch, by way of Mak and Shan's review of 2D semiconductor photonics, it reaches surveys of transition-metal dichalcogenides in two generations; from the bundler, by way of Rainò's superfluorescence of perovskite superlattices, the chemistry of perovskite nanocrystals. That is a finding of its own—where a paper's influence went—but to stay in a field, name it.
Output
Besides the tree, -f wiki gives it as a nested list of {{key}}, ready for this wiki like the output of VIPfrom -cs (a DOI link for the works with no key); -f keys the keys, to pipe into keys2wiki; -f ids the DOIs of the works not yet in my bib files, for doi2bib; -f json everything, including every citation between the works of the tree; -f dot the source of the drawing, which -o draws when the file ends in .svg, .pdf or .png.
Cost and limits
The citations by which the works are ranked all come from OpenAlex, so that they compare with one another. But OpenAlex misses some citers—a reference it could not read—so when it lists every citer of a work (no more than --pool), vip also asks OpenCitations and Semantic Scholar, joins their lists to OpenAlex's as citing does, and adds the citers OpenAlex had not linked; the count of citers is then the whole of them. Khalid's Perfect single-photon sources (2024) has 13 citers in OpenAlex and 15 in all, and one of the two it had missed, the detector of Li et al. that surpasses 99% efficiency, comes third in its tree. Google Scholar (21) and Dimensions (19, the count shown on the Nature page) find more still, from theses, preprints and books, but give no list that a program may read. --openalex-only keeps to OpenAlex. OpenAlex gives each address 1000 credits a day. A citation query costs one, so a tree of thirty works takes ten to twenty, but a --topic query costs ten. The answers are kept a week in ~/.cache/vip, so that trying other cuts of the same tree costs next to nothing, and the summary line says how many credits are left. The bib files are read from ~/.viprc, or else from the ~/.citingrc of citing, and so is a Semantic Scholar key.
OpenAlex marks few works as reviews—14 of the 2547 citing Weisbuch, while Carusotto and Ciuti's review in Reviews of Modern Physics is an "article"—so --type review finds some reviews, not all. The first author of a work that is not in my bib files is the last word of the name OpenAlex gives: "Abajo" for García de Abajo.
Options
vip -- very important papers: from one paper, the tree of the most important
works that cite it, of the most important works that cite those, and so on.
Usage:
vip PAPER [options]
PAPER is a DOI, an arXiv id, an OpenAlex id (W...), or a key of the --bib
files, as for citing (https://github.com/laussy/citing, which vip needs).
How the tree grows: the works citing the paper are read from OpenAlex, ranked,
and the best N are kept; each of them is treated in the same way, down to the
given depth. A work is shown once, at its first place in the tree; when it is
again among the best citers of another work, a "see [n]" line says so. The
citations of each work, by which it is ranked, come from OpenAlex, so that
they can be compared with one another.
OpenAlex misses some citations (a reference it could not read). When it has
listed every citer of a work (no more than --pool of them), the citers that
OpenCitations and Semantic Scholar give and OpenAlex does not are added, as
citing gathers them, when OpenAlex has a record of them; the number of citers
reported, and the paper's own count, is then the whole of them. Google Scholar
and Dimensions (the count on a Nature page) find more still, from theses,
preprints and books, but give no list that a program may read.
Truncation:
-d, --depth D generations below the paper (default 2)
-n, --top N[,N2,...] works kept under each work; a list gives one number
per generation, and the depth is then its length
unless -d says otherwise (default 5)
-r, --rank BY rate: citations per year since publication (default)
recent: citations in this year and the two before
cites: all citations
--min-cites N leave out works cited fewer than N times
--since YEAR leave out works published before YEAR
--until YEAR leave out works published after YEAR
--type TYPES only these OpenAlex types, comma-separated (review,
article, preprint, book-chapter, ...). By default all
but erratum, paratext and retraction
--topic WORDS only works with WORDS in their title or abstract
(an OpenAlex search: it costs more of the budget)
--no-self leave out works sharing an author with the work
they are placed under
--max N stop when the tree holds N works (default 300)
--pool N citing works read per work before ranking: the N
most cited, plus, unless --rank cites, the N most
cited of the last five years (default 200)
--openalex-only no citers from OpenCitations and Semantic Scholar
Output:
-b, --bib FILES BibTeX file(s), comma-separated: works get their key
-f, --format FMT text (default), json, dot (Graphviz), wiki (nested
list of {{key}}), keys (bib keys, one per line), ids
(DOI or arXiv:ID of works with no key)
-o, --output FILE write there instead of stdout; a .svg, .pdf or .png
is drawn by Graphviz's dot from the dot format
--no-links no "see [n]" lines, and in the drawing only the
tree's own edges
-W, --width N cut text lines at N characters (default: the
terminal's width; lines are not cut when piped)
--fresh do not use the cache (~/.cache/vip, answers younger
than 7 days are reused)
--mailto EMAIL contact for the OpenAlex polite pool (or env
VIP_MAILTO); nothing is sent without it
--openalex-key K OpenAlex API key (or env OPENALEX_API_KEY)
--s2-key K Semantic Scholar API key (or env S2_API_KEY)
-q, --quiet no progress or summary on stderr
-V, --version the version
-h, --help this message
Text columns: [n] number in the tree, year, citations, the ranking figure
(per year, or recent), marks (R review, P preprint, * shares an author with
the work above it), then the bib key (or the DOI), first author and title.
Defaults: ~/.viprc (or the file named by env VIP_RC) holds options as on the
command line; --bib, --mailto and --openalex-key are also taken from
~/.citingrc when ~/.viprc does not give them, and so is --s2-key.
Version
- 1.0.0 (15 October (2026)): first version.
- 1.0.1 (15 October (2026)): a margin around the drawing, whose years this wiki's renderer cut off.
- 1.1.0 (24 October (2026)): the citers OpenAlex has not linked, from OpenCitations and Semantic Scholar; --openalex-only, --s2-key.
The file
#!/usr/bin/env python3
# _
# __ _(_)_ __
# \ \ / / | '_ \
# \ V /| | |_) |
# \_/ |_| .__/
# |_|
#
# F. P. Laussy -- https://github.com/laussy/vip
# v1.0.0 Thu 15 Oct 2026 - first version: the tree of the most important
# works citing a paper, generation after generation,
# from OpenAlex; text, JSON, Graphviz, wiki, keys.
# v1.0.1 Thu 15 Oct 2026 - a margin around the drawing: MediaWiki's SVG
# renderer, with a wider font, cut off the years.
# v1.1.0 Sat 24 Oct 2026 - the citers OpenAlex has not linked: those that
# OpenCitations and Semantic Scholar list are added
# (khalid24a: 13 in OpenAlex, 15 in all); the paper's
# own count is the whole of them. --openalex-only.
"""
vip -- very important papers: from one paper, the tree of the most important
works that cite it, of the most important works that cite those, and so on.
Usage:
vip PAPER [options]
PAPER is a DOI, an arXiv id, an OpenAlex id (W...), or a key of the --bib
files, as for citing (https://github.com/laussy/citing, which vip needs).
How the tree grows: the works citing the paper are read from OpenAlex, ranked,
and the best N are kept; each of them is treated in the same way, down to the
given depth. A work is shown once, at its first place in the tree; when it is
again among the best citers of another work, a "see [n]" line says so. The
citations of each work, by which it is ranked, come from OpenAlex, so that
they can be compared with one another.
OpenAlex misses some citations (a reference it could not read). When it has
listed every citer of a work (no more than --pool of them), the citers that
OpenCitations and Semantic Scholar give and OpenAlex does not are added, as
citing gathers them, when OpenAlex has a record of them; the number of citers
reported, and the paper's own count, is then the whole of them. Google Scholar
and Dimensions (the count on a Nature page) find more still, from theses,
preprints and books, but give no list that a program may read.
Truncation:
-d, --depth D generations below the paper (default 2)
-n, --top N[,N2,...] works kept under each work; a list gives one number
per generation, and the depth is then its length
unless -d says otherwise (default 5)
-r, --rank BY rate: citations per year since publication (default)
recent: citations in this year and the two before
cites: all citations
--min-cites N leave out works cited fewer than N times
--since YEAR leave out works published before YEAR
--until YEAR leave out works published after YEAR
--type TYPES only these OpenAlex types, comma-separated (review,
article, preprint, book-chapter, ...). By default all
but erratum, paratext and retraction
--topic WORDS only works with WORDS in their title or abstract
(an OpenAlex search: it costs more of the budget)
--no-self leave out works sharing an author with the work
they are placed under
--max N stop when the tree holds N works (default 300)
--pool N citing works read per work before ranking: the N
most cited, plus, unless --rank cites, the N most
cited of the last five years (default 200)
--openalex-only no citers from OpenCitations and Semantic Scholar
Output:
-b, --bib FILES BibTeX file(s), comma-separated: works get their key
-f, --format FMT text (default), json, dot (Graphviz), wiki (nested
list of {{key}}), keys (bib keys, one per line), ids
(DOI or arXiv:ID of works with no key)
-o, --output FILE write there instead of stdout; a .svg, .pdf or .png
is drawn by Graphviz's dot from the dot format
--no-links no "see [n]" lines, and in the drawing only the
tree's own edges
-W, --width N cut text lines at N characters (default: the
terminal's width; lines are not cut when piped)
--fresh do not use the cache (~/.cache/vip, answers younger
than 7 days are reused)
--mailto EMAIL contact for the OpenAlex polite pool (or env
VIP_MAILTO); nothing is sent without it
--openalex-key K OpenAlex API key (or env OPENALEX_API_KEY)
--s2-key K Semantic Scholar API key (or env S2_API_KEY)
-q, --quiet no progress or summary on stderr
-V, --version the version
-h, --help this message
Text columns: [n] number in the tree, year, citations, the ranking figure
(per year, or recent), marks (R review, P preprint, * shares an author with
the work above it), then the bib key (or the DOI), first author and title.
Defaults: ~/.viprc (or the file named by env VIP_RC) holds options as on the
command line; --bib, --mailto and --openalex-key are also taken from
~/.citingrc when ~/.viprc does not give them, and so is --s2-key.
"""
import datetime
import hashlib
import importlib.machinery
import importlib.util
import json
import os
import re
import shutil
import subprocess
import sys
import time
__version__ = '1.1.0'
NEEDS_CITING = (2, 0, 0)
def load_citing():
"""citing's functions: from the file next to vip, on the PATH, or in a
sibling checkout."""
here = os.path.dirname(os.path.realpath(__file__))
for p in (os.path.join(here, 'citing'), shutil.which('citing'), os.path.join(here, '..', 'citing', 'citing')):
if p and os.path.isfile(p):
loader = importlib.machinery.SourceFileLoader('citing', p)
spec = importlib.util.spec_from_loader('citing', loader)
m = importlib.util.module_from_spec(spec)
loader.exec_module(m)
if tuple(int(x) for x in m.__version__.split('.')) < NEEDS_CITING:
sys.exit('vip: needs citing %s or later, %s has %s' % ('.'.join(map(str, NEEDS_CITING)), p, m.__version__))
return m
sys.exit('vip: needs citing (https://github.com/laussy/citing) next to it or on the PATH')
C = load_citing()
say = C.say
TODAY = datetime.date.today()
CACHE = os.path.join(os.environ.get('XDG_CACHE_HOME') or os.path.expanduser('~/.cache'), 'vip')
CACHE_DAYS = 7
FRESH = False
STATS = {'cached': 0}
SEL = ('id,doi,title,publication_year,publication_date,cited_by_count,counts_by_year,'
'authorships,primary_location,type,ids,referenced_works')
# --------------------------------------------------------------------------
# OpenAlex, with a cache
# --------------------------------------------------------------------------
def oa_query(params):
"""A /works query; answers younger than CACHE_DAYS come from the cache."""
key = hashlib.sha1(json.dumps(params, sort_keys=True).encode()).hexdigest()
path = os.path.join(CACHE, key + '.json')
if not FRESH:
try:
if time.time() - os.path.getmtime(path) < CACHE_DAYS * 86400:
with open(path, encoding='utf-8') as fh:
STATS['cached'] += 1
return json.load(fh)
except (OSError, ValueError):
pass
p = dict(params)
if C.MAILTO:
p['mailto'] = C.MAILTO
d = C.get('https://api.openalex.org/works', p)
if d is not None:
try:
os.makedirs(CACHE, exist_ok=True)
tmp = path + '.tmp'
with open(tmp, 'w', encoding='utf-8') as fh:
json.dump(d, fh)
os.replace(tmp, path)
except OSError:
pass
return d
def short(wid):
return (wid or '').rsplit('/', 1)[-1]
TEX_ACC = {"'": '\u0301', '`': '\u0300', '^': '\u0302', '"': '\u0308', '~': '\u0303', '=': '\u0304',
'.': '\u0307', 'u': '\u0306', 'v': '\u030c', 'H': '\u030b', 'c': '\u0327', 'k': '\u0328',
'r': '\u030a', 'd': '\u0323', 'b': '\u0331'}
TEX_SYM = {'ss': '\u00df', 'o': '\u00f8', 'O': '\u00d8', 'l': '\u0142', 'L': '\u0141', 'aa': '\u00e5',
'AA': '\u00c5', 'ae': '\u00e6', 'AE': '\u00c6', 'oe': '\u0153', 'OE': '\u0152', 'i': '\u0131'}
def tex2uni(s):
"""BibTeX accents to Unicode, braces off: {S\\'anchez Mu\\~noz} -> Sánchez Muñoz."""
import unicodedata
s = re.sub(r'\\([`\'^"~=.]|[uvHckrdb](?=[\s{]))\s*\{?\\?([a-zA-Z])\}?',
lambda m: m.group(2) + TEX_ACC[m.group(1)], s or '')
s = re.sub(r'\\(ss|aa|AA|ae|AE|oe|OE|o|O|l|L|i)\b\s*', lambda m: TEX_SYM[m.group(1)], s)
s = s.replace('{', '').replace('}', '').replace('~', ' ')
return unicodedata.normalize('NFC', ' '.join(s.split()))
def bib_surname(name):
"""The surname of a BibTeX author, in Unicode; a braced group is one surname."""
m = re.search(r'\{((?:[^{}]|\{[^{}]*\})*\s(?:[^{}]|\{[^{}]*\})*)\}\s*$', name)
if m:
return tex2uni(m.group(1))
name = tex2uni(name)
if ',' in name:
return name.split(',')[0].strip()
parts = name.split()
# BibTeX's "von" part: lowercase words before the last name (del Valle)
for i in range(1, len(parts) - 1):
if parts[i][:1].islower():
return ' '.join(parts[i:])
return parts[-1] if parts else '?'
class Work:
"""An OpenAlex work, with what the tree needs."""
def __init__(self, x):
self.wid = short(x['id'])
self.doi = C.ndoi(x.get('doi'))
self.arxiv = C.arxiv_of_doi(self.doi)
loc = x.get('primary_location') or {}
lp = loc.get('landing_page_url') or ''
if not self.arxiv and 'arxiv.org/abs/' in lp:
self.arxiv = C.narxiv(lp.split('/abs/')[-1])
self.title = ' '.join((x.get('title') or '').split()) or None
self.year = x.get('publication_year')
self.date = x.get('publication_date')
self.cites = x.get('cited_by_count') or 0
self.by_year = {c['year']: c['cited_by_count'] for c in x.get('counts_by_year') or []}
self.authors = [a['author'].get('display_name') or '' for a in x.get('authorships') or []]
self.author_ids = {a['author'].get('id') for a in x.get('authorships') or [] if a['author'].get('id')}
self.venue = (loc.get('source') or {}).get('display_name')
self.type = x.get('type')
self.refs = {short(r) for r in x.get('referenced_works') or []}
self.nt = C.ntitle(self.title)
self.key = None
self.children = []
self.links = [] # works already in the tree, among the best citers
self.parent = None
self.depth = 0
self.self_cite = False
self.n = None
@property
def rate(self):
try:
d = datetime.date.fromisoformat(self.date)
except (TypeError, ValueError):
d = datetime.date(self.year or TODAY.year, 7, 1)
age = max((TODAY - d).days / 365.25, 1.0)
return self.cites / age
@property
def recent(self):
return sum(c for y, c in self.by_year.items() if y >= TODAY.year - 2)
def ident(self):
if self.key:
return self.key
if self.arxiv:
return 'arXiv:' + self.arxiv
return self.doi or self.wid
def set_key(self, bib, key=None):
self.key = key or (bib.match(self.as_rec()) or [None])[0]
if self.key and bib.authors.get(self.key):
self.bib_first = re.split(r'\s+and\s+', bib.authors[self.key])[0]
def first(self):
if getattr(self, 'bib_first', None):
return bib_surname(self.bib_first)
return C.surname(self.authors[0]) if self.authors else '?'
def as_rec(self):
r = C.Rec()
if self.doi:
C.add_doi(r, self.doi)
if self.arxiv:
r.arxiv.add(self.arxiv)
r.title = self.title
return r
# --------------------------------------------------------------------------
# The tree
# --------------------------------------------------------------------------
def filters(o, wid, recent_only=False, ids=None):
f = ['cites:' + wid] if wid else ['doi:' + '|'.join(ids)]
if o['type']:
f.append('type:' + '|'.join(o['type']))
else:
f += ['type:!erratum', 'type:!paratext', 'type:!retraction']
since = o['since']
if recent_only:
since = max(since or 0, TODAY.year - 4)
if since:
f.append('from_publication_date:%d-01-01' % since)
if o['until']:
f.append('to_publication_date:%d-12-31' % o['until'])
if o['min_cites']:
f.append('cited_by_count:>%d' % (o['min_cites'] - 1))
if o['topic']:
f.append('title_and_abstract.search:' + o['topic'].replace(',', ' '))
return ','.join(f)
def citers(o, node):
"""The pool of works citing node: the most cited, and (unless ranking by
all citations) the most cited of the last five years; when that is all of
them, also those only the other indexes know. Returns the pool, the number
of citers, and how many of them OpenAlex lists."""
wid = node.wid
out, total, whole = {}, 0, False
for recent in ((False, True) if o['rank'] != 'cites' else (False,)):
got, page = 0, 1
while got < o['pool']:
per = min(200, o['pool'] - got)
d = oa_query({'filter': filters(o, wid, recent), 'sort': 'cited_by_count:desc',
'per-page': per, 'page': page, 'select': SEL})
if not d:
break
if not recent and page == 1:
total = d['meta']['count']
for x in d['results']:
out.setdefault(short(x['id']), x)
got += len(d['results'])
if len(d['results']) < per:
break
page += 1
if not recent and got >= total:
whole = True
break # the whole list is in already
pool = [Work(x) for x in out.values()]
if not whole or o['openalex_only']:
return pool, total, total
more, extra = other_citers(o, node, pool)
for w in more:
if w.wid not in out:
pool.append(w)
return pool, total + extra, total
def other_citers(o, node, pool):
"""The citers of node that OpenCitations and Semantic Scholar list and
OpenAlex does not (joined as citing joins them), as OpenAlex works when
OpenAlex has them; and how many such citers there are."""
doi = node.doi if node.doi and not node.arxiv else None
if not (doi or node.arxiv):
return [], 0
quiet, C.QUIET = C.QUIET, True # citing's own progress lines
try:
oc = C.opencitations(node.doi) if node.doi else []
s2 = C.semanticscholar('DOI:' + doi if doi else 'ARXIV:' + node.arxiv, o['s2_key'])
recs = []
for w in pool:
r = w.as_rec()
r.src = {'A'}
recs.append(r)
works = C.fuzzy_join(C.merge_all(C.enrich(recs + oc + s2)))
finally:
C.QUIET = quiet
extra = [w for w in works if 'A' not in w.src and not (node.doi and node.doi in w.dois)
and not (node.arxiv and node.arxiv in w.arxiv)]
ids = sorted({d for w in extra for d in w.dois} | {'10.48550/arxiv.' + a for w in extra for a in w.arxiv})
ids = [d for d in ids if ',' not in d and '|' not in d]
found = {}
for k in range(0, len(ids), 50):
f = filters(o, wid=None, ids=ids[k:k + 50])
d = oa_query({'filter': f, 'per-page': 200, 'select': SEL})
for x in (d or {}).get('results', []):
found.setdefault(short(x['id']), Work(x))
more = list(found.values())
known = {d for w in more for d in (w.doi, w.arxiv and '10.48550/arxiv.' + w.arxiv) if d}
matched = [w for w in extra if (w.dois | {'10.48550/arxiv.' + a for a in w.arxiv}) & known]
# a citer OpenAlex has no record of passes no filter, but it is a citer
narrowed = o['type'] or o['since'] or o['until'] or o['min_cites'] or o['topic']
return more, len(extra) if not narrowed else len(matched)
def dedupe(works):
"""A preprint and its published version can both cite: keep one, the
published one (or the more cited)."""
best = {}
for w in works:
k = w.nt if len(w.nt) >= 20 else w.wid
b = best.get(k)
if b is None or ((b.type == 'preprint', -b.cites) > (w.type == 'preprint', -w.cites)):
best[k] = w
return list(best.values())
def score(o, w):
return {'rate': w.rate, 'recent': w.recent, 'cites': w.cites}[o['rank']]
def grow(o, seed, bib=None):
placed = {seed.wid: seed}
titles = {seed.nt: seed} if len(seed.nt) >= 20 else {}
level, depth = [seed], 0
count = 1
while level and depth < o['depth'] and count < o['max']:
nxt = []
top = o['top'][min(depth, len(o['top']) - 1)]
for node in level:
if count >= o['max']:
break
pool, total, in_oa = citers(o, node)
node.total, node.in_openalex = total, in_oa
pool = sorted(dedupe(pool), key=lambda w: (-score(o, w), -w.cites, w.wid))
for w in pool:
if len(node.children) >= top or count >= o['max']:
break
w.self_cite = bool(w.author_ids & node.author_ids)
if w.self_cite and o['no_self']:
continue
there = placed.get(w.wid) or (titles.get(w.nt) if len(w.nt) >= 20 else None)
if there is not None:
if there is not node and there is not seed and there not in node.links:
node.links.append(there)
continue
w.parent, w.depth = node, depth + 1
if bib:
w.set_key(bib)
node.children.append(w)
placed[w.wid] = w
if len(w.nt) >= 20:
titles[w.nt] = w
count += 1
nxt.append(w)
say(' [gen %d] %-24s %5d citers%s, %d kept' % (
depth + 1, node.ident()[:24], total,
' (%d in OpenAlex)' % in_oa if total != in_oa else '', len(node.children)))
level, depth = nxt, depth + 1
return placed
def walk(node):
yield node
for c in node.children:
yield from walk(c)
def number(seed):
for i, w in enumerate(walk(seed)):
w.n = i
def edges(seed):
"""Every citation between works of the tree: (citing, cited, in_tree)."""
nodes = list(walk(seed))
by = {w.wid: w for w in nodes}
out = []
for w in nodes:
for r in sorted(w.refs):
if r in by and by[r] is not w:
out.append((w, by[r], w.parent is by[r]))
if w.parent is not None and w.parent.wid not in w.refs:
out.append((w, w.parent, True)) # the tree edge, from the cites: filter
return out
# --------------------------------------------------------------------------
# Output
# --------------------------------------------------------------------------
def marks(w):
return ('R' if w.type == 'review' else 'P' if w.type == 'preprint' else ' ') + ('*' if w.self_cite else ' ')
def fig(o, w):
r = ('%5.1f' if w.rate < 999.95 else '%5.0f') % w.rate
return {'rate': r, 'recent': '%5d' % w.recent, 'cites': ' '}[o['rank']]
def text(o, seed, width):
head = {'rate': '/yr', 'recent': '3 yrs', 'cites': ''}[o['rank']]
lines = ['%5s %4s %6s %5s' % ('[n]', 'year', 'cites', head)]
pad = ' ' * 27
def cut(s):
return s if not width or len(s) <= width else s[:width - 1] + '…'
def rec(w, prefix, last, root):
branch = '' if root else (prefix + ('└─ ' if last else '├─ '))
lines.append(cut('%5s %4s %6d %5s %2s %s%s %s %s' % (
'[%d]' % w.n, w.year or '????', w.cites, fig(o, w), marks(w), branch,
w.ident(), w.first(), w.title or '')))
sub = '' if root else prefix + (' ' if last else '│ ')
items = [('c', c) for c in w.children] + ([('l', l) for l in w.links] if not o['no_links'] else [])
for i, (kind, c) in enumerate(items):
end = i == len(items) - 1
if kind == 'c':
rec(c, sub, end, False)
else:
lines.append(cut(pad + sub + ('└─ ' if end else '├─ ') + 'see [%d] %s' % (c.n, c.ident())))
rec(seed, '', True, True)
return '\n'.join(lines) + '\n'
def esc(s):
return (s or '').replace('\\', '\\\\').replace('"', '\\"')
def wrap(s, n=30, lines=2):
words, out, cur = (s or '').split(), [], ''
for wd in words:
if cur and len(cur) + 1 + len(wd) > n:
out.append(cur)
cur = wd
if len(out) == lines:
break
else:
cur = (cur + ' ' + wd).strip()
else:
if cur:
out.append(cur)
if len(out) == lines and ' '.join(out) != ' '.join(words):
out[-1] = out[-1].rstrip('.,;:') + '…'
return out
def dot(o, seed, desc):
nodes = list(walk(seed))
years = sorted({w.year for w in nodes if w.year})
L = ['digraph vip {',
# pad: a renderer whose font is wider than dot's cuts off what touches the edge
' graph [rankdir=TB, ranksep=0.35, nodesep=0.18, pad=0.3, fontname="Helvetica", fontsize=12, '
'labelloc=t, label="%s"];' % esc(desc),
' node [shape=box, style="rounded,filled", fillcolor="#ffffff", color="#555555", '
'fontname="Helvetica", fontsize=9, margin="0.08,0.04"];',
' edge [color="#444444", arrowsize=0.55];']
if years:
# the years that have works, a run of empty years shrunk to one row
axis = []
for y in years:
if axis and y - axis[-1][0] > 2:
axis.append((axis[-1][0] + 1, '\u22ee'))
elif axis and y - axis[-1][0] == 2:
axis.append((y - 1, str(y - 1)))
axis.append((y, str(y)))
L.append(' { node [shape=plaintext, style="", fontsize=11, fontcolor="#777777"];')
L.append(' ' + ' -> '.join('y%d' % y for y, _ in axis) + ' [style=invis]; }')
for y, lab in axis:
L.append(' y%d [label="%s"];' % (y, lab))
for y in years:
L.append(' { rank=same; y%d; %s }' % (y, ' '.join('n%d;' % w.n for w in nodes if w.year == y)))
for w in nodes:
lab = ['%s %s' % (w.first(), w.year or '')] + wrap(w.title)
lab.append('%s · %d cit.' % (w.key, w.cites) if w.key else '%d cit.' % w.cites)
attrs = ['label="%s"' % esc('\n'.join(lab))]
if w is seed:
attrs += ['fillcolor="#c9dcf0"', 'penwidth=2']
elif w.key:
attrs.append('fillcolor="#e6eff8"')
if w.type == 'review':
attrs += ['peripheries=2']
if w.doi:
attrs.append('URL="https://doi.org/%s"' % esc(w.doi))
attrs.append('tooltip="%s"' % esc('[%d] %s' % (w.n, w.title or '')))
L.append(' n%d [%s];' % (w.n, ', '.join(attrs)))
for a, b, tree in edges(seed):
# drawn from the cited work to the citing one: time runs down the page
if tree:
L.append(' n%d -> n%d;' % (b.n, a.n))
elif not o['no_links']:
L.append(' n%d -> n%d [style=dashed, color="#bbbbbb", penwidth=0.7, arrowsize=0.4, '
'constraint=false];' % (b.n, a.n))
L.append('}')
return '\n'.join(L) + '\n'
def wiki(o, seed):
out = []
for w in walk(seed):
stars = '*' * (w.depth + 1)
if w.key:
out.append('%s {{%s}}' % (stars, w.key))
else:
url = 'https://doi.org/' + w.doi if w.doi and not w.arxiv else (
'https://arxiv.org/abs/' + w.arxiv if w.arxiv else 'https://openalex.org/' + w.wid)
who = w.first() + (" ''et al.''" if len(w.authors) > 1 else '')
out.append("%s [%s %s], %s, ''%s'' (%s)" % (stars, url, (w.title or '').replace(']', ']'),
who, w.venue or w.type or '', w.year or ''))
return '\n'.join(out) + '\n'
def as_json(o, seed, desc):
nodes = list(walk(seed))
return json.dumps({
'vip': __version__, 'date': TODAY.isoformat(), 'title': desc,
'options': {k: o[k] for k in ('depth', 'top', 'rank', 'min_cites', 'since', 'until', 'type',
'topic', 'no_self', 'max', 'pool', 'openalex_only')},
'works': [{'n': w.n, 'depth': w.depth, 'parent': w.parent.n if w.parent else None,
'openalex': w.wid, 'doi': w.doi, 'arxiv': w.arxiv, 'key': w.key, 'year': w.year,
'date': w.date, 'title': w.title, 'authors': w.authors, 'venue': w.venue,
'type': w.type, 'cites': w.cites, 'rate': round(w.rate, 2), 'recent': w.recent,
'self_citation': w.self_cite, 'citers_total': getattr(w, 'total', None),
'citers_openalex': getattr(w, 'in_openalex', None),
'see': [l.n for l in w.links]} for w in nodes],
'citations': [{'citing': a.n, 'cited': b.n, 'tree': t} for a, b, t in edges(seed)],
}, indent=1, ensure_ascii=False) + '\n'
# --------------------------------------------------------------------------
# Main
# --------------------------------------------------------------------------
TAKES = {'-d': 'depth', '--depth': 'depth', '-n': 'top', '--top': 'top', '-r': 'rank', '--rank': 'rank',
'--min-cites': 'min_cites', '--since': 'since', '--until': 'until', '--type': 'type',
'--topic': 'topic', '--max': 'max', '--pool': 'pool', '-b': 'bib', '--bib': 'bib',
'-f': 'format', '--format': 'format', '-o': 'output', '--output': 'output',
'-W': 'width', '--width': 'width', '--mailto': 'mailto', '--openalex-key': 'openalex_key',
'--s2-key': 's2_key'}
FLAGS = {'--no-self': 'no_self', '--no-links': 'no_links', '--fresh': 'fresh', '-q': 'quiet', '--quiet': 'quiet',
'--openalex-only': 'openalex_only'}
def parse_args(argv):
o = {'depth': None, 'top': ['5'], 'rank': 'rate', 'min_cites': 0, 'since': None, 'until': None,
'type': [], 'topic': None, 'no_self': False, 'max': 300, 'pool': 200, 'bib': [],
'format': None, 'output': None, 'width': None, 'no_links': False, 'fresh': False,
'mailto': os.environ.get('VIP_MAILTO'), 'openalex_key': os.environ.get('OPENALEX_API_KEY'),
's2_key': os.environ.get('S2_API_KEY'), 'openalex_only': False, 'quiet': False}
for a in argv:
if a in ('-h', '--help'):
print(__doc__.strip())
sys.exit(0)
if a in ('-V', '--version'):
print('vip', __version__, '(citing %s)' % C.__version__)
sys.exit(0)
lists = {'bib': 'split', 'type': 'split', 'top': 'split'}
ints = ('depth', 'min_cites', 'since', 'until', 'max', 'pool', 'width')
pos = []
given = C.parse_options(argv, o, TAKES, FLAGS, lists, ints, 'vip', pos)
rc = dict(o)
for k in lists:
rc[k] = []
rc_given = C.parse_options(C.read_rc('VIP_RC', 'vip'), rc, TAKES, FLAGS, lists, ints, 'vip (~/.viprc)', None)
for k in rc_given - given:
o[k] = rc[k]
# what ~/.citingrc says of the bibliography and of OpenAlex
crc = {}
try:
cg = C.parse_options(C.read_rc('CITING_RC', 'citing'), crc, C.TAKES, C.FLAGS,
{'bib': 'split', 'own': 'split', 'hunt': 'raw'}, ('since', 'hunt_since', 'hunt_max'),
'vip (~/.citingrc)', None)
except SystemExit:
cg = set()
for k in ('bib', 'mailto', 'openalex_key', 's2_key'):
if k in cg and k not in given | rc_given:
o[k] = crc[k]
if not pos:
print(__doc__.strip())
sys.exit(1)
if len(pos) > 1:
sys.exit('vip: one paper at a time')
o['paper'] = pos[0]
try:
o['top'] = [int(x) for x in o['top']]
except ValueError:
sys.exit('vip: --top takes numbers, e.g. 5 or 10,5,3')
if any(t < 1 for t in o['top']):
sys.exit('vip: --top numbers must be 1 or more')
if o['depth'] is None:
o['depth'] = len(o['top']) if 'top' in given | rc_given and len(o['top']) > 1 else 2
if o['rank'] not in ('rate', 'recent', 'cites'):
sys.exit('vip: --rank is rate, recent or cites')
ext = os.path.splitext(o['output'] or '')[1].lower().lstrip('.')
if o['format'] is None:
o['format'] = {'json': 'json', 'dot': 'dot', 'gv': 'dot', 'svg': 'dot', 'pdf': 'dot', 'png': 'dot',
'wiki': 'wiki', 'txt': 'text'}.get(ext, 'text')
if o['format'] not in ('text', 'json', 'dot', 'wiki', 'keys', 'ids'):
sys.exit('vip: --format is text, json, dot, wiki, keys or ids')
if ext in ('svg', 'pdf', 'png') and o['format'] != 'dot':
sys.exit('vip: a .%s is drawn from the dot format' % ext)
o['render'] = ext if ext in ('svg', 'pdf', 'png') else None
return o
def main(argv):
global FRESH
o = parse_args(argv)
C.QUIET, C.MAILTO, C.OPENALEX_KEY = o['quiet'], o['mailto'], o['openalex_key']
FRESH = o['fresh']
if o['render'] and not shutil.which('dot'):
sys.exit('vip: drawing a .%s needs Graphviz (dot); -f dot writes the source' % o['render'])
bib = C.Bib(o['bib']) if o['bib'] else None
s = C.resolve(o['paper'], bib, prog='vip')
ident = s['wid'] or 'doi:' + s['doi']
x = C.openalex_work(ident, select=SEL)
if not x:
sys.exit('vip: OpenAlex has no work for %s' % C.label(s))
seed = Work(x)
if bib:
seed.set_key(bib, s['key'])
by = {'rate': 'citations per year', 'recent': 'citations in %d–%d' % (TODAY.year - 2, TODAY.year),
'cites': 'citations'}[o['rank']]
tops = ','.join(map(str, o['top']))
only = [x for x in (
o['topic'] and 'on "%s"' % o['topic'],
o['type'] and 'type %s' % '|'.join(o['type']),
o['since'] and 'from %d' % o['since'],
o['until'] and 'to %d' % o['until'],
o['min_cites'] and 'cited %d+ times' % o['min_cites'],
o['no_self'] and 'no self-citations') if x]
desc = '%s (%s %s): %d generation%s, top %s by %s%s, OpenAlex %s' % (
seed.ident(), seed.first(), seed.year or '', o['depth'], 's' if o['depth'] > 1 else '', tops, by,
' (%s)' % ', '.join(only) if only else '', '%d %s' % (TODAY.day, TODAY.strftime('%b %Y')))
say('vip %s: %s' % (__version__, desc))
grow(o, seed, bib)
# the paper is ranked against nothing: its count is all its citers
seed.cites = max(seed.cites, getattr(seed, 'total', 0) or 0)
number(seed)
nodes = list(walk(seed))
f = o['format']
width = None
if f == 'text':
if o['width']:
width = o['width']
elif not o['output'] and sys.stdout.isatty():
width = shutil.get_terminal_size((160, 40)).columns
out = text(o, seed, width)
elif f == 'json':
out = as_json(o, seed, desc)
elif f == 'dot':
out = dot(o, seed, desc)
elif f == 'wiki':
out = wiki(o, seed)
elif f == 'keys':
out = ''.join(w.key + '\n' for w in nodes if w.key)
else:
out = ''.join(w.ident() + '\n' for w in nodes if not w.key and (w.doi or w.arxiv))
if o['render']:
r = subprocess.run(['dot', '-T' + o['render'], '-o', o['output']], input=out.encode(), capture_output=True)
if r.returncode:
sys.exit('vip: dot failed: ' + r.stderr.decode(errors='replace').strip())
elif o['output']:
with open(o['output'], 'w', encoding='utf-8') as fh:
fh.write(out)
else:
sys.stdout.write(out)
links = sum(len(w.links) for w in nodes)
inbib = sum(bool(w.key) for w in nodes)
left = (C.BUDGET.get('api.openalex.org') or {}).get('x-ratelimit-remaining')
gens = max(w.depth for w in nodes)
say('%d works in %d generation%s%s%s; OpenAlex: %d queries, %d from the cache%s%s' % (
len(nodes), gens, '' if gens == 1 else 's',
', %d seen again ("see")' % links if links else '',
', %d in the bib files' % inbib if bib else '',
C.CALLS.get('api.openalex.org', 0), STATS['cached'],
', %s credits left today' % left if left else '',
'; written to ' + o['output'] if o['output'] else ''))
if __name__ == '__main__':
try:
main(sys.argv[1:])
except KeyboardInterrupt:
sys.exit(130)
except BrokenPipeError:
pass