arXiv2bib
arXiv2bib does for still-unpublished (or never published) papers from the arXiv what doi2bib does for published papers, i.e., it returns a bibTeX entry ready to be put in the bib collection (it does so with option -w).
For example:
laussy@azag:~$ ./arXiv2bib 2402.01627 -w
@Article{arXiv_casalengua24a,
author = {E. {Zubizarreta Casalengua} and F. P. Laussy},
title = {Spatial correlations of vortex quantum states},
journal = {arXiv:2402.01627},
year = {2024},
pdf = {arXiv/arXiv_casalengua24a},
url = {doi:10.48550/arXiv.2402.01627}
}
====
arXiv:2402.01627
arXiv_casalengua24a
arXiv_casalengua24a added at the top of /home/laussy/bib/arXiv.bib
This can then be exported for laussy.org with bib2wiki:
laussy@azag:~$ bib2wiki arXiv_casalengua24a
arXiv_casalengua24a
<wz tip="arXiv:2402.01627">[[File:arXXXiv.png|14px|link=]]</wz><u>[[arXiv_casalengua24a|Spatial correlations of vortex quantum states]]</u>. [[E. Zubizarreta Casalengua]] and [[F. P. Laussy]] in {{arXiv|2402.01627}} ([[2024]]).
which outputs:
Spatial correlations of vortex quantum states. E. Zubizarreta Casalengua and F. P. Laussy in arXiv:2402.01627 (2024).
Todo
Note that in the example above, as per my naming conventions, it should really be arXiv_zubizarettacasalengua24a but I'll leave it at that for now.
There is not yet a way to check for already existing keys, like doi2bib, so as to increment the last letter.
Insertion in the file is also not sorted.
Version
I worked out the first version from doi2bib with the help of both Grok and ChatGPT, and came up with the first archivable version as v°0.8.8. Since then:
- v°0.8.9 (6 August (2026)) — -a writes the entry (-w) and then pushes its template to the wiki with bib2wiki, as doi2bib -a does. A flag written before the id is no longer lost: perl's -s takes it, and it is read back from there.
- v°0.9.0 (11 August (2026)) — accented letters in the bib key are transliterated instead of deleted. Papuccio-Fernández gave papucciofernndez, because the strip of non-letters ran first and ate the UTF-8 bytes of the á, which left the accent substitutions after it as dead code. $\mathrm{\TeX}$ accent escapes are now stripped, then the name is NFKD-decomposed and its combining marks dropped, as doi2bib does.
- v°0.9.1 (11 August (2026)) — the initials are cut by one function for both the Crossref and the HTML path. J. -V. Vidal came out as J. -. V.; only the first given name was initialled on the Crossref path, so the bibnames rules, written for the full form, could never fire; a mononymous author made the whole call fail and lost every author at once; multi-word family names are braced on both paths.
- v°0.10.0 (11 September (2026)) — -h lists the options.
The current version, v°0.10.0:
#!/usr/bin/perl -X -s
# __ ___ ____ _ _ _
# __ _ _ _\ \/ (_)_ _|___ \| |__ (_) |__
# / _` | '__\ /| \ \ / / __) | '_ \| | '_ \
#| (_| | | / \| |\ V / / __/| |_) | | |_) |
# \__,_|_| /_/\_\_| \_/ |_____|_.__/|_|_.__/
# F.P. Laussy - fabrice.laussy@gmailcom
# Adapted from doi2bib by Grok - Sun Sep 21 2025
# v0.8.8 Sun Sep 21 2025 - Fixed extraneous "Invalid JSON" error on HTTP 404 by conditioning JSON processing
# on $http_status == 200 (in addition to -f && -s > 0). Retains "Failed to download"
# for non-200 responses.
# v0.8.9 Thu Aug 06 2026 - -a: write (-w) then push the new entry to the
# wiki via bib2wiki (as doi2bib -a does). Flags are
# now also read from $w/$a, which perl's -s (shebang)
# steals when the flag comes *before* the id.
# v0.9.0 Tue Aug 11 2026 - bib key: accented letters are now transliterated
# instead of deleted. "Papuccio-Fernández" used to
# give papucciofernndez, because s/[^a-zA-Z]//g ran
# *first* and ate the UTF-8 bytes of á (the accent
# substitutions that followed were dead code, they
# could never match). Now bibkey() strips TeX accent
# escapes (\'a, \"u, {\ss}...), then NFKD-decomposes
# and drops the combining marks, as doi2bib does.
# v0.9.1 Tue Aug 11 2026 - author initials are now cut by initials(), shared by
# both the Crossref and the HTML path, which fixes:
# * "J. -V. Vidal" (arXiv writes a compound given name
# with a space before the hyphen) came out as
# "J. -. V." -- substr($_,0,1) took the hyphen;
# * only the *first* given name was initialled on the
# Crossref path, so "Juan Camilo L\'opez" gave "J."
# and the bibnames rules, which are written for the
# full "J. C. L." form, could never fire;
# * a mononymous author (family, no given) made the
# whole jq call fail, losing every author at once;
# * multi-word family names are braced on both paths.
# bibnames is applied through a file: sh's echo ate
# the backslashes of the TeX escapes.
# v0.10.0 Fri Sep 11 2026 - -h lists the options. Written before the id it is
# eaten by perl's -s and read back from $h, as -w and
# -a already are.
#
# Usage:
# arXiv2bib [-w] id
# arXiv2bib id [-w]
# arXiv2bib -a id # -w, then bib2wiki <newkey>
use strict;
use warnings;
use Encode qw(decode_utf8);
use Unicode::Normalize qw(NFKD);
my $VERSION = '0.10.0';
# --- the options -----------------------------------------------------------
sub usage {
print <<"END";
arXiv2bib -- v$VERSION -- an arXiv id becomes a BibTeX entry in ~/bib/arXiv.bib
arXiv2bib ID show the entry, write nothing
arXiv2bib -w ID write: put it at the TOP of arXiv.bib, newest first
arXiv2bib -a ID -w, and then push Template:<Key> with bib2wiki -- which
creates the page but, unlike doi2bib -a, does not update
one that is already there
-h this list
A flag may be written on either side of the id. The id may be bare, prefixed
with arXiv:, versioned (2409.12952v2) or in the old scheme (cond-mat/0605294).
When the paper has since been published, the doi is followed and the entry is
made from the journal's metadata instead of arXiv's.
END
exit 0;
}
# "Alejandro Andres" -> "A. A.", "Xing-Gang" -> "X.-G.", "Juan Camilo L\'opez"
# -> "J. C. L." (which is the form the bibnames sed file expects). Case is left
# alone on purpose, so "Elena del" gives "E. d." and rule 3 of bibnames fires.
sub initials {
my ($given) = @_;
return "" unless defined $given;
$given =~ s/^\s+|\s+$//g;
return "" if $given eq "";
# arXiv and Crossref write a compound given name as "J. -V.", with a space
# before the hyphen: glue such a piece back onto the one before it
$given =~ s/\s+-/-/g;
my @out;
foreach my $part (split /\s+/, $given) {
my @bits = grep { $_ ne "" } split /-/, $part;
next unless @bits;
push @out, join("-", map { substr($_, 0, 1) . "." } @bits);
}
return join(" ", @out);
}
# one BibTeX author: initialled given name plus family name, the latter braced
# when it is made of several words, so BibTeX does not mistake "de la Pradilla"
# for a given name. A mononymous author is just the family name.
sub format_author {
my ($given, $family) = @_;
$family = "" unless defined $family;
$family =~ s/^\s+|\s+$//g;
return "" if $family eq "";
$family = "{$family}" if $family =~ /\s/ && $family !~ /^\{.*\}$/;
my $ini = initials($given);
return $ini eq "" ? $family : "$ini $family";
}
# run the author list through the bibnames sed file (accents -> TeX escapes,
# plus the hand-written rules for the group's own compound surnames). Through a
# file rather than `echo "..." | sed`: sh's echo mangles the backslashes that
# the earlier rules have just put in, and the names are full of them.
sub tex_names {
my ($text, $sedfile) = @_;
return $text unless defined $text && $text ne "" && -f $sedfile;
open(my $t, '>', "authors.txt") or return $text;
print $t $text, "\n";
close $t;
my $out = qx{sed -f $sedfile authors.txt 2>/dev/null};
unlink "authors.txt";
chomp $out;
return $out ne "" ? $out : $text;
}
# turn a family name -- raw UTF-8 ("Fernández") or already TeX-escaped by
# bibnames ("Fern\'andez") -- into the bare ASCII stem of a bib key
sub bibkey {
my ($name) = @_;
return "unknown" unless defined $name && $name ne "";
# dotless i and j, which accent commands are built on: \u{\i} -> \u{i}
# (keep the braces, so the accent rules below still see \u{...})
$name =~ s/\\i(?![a-zA-Z])/i/g;
$name =~ s/\\j(?![a-zA-Z])/j/g;
# TeX accents: \'a \"{u} \^o \~n \c{c} \v{s} \k{a} \.{z} ... keep the letter
$name =~ s/\\[`'"^~=.uvHtcdbrk]\s*\{(\w)\}/$1/g;
$name =~ s/\\[`'"^~=.]\s*(\w)/$1/g;
$name =~ s/\\[uvHtcdbrk]\s+(\w)/$1/g;
# TeX ligatures with no ASCII letter of their own
$name =~ s/\{?\\ss\}?/ss/g;
$name =~ s/\{?\\(?:ae|AE)\}?/ae/g;
$name =~ s/\{?\\(?:oe|OE)\}?/oe/g;
$name =~ s/\{?\\[oO]\}?/o/g;
$name =~ s/\{?\\[lL]\}?/l/g;
$name =~ s/\{?\\(?:aa|AA)\}?/a/g;
# interpret the bytes as UTF-8 (qx// hands us bytes; be safe if it does not)
$name = decode_utf8($name) unless utf8::is_utf8($name);
$name = NFKD($name); # é -> e + combining acute
$name =~ s/\p{NonspacingMark}//g; # drop the diacritics
# letters NFKD does not decompose, because the stroke/hook is part of the
# glyph. Written \x{} rather than literally: this file is not "use utf8",
# so an accented character typed here would be *bytes* and match nothing.
$name =~ s/[\x{0110}\x{0111}\x{00D0}\x{00F0}]/d/g; # Đ đ Ð ð
$name =~ s/[\x{00D8}\x{00F8}]/o/g; # Ø ø
$name =~ s/[\x{0141}\x{0142}]/l/g; # Ł ł
$name =~ s/[\x{0131}]/i/g; # ı dotless i
$name =~ s/[\x{00DF}\x{1E9E}]/ss/g; # ß ẞ
$name =~ s/[\x{00C6}\x{00E6}]/ae/g; # Æ æ
$name =~ s/[\x{0152}\x{0153}]/oe/g; # Œ œ
$name =~ s/[\x{00DE}\x{00FE}]/th/g; # Þ þ
$name =~ s/[^a-zA-Z]//g; # braces, hyphens, spaces, leftovers
return lc($name) || "unknown";
}
# the -s on the shebang line makes perl itself eat a *leading* -w or -a and
# set $w / $a instead, so a flag put before the identifier never reaches
# @ARGV: honour both spellings, whichever side of the id the flag is on
our ($w, $a, $h);
usage() if $h;
my $write_flag = $w ? 1 : 0;
my $wiki_flag = 0;
my $id = '';
if ($a) { $write_flag = 1; $wiki_flag = 1; }
# Parse arguments
foreach my $arg (@ARGV) {
if ($arg eq '-w') {
$write_flag = 1;
} elsif ($arg eq '-a') {
$write_flag = 1;
$wiki_flag = 1;
} elsif ($arg eq '-h' || $arg eq '--help') {
usage();
} elsif ($arg =~ /(?:arXiv:)?((?:[\w\-]+\/)?\d+(?:\.\d+)?)(?:v\d+)?/i) {
$id = $1;
}
}
if ($id eq '') {
print STDERR "Please provide me with an arXiv identifier, e.g.,\n";
print STDERR "arXiv2bib 2409.12952\n";
print STDERR "arXiv2bib -h lists the options.\n";
exit(-1);
}
# path to important substitution files
my $iso4 = "/home/laussy/bib/doi2bib/iso4";
my $bibnames = "/home/laussy/bib/doi2bib/bibnames";
my $bibfile = "/home/laussy/bib/arXiv.bib";
# make a temporary directory to store files
my $dest = "arXiv2bib-".`uniqname`;
chomp $dest;
mkdir $dest or die "Cannot create directory $dest: $!";
chdir $dest or die "Cannot change to directory $dest: $!";
# set initial DOI for arXiv
my $doi = "10.48550/arXiv.$id";
# download the bibliographic data for arXiv DOI using Crossref REST API
my $curl_cmd = "curl -s -L -H \"Accept: application/json\" -H \"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\" \"https://api.crossref.org/works/$doi\" -o from_doi.json -w \"%{http_code}\"";
my $http_status = `$curl_cmd 2>&1`;
chomp $http_status;
if ($http_status != 200 || ! -f "from_doi.json" || -s "from_doi.json" == 0) {
# Silently skip printing the error, as fallback will handle it
}
# always fetch the arXiv abstract HTML for additional info
my $html_cmd = "curl -s -A \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\" \"https://arxiv.org/abs/$id\" -o article.html 2>/dev/null";
`$html_cmd`;
if (! -f "article.html" || -s "article.html" == 0) {
print STDERR "Error: Failed to download arXiv abstract page for $id\n";
chdir "..";
system("rm -rf $dest");
exit(-1);
}
# extract published DOI if present
my $published_doi = `grep -oP '(?<=name="citation_doi" content=")[^"]+' article.html 2>/dev/null`;
chomp $published_doi;
my $reretched = 0;
if ($published_doi ne "") {
$doi = $published_doi;
$curl_cmd = "curl -s -L -H \"Accept: application/json\" -H \"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\" \"https://api.crossref.org/works/$doi\" -o from_doi.json -w \"%{http_code}\"";
$http_status = `$curl_cmd 2>&1`;
chomp $http_status;
if ($http_status == 200 && -s "from_doi.json" > 0) {
$reretched = 1;
print STDERR "Found published DOI $published_doi, using published metadata\n" if $write_flag;
} else {
print STDERR "Warning: Failed to fetch JSON for published DOI $published_doi, falling back to arXiv metadata\n" if $write_flag;
}
}
# extract journal reference if present
my $journalref = `grep -oP '(?<=Journal reference: ).*?(?=<)' article.html 2>/dev/null`;
chomp $journalref;
# generate bib key base
my $redirect = $write_flag ? "2>&1" : "2>/dev/null";
my $json_failed = 1;
my $bibauthors;
my $bibtitle;
my $bibyear;
my $bibdoi = "10.48550/arXiv.$id"; # Initialize bibdoi
my $bibkey;
if ($http_status == 200 && -f "from_doi.json" && -s "from_doi.json" > 0) {
# Check if JSON is valid
my $json_check = qx{jq -e . from_doi.json >/dev/null 2>&1; echo \$?};
chomp $json_check;
if ($json_check == 0) {
# JSON-based extraction (now using .message paths). jq only hands over
# the raw given/family pair, one author per line; the initials are cut
# in perl, by the same initials() the HTML fallback below uses.
my @rows = split /\n/, qx{<from_doi.json jq -r '.message.author[] | [(.given // ""), (.family // "")] | \@tsv' 2>/dev/null $redirect};
my @people;
foreach my $row (@rows) {
my ($given, $family) = split /\t/, $row, 2;
my $one = format_author($given, $family);
push @people, $one if $one ne "";
}
$bibauthors = tex_names(join(" and ", @people), $bibnames);
chomp $bibauthors;
if ($write_flag) {
print STDERR "Raw author output from jq: $bibauthors\n";
}
if ($bibauthors eq "" || $bibauthors eq "null") {
print STDERR "Error: Failed to extract authors for arXiv $id\n" if $write_flag;
$bibauthors = "null";
}
$bibtitle = qx{<from_doi.json jq -r '.message.title[0]' 2>/dev/null $redirect};
chomp $bibtitle;
if ($bibtitle eq "" || $bibtitle eq "null") {
print STDERR "Error: Failed to extract title for arXiv $id\n" if $write_flag;
$bibtitle = "null";
}
$bibyear = qx{<from_doi.json jq -r '.message.published."date-parts"[0][0]' 2>/dev/null $redirect};
chomp $bibyear;
if ($bibyear eq "" || $bibyear eq "null") {
print STDERR "Error: Failed to extract year for arXiv $id\n" if $write_flag;
$bibyear = "0000";
}
$bibdoi = qx{<from_doi.json jq -r '.message.DOI' 2>/dev/null $redirect};
chomp $bibdoi;
if ($bibdoi eq "" || $bibdoi eq "null") {
$bibdoi = "10.48550/arXiv.$id";
}
# Extract first author's last name from formatted bibauthors
my @authors = split / and /, $bibauthors;
$bibkey = "unknown";
if (@authors && $bibauthors ne "null") {
my $first_author = $authors[0];
if ($first_author =~ /.*?\s+([^{}\s]+|\{[^}]+\})$/) {
$bibkey = $1;
$bibkey =~ s/[{}]//g; # Remove braces for compound names
}
}
$json_failed = ($bibauthors eq "null" || $bibtitle eq "null" || $bibkey eq "unknown");
} else {
print STDERR "Error: Invalid JSON for arXiv $id\n" if $write_flag;
}
}
# HTML-based extraction as fallback
if ($json_failed) {
# Extract authors
my @authors = split /\n/, `grep -oP '(?<=name="citation_author" content=")[^"]+' article.html 2>/dev/null`;
my @formatted_authors;
my $first_family = "";
foreach my $author (@authors) {
chomp $author;
# "Papuccio-Fern\xc3\xa1ndez, I." -- and, for a mononym, no comma at all
my ($surname, $given) = $author =~ /^([^,]+),\s*(.*)$/ ? ($1, $2) : ($author, "");
$surname =~ s/^\s+|\s+$//g;
next if $surname eq "";
$first_family = $surname if $first_family eq "";
# the accents are left as they are: tex_names() escapes them below,
# with the whole of bibnames rather than the two rules that used to be
# hard-coded here (and which corrupted the key they were cut from)
push @formatted_authors, format_author($given, $surname);
}
$bibauthors = join(" and ", @formatted_authors);
if ($bibauthors eq "") {
print STDERR "Error: Failed to extract authors from HTML for arXiv $id\n" if $write_flag;
$bibauthors = "null";
} else {
$bibauthors = tex_names($bibauthors, $bibnames);
}
# Extract title
$bibtitle = `grep -oP '(?<=<meta name="citation_title" content=")[^"]+' article.html 2>/dev/null`;
chomp $bibtitle;
if ($bibtitle eq "") {
$bibtitle = `grep -oP '(?<=<h1 class="title mathjax">\\s*<span class="descriptor">Title:</span>\\s*)[^<]+' article.html 2>/dev/null`;
chomp $bibtitle;
}
if ($bibtitle eq "") {
print STDERR "Error: Failed to extract title from HTML for arXiv $id\n" if $write_flag;
$bibtitle = "null";
}
# Extract year
$bibyear = `grep -oP '(?<=<meta name="citation_date" content=")[0-9]{4}' article.html 2>/dev/null`;
chomp $bibyear;
if ($bibyear eq "") {
$bibyear = `grep -oP '(?<=<div class="submission-history">.*Submitted.*?)\\d{4}' article.html 2>/dev/null`;
chomp $bibyear;
$bibyear =~ s/.*(\d{4})$/$1/;
}
if ($bibyear eq "") {
print STDERR "Error: Failed to extract year from HTML for arXiv $id\n" if $write_flag;
$bibyear = "0000";
}
$bibkey = $first_family || "unknown";
}
# Clean bibkey for key format (transliterate accents, then keep letters only)
$bibkey = bibkey($bibkey);
# Exit if critical fields are missing
if ($bibauthors eq "null" || $bibtitle eq "null" || $bibyear eq "0000" || $bibkey eq "unknown") {
print STDERR "Error: Failed to extract valid author, title, or year for arXiv $id\n";
chdir "..";
system("rm -rf $dest");
exit(-1);
}
my $name = $bibkey;
my $bibyear_short = substr($bibyear, -2); # Use last two digits of year
my $basekey = "arXiv_" . $name;
# determine next letter
my %letters;
my $content = '';
my $existing_key = '';
my $exists = 0;
if (-f $bibfile) {
open(my $fh, '<', $bibfile) or die "Cannot open $bibfile: $!";
{ local $/; $content = <$fh>; }
close $fh;
while ($content =~ /\@(\w+)\{([^,]+),\s*(.*?)\n\s*\}/gxis) {
my $type = $1;
my $key = $2;
my $fields = $3;
if ($type !~ /string/i && ($fields =~ /url\s*=\s*\{doi:\Q$bibdoi\E\}/i)) {
$existing_key = $key;
$exists = 1;
last;
}
if ($type !~ /string/i && $key =~ /\QarXiv_$name\E(\d{2})([a-z])/i) {
my $captured_year = $1;
my $captured_letter = $2;
if ($captured_year eq $bibyear_short) {
$letters{lc $captured_letter} = 1;
}
}
}
}
# check for existing entry
if ($exists) {
print STDERR "Entry for arXiv $id already exists with key $existing_key\n";
chdir "..";
system("rm -rf $dest");
exit(0);
}
my @used_letters = sort keys %letters;
my $letter = 'a';
while (grep { $_ eq $letter } @used_letters) {
$letter = chr(ord($letter) + 1);
}
my $bibkey_final = "arXiv_" . $name . $bibyear_short . $letter;
# construct entry
my $entry = "\@Article\{$bibkey_final,\n";
$entry .= " author = {$bibauthors},\n";
$entry .= " title = {$bibtitle},\n";
$entry .= " journal = {arXiv:$id},\n";
$entry .= " year = {$bibyear},\n";
if ($journalref ne "") {
$entry .= " note = {$journalref},\n";
}
$entry .= " pdf = {arXiv/$bibkey_final},\n";
$entry .= " url = {doi:$bibdoi}\n}";
# output entry to stdout
print "$entry\n";
# output to stderr
print STDERR "\n====\n\n";
print STDERR "arXiv:$id\n";
print STDERR "$bibkey_final\n";
if ($write_flag) {
$content = $entry . "\n\n" . $content;
open(my $out, '>', $bibfile) or die "Cannot write to $bibfile: $!";
print $out $content;
close $out;
print STDERR "$bibkey_final added at the top of $bibfile\n";
}
if ($wiki_flag) {
system("bib2wiki", $bibkey_final);
}
# back to initial directory and remove temporary one
chdir "..";
system("rm -rf $dest");