Show HN: The scholarref tools: never deal with journal webpages again, Hacker News

Rationale

During the writing phase of an academic paper, common tasks include downloading PDFs of publications and getting their references into your bibliography. However, I am not a fan of navigating the slow, bloated, tracker-filled, and distracting webpages of academic journals and publication aggregators. For some reason, many publishers decided that clicking the “Download PDF” link should redirect the user to an unusable in-browser PDF viewer instead of providing the PDF file directly. While the majority of journal webpages provide formatted citations for their publications, these are inconsistent in style and content.

For these reasons, I constructed a set of shell tools calledscholarrefthat allow me to perform most of the tasks without having to open a web browser. As the title of this post indicates, the goal of the toolset is to provide as much functionality a person might need during scientific writing without leaving the command line. The tools are undercontinuous development. At present I avoid roughly 90% of visits to journal webpages. I hope to get to 100% someday.

Thescholarrefdesign goals are the following:

• Written as POSIX shell scripts with minimal external dependencies: Ensures maximum flexibility and portability.
• Aim for simplicity: Fewer lines of code make the programs easier to understand, maintain, and debug.
• Each tool should do one thing, and do it well: Let the users piece the components together to fit their workflow.
• Return references in BibTeX format.

DISCLAIMER:The functionality provided by these programs depends on communication with third party webpages, which may or may not be permitted by law and the terms of service upheld by the third parties. What is demonstrated here are examples only. Use of the tools is entirely your own responsibility.

Installation

$ git clone git: // src. adamsgaard.dk/scholarref $ cd scholarref # make install

Themake installcommand may require superuser priviledges to install the tools to/ usr / local. Prefix withdoasorsudo, whatever is appropriate for the target system.

The scholarref toolset

The core functionality is provided by the scriptsgetdoi,getref, andshdl. All programs accept input as command-line arguments or from standard input (stdin). The programs come with several OPTIONS, and it is encouraged to explore the help text (invoke with option– h). The– toption may be of particular interest, since it tunnels all communication throughTorviatorsocks(if available on the system).

Getdoi

This tool accepts either names of PDF files or arbitrary search queries. If a PDF file name is supplied,getdoiscans the PDF text in order to find the first occuring DOI entry, which typically is the DOI of the publication itself. If an arbitrary query is supplied, theCrossRef APIis used to find the DOI of the closest publication match. You can supply author names, parts of the title, ORCID, journal name, etc. Examples:

$ getdoi damsgaard 2018 .pdf 10. 1029 /  (ms)  $ getdoi 'damsgaard sergienko adcroft journal advances modeling earth systems' 10. 1029 /  (ms)

Getref

Thegetreftool fetches the BibTeX citation for a given DOI fromdoi.org. By default, the journal names and author first names are abbreviated, which is what most journals want. I have taken most abbreviations from theCaltech Library list of Journal Title Abbreviations. Thegetrefruleset of journal-title abbreviations is incomplete, and is expanded on a per-need basis. If desired, the abbreviation functionality can be disabled. Seegetref -hfor details.

$ getref 10. 1029 / 2018 ms 001299 @article {Damsgaard 2018,         doi={10. 1029 /  (ms) },         year=2018,         publisher={American Geophysical Union ({AGU})},         volume={10},         number={9},         pages={2228 - 2244},         author={A. Damsgaard and A. Adcroft and O. Sergienko},         title={Application of Discrete Element Methods to Approximate Sea Ice Dynamics},         journal={J. Adv. Mod. Earth Sys.} } $ getref -j 10. 1029 /  (ms)  # do not abbreviate journal title @article {Damsgaard 2018,         doi={10. 1029 /  (ms) },         year=2018,         publisher={American Geophysical Union ({AGU})},         volume={10},         number={9},         pages={2228 - 2244},         author={A. Damsgaard and A. Adcroft and O. Sergienko},         title={Application of Discrete Element Methods to Approximate Sea Ice Dynamics},         journal={Journal of Advances in Modeling Earth Systems} }

SHDL

This tool takes a DOI as input and attempts to download the corresponding publication as a PDF throughsci-hub. Unfortunately, the sci-hub web interface often puts up captias to restrict automated downloads. If that’s the case,shdlopens the tor browser (if installed) or the system web browser in order to manually complete the download. Output PDF files are saved in the present working directory.

Usage examples

Thescholarreftools are meant to be chained together. For example, if you want a BibTeX reference a search query, simply use UNIX pipes to send thegetdoioutput as input togetref:

$ getdoi 'damsgaard egholm ice flow dynamics' | getref @article {Damsgaard 2016,         doi={10. 1002 / 2016 GL 071579},         year=2016,         publisher={American Geophysical Union ({AGU})},         volume={43},         number={23},         pages={12, 165 - 12, 173},         author={A. Damsgaard and D. L. Egholm and L. H. Beem and S. Tulaczyk and N. K. Larsen and J. A. Piotrowski and M. R. Siegfried},         title={Ice flow dynamics forced by water pressure variations in subglacial granular beds},         journal={Geophys. Res. Lett.} }

Thescholarrefprogram itself is an aggregation of thegetdoiandgetrefcommands. If called with the– aoption, the reference is directly inserted into the system bibliography. The full path to the bibliography file (.bib) is assumed to be set in the$ BIBenvironment variable, for instance defined in the user~ / .profile.

$ echo $ BIB /home/ad/articles/own/BIBnew.bib $ scholarref -a 'damsgaard egholm ice flow dynamics' Citation Damsgaard 2016 added to /home/ad/articles/own/BIBnew.bib

Integrating into your favorite $ EDITOR

Thescholarreftool is particularly useful if called from within a text editor. Below I demonstrate how keyboard bindings can be bound in various editors to provide scholarref functionality.

vi

My editor of choice is the plain, old, and simple (vi) 1). I have the following binding in my~ / .exrc, including a trailing space:

map qr: r! scholarref

The rest of my editor configuration can be found under mydotfiles source code repository.

VIM

You can add the following bindings to~ / .vimrcor~ / .vim / vimrcin order to get scholarref functionality withinvim (1):

nnoremapR: R! Scholarref"insert reference into current buffer nnoremapR: r! scholarref --add"append reference into $ BIB file

vis

Thevis (1)editor is an interesting combination of modal editing and structural regular expressions from the plan9 editorsam (1). However, after using it exclusively for three months, I concluded that it is too immature for general use. If desired, add the following binding to~ / .config / vis / visrc.lua:

vis: map (vis.modes.NORMAL , leader .. 'r', ':

emacs

Don’t know, figure it out yourself.

Integrating into your pdf viewer

My PDF viewer of choice iszathura (1), which has a minimal graphical user interface and is keyboard-centric. The following configuration callsgetdoion the currently open file if I pressCtrl-i. The resultant DOI is copied to the clipboard. Similarly,Ctrl-stries to extract the DOI in the same manner, but fetches the accompanying reference and adds it directly to the bibliography.

mapfeedkeys ": exec getdoi --notify --clip '$ FILE'" mapfeedkeys ": exec scholarref --add '$ FILE'"

My full zathura configuration is availablehere.

Questions / bugs / feedback / improvements

Pleaseget in touchif you encounter any. Improvement suggestions are best sent as patches by e-mail.