This R Markdown Notebook demonstrates how to use the paperfetcher Python package to perform handsearching in R through the reticulate interface to Python.
To execute a code chunk in RStudio, click the Run button within the chunk or place your cursor inside it and press Cmd+Shift+Enter.
We first need to run a couple of lines of code to install reticulate, Python, and paperfetcher.
# Install reticulate if not already installed
if(!require("reticulate")){
install.packages("reticulate")
library(reticulate)
}Loading required package: reticulate# Install Python if not already installed, and create a new virtualenv for paperfetcher
# (Uncomment the lines below to run code)
#install_python("3.7:latest")
#virtualenv_create("paperfetcher", version="3.7:latest")
#use_virtualenv("paperfetcher")
# Install paperfetcher
# (Uncomment the lines below to run code)
#py_install("paperfetcher", envname="paperfetcher")
# Import the paperfetcher package
paperfetcher <- import("paperfetcher")Let’s perform a simple task: to search for all journal articles published in the journal Science between January 01, 2023 and January 31, 2023.
A quick Google search reveals that the ISSN for the online edition of Science is 1095-9203.
Now let’s use this information to create a search object:
search <- paperfetcher$handsearch$CrossrefSearch(ISSN="1095-9203", from_date="2023-01-01", until_date="2023-02-01")
search()2023-02-12 21:44:36.343 INFO paperfetcher.handsearch: Fetching 144 works.
Fetching 8 batches of 20 articles: 0%| | 0/8 [00:00<?, ?it/s][A
Fetching 8 batches of 20 articles: 12%|█▎ | 1/8 [00:06<00:48, 7.00s/it][A
Fetching 8 batches of 20 articles: 25%|██▌ | 2/8 [00:32<01:48, 18.08s/it][A
Fetching 8 batches of 20 articles: 38%|███▊ | 3/8 [00:42<01:10, 14.13s/it][A
Fetching 8 batches of 20 articles: 50%|█████ | 4/8 [00:45<00:39, 9.99s/it][A
Fetching 8 batches of 20 articles: 62%|██████▎ | 5/8 [00:48<00:22, 7.34s/it][A
Fetching 8 batches of 20 articles: 75%|███████▌ | 6/8 [00:58<00:16, 8.12s/it][A
Fetching 8 batches of 20 articles: 88%|████████▊ | 7/8 [01:11<00:09, 9.85s/it][A
Fetching 8 batches of 20 articles: 100%|██████████| 8/8 [01:21<00:00, 9.90s/it][A
Fetching 8 batches of 20 articles: 100%|██████████| 8/8 [01:21<00:00, 10.20s/it]How many articles did our search return?
py_len(search)[1] 144This was rather slow… Can we speed this up?
Yes, we can!
Why was the search so slow?
Paperfetcher retrieved all the metadata available on Crossref for each paper. Each paper can have a lot of metadata (abstract, citations, keywords, funding information, etc.) deposited on Crossref, and retrieving all this informating can take a lot of time (and also, memory!).
How do we make it faster?
By retrieving only the metadata we need!
For example, let’s say we need DOI, URL, article title, author list,
and publication date. As per the Crossref API, these fields are:
DOI, URL, title,
author, and issued. We can ask paperfetcher to
only select these fields using the ‘select_fields’ parameter:
search <- paperfetcher$handsearch$CrossrefSearch(ISSN="1095-9203", from_date="2023-01-01", until_date="2023-02-01")
search(select=TRUE, select_fields=list('DOI', 'URL', 'title', 'author', 'issued', 'abstract'))2023-02-12 21:46:19.861 INFO paperfetcher.handsearch: Fetching 144 works.
Fetching 8 batches of 20 articles: 0%| | 0/8 [00:00<?, ?it/s][A
Fetching 8 batches of 20 articles: 12%|█▎ | 1/8 [00:07<00:52, 7.48s/it][A
Fetching 8 batches of 20 articles: 25%|██▌ | 2/8 [00:12<00:35, 5.96s/it][A
Fetching 8 batches of 20 articles: 38%|███▊ | 3/8 [00:20<00:35, 7.08s/it][A
Fetching 8 batches of 20 articles: 50%|█████ | 4/8 [00:22<00:19, 4.81s/it][A
Fetching 8 batches of 20 articles: 62%|██████▎ | 5/8 [00:31<00:19, 6.54s/it][A
Fetching 8 batches of 20 articles: 75%|███████▌ | 6/8 [00:36<00:11, 5.89s/it][A
Fetching 8 batches of 20 articles: 88%|████████▊ | 7/8 [00:49<00:08, 8.41s/it][A
Fetching 8 batches of 20 articles: 100%|██████████| 8/8 [01:09<00:00, 11.87s/it][A
Fetching 8 batches of 20 articles: 100%|██████████| 8/8 [01:09<00:00, 8.65s/it]Filtering articles by keywords
We can also pass a list of keywords to paperfetcher to refine our search:
search <- paperfetcher$handsearch$CrossrefSearch(ISSN="1095-9203", from_date="2022-01-01", until_date="2023-01-01", keyword_list=list("COVID"))
search(select=TRUE, select_fields=list('DOI', 'URL', 'title', 'author', 'issued', 'abstract'))2023-02-12 21:49:08.712 INFO paperfetcher.handsearch: Fetching 33 works.
Fetching 2 batches of 20 articles: 0%| | 0/2 [00:00<?, ?it/s][A
Fetching 2 batches of 20 articles: 50%|█████ | 1/2 [00:36<00:36, 36.15s/it][A
Fetching 2 batches of 20 articles: 100%|██████████| 2/2 [00:58<00:00, 27.90s/it][A
Fetching 2 batches of 20 articles: 100%|██████████| 2/2 [00:58<00:00, 29.14s/it]paperfetcher provides many different ways to access the search result data, using special data structures called Datasets.
For example, we can make a Dataset of DOIs from the search results:
doi_ds <- search$get_DOIDataset()We can display this as a DataFrame:
doi_ds$to_df()We can also make a Dataset containing all the fields we retrieved, and display this as a DataFrame:
parsers <- import("paperfetcher.parsers", convert=FALSE)
ds <- search$get_CitationsDataset(field_list=list('DOI', 'URL', 'title', 'author', 'issued'),
field_parsers_list=list(NULL, NULL, parsers$crossref_title_parser,
parsers$crossref_authors_parser,
parsers$crossref_date_parser))
ds$to_df()We can save the search results to
ds$save_txt("out/handsearching_citations.txt")ds$save_csv("out/handsearching_citations.csv")ds$save_excel("out/handsearching_citations.xslx")Citation data stored in the RIS (Research Information Systems) file format can easily be imported into systematic review screening tools (such as Covidence) and citation management software (such as Zotero). Paperfetcher can export search results to RIS files. Let’s take a look:
Exporting to RIS format without abstracts
Paperfetcher uses Crossref’s content negotiation service to get RIS data for each DOI. Unfortunately, this does not contain abstracts. However, there is a workaround, which we’ll get to in a bit.
First, let’s see how to export data to RIS format without abstracts:
ds <- search$get_RISDataset()
Converting results to RIS format.: 0%| | 0/33 [00:00<?, ?it/s][A
Converting results to RIS format.: 3%|▎ | 1/33 [00:00<00:08, 3.64it/s][A
Converting results to RIS format.: 6%|▌ | 2/33 [00:00<00:07, 3.93it/s][A
Converting results to RIS format.: 9%|▉ | 3/33 [00:00<00:06, 4.29it/s][A
Converting results to RIS format.: 12%|█▏ | 4/33 [00:00<00:06, 4.57it/s][A
Converting results to RIS format.: 15%|█▌ | 5/33 [00:01<00:06, 4.31it/s][A
Converting results to RIS format.: 18%|█▊ | 6/33 [00:01<00:05, 4.62it/s][A
Converting results to RIS format.: 21%|██ | 7/33 [00:01<00:05, 4.66it/s][A
Converting results to RIS format.: 24%|██▍ | 8/33 [00:01<00:05, 4.53it/s][A
Converting results to RIS format.: 27%|██▋ | 9/33 [00:02<00:05, 4.11it/s][A
Converting results to RIS format.: 30%|███ | 10/33 [00:02<00:05, 4.29it/s][A
Converting results to RIS format.: 33%|███▎ | 11/33 [00:02<00:05, 4.23it/s][A
Converting results to RIS format.: 36%|███▋ | 12/33 [00:02<00:04, 4.51it/s][A
Converting results to RIS format.: 39%|███▉ | 13/33 [00:02<00:04, 4.69it/s][A
Converting results to RIS format.: 42%|████▏ | 14/33 [00:03<00:04, 4.67it/s][A
Converting results to RIS format.: 45%|████▌ | 15/33 [00:03<00:03, 4.70it/s][A
Converting results to RIS format.: 48%|████▊ | 16/33 [00:03<00:03, 4.40it/s][A
Converting results to RIS format.: 52%|█████▏ | 17/33 [00:03<00:03, 4.59it/s][A
Converting results to RIS format.: 55%|█████▍ | 18/33 [00:04<00:03, 4.66it/s][A
Converting results to RIS format.: 58%|█████▊ | 19/33 [00:04<00:03, 4.45it/s][A
Converting results to RIS format.: 61%|██████ | 20/33 [00:04<00:02, 4.56it/s][A
Converting results to RIS format.: 64%|██████▎ | 21/33 [00:04<00:02, 4.81it/s][A
Converting results to RIS format.: 67%|██████▋ | 22/33 [00:04<00:02, 4.93it/s][A
Converting results to RIS format.: 70%|██████▉ | 23/33 [00:05<00:02, 4.65it/s][A
Converting results to RIS format.: 73%|███████▎ | 24/33 [00:05<00:01, 4.73it/s][A
Converting results to RIS format.: 76%|███████▌ | 25/33 [00:05<00:01, 4.71it/s][A
Converting results to RIS format.: 79%|███████▉ | 26/33 [00:05<00:01, 4.67it/s][A
Converting results to RIS format.: 82%|████████▏ | 27/33 [00:05<00:01, 4.81it/s][A
Converting results to RIS format.: 85%|████████▍ | 28/33 [00:06<00:01, 4.82it/s][A
Converting results to RIS format.: 88%|████████▊ | 29/33 [00:06<00:00, 5.00it/s][A
Converting results to RIS format.: 91%|█████████ | 30/33 [00:06<00:00, 4.86it/s][A
Converting results to RIS format.: 94%|█████████▍| 31/33 [00:06<00:00, 4.95it/s][A
Converting results to RIS format.: 97%|█████████▋| 32/33 [00:06<00:00, 5.03it/s][A
Converting results to RIS format.: 100%|██████████| 33/33 [00:07<00:00, 4.84it/s][A
Converting results to RIS format.: 100%|██████████| 33/33 [00:07<00:00, 4.62it/s]ds$save_ris("out/handsearching.ris")Exporting to RIS format with abstracts
Recall that we have already retrieved abstracts during our search. We can insert these abstracts as an extra field into the RIS dataset. Here’s how:
ds <- search$get_RISDataset(extra_field_list=list("abstract"),
extra_field_parser_list=list(NULL),
extra_field_rispy_tags=list("notes_abstract"))
Converting results to RIS format.: 0%| | 0/33 [00:00<?, ?it/s][A
Converting results to RIS format.: 3%|▎ | 1/33 [00:00<00:07, 4.45it/s][A
Converting results to RIS format.: 6%|▌ | 2/33 [00:00<00:06, 4.63it/s][A
Converting results to RIS format.: 9%|▉ | 3/33 [00:00<00:06, 4.97it/s][A
Converting results to RIS format.: 12%|█▏ | 4/33 [00:00<00:06, 4.45it/s][A
Converting results to RIS format.: 15%|█▌ | 5/33 [00:01<00:06, 4.55it/s][A
Converting results to RIS format.: 18%|█▊ | 6/33 [00:01<00:06, 4.48it/s][A
Converting results to RIS format.: 21%|██ | 7/33 [00:01<00:05, 4.71it/s][A
Converting results to RIS format.: 24%|██▍ | 8/33 [00:01<00:05, 4.91it/s][A
Converting results to RIS format.: 27%|██▋ | 9/33 [00:01<00:04, 4.93it/s][A
Converting results to RIS format.: 30%|███ | 10/33 [00:02<00:04, 4.94it/s][A
Converting results to RIS format.: 33%|███▎ | 11/33 [00:02<00:04, 4.63it/s][A
Converting results to RIS format.: 36%|███▋ | 12/33 [00:02<00:04, 4.83it/s][A
Converting results to RIS format.: 39%|███▉ | 13/33 [00:02<00:04, 4.91it/s][A
Converting results to RIS format.: 42%|████▏ | 14/33 [00:02<00:03, 4.92it/s][A
Converting results to RIS format.: 45%|████▌ | 15/33 [00:03<00:03, 4.97it/s][A
Converting results to RIS format.: 48%|████▊ | 16/33 [00:03<00:03, 5.11it/s][A
Converting results to RIS format.: 52%|█████▏ | 17/33 [00:03<00:03, 4.95it/s][A
Converting results to RIS format.: 55%|█████▍ | 18/33 [00:03<00:03, 4.67it/s][A
Converting results to RIS format.: 58%|█████▊ | 19/33 [00:04<00:03, 4.33it/s][A
Converting results to RIS format.: 61%|██████ | 20/33 [00:04<00:03, 4.29it/s][A
Converting results to RIS format.: 64%|██████▎ | 21/33 [00:04<00:02, 4.54it/s][A
Converting results to RIS format.: 67%|██████▋ | 22/33 [00:04<00:02, 4.55it/s][A
Converting results to RIS format.: 70%|██████▉ | 23/33 [00:04<00:02, 4.26it/s][A
Converting results to RIS format.: 73%|███████▎ | 24/33 [00:05<00:02, 4.23it/s][A
Converting results to RIS format.: 76%|███████▌ | 25/33 [00:05<00:01, 4.17it/s][A
Converting results to RIS format.: 79%|███████▉ | 26/33 [00:05<00:01, 4.15it/s][A
Converting results to RIS format.: 82%|████████▏ | 27/33 [00:05<00:01, 4.14it/s][A
Converting results to RIS format.: 85%|████████▍ | 28/33 [00:06<00:01, 4.31it/s][A
Converting results to RIS format.: 88%|████████▊ | 29/33 [00:06<00:00, 4.63it/s][A
Converting results to RIS format.: 91%|█████████ | 30/33 [00:06<00:00, 4.37it/s][A
Converting results to RIS format.: 94%|█████████▍| 31/33 [00:06<00:00, 4.47it/s][A
Converting results to RIS format.: 97%|█████████▋| 32/33 [00:07<00:00, 4.43it/s][A
Converting results to RIS format.: 100%|██████████| 33/33 [00:07<00:00, 3.99it/s][A
Converting results to RIS format.: 100%|██████████| 33/33 [00:07<00:00, 4.50it/s]ds$save_ris("out/handsearching_abstracts.ris")