Restoring execution environments of Jupyter notebooks

Jiawei Wang; Li Li; Andreas Zeller

doi:10.1109/ICSE43902.2021.00144

Restoring execution environments of Jupyter notebooks

Jiawei Wang
, Li Li
, Andreas Zeller

Department of Software Systems & Cybersecurity

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

51 Citations (Scopus)

Abstract

More than ninety percent of published Jupyternotebooks do not state dependencies on external packages. This makes them non-executable and thus hinders reproducibility of scientific results. We present SnifferDog, an approach that1) collects the APIs of Python packages and versions, creating a database of APIs; 2) analyzes notebooks to determine candidates for required packages and versions; and 3) checks which packages are required to make the notebook executable(and ideally, reproduce its stored results). In its evaluation, we show thatSnifferDogprecisely restores execution environments for the largest majority of notebooks, making them immediately executable for end users.

Original language	English
Title of host publication	Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021
Editors	Arie van Deursen, Tao Xie
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	1622-1633
Number of pages	12
ISBN (Electronic)	9780738113197
ISBN (Print)	9781665402965
DOIs	https://doi.org/10.1109/ICSE43902.2021.00144
Publication status	Published - 2021
Event	International Conference on Software Engineering 2021 - Online, Madrid, Spain Duration: 25 May 2021 → 28 May 2021 Conference number: 43rd https://conf.researchr.org/committee/icse-2021/icse-2021-organizing-committe https://conf.researchr.org/home/icse-2021 https://ieeexplore.ieee.org/xpl/conhome/9401807/proceeding (Proceedings)

Publication series

Name	Proceedings - International Conference on Software Engineering
Publisher	The Institute of Electrical and Electronics Engineers, Inc.
ISSN (Print)	0270-5257
ISSN (Electronic)	1558-1225

Conference

Conference	International Conference on Software Engineering 2021
Abbreviated title	ICSE 2021
Country/Territory	Spain
City	Madrid
Period	25/05/21 → 28/05/21
Internet address	https://conf.researchr.org/committee/icse-2021/icse-2021-organizing-committe https://conf.researchr.org/home/icse-2021 https://ieeexplore.ieee.org/xpl/conhome/9401807/proceeding (Proceedings)

Keywords

API
Environment
Jupyter Notebook
Python

Access to Document

10.1109/ICSE43902.2021.00144

Cite this

Wang, J., Li, L., & Zeller, A. (2021). Restoring execution environments of Jupyter notebooks. In A. van Deursen, & T. Xie (Eds.), Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021 (pp. 1622-1633). (Proceedings - International Conference on Software Engineering). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICSE43902.2021.00144

Wang, Jiawei ; Li, Li ; Zeller, Andreas. / Restoring execution environments of Jupyter notebooks. Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021. editor / Arie van Deursen ; Tao Xie. Piscataway NJ USA : IEEE, Institute of Electrical and Electronics Engineers, 2021. pp. 1622-1633 (Proceedings - International Conference on Software Engineering).

@inproceedings{04052ef900634498bf56fb55ca8b8236,

title = "Restoring execution environments of Jupyter notebooks",

abstract = "More than ninety percent of published Jupyternotebooks do not state dependencies on external packages. This makes them non-executable and thus hinders reproducibility of scientific results. We present SnifferDog, an approach that1) collects the APIs of Python packages and versions, creating a database of APIs; 2) analyzes notebooks to determine candidates for required packages and versions; and 3) checks which packages are required to make the notebook executable(and ideally, reproduce its stored results). In its evaluation, we show thatSnifferDogprecisely restores execution environments for the largest majority of notebooks, making them immediately executable for end users. ",

keywords = "API, Environment, Jupyter Notebook, Python",

author = "Jiawei Wang and Li Li and Andreas Zeller",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; International Conference on Software Engineering 2021, ICSE 2021 ; Conference date: 25-05-2021 Through 28-05-2021",

year = "2021",

doi = "10.1109/ICSE43902.2021.00144",

language = "English",

isbn = "9781665402965",

series = "Proceedings - International Conference on Software Engineering",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

pages = "1622--1633",

editor = "\{van Deursen\}, Arie and Xie, \{Tao \}",

booktitle = "Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021",

address = "United States of America",

url = "https://conf.researchr.org/committee/icse-2021/icse-2021-organizing-committe, https://conf.researchr.org/home/icse-2021, https://ieeexplore.ieee.org/xpl/conhome/9401807/proceeding",

}

Wang, J, Li, L & Zeller, A 2021, Restoring execution environments of Jupyter notebooks. in A van Deursen & T Xie (eds), Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021. Proceedings - International Conference on Software Engineering, IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 1622-1633, International Conference on Software Engineering 2021, Madrid, Spain, 25/05/21. https://doi.org/10.1109/ICSE43902.2021.00144

Restoring execution environments of Jupyter notebooks. / Wang, Jiawei; Li, Li; Zeller, Andreas.
Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021. ed. / Arie van Deursen; Tao Xie. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2021. p. 1622-1633 (Proceedings - International Conference on Software Engineering).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Restoring execution environments of Jupyter notebooks

AU - Wang, Jiawei

AU - Li, Li

AU - Zeller, Andreas

N1 - Conference code: 43rd

PY - 2021

Y1 - 2021

N2 - More than ninety percent of published Jupyternotebooks do not state dependencies on external packages. This makes them non-executable and thus hinders reproducibility of scientific results. We present SnifferDog, an approach that1) collects the APIs of Python packages and versions, creating a database of APIs; 2) analyzes notebooks to determine candidates for required packages and versions; and 3) checks which packages are required to make the notebook executable(and ideally, reproduce its stored results). In its evaluation, we show thatSnifferDogprecisely restores execution environments for the largest majority of notebooks, making them immediately executable for end users.

AB - More than ninety percent of published Jupyternotebooks do not state dependencies on external packages. This makes them non-executable and thus hinders reproducibility of scientific results. We present SnifferDog, an approach that1) collects the APIs of Python packages and versions, creating a database of APIs; 2) analyzes notebooks to determine candidates for required packages and versions; and 3) checks which packages are required to make the notebook executable(and ideally, reproduce its stored results). In its evaluation, we show thatSnifferDogprecisely restores execution environments for the largest majority of notebooks, making them immediately executable for end users.

KW - API

KW - Environment

KW - Jupyter Notebook

KW - Python

UR - https://www.scopus.com/pages/publications/85104223345

U2 - 10.1109/ICSE43902.2021.00144

DO - 10.1109/ICSE43902.2021.00144

M3 - Conference Paper

AN - SCOPUS:85104223345

SN - 9781665402965

T3 - Proceedings - International Conference on Software Engineering

SP - 1622

EP - 1633

BT - Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021

A2 - van Deursen, Arie

A2 - Xie, Tao

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

T2 - International Conference on Software Engineering 2021

Y2 - 25 May 2021 through 28 May 2021

ER -

Wang J, Li L, Zeller A. Restoring execution environments of Jupyter notebooks. In van Deursen A, Xie T, editors, Proceedings - 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE 2021. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers. 2021. p. 1622-1633. (Proceedings - International Conference on Software Engineering). doi: 10.1109/ICSE43902.2021.00144

Restoring execution environments of Jupyter notebooks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Cite this