This presentation from Joel Grus on the problems inherent with Jupyter notebooks is great.
I’ve experienced a lot of these problems first-hand, especially issues related to running notebook cells out of order, code organization, and Jupyter notebooks’ poor ergonomics compared any modern code editor. My rule of thumb: anything that needs tests or code review should be moved out of a notebook into a .py file. Same for anything producing data for external consumption or used in making a decision. In other words, Jupyter notebooks are for exploration only.
As a counterpoint, Netflix has two posts on how they’re using Jupyter notebooks extensively. They have had to throw engineering resources at some of the problems described by Grus to make Jupyter notebooks viable for them, however.
Just like any tool, Jupyter notebooks have pros and cons. We run into problems when the low friction (one of the pros!) masks the downsides of working extensively in notebooks, especially when downsides like poor reproducibility are only evident later.
