An Introduction to Pluto(lwn.net)
lwn.net
An Introduction to Pluto
https://lwn.net/SubscriberLink/835930/47c363bef07134a4/
27 comments
Besides language design, existence (and starting adoption) of such autonomous (in this case from Jupyter) projects in Julia makes me enthusiastic about this language future.
Sure, Python is not going anywhere (it is a nice language after all) but it wasn't made with scientific computations in mind and it shows. I think sooner or later it will make way for Julia in that domain (and hopefully also some others).
Sure, Python is not going anywhere (it is a nice language after all) but it wasn't made with scientific computations in mind and it shows. I think sooner or later it will make way for Julia in that domain (and hopefully also some others).
It is linked in the article, but I would like to draw attention to the wonderful JuliaCon talk presenting Pluto:
https://www.youtube.com/watch?v=IAF8DjrQSSk
https://www.youtube.com/watch?v=IAF8DjrQSSk
Pluto is like Jupyter, but with no hidden global state, and it’s git friendly. It’s specialized for Julia.
Ironically, it actually has more global state than Jupyter. The way that they make it feel like it doesn't have state is by adding more state. It's pretty clever actually.
I think the key is that the state is not hidden: what you see is what you get.
There is indeed hidden state in a Pluto notebook. Look at the .jl file it generates, there is hidden state there describing the cell ordering.
But those are comments (non-executable) that keep track of the order in which the cells are presented in the notebook. There is no hidden state relevant to the computation.
Those comments control how information is presented to you and are state from the point of view of the notebook. However, the comments are not the only hidden state in pluto.
Beyond that, Pluto has a huge amount of state that's hidden from you in that it tries to keep track of all the variables you define and where so that it can control which cells need to be re-run when you change something. This lets it avoid re-running all the cells when you make a slight adjustment.
However, yes I agree. As I originally said, they added state that makes it work more like it doesn't have state. Or in other words, they solved Jupyter's state problem by adding more state.
Beyond that, Pluto has a huge amount of state that's hidden from you in that it tries to keep track of all the variables you define and where so that it can control which cells need to be re-run when you change something. This lets it avoid re-running all the cells when you make a slight adjustment.
However, yes I agree. As I originally said, they added state that makes it work more like it doesn't have state. Or in other words, they solved Jupyter's state problem by adding more state.
You are talking about state relevant to the implementation. OP was talking about state from a conceptual point of view. To the user there is no state. The user does not need to keep track of what they evaluated and in what order as in Jupytr. That is a kind of state that exits conceptually and which is very visible to the user.
That's what I've been saying in this thread the entire time. It adds extra state so that it feels like there's less state.
I disagree on your second paragraph. It's literally true but that state, again, has no bearing on the actual result of the computation. It's just an optimization.
There is no hidden state in the editor that can change the values between runs.
There is no hidden state in the editor that can change the values between runs.
It may be that we are using the word in different senses, but I would say that Pluto has no hidden state whatsoever. The comments for cell ordering control what you see, so they are the opposite of hidden state. The execution order is determined by a static analysis of the dependencies among your code cells, and is determined only by what you see in those cells. There is no hidden state anywhere.
That's to make sure the generated .jl file executes correctly (since that does still run in a linear fashion). It doesn't correspond to the cells index; moving a cell does not change the exported .jl file.
Does this mean that if we change the .jl file in an editor it becomes unreadable by Pluto? That would contradict the claim in the article that Pluto files are just regular julia files.
If you change it the .jl file in a editor you could quite easily make it unreadable to pluto.
> That would contradict the claim in the article that Pluto files are just regular julia files.
Not really, no. The files pluto generates are a subset of valid julia files. All Pluto files are valid julia files; not all valid julia files are valid Pluto files.
> That would contradict the claim in the article that Pluto files are just regular julia files.
Not really, no. The files pluto generates are a subset of valid julia files. All Pluto files are valid julia files; not all valid julia files are valid Pluto files.
There is another Julia package called Revise.jl : https://github.com/timholy/Revise.jl
When loading Revise.jl in a Julia REPL session, it keeps track of all packages loaded afterwards and if the code is changed in one of the packages it automatically updates the functions provided by that package. As far as I know this does not work (yet) for structures that have been changed though.
The cool thing is now, that it seems as if one can also use Revise.jl in a Pluto notebook - when the code of a package loaded in a Pluto notebook is changed, then also the cells in that notebook are run again with the updated code.
When loading Revise.jl in a Julia REPL session, it keeps track of all packages loaded afterwards and if the code is changed in one of the packages it automatically updates the functions provided by that package. As far as I know this does not work (yet) for structures that have been changed though.
The cool thing is now, that it seems as if one can also use Revise.jl in a Pluto notebook - when the code of a package loaded in a Pluto notebook is changed, then also the cells in that notebook are run again with the updated code.
Really cool since I often migrate code from notebooks to modules.
> Much of Pluto's interactivity and visualization is based on Observable, an interactive notebook for JavaScript, which the Pluto authors credit for being their inspiration.
Observable (https://observablehq.com/) is a spiritual successor to Mike Bostock's (d3) https://bl.ocks.org/ that's very useful for data visualization or just general JS-based data exploration.
Observable (https://observablehq.com/) is a spiritual successor to Mike Bostock's (d3) https://bl.ocks.org/ that's very useful for data visualization or just general JS-based data exploration.
In case you work with Jupyter, there's nbdime[0] for better integration with git.
Here is how we do things (in the context of machine learning as a team working together on a project using our platform[1]. This may not be relevant for solo/portfolio projects with few or no expected deliverables):
We use notebooks interactively just to get something we can schedule as a long-running notebook. We don't run training jobs in interactive mode: we schedule the notebook and it's pretty much fire and forget and no hidden state. It runs against a Docker image, and you can close your browser if you want at that point, we have both the computation and the output as it runs. Or you can keep tweaking the notebook, schedule that tweak, and keep on tweaking. We do that to decouple experiments from human time. Users can publish AppBooks, and then try different params using an automatically generated form from the notebooks so we can fiddle with params in a form without opening the notebook. We do that to be able to try things from a form, as in on our phone or allowing a domain expert to train models without the whole notebook experience overhead. This is particularly useful when we use features that the domain expert wants to be able to tweak. We just expose them and let them do their thing.
We do this to track experiments as we automatically detect models, parameters, and metrics without having to write tracking/saving code. All this allows us to have ready models for deployment, which we can do easily. We do that to allow people to deploy their own models instead of asking someone, and be able to use, and let others use the models simply by sending HTTP requests or using a form, instead of bothering with models and their dependencies.
For the collaboration aspect, we use near real-time collaboration to help someone troubleshoot or go over code with someone, and then commit changes. We automatically save checkpoints so people can go back and forth.
We mainly use Python and we built around Jupyter. We took a look at other notebooks but, in our experience, stylesheets aren't what held our machine learning projects back in the past and we decided to keep using Jupyter adding functionality as "natively" as possible. More embarrassingly, we didn't come from a front-end background but from an ML background, and built this around our workflow to enable us to execute projects.
- [0]: https://github.com/jupyter/nbdime
- [1]: https://iko.ai
Here is how we do things (in the context of machine learning as a team working together on a project using our platform[1]. This may not be relevant for solo/portfolio projects with few or no expected deliverables):
We use notebooks interactively just to get something we can schedule as a long-running notebook. We don't run training jobs in interactive mode: we schedule the notebook and it's pretty much fire and forget and no hidden state. It runs against a Docker image, and you can close your browser if you want at that point, we have both the computation and the output as it runs. Or you can keep tweaking the notebook, schedule that tweak, and keep on tweaking. We do that to decouple experiments from human time. Users can publish AppBooks, and then try different params using an automatically generated form from the notebooks so we can fiddle with params in a form without opening the notebook. We do that to be able to try things from a form, as in on our phone or allowing a domain expert to train models without the whole notebook experience overhead. This is particularly useful when we use features that the domain expert wants to be able to tweak. We just expose them and let them do their thing.
We do this to track experiments as we automatically detect models, parameters, and metrics without having to write tracking/saving code. All this allows us to have ready models for deployment, which we can do easily. We do that to allow people to deploy their own models instead of asking someone, and be able to use, and let others use the models simply by sending HTTP requests or using a form, instead of bothering with models and their dependencies.
For the collaboration aspect, we use near real-time collaboration to help someone troubleshoot or go over code with someone, and then commit changes. We automatically save checkpoints so people can go back and forth.
We mainly use Python and we built around Jupyter. We took a look at other notebooks but, in our experience, stylesheets aren't what held our machine learning projects back in the past and we decided to keep using Jupyter adding functionality as "natively" as possible. More embarrassingly, we didn't come from a front-end background but from an ML background, and built this around our workflow to enable us to execute projects.
- [0]: https://github.com/jupyter/nbdime
- [1]: https://iko.ai
Would love to see Pluto for Python. This fixes key annoyances I have with Jupyther.
And considering the origin of the name Jupyter (Ju:Julia, pyt:Python, er:R) I really thought that Pluto, starting with a P, would be for Python.
P:Python, lu:Lua, to:Go as prounounced by a baby - given Pluto's size...
In many cases, Pluto's execution model would be prohibitively obscure (despite being 'pure'). E.g., at work I quite often have 100gb-ram Jupyter notebook kernels, where running one extra command or running commands in a wrong order would simply cause the box to crash.
"Pluto is a new computational notebook for the Julia programming language."
The first sentence of the article, excellent. Many times, something is mentioned here on HN and even after clicking sound on a number of with pages, I still don't know what they are talking about. This look at Pluto is exactly what it says: a good introduction to a Jupyter system for the Julia programming language.
Maybe the title of this HN post could change to say that?
The first sentence of the article, excellent. Many times, something is mentioned here on HN and even after clicking sound on a number of with pages, I still don't know what they are talking about. This look at Pluto is exactly what it says: a good introduction to a Jupyter system for the Julia programming language.
Maybe the title of this HN post could change to say that?
> [...] a good introduction to a Jupyter system for the Julia programming language.
To clarify: it is not Jupyter system for Julia programming language. I mean you can use Julia in Jupyter, but this article is not about that.
It's about another, independent computational notebook system written from scratch in Julia for Julia language.
To clarify: it is not Jupyter system for Julia programming language. I mean you can use Julia in Jupyter, but this article is not about that.
It's about another, independent computational notebook system written from scratch in Julia for Julia language.
“ Briefly, these are its global hidden state and its incompatibility with version control.” If so, other language?
They have checks, hints, and interactive plots that update based on your answers. It's really satisfying to get this immediate feedback. The content is great, too, which helps.