What if you want to ensure that the same packages are available, built in a similar way, between your Mac and some linux servers? What if you need to share or ensure that your projects work between Linux and Windows?
What if you are supporting a lot of less-sophisticated users across a number of different OSes, or even different versions of the same OS?
There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.
The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.
You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.
TLDR - we are working to clean up this language to leave it clear that educational institutions are exempt, and that these commercial terms do not apply to third-party channels hosted at anaconda.org (which includes conda-forge).
Fun joke but Anaconda has a track record of creating OSS and then turning it over to community governance. This includes the conda tool itself, libraries like bokeh, dask, numba, jupyterlab, and many more. And while PyScript project governance isn't in NumFOCUS, all of the code is permissively-licensed BSD/MIT.
The commercial licenses for the products and commercial repository is what supports all of this OSS development work.
Anaconda Code is what you're looking for! It's Python (via PyScript) running as an Excel plug-in, that has full access to the spreadsheet and can harness a big part of the core PyData stack (including matplotlib, sklearn, pandas, etc.)
I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
This is one of the reasons people use Anaconda/miniconda for non-data science work: conda environments are self-contained Python installs, so if you conda/pip install packages into those environments, they will not break each other. This design requirement arose from the specific needs of numerical computing (which always drags in a ton of system-level C/C++/FORTRAN dependencies), but is a generically useful design construct.
Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time.
Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)
FWIW, we are soon going to be releasing a much faster depedency resolver. We are also thinking hard about how best to address the "growing ecosystem" problem, in a future-proof way.
You only need a commercial license under certain circumstances, and it only pertains to accessing the default package repository at repo.anaconda.com. You can still use conda-forge etc. Conda itself (the packaging tool) remains open source.
Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.
You keep reposting this Nature article but it is vacuous. There are many "guilt by association to Trump", "false equivalency to outrageous conspiracy theories", etc. and other non-good-faith argumentation that the author relies upon. This does not give confidence in her motives.
There are a few much more substantive sites with analysis into the genetics and circumstances around the virus, which emerged since the April 2020 which your Nature article cites as its primary source.
It's tempting to say that Lem was way ahead of his time, but then we look at his contemporary philosophers of politics, technology, society like Jacques Ellul, Marshall McLuhan, Gilbert Simondon, and realize that the mid-20th century was awash in brilliant foresight about the unpleasant implications of a technological society.
IMO this nuanced thought was simply lost in the craziness of the late-60s and the sex/drug/rock&roll hedonism of the 70s, which then matured into stockbroker 80s, before a second wave of tech-capital-blindness emerged in the 1990s.
And now as these waves have transformed the entirety of modernity, we are faced with the unpleasant question of "where does it go from here, now that the Boomers -- whose narcissism birthed Consumerism -- are dying off?"
There is a factual inaccuracy in the article: Beaker Browser is not "actually under the control of governments and big corporations". It is an independent open-source project.
Exactly. Ads are a legacy business model of a legacy distribution system. Too few people have read JPB's excellent "Selling wine without bottles", but that is in full effect here.
The problem is that new content-centric business models have not yet emerged. So, some people remain chained to the old paradigms.
The crisis we have is a gap in vision. People don't realize that driving cost of information distribution to zero means that it no longer has enough scarcity to force the economic transfer of other scarce resources. Basically, what OSS did for commercial software, the Internet did for anything that fits within a 2D screen. Netflix, NYTimes, Fortnite, JK Rowling, and Jenna Jameson are all competitors in a space whose Pixels*Seconds value is commoditized.
Capitalism doesn't thrive unless there is an exponentiating dynamic. The only one available on the Internet is bandwidth capture. Which, for now, translates into attentional capture.
With the imminent arrival of P2P web software (e.g. Beaker Browser) and mesh networking, the tides will turn back towards a creator-centric peer network.
> It looks like that paper is about the transformations of visualizations for higher dimensional data, not rendering accuracy, so these two things are being conflated even though they are completely separate concepts.
Actually, no. The paper may not have been explicitly clear about this, but the ENTIRE point of a "data visualization" system is to transform potentially high-dimensional datasets, with a large number of columns, into meaningful images by a series of steps. You seem to be interpreting this narrowly, and imagining that geometry is already pre-defined in the dataset, so then of course this looks like a fairly trivial 2D accumulator.
That is not the intent, nor is the common use case.
For data visualization, the question of "how do I accurately aggregate or accumulate the 25 - 1million points in this bucket" is a deep one. There is NO data visualization system that programmatically gives access to this step of the viz pipeline to a data scientist or statistician. Most "infoviz" tools gloss over this problem - they do simple Z buffering, or cheesy automatic histograms of color/intensity, etc. These are almost always "wrong" and produce unintended hallucinators.
Your first comment - about "not needing all the geometry present" - indicates that you are not understanding the nature of the problem datashader was designed to solve. There is no simple "cull" function for data science; there is no simple "Z" axis on which to sort, smush, blend, etc. At best, your data points can be projected into some kind of Euclidean space on which you can implement a fast spatial subdivision or parallel aggregation algorithm. But once that's done, you're still left holding millions of partitions of billions of points or primitives, each with dozens of attributes.... what then?
Yes, datashader actually gives you the ability to dial-in as much gamma compensation as you want, to account for the human visual system's nonlinear response to luminance.
I think you are deliberately trying to misunderstand what is being done in this project.
It's not about what APIs are being used to render whatever. At that level of analysis, all that anybody is every doing, is just doing memcpy and bitblt. Rather, datashader provides a framework for applying semantically meaningful, mathematical transformations on datasets as they're being accumulated, as those accumulations are converted into aesthetic/geom primitives, and as those primitives are rendered into colors. It really is "renderman for data", along with arbitrary vertex/texture shaders, driven by a dynamic rasterizer that can use whatever bins in data-space (not merely physical pixels).
BTW "Out of core" does NOT come from raytracing; in fact its history in computing is a term for anything that exceeds physical memory. We use it all the time in scientific/HPC and data science because datasets are frequently much larger than available memory.
It's actually not merely an accumulation buffer. It's a shader pipeline that allows for arbitrary Python code to be executed at each stage of data processing. It's actually very much like "renderman for data", but with Python (via Numba, Dask for performance).
The pipeline is also built in such a way that it permits front-end JS viewers like Bokeh to drive a very dynamic experience.
There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.
The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.
You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.