onalark · HackerTrans

onalark

no profile record

comments

7 lat temudiscuss

It's complicated. The Army Corps of Engineers has had a civilian mandate to support flood control prevention since 1917 [1]. Beyond that, they are also involved in large public works projects such as the building of roads and bridges, and Superfund clean-up sites. On top of this, they regularly receive large pork barrel grants from Congress that can siphon money into a senator's state or a congressperson's district. They do have a large contracting arm and are actually pretty well-regarded for their comprehensive procurement and management process for these large public works projects.

So it's scale, politics, and history/momentum at this point.

[1] https://en.wikipedia.org/wiki/U.S._Army_Corps_of_Engineers_c...

8 lat temudiscuss

Ilan Schnell is not "some guy". He's the original primary author of the Anaconda distribution. One of the main reasons that so many data scientists use Python.

10 lat temudiscuss

NumPy is a library that provides typed multidimensional arrays and functions that run atop them. It does provide a built-in LAPACK/BLAS or can link externally to LAPACK/BLAS, but that's a side effect of providing typed arrays and is nowhere near the central purpose of the library.

Also, NumPy is implemented completely in C and Python, and makes extensive use of CPython extension hooks and knowledge of the CPython reference counting implementation, which is part of the reason why it is so hard to port to other implementations of Python.

10 lat temudiscuss

Except that Docker containers play terribly with virtualization solutions. Still, some sort of configuration/infrastructure-as-code would go a long way.

10 lat temudiscuss

This is a really interesting article and I'm glad this is getting attention. It's especially refreshing to see a theoretical treatment bridging algorithms with more "modern" hardware implementations.

One opportunity I do find in this article is real-world performance tests and determining the importance of constants in performance tuning. Prakop et al. did some really interesting work in 1999 on cache-oblivious algorithms, with applications to kernels such as the FFT and matrix multiplication. The work is theoretically extremely interesting, but in practice, hand-written or machine-generated algorithms continue to dominate. The most famous example of this is probably Kazushige Goto, whose hand-optimized GotoBLAS (now under open source license as OpenBLAS) still provides some of the fastest linear algebra kernels in the world.

If you're interested in learning more about how the differences between the two approaches shake out in linear algebra, I recommend "Is Cache-Oblivious DGEMM Viable?" by Gunnels et al.

10 lat temudiscuss

I don't think this would work for a number of reasons. If it's a database that you're modifying, you can see that a lot of operations (increment, delete, etc...) will do the wrong thing if they're called twice. If the operations themselves are idempotent, you wouldn't be able to verify that the intended side effect was correct. This is one reason developers spend a lot of time building mock objects: to capture "side effects".

10 lat temudiscuss

Awesome, I'm a huge fan of new and innovative tools that help improve the process of refactoring and improving existing code. This looks like a really promising tool for Ruby developers, and I'm always grateful when companies and their employees invest the time and effort to release their tools to the community. I really liked the point about "buggy data" as opposed to just buggy code, I think that's a really important point.

A few reactions from reading through the release:

Scientist appears to be largely limited to situations where the code has no "side effects". I think this is a pretty big caveat, and it would have been helpful in the introduction/summary to see this mentioned. Similarly, I think it would be nice to point out that Scientist is a Ruby-only framework :)

You don't mention "regression test" at any point in the article, which is the language I'm most familiar with for referring to this sort of testing. How does a scientist "experiment" compare to a regression test over that block of code?

Anyway, thanks again for writing this up, I'll be thinking more about the Experiment testing pattern for my own projects.

10 lat temudiscuss

Shiny, and I love the interface/layout!

This looks a lot like the Jupyter/IPython Notebook, which is a free and open source "scientist's notebook". If you're interested in mixing LaTeX, Markdown, and code from almost any language (Python, R, and Julia are very well-supported but there's an open kernel spec), then this might be a more appropriate tool for you to use.

The Jupyter/IPython notebook default storage format is JSON, which makes it a little more friendly for text-based version control, and also enables a static HTML view of notebooks (http://nbviewer.jupyter.org/github/ketch/teaching-numerics-w...) on GitHub.

Helen Shen wrote up a great article for Nature (http://www.nature.com/news/interactive-notebooks-sharing-the...) on how scientists are using the notebook, but it also provides a good overview of how you might use it, as well as a free interactive demo.

11 lat temudiscuss

Have you looked at conda and http://anaconda.org? We spent a lot of time curating the most important Python packages for data science into the Anaconda distribution, and conda packages are a great format for distributing complicated software.

11 lat temudiscuss

Thanks, I clearly wasn't being careful. I'll update my Gist...

edit: On reviewing, I think the intent of the original blog post was to modify images in place (or at least to do it as quickly as possible with in-place filtering ok). In that case, I think my comparison is fair, since NumPy doesn't offer a faster way to do the requested operation. I didn't try out einsum, but I think Numba would outperform that as well.

11 lat temudiscuss

Agreed, for some reason when I was looking at this last night I thought I couldn't use broadcasting, I've added the example to the gist: https://gist.github.com/ahmadia/c1f8be119f3cb2d2b8e5

Would you believe that Numba is 4 times faster for the sort of simple transformations described in the blog post?

(See aktiur's response below, some performance gains come from avoiding a copy)

11 lat temudiscuss

Great post, William. I really appreciated your exposition on both the challenges you folks were facing and your solution to the problem.

As a few others have pointed out, sending a lambda function through NumPy is almost always the last thing you want to do. Unfortunately, you were in a situation where you were either going to have to do something really painful like using einsum: https://stackoverflow.com/questions/29989059/matrix-multipli... or writing your own ufunc:https://docs.scipy.org/doc/numpy-dev/user/c-info.ufunc-tutor...

I suspect that your primary limitation was the Google Compute Engine infrastructure. I'm not familiar with the limitations there, but a quick search on Google turns up a fairly limited set of libraries indeed.

I thought it would be interesting to adapt your code slightly to use Numba acceleration. Here's what it looks like:

  from numba import jit

  def avg_transform(image):
      m, n, c = image.shape
      for i in range(m):
          xi = image[i]
          for j in range(n):
              avg = xi[j].sum()/3
              xi[j][:] = avg
      return image

  fast_avg_transform = jit(avg_transform, nopython=True)

I observed 25ms per image https://gist.github.com/ahmadia/c1f8be119f3cb2d2b8e5 processing times on my laptop on 1280x720 pixels.

Re-reading your post, I suspect that einsum might actually be your cup of tea, but I really enjoy the simplicity and performance of using Numba for these sort of tasks.

11 lat temudiscuss

Yes, I'm thinking of discretizations of elliptic 2D/3D PDEs. They don't scale in the weak or strong sense, and they can't hold O(n log n) asymptotic complexity due to fill-in from Cholesky/LU-style factorizations.

11 lat temudiscuss

That's because direct solvers can't scale. If you want to solve a large (distributed over hundreds of nodes) sparse linear algebra problem as fast as possible, decades of research have been poured into efficient techniques (Krylov methods, Multigrid, preconditioners) for solving them iteratively.

11 lat temudiscuss

Hi Niels. I have contacts within the DoD HPCMP and have run Python 2.7 on several of the heavy machines available on the Army side of the fence. I work for Continuum, which provides both a free and a commercial Python solution that are basically "drag-and-drop" installs on x86 hardware, and when I was at the Army Corps of Engineer Research Lab I worked on an open source tool called HashDist that, among other things, simplifies the deployment of Python stacks on HPC hardware. Feel free to get in touch with me via my contact information at http://aron.ahmadia.net if I can be any help.

11 lat temudiscuss

Let's hope this is not just bluster :(

11 lat temudiscuss

I believe "conda install seaborn" will do what you need :)

11 lat temudiscuss

Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.

11 lat temudiscuss

I agree, this is the first thing that jumped out at me when I was looking at the landing page. I don't know of quantitative evidence that this sort of thing causes harm, but I don't see any harm in mixing it a bit.

12 lat temudiscuss

Yeah, Kyle is a rock star!