S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.
Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.
I expect Photoshop to contain many highly optimized function routines for photo processing. These routines are optimized by hand and usually use architecture-specific features such as SSE/AVX which is not portable.
Unoptimized (more portable) versions of theses routines exists, but I suspect they will perform well even with M1.
It is because Google wants a monorepo, then Google choose to use Perforce (and later Piper). It is not that Google uses Perforce and thus are limited to a monorepo.
The core value behind monorepo (and monorepo-like approaches) explained in the book is that dependency management is harder than version control.
For anyone interested in why monorepo works, I'd recommend the book Software Engineering at Google: Lessons Learned from Programming Over Time. It has detailed the reasons for the One Version Rule and Version Control over Dependency Management.
The site is using Let's Encrypt for https. Currently there is a hard limit of 2000 subdomains per week, meaning there will be at most 2000 simultaneous connections!
It seems that Wildcard support is coming Jan 2018.
1. Kernels (Functions in NNabla) are mostly implemented in Eigen.
2. Network Forward is implemented as sequential run of functions. No multi-threaded scheduling. No multi-GPU or distributed support.
3. Python binding is implemented in Cython.
4. Have some basic dynamic graph support: run functions as soon as you add them to the graph, and run backward afterwards. Somewhat similar to PyTorch.
5. No support for checkpointing and graph serialization, or I'm missing something.
I'm not sure why Sony is releasing this (yet another) deep learning framework. I don't see any new problems the project is trying to solve, compared to other frameworks like TensorFlow and PyTorch. The code is simple and clear, but nowadays people need high-performance, distributed, production-ready frameworks, not another toy-ish framework. Someone please shed some light on me?
I don't see the point of writing a machine learning package in Go.
Go is designed for system programming, which is the opposite of scripting. In scientific computing, interactivity is an important feature. Softwares like MATLAB and IPython(now Jupyter) become popular because of this. Go doesn't have a nice REPL like IPython, and it never will. Besides, Go doesn't allow operator overload, which leads to verbose programs. Compare `c = a @ b` with `c := Must(Mul(a, b))`, the former Python statement is much cleaner than the Go one.
Usually, researchers use MATLAB, R, and Python for prototyping and training. The final model will be put into production by using something like TensorFlow Serving or by hand written C++ for performance. Even if the final product uses Go for serving, the Go program mostly runs the model via cgo or RPC.
Write in C/C++, and wrap the library in Python and Go is a much saner option. TensorFlow and MXNet are both good examples.
Python really need something like Go's official libraries. I mean those with a prefix "golang.org/x/". Those libraries update more often than stdlibs but still provide a concrete common ground for developers. It is like "evergreen" stdlibs.
The spec hasn't been updated for 8 months with several PR outstanding. What can be do to push the spec forward when the original leaders seem to be absence?
S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.
Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.
[1]: https://github.com/pytorch/examples/blob/main/imagenet/extra...