In practice the functions just need to be piecewise differentiable. The RELU is the canonical example for deep learning. At kinks a subderivative is used.
Actually I'm pretty sure the Rome numbers are for double precision whereas most numbers quoted for GPUs are for single precision or less, making Rome's 3.4tf even more impressive.
I've got the 9550 and run Ubuntu on it. No issues whatsoever and I do very compute-intensive work on it.
One thing that I've found is very important is to clean out the fans often, otherwise dust builds up and prevents cooling. Just unscrew the plate on the underside of the laptop and blow / brush / pick out the dust that's built up (both the fan intake and the fan outlet). Doing this every few months has been a game changer for my 9550.
Essentially, RNNs and feed forward networks are very similar - RNNs are just "unrolled through time" and every timestep shares the same weights. The activations are slightly different as well, but the core concept is the same as feed forward networks; it's not a completely different concept or idea.
Note that it's not just the loss function. It's the loss function combined with a very specific problem formulation - namely a neural network with only linear activations (equivalent to a 0-layer network). Once you go to non-linear layers or a different loss it's no longer solved analytically.
I do see a lot of people writing tutorials like OP's. See for example:
The existence of these articles should not be taken as an indication of best practice. They often have the goal of teaching SGD in a simplified setting, not teaching best practice for LLS. I suppose only nice thing about using TF / SGD for such a simple problem is that you now have starting point for solving more complex problems (RELU activation, cross-entropy loss, more layers, etc.).
A few other points as to why you would never SGD for LLS:
1) it's always way slower than the closed form matrix solutions
2) if you're doing SGD instead of just GD, there's noise in which "rows" are in a given batch - as a result, repeated runs may not converge to exactly the same final weights. This never happens with the analytical solution which always gets exactly the same result.
3) if you're doing this as part of a data science pipeline which is likely the case in the real world, you'll likely want to do some cross-validation. In the SGD case you have to recompute the entire solution for each fold whereas in the LLS case you can immediately compute CVs once you've calculated the initial XTX / XTYs. This makes the process of using LLS even faster than SGD.
I find it hard to believe that SGD would be faster than the closed form solutions for linear regression (gels, gelsd etc.). The closed-form solutions give a lot of other benefits in practical settings as well which makes them more likely to be used if possible. SGD + related optimizers give benefits with non-convex or non-analytical loss functions or with non-linear layers / more than one layer.
Are physics engines not yet accurate enough to enable "virtual" pre-training / full training of the networks, lighting conditions, etc? If they are, exclusively using physical robots seems somewhat inefficient.
DAC is an order of magnitude cheaper than optical and might be fine for their use based on their comments of cat6 v cat6a and "short runs in their lab".
Also, if they are using two or three NICs per box, you'd need to factor in the cost of two / three of the X540's vs. a single 40gbe connection and you'd need to double the cost-per-port of the switch vs the 40gbe option.
Some claim that this is part of a broader trend of "generational theft" from the young to the old and have some interesting slides to demonstrate their thesis:
The pairing idea isn't new although it does seem to be very successful - there's a company called Thinkful (http://www.thinkful.com) that does online learning by pairing a mentor who gets paid with every student who signs up and pays for a course. They apparently have a very good amount of traction with thousands of paying students. Should be interesting to see how askadev's free model compares.
ZFS may have been, but not the 128 bit support. I mentioned that there are "admittedly much more relevant features" which are in fact what smugmug likes:
"ZFS is the most amazing filesystem I’ve ever come across. Integrated volume management. Copy-on-write. Transactional. End-to-end data integrity. On-the-fly corruption detection and repair. Robust checksums. No RAID-5 write hole. Snapshots. Clones (writable snapshots). Dynamic striping. Open source software."
Yes, this is theoretically cool and all, but when I hear something like:
"Let's start with the easy one: how do we know it's necessary?
Some customers already have datasets on the order of a petabyte, or 250 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away."
It raises a huge red flag from a business point of view. In 2004 "some customers" had datasets that would cause issues sometime around 2015 - a problem that ZFS purported to solve (among admittedly much more relavant features). In planning for a future that was possibly ten years away for "some customers" instead of focusing on what was relevant to the larger customer base, Sun managed to continue their slow profitability death march until Oracle finally snatched them up this year.
Something to keep in mind when you decide to add "cool theoretical feature X" or "unparalleled scalability" to your four month old startup...
Here's a few I've really liked that have a broader subject matter than only what you asked for. I figure that if you like to read you'd like these books.