I think this is a property spheres. It seems to me that any two spheres that are touching have a straight line from one center to the other center exactly through the point of contact. Try thinking of just two spheres and adding more in step-by-step.
Then the result follows because all the spheres are defined as centered on the cube/sub-cubes respectively.
There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet.
If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities?
It's definitely a challenge to get enough high-quality data to feed a 100B parameter version of such a mutli-modal model, but there don't seem to be any theoretically insurmountable issues towards this "dumb" way of giving the models more intuition.
Having more memory to address means you need more circuits that direct your reads/writes to the right place. Travelling a longer distance / through more complicated routes means the latency for each request will be higher on average.
The hidden state isn't storing the samples of your loop (or a compressed version of your loop). It's encoding a representation of how the output will change based on what the current state and input are. This might be strongly dependent on what the exact samples in the loop are, but it could also be more general. I think it's missing a bit of the representational power of an LSTM to see the state representation as just a buffer of the current input.
But, yeah, at some point your signal has such a complex behavior on long time scales that there isn't a good way to predict it based on a limited state size (or at least gradient descent can't find a function to predict it for you).
LSTM stands for Long Short Term Memory. It's a recurrent network that learns what and how long things should be kept in its internal state buffer. It doesn't have a fixed state size because it's just learning a nonlinear function that takes an input and a state to an output and a new state. Obviously it can't model all possible, infinite length recurrences, but it can definitely do a pretty good job of approximating long term recurrence relations in complex signals.
I think most human creativity is built around a seed of inspiration from outside sources.
In my experience serendipity and happy accidents are exactly what leads to the most creative and interesting outcomes.
This is a system that’s able to generate these inspirations/prompts for you in a way that’s more focused on what you’ve already written. I don’t see it as ironic because I don’t think it’s hijacking your creative potential, just augmenting it.
I think the larger models get, the more incentive there is for researchers to look into pruning/distilling them for practical use.
GPT-1,2,3 et al. have all shown that larger is better. While in the short term this means people will simply throw larger and larger clusters at the problem, in the longer term there needs to be inovation in making it more efficient on the clusters we have (as even the cloud has limits).
I think sheer parameter count is an important part of the equation in general intelligence, so it's important that there are labs that work on scaling up promising leads to trillions of parameters on top of labs thinking of new promising directions.
I think there's more than enough room in between bullshit and nobel prize.
This looks very promising, but as always the devil is in the details. The next steps are multiqubit gates, then linking those up into useful quantum circuits, and then hopefully doing actual quantum computations.
Personally I wouldn't expect a nobel prize until one of those last 2 steps. There's a lot of hurdles before then to run into some nasty problems.
I don't really understand why I'd use gin. From the example ipynbs it looks like pretty much the same amount of code but in a gin file and then it spookily fills in parameters for you in python. Why is this useful?
As far as I understand, it's because what it's simulating is a logical qubit which is different from the very noisy, almost instantaneously collapsing physical qubits present in current quantum computers.
Software simulates what's supposed to happen while the hardware only approaches it through many repeated trials.
Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point.
The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The reason some losses allow GANs to converge in certain situations while others don't isn't a complete mystery, there is theory that supports this.
Researchers designing new models are considering weak points in old approaches, identifying why they aren't working correctly, and proposing something new that solves a part of the problem. All of this is done by looking at the math behind all the operations in the network (or at least the parts relevant to a certain question).
That nobody really knows how AI works is one of those myths told by the media. Just because the model weights aren't interprable doesn't mean we don't know why that model works well. It just takes quite a bit of maths knowledge to really understand state of the art models. All that knowledge is also easily packaged into modern frameworks that make it easy to use without a deep knowledge of why it works. All of this contributes to the feeling that nobody really knows what's going on, while in reality it's onky the majority of people that don't know what's going on ;)
I think the approach is really cool but the processing time required is too much for this to be very useful at the moment.
On a 1080 Ti it takes 45-90 minutes to train networks for the various tasks on 256px images (depending on some quality parameters and which task). Each task also requires training individually so if you'd like to try them all for a given image you'll need to train 6 times.
Also the pyramid of GANs approach is very memory hungry. I was only able to get up to 724px images with 11 GB of VRAM. This was also only possible with a higher scale factor (sparser pyramid) which sacrifices a lot of quality and is incredibly noticeable at larger image sizes. I only tried for larger sizes with the animation task though, perhaps there is a way to combine the super resolution and animation task and achieve better results. Training on larger sizes was taking upwards of 6-8 hours.
All of this was tested with the official repo[1] about a month ago.
I had already completed this course before Stefan started his conquest of the CS bachelors courses, however, I did have some other courses that he taught. I think he has a great knack for explaining things in a way that doesn't make the formalism seem overbearing, but rather as the most obvious way of communicating a concept. Assuming he is also able to communicate this in writing I'd say this book is probably a great introduction to the formal side of computer science.
Detect when there are multiple apps that are asking for new permissions simultaneously and then ask the user if they want to go to the "bulk permission editing page" to manage them.
On that page there could be a grid of apps vs permissions with the permissions each app is requesting highlighted. One click to switch on each permission and even a power user could be on their way in a couple minutes.
LineageOS has a similar permissions list for all apps installed (albeit without a nice overview grid) that I think makes it very easy to manage what each app is allowed to do.
This is the same idea that underlies style transfers and metrics like the FID (which is used to judge generative networks' outputs on their similarity to the test set).
The idea is that the activations within an image recognition network are similar for images that are similar and so you can measure the distance between two images in a space that has some semantic meaning.
I think your camera example is a false equivalence that makes this seem as if it's not a computation. The camera is not running the same algorithm as the renderer and so you're comparing different things.
The experiment used a classical computer to randomly generate a circuit C, told the quantum computer to execute it, and recorded the result. Then they repeated this but executed each circuit in the most optimal way a classical computer can. Finally they compared the distributions to verify that the quantum computer's results matched the correct classically computed results.
This proves that the quantum computer is able to generate that distribution faster than a classical computer and isn't just doing some other spurious process that happens to be faster.
Yes, this random distribution isn't very useful and so this result is only really interesting as an experimental verification of theory. However, it is a computation that benefits from quantum speedup in real life!
Hopefully soon, algorithms will be found to generate more useful distributions (as it seems for the time being sampling is the only application of this type of QC that is practically doable). For example, Aaronson mentions that generating verifiably random bits is not much more difficult than the noise generated in this experiment and could have an impact in a variety of cryptographic applications.
I don't think it's a non-event even if you're only interested in factoring.
This proves the underlying principle of a quantum speedup is a physical reality. It might be something people already take for granted if they're interested in factoring, but it's a crucial prerequisite that was not yet irrefutable.
There are quite a few open source implementations of this e.g. https://github.com/outlines-dev/outlines