Yes, it feels like we have squeezed most of the performance out of current algorithms and architectures. OpenAI and deepmind have thrown tremendous compute against the problem with little overall progress (overall, alpha go is special). There was a big improvement in performance by bringing in function approximators in the form of deep networks. Which as you said can scale upwards nicely with more data and compute. In my opinion as an academic in the deep RL, it feels like we are missing some fundamental pieces to get another leap forward. I am uncertain what exactly the solution is but any improvement in areas like sample efficiency, stability, or task transfer could be quite significant. Personally I’m quite excited about the vein of learning to learn.
Mosh is great, but it does not support port forwarding. For some people, including myself, this is a deal breaker. It has been an open issue/feature request since 2012[1] and even has a ~600 USD bounty on it[2].
Working on research for my thesis, which will be wrapped up by years end. The current project focuses on improving sample efficiency in deep reinforcement learning. I am researching how best to merge the options framework with the adaptability of meta-reinforcement learning.
In my spare time I write and research algorithmic trading strategies. I’ve been sticking to the traditional techniques, with a small toe into statistics for modeling.
With whatever time is left, I’ve been learning rust and have enjoyed it quite a bit so far.
From my viewpoint, as both a researcher and someone who has built frameworks around environments/games:
- Each step within the game has to be extremely fast. I.e the game should be able to be run as fast as the machine allows while keeping physics etc. consistent.
- Runnable via library import such that there is no drawing to the screen.
- Should be easy to reset the environment to an initial state.
- RNG state should be seedable.
- I highly recommend supporting an identical interface found in OpenAI's gym. Check their docs out. Even better would be to have your game importable as an environment in gym.
- Configurable screen resolution would be great (eg. output 120x100)
- The environment is "hackable" eg. the maps or levels can be modified or loaded say via some ascii map.
- Should support multiple copies of the game running at once.
- A nice to have would be if the current environment state could be exported and loaded later.
- Expose some information/signals such that a reward signal can be created. Or better yet you define one as the game creator.
No they all use the same general principal of backpropagation to do the training. Different flavours of optimizers exist with different tweaks and additions to speed training up.
Thought I'd share a library I've been using with my personal work and projects. Its an interface around PyGame which makes it painless to start doing RL based work. Deciding to share it now as I'd like to get feedback from others on how to adjust the library and make it useful. Plus I was afraid of developing "just-one-more-thing" syndrome causing further delays.
If you want to start using it right away General Deep Q RL[1] currently supports PLE out of the box.
Not OP but I believe the low score is due to not enough training time and incorrect parameters such as frameskip. Space Invaders was mentioned as one of the few games they needed to lower the frameskip (from 4 to 3 or 2?) on because of the flashing lasers. I'm assuming OP left the parameters as is from the implementation by Nathan Sprague, which has frameskip at 4, and trained for a few epochs.
I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then:
Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850
Older one, but important to understand deeply since other recent ideas have come from this!
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434
State of the Art Control of Atari Games Using Shallow Reinforcement Learning - http://arxiv.org/abs/1512.01563
Interesting discussion in section 6.1 on the shortcomings/issues of DQN done by Deepmind
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) - http://arxiv.org/abs/1511.07289
I wish they did more comparisons between similar network architecture with only the units swapped out. Eg. Alexnet, Relu vs Alexnet, Elu.
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models - http://arxiv.org/abs/1511.09249
"The final use case that we thought of was saving bandwidth. A smaller image could be sent to the client which would run a client side version of this model to gain a larger image."
Can be applied to gifs and videos, but this really depends on the usage case and if the client would tolerate such a thing.
Heads up, I interviewed with these guys before and was shafted on reimbursement for travel when I told them I took another offer. It wasn't a ton of money but its annoying as a college student.
You can get them if you look through the network request the browser makes. Largest I saw was 4096x3096, which is just perfect for desktop backgrounds!
For anyone actually interested in implementing DNN's I wrote up a quick blog post (essentially a brain dump) of general guidelines to adhere to when training DNNs. The source for this information is primarily from videos given by Geoffrey Hinton as well as various papers.
+1 on this. I've used it in a previous project of mine. I did experiment with the other Redis-go libraries but Gary's felt the most complete and solid.