I think the idea is not to replace learning the theory or math, but rather to just postpone it. Learning the practical aspects of an engineering discipline can provide the necessary motivation to study the theory/math, and this is the oft ignored factor. I also think there is some benefit in learning a topic using the tool that you will actually use in production (if that is possible without adding unnecessary complexity in the syllabus).
I personally learned DRL from David Silver's course and Sutton & Burto back in the days. They were the only good resources around and I liked them very much. But I think that with the advent of high-level frameworks in DRL, there are better learning paths.
I do intend to teach the theory/math in a later installment of this series, but I wanted to do it by showing students how to implement the various classes of algorithms e.g. Q-learning (DQN/Rainbow), policy gradients (PPO) and model-based (AlphaZero) using RLlib. This would kill two birds with one stone: you can simultaneously pick up the theory/math and the lower level API of the tool that you will be using in the future anyway.
Depends on what you mean by faster. Do you mean "time to solution" or "time to inference"? I think there are also more factors to take into consideration when considering the merit of the method e.g. performance, robustness, ability to handle non-linearity, ability to solve the full online problem etc.
When all these factors are taken into account, I have encountered situations where Deep RL performed better.
There are also very public examples of this e.g. Google's data center cooling [0] and competitive sailing [1].
I think the levels (high, low etc.) are relevant for the Deep RL algorithm, not the environment. The lower level version of OpenAI Gym canned environments would be custom Gym environments. I don't see much reason to go any lower than that.
The situation looks different for Deep RL algorithm. You can implement them from scratch yourself using Tensorflow or any other similar library. Otherwise, you could just use a higher-level library like RLlib which implements the algorithm using modular components and exposes hyperparameters as configuration parameters.
In many real world use cases, all one needs to do is to use RLlib's implementation and then tune the hyperparameters. In that way RLlib is to Deep RL what Keras is to Deep Learning.
This course uses RLlib. Does that answer your question?
Maybe this would help you differentiate: GPT-3, DALL-E 2 etc. uses transformers, while AlphaGo, OpenAI Five etc. uses Deep Reinforcement Learning. They are not mutually exclusive, but just different things.
Thanks. I learned it from an Udemy course [0]. Took just a couple of weeks to pick up. Regarding overlays, Sketchbook supports the idea of layers. I simply put different elements in the illustration in different layers. Sketchbook gives me PSD files that can be imported in GIMP. I then export many PNG files by progressively selecting more layers in GIMP. These PNG files go into Beamer like this:
My experience matches yours. Recently, I was trying to solve an optimization problem using Deep RL. As usual, I had to run many experiments over several days using various tricks and hyperparameters. Finally, it turned out something related to the symmetry of the action space made a huge difference in learning.
Anyhow, the experimentation stage requires a certain discipline and feels tedious at times. But the moment when learning takes off, it feels great, and for me personally, compensates for the tedious phase before.
It's certainly not fun for everyone, but I guess it could be fun for the target audience of the course (ML engineers/Data Scientists).
Regarding frameworks, my experience has been different. I find RLlib to be more modular and adaptable than SB3. But the learning curve is certainly steeper. The biggest differentiating factor for me is production readiness. Assuming that we are learning something in order to actually use it, I would recommend RLlib over SB3. The equation for researchers may be different though.
I am glad you like it. The coding exercises don't require a GPU. Thankfully, most RL problems (and certainly the ones used in the course) require small neural nets which can be trained in reasonable time using a CPU.
Hi all, creator of PythonBooks here. PythonBooks is running since 2017. During launch, it was on the front page of HN [0]. Thanks to the HN community for showing so much interest in the website. There has been steady traffic ever since and I have tried my best to keep the website updated with the latest books. Starting this month, there's a new page which lists the best Python books published every month - based on popularity, topic and novelty. I thought that it could be an useful tool for Python developers to cut through the noise and keep abreast of the best recent books. Feedback is welcome :-)
The article claims that catching up on sleep during the weekend doesn't work. This is most likely wrong information, as shown in this very recent study [0].
You are right. Discovery, fraud protection and many other functions need to be there either way. When I say cost reduction, I mean getting rid of any additional premiums which eBay (or any corporation for that matter) levies because of their strong position in the market.
As for the problems of online poker, the Virtue Poker whitepaper is a good starting point about why blockchain is indeed that magic pixie dust[0]
A lot of valid points here, but ultimately very short sighted. Firstly, the author does not mention many of the highly plausible use cases for trustless and permissionless services such as replacing rent seeking P2P marketplaces like eBay (where the advantage is cost reduction) or online Poker sites (where the advantage is manipulation resistance). He does not mention DAOS, where strangers can trustlessly enter into contractual obligation to create e.g. a crowdsourced Venture fund. He does not mention the potential to provide financial services to the 2 billion unbanked people in the world. He does not see the potential for innovative and completely new financial products coming out because of the permissionless nature of public blockchains.
The use cases that he sites include buying an e-book using a smart contract or its use in the supply chain or voting. The e-book example is slightly misguided, because this is not an use case where decentralization provides much value. So the entire discussion about auditing smart contracts is rather unnecessary. But it is nonetheless an important objection that may become important in DAOs. But it's short sighted to think that this problem won't be solved. Some companies are already starting to tackle the smart contract to natural language translations (e.g. Iolite).
Supply chain is yet another use case where it is unclear whether blockchains will add value. Perhaps cost saving. But note that the blockchains used in supply chains will most likely be private, therefore permissioned. It's unfair to put private and public blockchains in the same basket, because they are very different beasts.
Finally the use case of voting. The author thinks that whoever is in power might distribute a few additional addresses to their cronies to manipulate the system. He is missing the point that all data is public in public blockchains. So anyone can audit the voting results later if they want and discover that those addresses do not belong to any real people (assuming there is a KYC type identification system built-in). Voting is an use case where blockchains are sorely required. All we need is for the knowledge to reach public consciousness and for the people to tell their governments that they want it.
I love how the SpaCy related websites are always so well designed. Their dependency graph visualizer is just amazing. I know that Ines is behind that one, but don't know about the other stuff.
Now coming back to the topic, I have so far just used Jupyter Notebooks and spreadsheets to do annotations and by golly, it is an extremely boring and tedious process. This looks like a fun tool to try out for my next NLP related project. Might spice things up!
But I hope that like all SpaCy related ideas, it doesn't assume too much about the problem at hand. I usually use NLTK instead of SpaCy because it allows me to be very flexible, except for the sentence tokenizer, where SpaCy's accuracy is hard to beat.
> Rarely on top but still a high ranking relatively speaking(2-4 etc.)
If you look at the beginner book section, the kid's section and the application walkthrough section, there are almost no Packt books there. In the reference book section, the Packt book ranks last among 6 listed books.
In the individual topical sections, there are on an average 5 books listed per topic, so 3-4 is a pretty low rank.
The only two exceptions are the ML book by Sebastian Raschka and the high performance book by Gabriele Lanaro which are ranked in the 2nd position. Both of these are outstanding books in their own domain and deserve their spot.
I see your point about Powerful Python having another retail outlet, but I think the same is true for all the other books because they too have other retail outlets.