The question is: what tool to use? I'm a solo developer running a very small kube cluster for my hobby project. I very much wanted to have a declarative version controlled state of my cluster. Every time I try googling solutions I get flooded with some enterprise Saas offerings that do nothing I want.
I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky.
Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.
Seems like an awesome idea, but the first thing I see when looking at it is that's one big chicken and egg problem - it'll be an awesome platform if it gathers a critical mass of bands and users, but before then probably will feel like a dead place for a long time. What's your plan for overcoming that?
> This is like if the world's best chess AI had gone from losing high school tournaments to being competitive with Kasparov in less than 3 years.
I don't think it's like that at all. On the high level, there is no "chess AI", "go AI", "image classification AI" and "dexterous manipulation AI". These are all sides of the same coin, that gets significantly better every year. Adding support for the new game or new "environment" to existing deep learning based backbone still requires a bit of engineering work and a few creative tricks to unlock the best possible performance, but the underlying fundamentals are already there and are getting better and better understood.
There is a reason why the progress in AI is so hard to measure. Anytime a next task is solved, there is a crowd saying it's not a "real AI" and that scientists are solving "toy problems". Both statements are totally true. But the underlying substance is that each of these toy problems is of increasing complexity and brings us closer and closer to solving the "real problems", which are mostly so undeniably complex that we couldn't attack them upfront. Still, the speed of progress in the field of AI research is staggering and it's hard to keep up with it even for professional researchers who spend all their waking hours working on these things.
6 years ago we were able to solve some Atari games from pixels. Today, that feels like a trivial exercise compared to modern techniques. With billions of dollars of investment pouring in and steady supply of fresh talent, it is very hard to predict what the pace of research will be in the coming years. It is entirely possible we'll encounter a wall we won't be able to overcome for a very long time. It is also possible that we won't, and in that case we're in for a very interesting next few decades.
I've used both in production and I must say I'm a very big proponent of both. Actor model of concurrent computation I think maps very well to, at least my, way of thinking and I think is the most productive way of writing distributed systems.
Elixir/Erlang/OTP:
+ Very mature, very well thought out. While the newer stuff may feel a still under construction (string handling, date handling) all the concurrency primitives are rock solid, and by rock I mean diamond.
+ Elixir is simply a great language and you can get quickly very productive in it once you grasp the actor/process model of BEAM
+ Has one very big advantage over akka that actors can receive messages in different order than they were sent. That of course can cause some headaches if not handled carefully, but 99% of the time straight up leads to nicer and simpler programs. Really a lot of akka code very often is just written to deal with the order in which the messages may arrive.
+ Truly resilient with very good error recovery design once you know how to work with it. I still don't know more graceful and productive way of recovering from failures in a running system.
- For doing any expensive computations it's slow and it's a fact. Not much can be done about it.
- Library coverage is 7/10 or 8/10, but these few missing points can sometimes make a big difference.
Scala/Akka:
+/- I love Scala language and static typing, but one must be honest that it's much more complex. You can learn Elixir (without macros) in an afternoon. One really needs to take some time and think it through to utilize Scala properly and a true mastery lies even further. To be fair proficiency with Elixir macros also requires a considerable effort, but one can go very far with Elixir without writing macros, while with Scala already the upfront cost is pretty high.
+ Up to my best knowledge Akka Streams is completely unique and completely amazing library that is gaining support throughout library ecosystem. This is one point where Scala/Akka completely outshines everything else. Streams are such a great abstraction and gave a huge productivity boost to most of the projects I was working on. Compared to that, elixir GenStage feels much less robust and polished.
+ Speed of JVM should be enough for 99% of applications, and with that respect it wins with BEAM.
+ Java/Scala library ecosystem is very deep and is simply much more comprehensive than Elixir/Erlang one.
- There are places around akka that still feel a bit immature/adhoc, but the library is steadily improving. It just is not as mature as BEAM/OTP.
- Over small and mid size projects I think I was more productive with Elixir. Meaning, if I had the same amount of time, I could implement more features in less time using Elixir. But it could be just a personal thing.
Overall I think both are fantastic platforms and I'm happy to have both of them to choose for each project. If I were to chose what to select today:
- Choose Elixir/OTP for a system where we need to do a lot of io but not much computation and we're sure existing libraries cover our needs. Very big plus if we need it to be resilient.
- Choose Scala/Akka if we need speed or call any existing JVM libraries. Very big plus if your project could use Akka Streams.
I can share with you one hiring insight I have that goes against your sentiment. I've been hiring for hedge fund research roles for the last three years interviewing ~30 candidates over that period.
What I've learned is that it is a very weak signal to talk to candidate about "their research" unless it's immediately and directly applicable to what we were doing as a company, so that we can talk about it as equal experts, which never really happened in our case. Previous research done by candidate was, like research almost always is in a narrow and specialize niche. Every candidate I was talking to was very good with talking about their research and from my perspective at the time of the interview it was impossible to validate and evaluate their claims. That conversation could convey the character of the candidate but I couldn't evaluate their ability to do new research.
Instead, presenting candidates with the same problem, that in our case took around 3 hours to solve, in our case proved a good way to compare candidates a bit more objectively and test how they approach solving problems they haven't seen before which is more applicable to what they'll be doing on the job.
The downside of that process is that during three hours you can only test so much, but hiring is a really hard thing to do right and we didn't want to take more of candidates time as we did value it highly and found that to be the sweet spot for us.
In my opinion PyTorch code is easier to understand and debug for newcomers. Code is definitely lacking in documentation, but whenever there was a tradeoff between clarity and modularity in the end I've chosen modularity. Ideally I would like others to be able to take bits and pieces and incorporate into their projects to speed up time to delivery of their ideas.
I’d be very interested to hear for which similar problems (games, control problems) reinforcement learning is not the best approach (there exists other, qualitatively better method£. We hear a lot about success stories but not a lot about limitations of RL
1. Yes, that happens, although constitutes around 10-15% of work being done.
2. I am certainly listening to any suggestions, but don't necessarily actively encourage it before proposing a solution. Perhaps this is a direction I need to move towards.
3. Mostly yes, as overwhelming part of the complexity just comes from knowing how various parts of the system interact. From time we have an open-ended 'research' (where the outcome is not known upfront) project, but these have a very mixed success rate and tend to overextend the time budget.
I would like to get a bit of HN feedback on the matter. I am exactly the person mentioned here - an engineer who knows nothing about the management, but because company is growing and we're hiring new people I'm slowly transitioning towards a management position. Just like you've stated I know nothing about the management and all I have is a common sense to guide me which I'm afraid is not enough on multiple occasions.
I'm especially afraid of too much micromanagement, but I don't really know how to handle the process better. Here is my style of management, briefly summarized:
I have a very deep and detailed knowledge of all of our codebase, which is in it's entirety shared by noone else at the company. I've written half of it myself and the other half has been written by other people. Most of our hires have been hired between a year and few months ago (company is very young by itself).
What is the usual process for me is that I discuss the general direction and required features with top-level management, who is entirely nontechnical and make a roadmap in my head with plans for the "subprojects" which are more or less self-contained units of work, their time estimates and relative priorities with respect to company-wide deadlines.
Based on the last two, I dispatch the tasks to relevant people, waiting when they finish previous tasks. I try not to stress people with too many tasks at once, but treat "tasks" more like a priority queue, where when a person becomes available a current highest-priority task is taken out from a queue and assigned to a person, taking into account personal technical strengths and weaknesses as well.
Most of the time, when giving a task to a person, I already have solution implemented "in my head" and the process mostly revolves around me explaining the relevant part of the codebase to a person and detailing the solution I've envisioned with special emphasis on the parts where a mistake is more likely to be made. After the implementation is done, I take part in testing and perform a code review before merging the code in the production branch.
Sometimes I wish I would give people more freedom, but on the other hand maintaining project coherency and keeping a single direction for the whole team I feel requires for me to make final decisions quite often. I plan to hand off ownership of certain "modules" in some time in the future, but first I'm waiting for deeper understanding of the codebase to develop in the reports.
On the outside, to me, it seems that the process works. People are not complaining, team makes progress, features are done on time and seem to be of satisfactory quality. But like you can read above, I'm base strongly on detailed technical knowledge and basically "planning the work ahead" for the people.
I would really appreciate if anyone could share their experience with making that transition.
Hi, I'm a professional (applied) mathematician and a website like that for mathematics is what I always dreamt of making ;) Even tried to apply to YC once with the idea but didn't really get very far with it. The need to provide a paycheck for my family unfortunately makes it quite hard to find enough free time to proceed with the plan but maybe if there are enough people here who are interested, we can do it together?
If interested, drop me a message at jerry at millionintegrals com.
I don't know if I'm the only one, but I really see the basic income as a strong force for increasing the inequality, leading to society stagnation and disallowing any social mobility effectively setting in stone current power structures.
I don't think we can ever say that the "jobs are gone". There are always things that people can do that are beneficial to the society but they can be either
a) not profitable in the current economic framework or
b) require highly skilled workforce that is currently not available on the job market.
If a) is the case, that means we are not really yet in the post scarcity world and need to first work more on providing "basic" goods so that we can afford having these less essential ones.
If b) is the case that means we need to put more resources into educating and training people. There is a sentiment that modern day universities are not worth the cost, but I highly disagree with that - society is only as good as people are educated and the more knowledge and highly developed skills people have the more they can participate in the economy.
In the world of the future, the "easy" jobs will be gone, but there will still be lots of things to do for smart, educated and skilled people. The only problem is that learning process and guiding a young human from newborn to such person is a costly process to the society. Providing basic income of barely "survival minimum" disincentives people to take that cost and participate in the economy. Until we have the AI that is better than humans at doing basic research, there is still a ton of things to do for humans, and I don't see anything good with excluding parts of the society from the process.
I have very little knowledge about the actual license law, but what is the right way to do that?
Let's say there is an open source project published with a very permissive license that has one function you need. Putting the whole project as a dependency is an overkill and rewriting the function with your own variable names and indentation doesn't sound like an improvement.
Would adding a copyright notice somewhere in the repository be enough?
Amsterdam, Netherlands | Full time | Onsite, relocation
Independent View is an asset management firm focusing on quantitative investment strategies. We are a young, energetic, research-driven company consistently outperforming benchmark returns. For this, we were recently awarded Best Newcomer Commodity trading advisor in 2015 by CTA Intelligence Magazine. We are keen to build on our early success and are looking for talent to expand our Research and Development team.
We are hiring for a very hands-on developer position with direct ownership of multiple internal products. Pace of work is fast, decisions are made quickly and communication is efficient within our tight-knit team.
If you see yourself growing together with a very promising young company, this may be just the right place for you.
We are a small, highly multinational team, communicating mostly in English. We value our casual, creative atmosphere, with emphasis put on tangible results and technical arguments.
If you’d like to meet, have a chat and get some more information about the company email me jerry at independentview.com. When found the right match, we’ll be more than happy to help with relocation to Amsterdam, which is a lovely city to live in.
I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky.
Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.