willsewell · HackerTrans

willsewell

no profile record

Submissions

Monzo does code migrations across 2800 microservices

infoq.com

1 pointsby willsewellhace 2 años0 comments

We run migrations across 2,800 microservices

monzo.com

29 pointsby willsewellhace 2 años12 comments

Argo Rollouts at scale: Bringing Automated Rollbacks to 2,100 services at Monzo

monzo.com

9 pointsby willsewellhace 4 años0 comments

We deploy to production over 100 times a day

monzo.com

36 pointsby willsewellhace 4 años16 comments

What are your company's anti-values?

willsewell.com

294 pointsby willsewellhace 4 años260 comments

Counting Connections at Scale

blog.pusher.com

1 pointsby willsewellhace 5 años0 comments

How we unified our dashboards (and why we had two in the first place)

blog.pusher.com

1 pointsby willsewellhace 6 años0 comments

Debugging MySQL replication lag by diving into the internals

making.pusher.com

1 pointsby willsewellhace 6 años0 comments

Ephemeral port exhaustion and how to avoid it

making.pusher.com

2 pointsby willsewellhace 8 años0 comments

Alternatives to sum types in Go

making.pusher.com

5 pointsby willsewellhace 8 años2 comments

Per-IP rate limiting with iptables

making.pusher.com

1 pointsby willsewellhace 9 años0 comments

Golang's Realtime GC in Theory and Practice

blog.pusher.com

8 pointsby willsewellhace 10 años0 comments

comments

hace 2 añosdiscuss

I think this depends on the library. There are some libraries that are just used in a handful of services, and in those cases I don't think that wrapping is worth the overhead.

However for some of the core platform functionality depended on by all services, we've seen a lot of value in wrapping because it:

- We can provide a more opinionated interface (a third party library is typically more flexible because it has vastly more variety in its use cases) - We can hook in our own instrumentation easily (e.g. metrics, logging) or change functionality with config/feature flags - We can transparently change the implementation in the future

hace 2 añosdiscuss

There was previous discussion related to our microservices architecture here: https://news.ycombinator.com/item?id=22725989.

hace 2 añosdiscuss

I think we've gone back on forth on this over the years. The rate of new services is decreasing, so I think we have shifted more from lots of very small services (low thousands lines of code) to bigger ones that are more like an entire product (e.g. 100k LoC+). But I wouldn't be surprised if the pendulum shifts back again in the future - there are downsides with the larger services, like greater contention withe other engineers.

hace 4 añosdiscuss

I've noticed these tools increase my reading speed:

* Sprint reading: http://www.sprintreader.com/. You can install a Chrome extension here: https://chrome.google.com/webstore/detail/sprint-reader-spee...

* "Bionic Reading": https://www.reddit.com/r/Damnthatsinteresting/comments/usdy2...

hace 4 añosdiscuss

This is a potential downside of this architecture. As others have already said, there are mitigations, but fundamentally each edge request is going to be more costly (time or money) to serve than having it served by a single machine/DB.

One of our engineering principals is to avoid premature optimisation, which is possibly one of the reasons our architecture has grown in this way. So far, whenever we've needed to fix a performance issue we've been able to solve it locally rather than change the architecture.

At the business level, we've been optimising for growth rather than costs, but this could change in the future, at which point we may need to reconsider our architecture. But for now it's working for us.

hace 4 añosdiscuss

I like this suggestion from another sub-thread:

> bias for action when it's a reversible decision

https://news.ycombinator.com/item?id=30376853

hace 4 añosdiscuss

I think the other responses have explained the definition I had in mind for "anti-value".

"Frugality" is a bit less explicit than "move fast _and break things_. I think the reason it could be considered an anti-value is because it implies obvious _costs_. For example it can cost time in "shopping around", or it could mean that you miss out on opportunities - for example missing out on a great employee because you pay below market rate.

hace 4 añosdiscuss

Thank you for raising the point about contradictions in values. I tried to work it into the original blog post, but I felt like it detracted from my main point.

I previously felt these contradictions are problematic because they make it harder to use values for prioritisation. It's a good point you make about different values being applicable in different contexts. I wonder whether the context can be incorporated into the value?

For example "Bias for action when the cost of failure is low". In an oncall incident, restarting a stateless service is worth trying even before the problem is understood in depth, because risk of failure is low. There are potential actions in an oncall incident that could quite easily make things much worse - then it's probably worth diving deeper before taking action.

It might not be pithy enough for the value itself, but I think it's at least adding this kind of context to the subtext like Amazon have done in the page you linked to.

hace 5 añosdiscuss

This are beautiful.

I had a go at doing roughly the same thing a couple of years ago [0]. It was mostly straightforward, but the part I really struggled with were rendering the sea. The encoding of OSM coastline is quite quirky [1]. When the edge of the rendering intersects with the coastline it's very tricky to compute which side of a coastline is sea and which is land. As an example - how would you render this [2]?

I wasn't sure how I could solve this, so I wrote up a more abstract formulation of the problem here [3], and asked for help. I think the proposed solution make sense, but I think I would have to implement it by rendering individual pixels and wouldn't be able to lean on a higher level graphics library.

I'm looking forward to seeing how the author solved this problem.

[0] https://twitter.com/willsewell_/status/1172523752699113473

[1] https://wiki.openstreetmap.org/wiki/Tag:natural%3Dcoastline#...

[2] https://www.openstreetmap.org/#map=15/22.0330/88.8819

[3] https://leetcode.com/discuss/general-discussion/1104642/im-s...

hace 6 añosdiscuss

I see this sentiment a lot in the Go community. I think it is reasonable in some cases, but there are many use cases (vanilla CRUD web apps) where a web framework is really helpful.

The standard library is very low level. Want sessions? DIY. Want user auth? DIY. Want CSRF protection? DIY. The list goes on.

It feels like a waste of time implementing these "solved problems" from scratch, but the biggest problem is how easy it is to introduce security vulnerabilities when implementing from scratch, or forgetting to do so.

It’s nice to learn concepts from first principals by using the standard library. But once I know how these things work, I’d rather rely on someone else’s battle tested code and best practices.

Yes, you can add in separate libraries to solve these specific problems, but they are less likely to compose as well as they would in a framework. On top of this, each time you pull in a new library you have to spend time evaluating it. When I use a framework I don't have to think.

hace 6 añosdiscuss

Pusher | https://pusher.com/ | Ruby on Rails Engineer | £50-£70k + equity | London office or remote (EU timezone preferred) | Full-Time

Pusher’s APIs provide realtime capabilities to thousands of developers around the globe. Every day billions of messages are sent through millions of WebSocket connections to our servers. Through our many SDKs we make it easy for developers to make amazing realtime features like chat, live updates, and various collaborative tools.

We are looking for a RoR to help build features for our dashboard. Our customers use our dashboard to manage their apps and get insights on their usage. We have a fairly large RoR code base, which recently got refreshed (upgraded to RoR 5.2) and have begun using view-components and webpack. We run it all on Heroku.

There are still opportunities to modernise and simplify the system as well as adding new exciting features to the dashboards and this role will be essential to achieving those goals. You will work closely together with our Front End engineer on new features and app performance.

To apply: https://apply.workable.com/pusher/j/DE300A6018/

hace 8 añosdiscuss

Somewhat off topic, but the idea to show the examples as diffs against the prior one was based on this block post: https://jameshfisher.com/2017/11/23/diff-views.

I would be interested to hear people's thoughts on the effectiveness. I like how it is clear what is important in each example, but there is perhaps too much noise and it's a shame the regular syntax highlighting does not work.

hace 9 añosdiscuss

I think what is often missed with values, is the "anti values". Often when I see a list of company values, it looks as if they can have their cake and eat it too. They are not explicit about the trade offs being made; e.g. "more perks (e.g. beer), lower salary". I think if it's hard to find one of these opposing anti-values, then it's not really an interesting choice to make, i.e. at worst it's probably a nice to have.

Last time we discussed our values at work, we thought about what we were trading off and this helped us decide whether they really were important to us.

hace 9 añosdiscuss

Pusher: https://making.pusher.com/

Disclosure: I work there and have written articles for the blog.

At the moment we mainly blog about our experiences with Golang (our language of choice right now). But really it's open to any topic someone in our engineering team is interested in writing about.

We aim to keep things visual, interactive and example-based. For example Jim Fisher created an interactive animation of Golang's GC algorithm here: https://making.pusher.com/golangs-real-time-gc-in-theory-and.... We also managed to embed Golang's trace visualiser within one of the posts: https://making.pusher.com/go-tool-trace/#tour-of-the-go-tool... (using some pretty dirty tricks).

hace 9 añosdiscuss

> How active is GHC's development?

There are two active areas of development which could solve this problem (long pause times with a large working set):

* Compact regions -- which will provide off-heap storage that can be manually memory managed. These are shipping with GHC 8.2. See the paper here if you're interested: http://ezyang.com/compact.html

* Linear types -- A type system extension which allows the programmer to specify values which can only be used a single time. This allows a king of type-safe manual memory management where the compiler statically check that values are freed. This work is further off, but there is currently a prototype being developed in GHC. Check out this blog post for more information: http://blog.tweag.io/posts/2017-03-13-linear-types.html

> Would it be possible to efficiently run Haskell using Go's runtime environment, i.e. by changing the compiler backend?

It's an interesting thought, although I suspect this would be very challenging in practice. Haskell's lazy evaluation, with thunks instead of concrete values, probably wouldn't play nicely with Go's heap layout. I would also suspect Go's runtime has been quite special-cased to Go the language. It would nice if the Go runtime system was opened up in the same way the JVM is.

At this point it might be simpler to translate Haskell source syntax into Go.

My overall feeling is that putting more work into the standard GHC runtime system would be a more efficient use of time.

hace 9 añosdiscuss

We're asked this question a lot (I work at Pusher). I wrote up some of my thoughts on this in reply to a comment on one of our previous blog posts: https://www.reddit.com/r/programming/comments/5fyhjb/golangs....

Right now I'm keen to get more personal experience with Rust, and hopefully in the wider team. I'm particularly interested in [Tokio](https://tokio.rs/), which should address some of our concerns about the existing concurrency support.

Also, in same reddit comments there's a longer (quite argumentative) discussion on Rust concurrency that you might be interested in: https://www.reddit.com/r/programming/comments/5fyhjb/golangs...

hace 9 añosdiscuss

I worked with Michael on the same project, after also working with Haskell previously. On the whole I agree with the pros/cons stated in the article. Having said that, my conclusion would be a bit different: I would err on the side of Go for the majority of commercial projects.

The article mentions the impressive worst case pause times of Go's GC. Since then we have performed some additional benchmarking. The conclusion was: it is impressive, but there are still a couple of issues that break the sub 1ms claims. We blogged about this here: https://making.pusher.com/golangs-real-time-gc-in-theory-and.... It's hard to guarantee low latency in all cases...

Michael also mentions that there is nothing like ThreadScope, or at least nothing that's easy to find. The latter is true. There is an impressive runtime system event visualiser which can be opened with `go tool trace` https://golang.org/cmd/trace/. You can see a screenshot of this in the GC blog post I linked to above. Unfortunately the only documentation is this Google Doc: https://docs.google.com/document/d/1FP5apqzBgr7ahCCgFO-yoVhk... which is tricky to find, and could be more in-depth.

I'm in the middle of writing a blog post on how to use this too. Watch out for it on our engineering blog in the next month or two. https://making.pusher.com/

hace 10 añosdiscuss

It's a shame -- and slightly surprises me -- that ITAR applies to roles like this. Does it apply to entire orgs, or is SpaceX to small to be worth the trouble of creating separate legal categories of employees?

hace 10 añosdiscuss

How long is not "anytime soon"?

Quoting from the excellent "Wait But Why?" blog post The AI Revolution: Our Immortality or Extinction (http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...):

> In 2013, Vincent C. Müller and Nick Bostrom conducted a survey that asked hundreds of AI experts at a series of conferences the following question: “For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such [Human-Level Machine Intelligence] to exist?” It asked them to name an optimistic year (one in which they believe there’s a 10% chance we’ll have AGI), a realistic guess (a year they believe there’s a 50% chance of AGI—i.e. after that year they think it’s more likely than not that we’ll have AGI), and a safe guess (the earliest year by which they can say with 90% certainty we’ll have AGI). Gathered together as one data set, here were the results:

  > Median optimistic year (10% likelihood): 2022
  > Median realistic year (50% likelihood): 2040
  > Median pessimistic year (90% likelihood): 2075

> So the median participant thinks it’s more likely than not that we’ll have AGI 25 years from now. The 90% median answer of 2075 means that if you’re a teenager right now, the median respondent, along with over half of the group of AI experts, is almost certain AGI will happen within your lifetime.

> A separate study, conducted recently by author James Barrat at Ben Goertzel’s annual AGI Conference, did away with percentages and simply asked when participants thought AGI [Artificial General Intelligence] would be achieved—by 2030, by 2050, by 2100, after 2100, or never. The results:

  > By 2030: 42% of respondents
  > By 2050: 25%
  > By 2100: 20%
  > After 2100: 10%
  > Never: 2%

This might be longer than "anytime soon". But when I read it (as a layperson), I was surprised how relatively first these experts believed it would happen.

hace 10 añosdiscuss

From my understanding, the weak generational hypothesis holds for the majority of programs in general. In fact, it is particularly true of language that encourage immutable data ( because new data is created rather than existing data updated). I don't see why it would apply particularly to OO languages, in fact, I would assume that passing pointers around would result in less short-lived objects due to less copying.