jmoiron · HackerTrans

I played from 1995~2001/2ish. Probably to this day the most rich online gaming experience I've had; the book version of a succession of inferior films.

jmoiron

10 anni fadiscuss

97th to 104th between 2nd & 3rd is basically all public housing, and there's public housing along the east river that will also have closer, less crowded stations.

A lot of the tunnels for Phase 2 (extending the Q to 125th) were built or partially built in the 1970s, and this phase already has funding commitments.

jmoiron

10 anni fadiscuss

What? He also predicted how and why it would fail. Sun was a big enough player then that it could have survived plenty of other ways. Apple of today looks nothing like the company in 2000, but Sun got caught out more or less exactly as described and never adapted.

jmoiron

11 anni fadiscuss

Same author as GORM so its to be expected.

jmoiron

11 anni fadiscuss

A few comments on some things I'm seeing in these.. comments.

- What about Cassandra/PostgreSQL/Redis?

One of the implications of "99% of data is never read" is that it's incredibly wasteful to keep it all in memory. You might be assuming that you are letting the data expire eventually, but I'm actually not; expiry is a secondary concern to storage.

Once you start to involve the disk (PostgreSQL and Cassandra), you start to get into locality issues that these databases weren't really designed for.

For a more concrete description, lets say I have 2000 machines. Our app is Python, so they run 16 dockerized processes each, with each container reporting 10 simple system metrics every 10 seconds. These metrics are locally aggregated (like statsd) into separate mean, max, min, median, and 90pctile series. I've not instrumented my app yet and that's already 160k writes/sec on average; if our containers all thunder at us it's 1.6m, and frankly this is the "simple" case as:

* we've made some concessions on the resolution

* we only have dense series

Anyone who has used graphite at scale knows that this is a really painful scenario, but these numbers are not particularly big; anywhere you could take an order of magnitude there are a few other places you can add one.

I'm also assuming we are materializing each of these points into their own timeseries, but that's more or less a necessity. It gets back to the locality issues; If we wanted to see the "top 10 system load per node", it's actually quite imperative that we aren't wasting IO cycles on series that _aren't_ system load; we need all we've got to handle the reads.

(As a side point, this is why people in the cloud are adopting so much Go so quickly; it's proven to be easy to write AND read, and also to reduce orders of magnitude in several of the dimensions above, eg. "we can use 1 process per box not 16 containers, and we can get by with 1000 machines not 2000." Having to write your own linked list or whatever doesn't register in the calculus.)

- 1s resolution isn't dense:

No, not always. It's hard to please everyone with these things. In my world, 1s is good (but not great), but 10s seems to more or less be accepted universally, and much sparser (5m, 30m, 6h) is not actually uncommon. At the other end of the spectrum, you can be instrumenting at huge cardinality but very sparsely (think per-user or per-session), perhaps only a few points per timeseries, and the whole of what I've described above kinda gets flipped on its head and a new reality emerges. For what I've described, I quite like the Prometheus approach, but for my very specific use case 1-file-per-metric only beats the filesize block overhead often enough for very long timeframes; to long.

- Why are all TSDB over-engineered?

I hope some of the above has made explicit some of the difficulties in collecting and actually making this data readable. I've only actually thusfar discussed the problem of "associate this {timestamp,value} pair with the correct series in storage"; there are also the following problems:

* you can't query 3 months of 1s resolution data in a reasonable amount of time, so you need to do rollups, but the aggregations we want aren't all associative so you have to do a bunch of them or else you lose accuracy in a huge way (eg, if you do an avg rollup across min data, you flatten out your mins.. which you don't want); this means adding ANOTHER few dimensions to your storage system (time interval, aggregator)

* eventually you have to expire some of this junk, or move it to cold storage; this is a process, and processes require testing, vigilance, monitoring, development, etc.

* you need an actual query system that takes something useful and readable by a human ("AVG system.cpu.usage WHERE env=production AND role=db-master") and determines what series' actually fall into those categories for the time interval you're querying. Anything holistic system that _doesn't_ do this is an evolutionary dead end; eventually, something like Prometheus or Influx will replace them.

These are minimum requirements once you "solve" storage, which is always a very tricky thing to have claimed. If you get here, you've reached what decent SaaS did 4 years ago and what very expensive proprietary systems handled 10 years ago.

- What about Prometheus/InfluxDB/Kdb+/Et al.

Kdb+ is very expensive, its open source documentation is difficult, and its source is unintelligible. It is basically from a different planet that I'm from. Even recently, when I encounter people from, say, the C# world and tell them I work with Python and Go, they ignore Go and say "Wow, there are like no jobs for Python", which I find utterly bewildering. Of course, I never encounter any jobs using C#, either. This is how little some of these spheres overlap sometimes. Someone from the finance world is going to have to come in and reproduce the genius of K and Q for us mortals in a language we understand.

As for Prometheus and InfluxDB, I follow these more closely and have a better understanding of how they operate. I think that they are both doing really valuable work in this space.

From a storage aspect, I think the Prometheus approach is closer to the one that I need for my particular challenges than the InfluxDB one is, and in fact it looks a bit like things we've already had (but also Catena, Parquet, et al..) For most people, storage actually isn't important so long as it's fast enough.

And this is kind of the point of my article. There's starting to be a bit of a convergence among a few Open Source TSDB, and I've taken I've tried to highlight some issues in those approaches and suggest that there's room for improvement. I have my own ideas about what these improvements might look like based on my work at Datadog, and once they're proven (or even disproven) they'll be published somehow.

jmoiron

11 anni fadiscuss

I have actually heard of Kx and Kdb+ and Q and have looked into those products. You're right that I should have mentioned them, because they are important in that space, and in many ways they are miles ahead. The state of time series query languages is quite poor (outside of Influx's effort), so there is a lot to learn from Kdb+ as well.

I was however focusing on recent Open Source efforts and on the general approaches. Hence, I didn't really discuss in detail my own tsdb.

I also find virtually everything to do with K and Kdb to be be simultaneously impressive and utterly unfathomable:

http://code.kx.com/wsvn/code/kx/kdb%2B/s.k

This is a cheap jab to make, but it makes these systems pretty impenetrable from a source level.

jmoiron

11 anni fadiscuss

I have a lot of experience with Django's ORM and I would say that the essential character of these heavier ORMs is missing from the afore mentioned libraries. They can be used in the way you describe, but that's not their typical usage.

AR/Django encourage you to describe your entire schema, _including_ the relationships between tables, as attributes on your model objects. Upon doing this, you get simple and reliable programmatic access to some basic access and storage patterns.

Using this knowledge of your data model, the ORM can now provide you with more advanced tools: it can automatically join across tables (`select_related`), lazily load dependent data[1], generate SQL schema for you, transparently cache queries[2], automatically provide HTML form validators, and even automate database schema migration for you. The more completely you model the system, the more it can do for you.

In these systems, the database is subservient to the model. This is a problem, because the database is reality and the model is a model.

Dropping to "opaque SQL strings" is discouraged, both because it's considered error prone ("You should leave SQL to the professionals! There are lots of eyes on this!") and because there is often no graceful way to integrate custom query code with your model layer; instead, you investigate how to do so within the confines of the ORM. For every case where you can eventually find what you need (`select_related("user__friends__email")`), there are a dozen where you can't.

People start writing things in application code that could be handled easily and more efficiently by the database: aggregations are a classic, since ORMs support is either missing or incredibly complex. As soon as the application becomes non-trivial[3], the problems magnify. They don't do this because they are stupid, or bad developers, they do this because it's what the tools encourage: they encourage simplistic CRUD access and mistrust/suspicion/fear of SQL.

SQLAlchemy is quite different from this, because its primary focus is to model databases, not to provide some kind of declarative object language with its own set of semantics that do not exist in SQL.

Because Go can't do things like meta classes or creating/modifying types at runtime, a lot of this simply isn't there, even though people want it.

Sqlx does like 2 things; it adds named query parameter support, and it marshals rows into structs. Squirrel is just a query builder. The whole philosophy that the database must somehow be modeled and that access to the database is done via that model is absent.

[1] This is especially durable tarmac with which to pave your road to hell.

[2] http://github.com/jmoiron/johnny-cache

[3] This can mean "lots of requests", or "complex schema", or "complex reporting requirements"... all sorts of things.

jmoiron

11 anni fadiscuss

> The belief that ORMs are evil is precisely the belief that this sort of code should be repeated everywhere database access is performed. If you have generalized routines for interacting with the database with more comfortable abstractions then string concatenation, you are using an ORM

This is incorrect.

The sqlx library, included in OP, has generalized routines for interacting with the database, but is not an ORM. The squirrel[1] library for Go lets you produce queries without "string concatenation", but is also very much not an ORM.

An ORM is a specific style of library that attempts to map object oriented data patterns to relational concepts. That's why it's called an Object-Relational Mapping. There are good reasons[2] why people find this approach problematic, which aren't down to cargo culting them as "evil" or believing that everyone should repeat data access code in all projects.

[1]http://github.com/lann/squirrel [2]http://en.wikipedia.org/wiki/Object-relational_impedance_mis...

jmoiron

11 anni fadiscuss

When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away.

However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use.

When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string keys.

The effects were pretty large for our dataset, which was all API results from various popular websites and featured lots of lists of things with repeating keys; on a lot of large documents, the loaded mem object was sometimes 100M for ujson and 50M for simplejson. We ended up switching back because of this.

[1] http://jmoiron.net/blog/python-serialization/

jmoiron

12 anni fadiscuss

Chopsticks are superior for lots of traditional chinese dishes.

jmoiron

12 anni fadiscuss

What tulip should have been.

Being a long time gevent user, I found goroutines pretty familiar and even unspectacular (no pool.map?!). What really converted me to Go was all of the other ways I found writing Go programs to be more pleasant and less error prone than Python programs:

* consistent formatting for all code

* superior distribution story

* compiler magnitudes faster and more effective than pylint

* programs run faster and use far less memory

* far simpler semantics make it easy to read

* higher quality standard components (net/http, eg)

What I traded for this was a 10-20% drop in productivity, which I was fine with. I use Python for all sorts of quick & dirty tasks still, including some batch processing where there's a big discovery phase, but I write all my software in Go.

jmoiron

12 anni fadiscuss

There is no type hierarchy and there are no subtypes. Interfaces do not describe type relationships. The conversion required between `[]int` and `[]interface{}` further demonstrates the significance of this distinction.

As for losing type safety when writing generic containers, this is true, and another reason why people don't do this.

jmoiron

12 anni fadiscuss

Go does not have a type hierarchy, so there is no top type.

You can easily use interface{} to make generic data structures, but interfaces have a time/space/code-complexity cost which makes them not worth it most of the time.

jmoiron

12 anni fadiscuss

The way you describe it ends up being more awkward, because you have to change code to reference the new path, both in your own code, and potentially in the library you've forked and other dependencies that use it. Vendoring by just maintaining a GOPATH for your dependencies that you control is the preferred way.

jmoiron

12 anni fadiscuss

Python 2.7 was released after 3.1 and contains many features backported from the 3.x series in order to ease migration.

jmoiron

12 anni fadiscuss

I spent years out in the woods with my own projects page that nobody could find and very few cared about, happily cutting trees down with nary a witness. I wanted control over backups, presentation, availability. I was terrified of relying on services that could disappear. I found self promotion distasteful, and I was happy to do the work just for its own sake, for the enjoyment of that process.

But then I wrote something people actually used. And they wanted it on Bitbucket, and then on Github, so they could contribute to it and track its progress.

Now that I've come in from my Zen training out in the wilderness, I feel that the old me who wanted "control" over all of my creations was immature and selfish, abstaining from participation in a community and helping no one but myself. I think that the social aspect of hosted coding sites, both for collaboration and exposure, is much more valuable than the control you get from running the whole show yourself.

jmoiron

13 anni fadiscuss

Yeah, I think that bold looks better with sans serif typefaces than underline, so that was a design rather than an ergonomic decision; perhaps not a correct one. I think underlined blue text is probably the "most linky" looking text, given the history of the web.

jmoiron

13 anni fadiscuss

I've independently come up with the same decisions on many of these points and, weirdly, a very similar blog design:

http://jmoiron.net

I decided even to dispatch with the header bar, putting navigation at the bottom of the main page; presumably, people will only care to see more of my writing if they've read the article. Not as good for people revisiting the page, but they can likely remember http://jmoiron.net/blog or bookmark it in order to get to the search.

My fonts are slightly smaller and slightly less contrast, and after looking at Gemmell's blog I wonder if I should change that decision. I wanted to have emphasized elements of the text easily distinguishable from the body while not distracting.

Also, I agree with Gemmell's "The basic tenets of hypertext should be left alone". One of these basic tenets, to me, are the semantic meanings of Blue, Purple, and Red text for normal, :visited, and :active hyperlinks. If you want to chose a link color, do not invert or violate these classical meanings.

jmoiron

13 anni fadiscuss

Unfortunately if you focus on beauty you sometimes break pragmatism. The JavaScript example in particular is not merely a typographical convention, but a way to avoid common errors.

    var i=1
      , j=2
      , k=3

You can remove any of the comma prefixed lines there (even the last one) and not introduce an error. You can add another similarly prefixed line anywhere to the list and not introduce an error. It's obvious if a comma is missing (which is good, because you don't have a compiler to let you know).

It can be difficult to spot the lack of a trailing comma, or the end of this declaration list having a comma instead of a semicolon (, vs ;), both of which will break the execution of your script.

So please, do not change your code to make it look better without understanding why it's like that in the first place.

The classic example is tchanging the following to allman/gnu style braces would break it in JavaScript:

    // works, returns {a:1,b:2}
    return {
        a: 1,
        b: 2
    }

    // semicolon inserted after return, returns undefined
    return
    {
        a: 1,
        b: 2
    }

jmoiron

14 anni fadiscuss

Redshift is a real physical phenomena describing the way light wavelengths get "shifted" (stretched, to visualize) towards the red as they are seen coming from something moving away from the observer:

http://en.wikipedia.org/wiki/Redshift

http://en.wikipedia.org/wiki/Hubbles_law