I don't really get it. Disclosure that I work at Looker which is an analytics company, though not really in the same space as Mixpanel. But I'm just confused about this latest trend of "you can get your data out!" from analytics providers that excel at one type of analysis (as Mixpanel does with funnels).
In my mind, the whole value of these providers is that they make that one kind of analysis really easy and pretty. But if you feel like you're outgrowing that and want to slice your data in new ways, I don't think you'd choose Mixpanel for their data capture alone, right? So is it just that the cost of Mixpanel is sunk and so might as well get the data they're already recording out?
Totally agree and the "write-only" nature of SQL is something that makes me crazy. Sounds like you're more of a dev. I'm an analyst, but same problem. For any analytic SQL I wrote more than a couple weeks ago, I end up throwing it out and starting from scratch because it's so hard to understand anything I wrote.
ORMs were a huge step forward for devs, but there wasn't really anything similar for analysts. Looker (where I now work after three years as a customer) is, in many ways, the equivalent of an ORM for analytics.
Agree that it's only one part of the data lifecycle, but if you don't have your own team of engineers to build all those connectors, you can't even get to the point of being able to transform your data.
We at Looker (disclosure: work there), are totally focused on making the transformations fast, flexible and powerful. But without the data to transform, there's nothing we can do to help customers. So we're super psyched that Segment is stepping in to fill this void and get the E and L done, so we can T.
As somebody who works on a data exploration platform (Looker <- disclosure), I'd say that far from making us freak out, it makes us super happy (which is why we're so excited to partner with Segment).
I can't tell you how many potential customers are crazily excited about the idea of centralizing the data that all their apps produce into one central warehouse and putting Looker on top of that, but are stymied by the middle step of actually getting the data OUT of the vendors' APIs and IN to their own warehouse. Being able to point them to an off-the-shelf solution for that problem is a big win for us.
That's kind of a broad question, but in theory, for sure. Once you've got the raw data in your own warehouse, you can roll up at any level of specificity. So that could be per salesperson, per product, per date, per office, whatever. The key is first getting the raw data out of the vendors' control and into your own warehouse.
I don't really agree. I mean, yeah, comprehensiveness is great, and it sounds like Segment is working towards it. But every integration they build is one less custom integration that your data engineers have to build.
I'm a huge proponent of the idea of centralizing the data model. That's the core idea behind Looker (where I just came to work after 3 years as a customer), and I agree it's a hugely powerful change from the world of everybody-in-their-own spreadsheet.
On your other point, though, to echo the build vs. buy discussion from above, I think it's a bit misleading to say "oh, we'll just use an open-source solution and that'll be cheaper." Because if open source means a couple of internal developers and an analyst, that's easily $300k+/year in salaries that you might not spend if you were using a vendor.
Anyway, given your particular statement of the problem you're facing, I'd humbly suggest you take a look at Looker. The data modeling layer that's core to Looker is meant to solve EXACTLY that problem, by leaving your data where it lives and then embedding your business logic in the layer that sits between end users and the data.
That's correct. We (I work at Looker) are written to leverage the analytical capabilities of SQL. So we support all kinds of SQL implementations, from totally standard (PG, MySQL, MS) to MPPs (Redshift, BigQuery) to SQL-on-Hadoop (Hive, Impala, Presto, Spark).
But the JDBC -> Druid connectors that exist look pretty janky. So if someone builds a stable connector, I suspect we'd support it. But for the moment, no Druid.
It's worth distinguishing between the tools that leave the data in your data warehouse (Caravel, Periscope, Mode, Looker (where I work)), and those that have their own data stores (Good Data, Qlikview, etc.) Tableau can connect directly to your datastore, but it's happier if it can operate on data that's stored locally in-memory.
Anyway, the ones where bring your own database can scale as far as the database can bring you.
Really depends on your needs. There are lots of options out there that are happy to talk to Postgres, but each has different strengths and weaknesses. If all you need is a way to basically share and visualize the output of SQL queries and everyone who's using the tool can write SQL well, then look at Periscope or Mode.
If you're ok pulling the data out of Postgres into memory locally and mostly care about manipulation and beautiful dataviz, then look at Tableau.
If you're mostly interested in more data sciency/ML stuff, then Shiny or something else that's R-based is a good option.
If you're interested in being able to embed your business logic into the tool so that non-SQL folks can build their own queries and everybody's relying on the same data definitions, that's where Looker (disclosure: where I work) excels.
Can't speak for Airbnb, but I'm not sure that any of the front-end clients that you mentioned (disclosure: I work at Looker), can talk to Druid. So if Airbnb already had a Druid warehouse in place, they may have decided it was easier to roll their own front-end than migrate to a different backend.
Yup, Looker is a Premier Partner for the GCP launch (https://cloud.google.com/partners/?q=Looker#search). And then we (I work at Looker) are specifically partnering with GCP for the Public Datasets project.
Because LookML (Looker's modeling language) makes it easy to explore and visualize big datasets, we're building out models and dashboards so visitors can get a sense of BigQuery/Looker's power and find insights from the datasets quickly (whether or not they write SQL).
Frankly, there's a ton of public data that's "available" in the sense that you can technically download and clean a CSV, but isn't actually easy to extract meaning from. So we figured we'd select some interesting datasets, do the cleaning, uploading and modeling, and then let folks have at it for free.
If there are specific datasets you'd be interested in seeing, let us know and we'll see what we can do.
Looks like the data goes up through 2015-10-13. I created a Look (disclosure: work for Looker) that shows story counts by day for the last 365 days here: https://looker.com/publicdata/looks/169?show=viz
Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.
And if you've got public data in BigQuery that you want to make explorable/shareable/visualizable to anybody (no SQL required), let Looker know and we'll see what we can do. (disclosure: work at Looker)
Great to see folks digging into this. Looker (where I work) is what you're seeing visualize and make the underlying datasets explorable. Our founder did a great blog post explaining how it works and how quickly these datasets yield interesting insights with just a few lines of code.
Full blog post explaining and walking through the process (and giving access to explore the data fully yourself, no SQL required) is here: http://looker.com/blog/hacking-hacker-news
Felipe and all the other folks at Google have done a great job getting this project off the ground and we're psyched to partner with them. We're working on some new public datasets now, but if you have particular ones you'd like to explore, let us know.
In my mind, the whole value of these providers is that they make that one kind of analysis really easy and pretty. But if you feel like you're outgrowing that and want to slice your data in new ways, I don't think you'd choose Mixpanel for their data capture alone, right? So is it just that the cost of Mixpanel is sunk and so might as well get the data they're already recording out?