How is this multiplayer? Claude-slackbot needs to inherit my permissions for accessing scoped context and data / tools. Sounds like I need to setup specific credentials for each instance of the agent.
More importantly, claude-slackbot automatically remembering sounds like it'll be company wide AI slop? 75% of the stuff on slack should not be remembered for the future. And 90% of the stuff worth having in context is not even on the slack thread.
The AI being wrong problem is probably not insurmountable.
Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed.
Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not.
If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.
You’re right in that it’s obviously not the only problem.
But without solving this seems like no matter how good the models get it’ll never be enough.
Or, yes, the biggest research breakthrough we need is reliable calibrated confidence. And that’ll allow existing models as they are to become spectacularly more useful.
Is it to build a copilot for a data analyst or to get business insight without going through an analyst?
If it’s the latter - then imho no amount of text to sql sophistication will solve the problem because it’s impossible for a non analyst to understand if the sql is correct or sufficient.
These don’t seem like text2sql problems:
> Why did we hit only 80% of our daily ecommmerce transaction yesterday?
> Why is customer acquisition cost trending up?
> Why was the campaign in NYC worse than the same in SF?
I totally agree with the sentiment here, but disagree with the conclusion.
1. The rest of the world doesn't have facebooks data layer. Ergo, GraphQL is very hard for everyone who's not FB.
2. Turns out GraphQL is actually a really good data API. Whether it's a postgres/mongo/graph/REST data source. Because it's a great mid-point between something as flexible as SQL but as controlled as a REST API.
Here are 3 things that GraphQL is absolutely great at:
- Select a 2 fields out of a 100 attributes in a data model? GraphQL is great.
- Gradually deprecate an old field in a database and replace with a new one? GraphQL is great.
- Want a flexible way to filter/join/paginate regardless of the underlying storage layer? GraphQL is great.
Other API formats suck at this.
Working with GraphQL ought to feel like you're working entirely with your database and sprinking the right bits of transform/validation/authz business logic and then the API that other teams can use is "free".
GraphQL is dying as a replacement to a REST API layer. I think GraphQL will see its second wind as a data API. Microsoft and Google announced their GraphQL data APIs recently.
GraphQL will probably only make sense when it's as close as a 1:1 map to a DB beneath it. Whether relational or NoSQL or graph.
In all other cases, the GraphQL juice is not worth the squeeze.
I’m the founder of Hasura - sharing some notes from how I’ve seen GraphQL usage evolve over the last few years.
1. GraphQL was and remains insanely hard to build without an underlying data layer that does the heavy lifting. Without projection push-down, predicate push-down, a data layer that can support heavy parallelism it’s untenable. Exactly the problems the OP highlights - needing to “hoist”…
2. GraphQL on REST is an anti-pattern. The hype takes you there, but the juice is not worth the squeeze. The problem was lack of types? Add openapi. The problem was custom aggregation endpoints? Make it super easy / cheap to build aggregate endpoints in REST. Use an AI copilot to write an aggregate REST endpoint and a openapi schema for it even. But maybe the last thing to do is to build annd mainatian another API layer with a completely different execution model.
It’s not what it was meant for perhaps, but GraphQL’s killer use-case is an API to access / operate on data. Especially when there are multiple consumers of data (services, apps) and they don’t own the underlying data sources or speak the underlying language of the database. This turns out to be the underlying problem behind a lot of api, data modernization / migration / decomposition efforts.
The cost of this approach is implementing
Graphql by building a compiler/planner with a robust authz system baked into it. A resolver based approach can never cut it, unless you resolve a query plan - aka build a compiler.
If you want to use graphql to just batch rest endpoints, it’s very likely going to become legacy tech as soon as the team that built it starts to move on.
I think there are 2 subtleties here on where you optimize perf:
1. Frontend makes n+1 requests to the api. Either use GraphQL or build a custom endpoint that shapes a perfect response.
2. Build an api server that fetches data from the db efficiently without n+1 hitting the DB.
The whole tension introduced with resolver/data-loader ways of building GraphQL, is that while it makes 1 easy, it makes 2 harder. Compared to bespoke REST-ish endpoints.
@alex_lav: Your SQL query here is exactly what I meant!
In SQL, this is easy, both to write and to optimize perf.
Implementing this with GraphQL resolvers (how folks typically write GraphQL servers) is hard - there's a users function and an articles function that are both called, so it's hard to implement those 2 functions contributing together towards a query plan. This was my point.
In fact a REST endpoint that takes a list of users and runs this query with the `IN` part parameterized is easier to build here. But the GraphQL version is a pain.
The problem is that it’s hard to get the articles resolver make one query to fetch all the recent articles. So you’ll end up with n+1 fetches from the db.
In the articles resolver, you have to:
Select from articles order by created where article.user_id = X limit 5.
And you’ll have to run this statement n times - once for each user.
Not sure how a data-fetching cache layer at the app server layer will help here.
> it's often much easier to scale the application server layer that the DB layer on most cases.
Hm…ignoring the specific design decision here on using json agg - I think the open question is - is that specific sentiment practically still the truth in 2023? Is it worth scaling app servers or is it better to use Postgres well and scale a managed PG offering?
Do we need to spend time writing boring code when we could instead just use DBs better? And then instead we can spend time where it matters in the app server.
> And in most situations you want to "expand" the data as close as possible to the final location.
What do you mean by “expand”?
From a perf pov: Imagine you’re fetching 100 authors and each had 1000 articles, would you rather a) fetch mxn Cartesian product or b) ask the DB to send you a json aggregation where each author has a 1000 articles? There’s a significant db perf, intermediate layer ser/de, and network latency difference right?
Totally feel the pain. Ironically, most DBs have started supported really good json aggregation functions.
Combined with lateral joins, it’s possible to even push this ser/de to the DB directly and have minimal unpacking and packing in sending that data back to the client!
IMHO: The way we often approach GraphQL execution is a little dated. It’s perhaps legacy thinking from times when we had data layers that couldn’t be expected to scale even for the majority of use cases. But that has completely changed over the last 5-10 years - DBs can scale pretty massively, for specific workloads with very little work required at the app server layer. So if we used DBs to actually model as close to our domain entities as possible (and yes, we’ll need multiple DBs sooner or later), then a huge part of our work is already technically done.
So - here’s another way to approach building GraphQL:
1. Build a “read graph” so that you get a highly composable API that can almost entirely be pushed down into a data layer (eg: Postgres). This would involve techniques similar to building a compiler. Which is hard, but the great news is that it’s easy to make reusable across people who share the same data layer (Eg: Postgres).
There are only 2 business logic concerns at layer, both of which can be pushed down, which helps simplify the problem.
1) modelling/transformation: which can be done in the language of the data layer and pushed down (like views)
2) authz: which can also be done in the language of the data layer and pushed down (like rls)
2. The “command graph”: a set of controllers written in whatever language you please that makes changes to the data layer (or delegates to an orchestration system) and returns a reference to the read graph after having made changes.
This is kind of how we approach the problem at Hasura - a connector SDK that makes it easy to build reusable query compilers for different DBs or API services.
And it’s been great to see similar ideas in the ecosystem - let’s make it easy to execute a tree of functions of create a query plan, instead of executing a tree of data fetch functions (like the OP). Grafast by the postgraphile folks comes to mind.
Came across this blogpost that happens to describe where something like hasura sits and how it decouples product dev (app + bff) from data api (microservice + db) dev.
Think of Hasura as a GraphQL BFD (backend for data) instead of a GraphQL BFF.
Scaling subscriptions is hard, but we work with our users/customers at scale to make sure settings are tweaked correctly.
We have users running 100k - 1M concurrent users in production for live-event type platforms. It's not completely trivial to benchmark and setup because query patterns, streaming vs live queries etc have an impact, but it works very reliably. No missing events, no problems disconnecting/reconnecting, no need for sticky sessions and so on.
An initial POC benchmark [1] should be a quick affair so if you're trying it out and run into any problems, please hit me up! Email on my bio.