I led the engineering team of a large adtech company (TripleLift - order of hundreds of billions of events/day) and we evolved from self hosting Kafka, to paying a vendor (Instacluster), to migrating to RedPanda.
RedPanda was a huge win for us. Confluent never made sense to us since we were always so cost conscious but the complexity/risk of managing a critical part of our infra was always something I worried about. RedPanda was able to handle both for us - cheaper than Kafka hosting vendors with significantly better performance. We were pretty early customers but was a huge win for us.
Twing.AI | AI/Full-Stack Engineer & Forward Deployed Engineer | Remote (US-friendly timezones) or NYC | Full-time or Part-time
We’re a lean (4 people, 3 of which are engineers), bootstrapped team building a private-equity automation platform (deal analysis, document processing, chat, investment theme research). Shipping fast, learning daily, and looking for hands-on engineers who want outsized impact.
- Why join: small team with direct ownership of core product, zero bureaucracy, Greenfield decisions and ability to learn extremely quickly, you’ll ship to paying users weekly.
AI / Full-Stack Engineer
- Own features end-to-end: data ingestion → AI pipelines → UI.
- Design and tune LLM/embedding flows; keep latency/cost in check.
- Requirements: strong TypeScript/Node, modern React, SQL + data modeling; comfort evaluating/using AI models; bias to ship.
Forward Deployed Engineer
- Work directly with customers (PE analysts/partners) to turn messy workflows into productized features.
- Rapidly prototype, measure impact, harden for production.
- Requirements: full-stack chops, great product sense, calm under ambiguity, willing to jump between code, data, and customer calls.
Happy to also talk to the self-taught. We care more about ability than credentials and value slope over intercept.
How to apply: Email [email protected] with “HN – [Role]” in the subject, a short note on something you’ve shipped, and any links (GitHub/portfolio). No recruiters.
I worked at an adtech company where we invested a bit in HA across AZ + regions. Lo and behold there was an AWS outage and we stayed up. Too bad our customers didn't and we still took the revenue hit.
Lesson here is that your approach will depend on your industry and peers. Every market will have their won philosophy and requirements here.
I'd reach out to your former manager that you got along with and ask for their advice. Seems you had a good rapport and they know the company dynamics.
Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.
Yea - the idea is that Snowflake will generate these after a query runs in order to help you look at multiple runs of the same query. So imagine you run a query that's "select a from b where c = 1" and you want to find all examples of that query running. That's where "query_hash" comes in. But Snowflake also says well what if we let you be generic about the parameters - so "where c=1" and "where c=2" and "where c=300000" all have the same query_parameterized_hash.
That's the intent but turns out it's only doing a very simple hashing and not actually looking at the canonical version of the query. For example it won't treat aliases/renames as the same even though it should. This makes it harder to look at all queries that are in essence doing the same thing.
Really cool stuff and a nice introduction but curious how much modern compilers do for you already. Especially if you shift to the JIT world - what ends up being the difference between code where people optimize for this vs write in a style optimized around code readability/reuse/etc.
Yea - I get that argument but these days it's just hard to do infra as true FOSS with the hyperscalers and current cloud economics. There is a community license and and the code is visible. Not saying it's ideal but Redpanda is further into the open source world than WarpStream.
Author here. Basic idea is you want some way of defining metrics. So something like “revenue = sum(sales) - sum(discount)” or “retention = whatever” which need to be generated via SQL at query time vs built in to a table. Then you can have higher confidence multiple access paths all have the same definitions for the metrics.
Yes but most data-heavy tasks are parallelizable. SQL itself is naturally parallelizable. There's a reason Apache RAPIDs, Voltron, Kinetica, Sqream, etc exist.
Full transparency I don't have huge amount of experience at working on this massive scale and to your point you need to understand the problem and constraints before you propose a solution.
Author here and there's nuance here but as a rule of thumb data size is a decent enough proxy. Audience here isn't everyone and the goal was to give less experienced data engineers and folk a sense of modern data tools and a possible approach.
But what did you mean by "Read the first paragraph of the `Cost` section"?
Not to change your direction but something I've been toying around is being able to support Algebraic types when defining tables. That way you can offload a lot of the error checking to the database engine's type system and keep application code simpler.
Author here but some ideas I was thinking about:
- An open source data pipeline built on top of R2. A way of keeping data on R2/S3 but then having execution handled in Workers/Lambda. Inspired by what https://www.boilingdata.com/ and https://www.bauplanlabs.com/ are doing.
- Related to above but taking data that's stored in the various big data formats (Parquet, Iceberg, Hudi, etc) and generating many more combinations of the datasets and choose optimal ones based on the workload. You can do this with existing providers but I think the cost element just makes this easier to stomach.
- Abstracting some of the AI/ML products out there and choosing best one for the job by keeping the data on R2 and then shipping it to the relevant providers (since data ingress to them is free) for specific tasks.
-
RedPanda was a huge win for us. Confluent never made sense to us since we were always so cost conscious but the complexity/risk of managing a critical part of our infra was always something I worried about. RedPanda was able to handle both for us - cheaper than Kafka hosting vendors with significantly better performance. We were pretty early customers but was a huge win for us.