How did you find your differentiation in a highly commoditized space? It's probably one of the most crowded spaces.
Even within YC, there are many competitors that do pretty much the same thing:
- Firecrawl
- BrowserUse
- Browserbase
- CloudCruise
- NotteLabs
- Intuned
- Expand.ai
- Reworkd
And then you have the extremely well-funded web retrieval players like Parallel and Exa.
How do you differentiate to all these?
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
I'm always genuinely curious on how startups navigate the founder maze as it helps to break the myth of an overnight success story.
Based on your YC page, you went through a couple of pivots over the last years:
- 4 years ago: Intuned - The data assistant for engineering leaders [0]
- 2 years ago: Intuned - The browser automation platform for developers and product teams [1]
- 1 year ago: Intuned Auth Sessions - Build authenticated scrapers and RPA [2]
What was kind of the evolution from YC S22 4 years ago till you arrived at today's launch? How did you find your differentiation in a highly commoditized space? Even within YC, there are many competitors like Firecrawl, Reworkd, BrowserUse, NotteLabs, Browserbase, etc.
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Intuned respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
Why is this not a Launch YC (or at least mention it?) since you seem to be part of the current batch?
The record/replay is definitely and interesting direction. The browser automation space is getting super crowded though (even within YC), so curious to hear how you differentiate from:
AI crawlers have lead to a big surge in scraping activity, and most of these bots don't respect any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits, user agents, etc.).
This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Webhound respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
The third quote is from a VC who has never founded a startup himself and has a clear interest in pushing founders to trade work-life balance for his own quick returns.
So none of these people worked on anything longer than 2 years. I wonder what will happen if we check back in 5–10 years. Will they still be doing and promoting 996, or will they be burned out and have changed their minds? Make your bets.
AI crawlers have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN (and experienced myself).
Do you have any built-in features that address these issues?
AI agents have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN.
Do you have any built-in features that address these issues?
I'm noticing a big increase in crawling activity on the sites I manage, likely from bots collecting data for LLMs. Most of them don't use proper user agents and of course don't stick to any scraping best practices that the industry has developed over the past two decades.
This trend is creating a lot of headaches for developers responsible for maintaining heavily scraped sites.
You have LinkedIn and Twitter examples, where you're very likely violating their TOS as they prohibit any scraping.
I also assume you don't check the robots.txt of websites?
I'm all for automating tedious work, but with all this (mostly AI-related) scraping, things are getting out of hand and creating a lot of headaches for developers maintaining heavily scraped sites.
Affected companies are becoming increasingly frustrated with the army of AI crawlers out there as they won't stick to any scraping best practices (respect robot.txt, use public APIs, no peak load).
It's not necessarily about copyright, but the heavy scraping traffic also leads to increased infra costs.
What's the endgame here? AI can already solve captchas, so the arms race for bot protection is pretty much lost.
Random anecdote: I've created a Suno song as an anniversary gift for my girlfriend. She was absolutely mindblown by it as it's an earworm song with many of our memories.
Always good to be aware of our small tech bubble here and that things we take for granted already, might not even be close to adoption :)
Looking at OpenAdapt, I'm wondering why they didn't integrate Tarsier into AgentGPT, which is their flagship github repo but doesn't seem to be under active development anymore.
> Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be.
Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.
Here's the Tarsier pipeline in a nutshell:
1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot
2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)
3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict
Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai).
By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!
Even within YC, there are many competitors that do pretty much the same thing:
- Firecrawl
- BrowserUse
- Browserbase
- CloudCruise
- NotteLabs
- Intuned
- Expand.ai
- Reworkd
And then you have the extremely well-funded web retrieval players like Parallel and Exa.
How do you differentiate to all these?
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?