Yeah for really big crawls your probably better off sticking it on a server or AWS, as much as anything so you don't need to leave your computer on for ages.
But Sitebulb is not resource hungry in the same was as other desktop crawlers. It saves to disk instead of using RAM, so you don't experience the same limitations.
I'm not sure what you mean about Google. There is no link between Sitebulb and Google - it doesn't visit Google at all, so there is no risk of banning. Using it on your 100 Mb work line would be ideal.
It should stick the subdomain you specify in the start URL, unless there is a redirect or something. Other subdomains won't be crawled, although it will HTTP status check links to subdomains. So possibly that is what you saw in the URL log on the crawl progress page?
If you want me to take a closer look send the subdomain over to [email protected] and I'll see what's going on.
- Email confirmation is required for the username/password, which is how free and trial licenses are controlled, and ultimately how paid licenses are doled out. So we need it for the licensing.
- No special characters at all! Excepts periods. Sorry!
- Agreed, we need to improve the settings switcher.
- Crawl Maps is not linked - you mean on the website right? I'll fix that.
- Running audits show on the main Dashboard, seemed kinda overkill to put it on Recent Audits as well. No?
- You can switch of 'Check external' in the Advanced Settings. Kinda 'hidden away' to keep the main settings UI cleaner (otherwise where does it end?!)
- "Filtered URL Lists" - they are there because people want them ('a big list of all the URLs') and kept missing them in our usability tests!
- Why no endless scrolling in tables? It's not easy to do because the data is written to disk, rather than stored in RAM (which is the reason it can typically crawl more pages), so it needs to go and fetch/filter/etc... every time.
I know right, SF is just too cheap for its own good! :p
We think it's a case of horses for courses. Sitebulb has the potential to save you a ton of time when auditing and reporting. If you don't do a lot of that then it might not be a good option for you. If you do, that's where a lot of the value lies.
There's a fully featured 2 week trial to give it a proper go, and the monthly billing means you have the option to switch it on/off as you need it.
Answered this one below already. For completeness:
The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc...
I've written a more comprehensive answer to this here:
The main difference from Screaming Frog (which is legitimately awesome) is the reporting. Once it has finished crawling it will do a lot of pre-processing for you and build graphs, lists of hints, etc...
That's frustrating, sorry. It's a 'reputation issue', that over-protective anti-virus software doles out to smaller software vendors like ourselves. Basically they don't know if it is good or bad because we haven't had millions of installations.
It has a lot more comprehensive reporting and data visualization than the likes of Scrutiny. I have no idea of the scale limitations of Scrutiny, but I'd be very surprised if it can handle ~500,000 URLs.
Absolutely. In development we tried different ways to make the Crawl Map also represent link data, and they were all just unintelligible. Even the Crawl Maps on big sites are hard to get your head around, and that's with Sitebulb sampling quite heavily.
I'd love for us to come up with some sort of solution for it, I just don't know how we'd do it!
Sorry, maybe I misread, but I kind of read the comment as 'what separates this from other cloud products on the market?'
So I wasn't trying to argue what is and isn't possible with cloud architecture, simply what is and isn't possible with (our) cloud-based competitors.
The process is along the lines of: 'Click Start', get taken to a screen which says 'Initializing' or similar, then maybe 2-3 minutes later you'll see something start to happen. But there is little to no data on which URLs are actually being crawled.
Sitebulb, and desktop crawlers in general, has a much quicker feedback loop.
Interesting comment regarding the App Store. To be honest I'm really surprised no one has ever said this to us before. We have another product - (http://urlprofiler.com/) - for Windows and Mac that we've been selling for 3 years and no one has ever given us this feedback.
We're not wedded to a price structure, although we're rolling with monthly for now. I'm pretty sure through weight of demand that we'll need to add Yearly plans in the next few months.
There's nothing preventing a Linux version (it's built in Electron) other than demand really. We'll do it if enough people want it, but we have a bunch of other features on our roadmap that are currently a higher priority.
Price is always a difficult one trying to get the cost/value balance right. We did some pricing sensitivity testing before launch so I'm hopeful we've not got it too wrong.
Although from your comment I think you might be thinking it is a link map, rather than a crawl map. So with the Crawl Map it is mapping out how each URL/node was found when the crawler traversed the site. So each node will only ever have one edge/link.
A link map ends up a LOT more messy, although it's on our roadmap to try and build one of these too!