you are right that a traditional fuzzer (especially a grey-box one like AFL or a white-box one) is superior in speed, cost per execution, and comprehensiveness.
The argument for using an LLM to generate a curated set of fuzz inputs isn't to replace traditional fuzzers, but to complement them by targeting a different class of bugs that traditional fuzzers are often poor at finding.
The goal of this tool is two fold.
1. give LLMs the ability to make use of traditional software testing tools
2. enhance some of the shortcomings in traditional software testing tools by selectively using LLMs (specifically their ability to understand the larger context the code is written in)
I also wish that I had more real world experience. It would help me a ton if I had 25 years of software testing experience.
It sounds like you do have experience, and I would love to learn from you. It would be awesome if you could help us build a tool that is truly useful for you and your work.
That is one of the obvious use cases. There are many others, you are welcome to install the bot and play around with it. I would love to hear your feedback.
The bugs shown in the "real bugs" section are real output from the tool. Are you referring to looking at the full table of bugs that we return? Sometimes we only find one bug in the PR, sometimes our clients don't want us to share other bugs that could expose their work.
I definitely agree that there's a lot of research happening in this space, and the false positive issue is a significant hurdle. From my own research and experimentation, I have also seen how challenging it is to get LLM-powered tools to consistently find real.
Our approach with Jazzberry is specifically focused on the dynamic execution aspect within the PR context. I am seeing that by actually running the code with the specific changes, we can get a clearer signal about functional errors. We're very aware of the need to demonstrate our ability to find those high-severity/exploitable bugs you mentioned, and that's a key metric for us as we continue to develop it.
Given your background, I'd be really interested to hear if you have any thoughts on what approaches you think might be most promising for moving beyond the false positive problem in AI-driven bug finding. Any insights from your work at MIT would be incredibly valuable.
You are right that static linters are incredibly fast and efficient for catching certain classes of issues.
Our focus with the dynamic sandbox execution is aimed at finding bugs that are much harder for static analysis to detect. These are bugs like logical flaws in specific execution paths and unexpected interactions between code changes.
I believe that the unprecedented scale of LLM-generated code will demand a novel approach to software review and testing. Human review may not be able to keep up (or will it become the bottleneck?)
The bet that I am making is that the system reduces its error rate by splitting a broad task into two more focused tasks.
However, it is possible that generating meaningful test cases is a harder problem (with a higher error rate) than producing code. If this is the case, then this idea I am presenting would compound the error rate.
I believe that I have unintentionally misled you. When I say "unbiased oracle" I am talking specifically about the test oracle being unbiased by how the software was implemented. ie. Black Box testing.
I don't think I made the point very clear in the blog (I will rectify that), but I am saying that because LLMs are so easily biased by their prompting that they sometimes perform better when doing black box testing tasks than they do when performing white box testing.