StructuredRAG: JSON Response Formatting with Large Language Models(arxiv.org)
arxiv.org
StructuredRAG: JSON Response Formatting with Large Language Models
https://arxiv.org/abs/2408.11061
3 comments
And OpenAI also has Structured Outputs[1] that has the same effect as Guidance. I use it to safely deserialize remote function calls based on a jsonschema[2]. It works very well.
1. https://openai.com/index/introducing-structured-outputs-in-t... 2. https://github.com/amoffat/manifest
1. https://openai.com/index/introducing-structured-outputs-in-t... 2. https://github.com/amoffat/manifest
To be fair, they first build a benchmark which they call "StructuredRAG" and it doesn't make sense to run constrained decoding against a benchmark, because it would always get you a 100% success chance. Once they have a benchmark, they try to figure out whether it is possible to prompt engineer your way to a 100% success rate and by using ORPO to generate the prompt, they did achieve that 100% success rate without relying on constrained decoding.
> It may additionally be promising to test Chain-of-Thought (CoT) prompting strategies. This entails adding a “rationale” key to the model’s output such that the additional reasoning improves the performance of the model. However, this will require the output to be a composite object with the additional "rationale" key and string-valued response. Our results suggest that this additional output structure may result in lower success rates. Similar in spirit to structured decoding methods, it may also be helpful to prefix the end of the prompt with “{“ or use the key, such as “{“paraphrased_questions”: [".
For most of my prompts I want CoT.
For most of my prompts I want CoT.
[deleted]
For practitioners, using a framework like Guidance which forces the models to write valid JSON as they generate text solves this trivially (https://github.com/guidance-ai/guidance)
For json in particular these frameworks have functions that take in json schemas or pydantic schemas https://guidance.readthedocs.io/en/latest/generated/guidance...