The claude provided skill-creator provides a decent jumping off point. It is easy enough to start with, but unless the skill is really simple I found it best to consider it a scaffold for building more tailored evals and reports.
The report leaves out a lot of detail. Several changes I found useful were: Pair with/without on same screen as left/right for easier viewing, token count for skill consumed, token used per run, time, pass rate, estimated cost, detailed aggregate stats, a parsed version of the conversation log (capturing the jsonl with each run, sometimes reading the log is the only way to find out why it's screwing up), work output logging (in my case screenshots and outputted script code), better formatting (syntax highlighting, log formatting).
Finally, I think the most useful thing was adding a self-reflection pass. After an eval is done, another agent looks at everything from that eval and tries to evaluate what went wrong along the way and what should be added to the skill, and conversely, from the without skill run what was in the skill that didn't need to be. It produces a skill change recommendation file for each eval. A further summary agent aggregates up all those recommendations in a way I can feed back to an agent.
fwiw, I just tried running the agent-skill they provide for fun to migrate an app-router based next 15 site and the end result is it entirely failed to start.
Vite just hangs when running vinext dev, with no output in logs whatsoever beyond printing`vinext dev (Vite 7.3.1)`.
The tool we use for our docs AI answers lets you mine that data for feature requests. It generates a report of what it didn't have answers for and summarizes them as potential feature gaps. (Or at least what it is aware it didn't have answers for).
People seem more willing to ask an AI about certain things then be judged by asking the same question of a human, so in that regard it does seem to surface slightly different feature requests then we hear when talking to customers directly.
We use inkeep.com (not affiliated, just a customer).
I feel like I should be writing with the goal that the end reader is actually an LLM. The LLM will be the one spitting out the answers to the actual users via things like co-pilot, but I am not sure how that should change my approach to structure or level of detail in docs. Heavier on the number of code examples?
I have the reverse problem with a bank atm in my city on google maps. I've marked it permanently closed several times. It will get marked permanently closed, then months later appear open again. The first time that happened, I actually went back thinking maybe they re-opened it, but nope still just a vacant wall with a bunch of 1 star reviews.
Typo's keyboard was very much a copy and probably infringed on even more then was listed.
I can't think of any of their design patents this would interfere with. There's a small chance of some internal mechanical or light guide related patents, but that would be pretty unlikely. Even more unlikely would be BlackBerry having anyone around still that would even know what to look for.
You can find many fake "UL certified" products on Amazon, with a counterfeit UL logo and all. I think the problem is companies like Amazon not being held liable for what they sell.
The report leaves out a lot of detail. Several changes I found useful were: Pair with/without on same screen as left/right for easier viewing, token count for skill consumed, token used per run, time, pass rate, estimated cost, detailed aggregate stats, a parsed version of the conversation log (capturing the jsonl with each run, sometimes reading the log is the only way to find out why it's screwing up), work output logging (in my case screenshots and outputted script code), better formatting (syntax highlighting, log formatting).
Finally, I think the most useful thing was adding a self-reflection pass. After an eval is done, another agent looks at everything from that eval and tries to evaluate what went wrong along the way and what should be added to the skill, and conversely, from the without skill run what was in the skill that didn't need to be. It produces a skill change recommendation file for each eval. A further summary agent aggregates up all those recommendations in a way I can feed back to an agent.