drothlis

> Claude's ability to count pixels and interact with a screen using precise coordinate

I guess you mean its "Computer use" API that can (if I understand correctly) send mouse click at specific coordinates?

I got excited thinking Claude can finally do accurate object detection, but alas no. Here's its output:

> Looking at the image directly, the SPACE key appears near the bottom left of the keyboard interface, but I cannot determine its exact pixel coordinates just by looking at the image. I can see it's positioned below the letter grid and appears wider than the regular letter keys, but I apologize - I cannot reliably extract specific pixel coordinates from just viewing the screenshot.

This is 3.5 Sonnet (their most current model).

And they explicitly call out spatial reasoning as a limitation:

> Claude’s spatial reasoning abilities are limited. It may struggle with tasks requiring precise localization or layouts, like reading an analog clock face or describing exact positions of chess pieces.

--https://docs.anthropic.com/en/docs/build-with-claude/vision#...

Since 2022 I occasionally dip in and test this use-case with the latest models but haven't seen much progress on the spatial reasoning. The multi-modality has been a neat addition though.

drothlis

2 年前議論

I noticed in your demo it generated the prompt "tap on the 'Log in' button located directly below the 'Facebook Password' field".

Does your model consistently get the positions right? (above, below, etc). Every time I play with ChatGPT, even GPT-4o, it can't do basic spatial reasoning. For example, here's a typical output (emphasis mine):

> If YouTube is to the upper *left* of ESPN, press "Up" once, then *"Right"* to move the focus.

(I test TV apps where the input is a remote control, rather than tapping directly on the UI elements.)

drothlis

2 年前議論

Beautiful.

drothlis

2 年前議論

https://en.wikipedia.org/wiki/Characterization_test

aka snapshot tests.

drothlis

3 年前議論

According to the ViperGPT paper their "ImagePatch.find()" uses GLIP.

According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application.

† https://arxiv.org/abs/2206.05836

drothlis

3 年前議論

Could you implement (some of) astroid's inference using stack graphs? [1],[2]

That would allow a lot of caching optimisations, as you can "index" each file in isolation.

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

[2]: https://github.com/github/stack-graphs

drothlis

3 年前議論

It side-steps the problem of git conflicts, I suppose. You'd have to use their tool (`touca diff`? I don't know if that exists) instead of `git diff`.

drothlis

3 年前議論

Some ideas I got from Jeremias Rõßler's talk: https://t.co/xWtA58Q9q5

- Snapshot testing is like version-control but for the outputs rather than the inputs (source code).

- Asserts in traditional unit tests are like "block lists" specifying which changes aren't allowed. Instead, snapshot testing allows you to specify an "allow list" of acceptable differences (e.g. timestamps).

drothlis

4 年前議論

Obviously a sensationalised title, but it's a neat illustration of how you'd apply the language models of the future to real tasks.

drothlis

4 年前議論

Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.

drothlis

4 年前議論

Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).

drothlis

4 年前議論

Related: I think it was Kernighan & Pike's "The Practice Of Programming" where I read the idea of testing a complex implementation by comparing its output against a simpler but less performant implementation.

drothlis

4 年前議論

Interesting thought, somewhat related to the articles on "snapshot testing" that have been trending on HN lately.

drothlis

4 年前議論

"Regression testing" can also refer to a process: When the QA team says they're doing regression testing, it means they're testing that existing functionality hasn't regressed (as opposed to testing a new feature).

I'm not particularly wedded to any of these terms, I'm just pointing out that "regression testing" has an established meaning, and it isn't snapshot testing (outside of certain industries, at least). I do find it amusing that one implementation of snapshot testing (https://pypi.org/project/pytest-regtest/) links to https://en.wikipedia.org/wiki/Regression_testing but that article doesn't describe snapshot testing at all! Maybe the article changed? Oh well, language changes too. ¯\_(ツ)_/¯

drothlis

4 年前議論

In the article they don't change /etc/machine-id, but APT::Machine-ID in apt.conf.

drothlis

4 年前議論

https://approvaltests.com/

drothlis

4 年前議論

...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test

"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/Regression_testing

drothlis

4 年前議論

I realised this is another example of "trees" as first-class citizens in a build system. In my comment above the tree we're passing around is a docker layer; in my LWN article it's an OSTree ref. We use the former for our cloud containers, the latter for our embedded device rootfs and systemd-nspawn containers.

I suppose we could use systemd-nspawn on our cloud servers too, instead of docker, but when we wrote the build system we were already using docker so it was the expedient thing to do at the time.

drothlis

4 年前議論

I suppose they're ideal for cases where there's a simple, obviously correct but slow implementation that you can use to test a more complex but faster implementation.

drothlis

4 年前議論

Fixed your link to the full article: https://lwn.net/Articles/821367/

OSTree is still working very well for us. At the time I wrote the article we had been using OSTree (and the build system I described in the article) for 4 years; 3 years later not much has changed. In those 3 years we have started building images for different architectures and our build system / OSTree handled it just fine, as you'd expect (it already handled cross-compilation, but for a single architecture).

Our build system also builds Docker containers (for our cloud services) but instead of a Dockerfile we use a custom yaml format that lists the base image (like "FROM" in a Dockerfile), the apt packages to install, and the commands to run (like "RUN"). Then we create a lockfile of all the apt packages, download them (into a local OSTree repository, but that's an implementation detail), and install them with a custom "docker run" command + "docker commit". We end up with the base layer + the apt layer + a single layer produced by concatenating all the RUN commands + a layer with our compiled Rust binaries and Python files. We use apt2ostree to generate the lockfile (really it's our patched version of aptly doing the work) but we use docker (not OSTree) to build the Docker layers. We use Docker's standard push/pull mechanisms to deploy these containers.

To hook this up to Ninja we use "marker files" (a file in the build directory) to track whether this work has been done (e.g. you need to regenerate the apt layer if that layer's "marker file" is older than the lockfile).

Our Python "configure" script (i.e. our build system) looks something like this:

    def build_docker_container(name):
        yamlfile = f"{name}/docker-base-image.yaml"
        with copen(yamlfile) as f:
            data = yaml.safe_load(f)
        ubuntu_version = data["ubuntu_version"]
        image = docker_pull("ubuntu:{ubuntu_version}")
        image = docker_apt_install(image, data.get("apt_dependencies"),
                                   lockfile="%s/Packages-%s-amd64.lock" % (
                                        name, ubuntu_version),
                                   ubuntu_version=ubuntu_version)
        cmd = "..."  # RUN commands from yaml
        deps = [...]  # dependencies from files listed in yaml
        return docker_mod(image, cmd, deps)

(Where "copen" is like "open" but it tracks which files have been read by "configure" itself, to detect if we need to re-run "configure").

What I really like about the Python+Ninja combo is that you can pass these targets around as Python variables — you don't have to come up with an explicit filename for each one (the target name is generated by each helper function, e.g. `docker_pull` returns "_build/docker/${name}"). This makes it so convenient to compose these build rules. And if you ever need to debug your build system, everything is very explicit in the generated ninja file.

We don't have a ton of these containers so it has worked well enough for us because the apt layer changes rarely, and the layer above it is small/fast. The main thing we get (that you don't get from vanilla docker/apt) is lockfiles and reproducibility.

投稿

GPT is all you need for the back end

Software testing, and why I'm unhappy about it

Cross-Branch Testing

Simple declarative schema migration for SQLite

Test.ai’s Appium plugin: How does it work, and how accurate is it?

Add visual verification to your Selenium tests with stb-tester

ErgoDox EZ: An incredible mechanical keyboard

[Mathematica] How do I draw a pair of buttocks?

If you're new to bash completion scripts...

Why Emacs

How to learn Emacs

3-minute video of what makes Smalltalk awesome

Helpful programming tools: Case studies

コメント