https://github.com/terryyin/lizard has been useful to track when functions get too convoluted or long, or when there's too much duplication -- in code generated by agents. Still have to see how well it works long term but it's caught things here and there, I have it in the build steps in my scripts so the agent sees its output.
here’s a basic impl that i trained on tinystories to decent effect: https://gist.github.com/nikki93/f7eae83095f30374d7a3006fd5af... (i used claude code a lot to help with the above bc a new field for me) (i did this with C and mlx before but ultimately gave into the python lol)
but overall it boils down to:
- tokenize the text
- embed tokens (map each to a vector) with a simple NN
- apply positional info so each token also encodes where it is
- do the attention. this bit is key and also very interesting to me. there are three neural networks: Q, K, V – that are applied to each token. you then generate a new sequence of embeddings where each position has the Vs of all tokens added up weighted by the Q of that position dot’d with the K of the other position. the new embeddings are /added/ to the previous layer (adding like this is called ‘residual’)
- also do another NN pass without attention, again adding the output (residual)
there’s actually multiple ‘heads’ each with a different Q, K, V – their outputs are added together before that second NN pass
there’s some normalization at each stage to keep the numbers reasonable and from blowing up
you repeat the attention + forward blocks many times, then the last embedding in the final layer output is what you can sample based on
i was surprised by how quickly this just starts to generate coherent grammar etc. having the training loop also do a generation step to show example output at each stage of training was helpful to see how the output qualitatively improves over time, and it’s kind of cool to “watch” it learn.
this doesn’t cover MoE, sparse vs dense attention and also the whole thing about RL on top of these (whether for human feedback or for doing “search with backtracking and sparse reward”) – i haven’t coded those up yet just kinda read about them…
now the thing is – this is a setup for it to learn some processes spread among the weights that do what it does – but what those processes are seems still very unknown. “mechanistic interpretability” is the space that’s meant to work on that, been looking into it lately.
I think a 'linear types' ish model where the compiler flags an error if you didn't write the explicit unlock call, and only compiles if some acceptable unlock call (or set of calls) is added, would be a good design. It can / would also prevent use-after-consume. I do want the static checks, but I don't think that means that implicit function calls need to be generated on a } or a = (assigning to a variable causes a drop) etc. This is what Rust already does with `.clone()` -- needing the explicit call -- and I think it makes sense for a lot of cases of implicit drop. I have seen discussions about implementing this for Rust and I have it implemented in a C borrow checker experiment I'm trying. ATS is also an example of this, and going further is something like Frama-C or seL4 and so on.
The main point being: the implicitness is not necessary for "compiler makes sure you don't forget". So the original comment about how usage of the explicitly named and paired APIs can clarify intent both for the writer and reader can still stand while not implying that forgetting is involved. I see this dichotomy often being drawn and I think it's important to consider the language design space more granularly. I think a reminder is a better fix for forgetting, rather than assuming what you wanted and doing it for you.
(the explicit calls also let you / work better with "go to defintion" on the editor to see their code, learn what to look up in a manual, step in with a debugger, see reasonable function names in stack traces and profiles, pass / require more arguments to the drop call, let the drop call have return / error values you can do something about (consider `fclose`), let the drop call be async, ...)
This would be nice also to have an API that's usable from C/C++ code running in Wasm. I see libraries often do this where they expose a C/C++ library like Postgres to Wasm and the main / documented API is JS and you have to dig a bit to find the C/C++ API if it's possible to access it that way.
You do have to tag struct fields with a macro, but you can attach contexpr-visitable attributes. There's also a static limit to how many reflectable fields you can have, all reflectable fields need to be at the front of the struct, and the struct needs to be an aggregate.
Nice! Ebiten is a super nice API for Go. Lots there to be inspired by in API design. Another API I like a lot is Love for Lua (which also actually can be used from C++).
Re: the comments on here about the GC etc. -- I've posted about this a couple times before but I've been using a custom Go (subset / some changes) -> C++ compiler for hobby gamedev, which helps with perf, gives access to C/C++ APIs (I've been using Raylib and physics engines etc.) and also especially has good perf in WebAssembly. Another nice thing is you can add in some reflection / metaprogramming stuff for eg. serializing structs or inspector UI for game entity properties. I was briefly experimenting with generating GLSL from Go code too so you can write shaders in Go and pass data to them with shared structs etc.
The compiler: https://github.com/nikki93/gx (it uses the Go parser and semantic analysis phases from the standard library so all it has to do is output C++) (it's a subset / different from regular Go -- no coroutines, no GC; just supporting what I've needed in practice -- typechecks as regular Go so editor autocomplete works etc.)
Very old video of a longer term game project I'm using it on which also shows state-preserving reload on rebuilds and also the editor inspector UI: https://youtu.be/8He97Sl9iy0?si=IJaO0wegyu-nzDRm (you can see it reload preserving state across null pointer crashes too...)
You do need some JS code that asks the browser to run the wasm blob. You can't eg. just have a script tag that refers to a wasm blob yet.
libc does help with things like having an allocator or string operations etc., or for using C libraries that use libc. And that's where emscripten becomes helpful.
Browser functionality like the console or making html elements is exposed through JS interfaces, and the wasm needs
to call out to those. But they may be directly exposed to wasm later (or they may already be at this point in new / experimental browser features).
I do also really like Go for various reasons, and have been working on a Go -> C++ transpiler and associated ECS libs to make a personal game project with. I used it to make a game for Raylib game jam earlier this year too: https://github.com/nikki93/raylib-5k You can see what the development workflow looks like in this video (the ECS stuff also has a built-in editor like a much more minimal version of Unity's): https://www.youtube.com/watch?v=8He97Sl9iy0
I'm trying to decide how much time I should devote to making this easier to set up / use by other people in the medium term, since it's just a side project for me. Might make a codespaces template so it's quick to get started.
I've been using my own little Go (subset / my own extensions) -> C++ compiler -- https://github.com/nikki93/gx -- and found it to be a fun way to add some guardrails and nicer syntax over C++ usage. You get Go's package system and the syntax analyzers / syntax highlighters etc. just work.
Re: perf for hobby gamedev, I basically agree for native builds, but lately I've felt like Wasm support seems key for hobby gamedev (so you can have more people play your game / without downloading it / it works directly on mobile too without dealing with app or play store). And Go perf in Wasm unfortunately is not so good (I was hitting big GC pauses when trying to make a game with Ebiten and large images).
I ended up writing a Go -> C++ compiler. Go's standard library parser/typechecker made it very doable. The games I've done with it don't use the GC at all but also don't manually manage memory -- they use an ECS api which helps. https://github.com/nikki93/gx -- the README links to development workflow video and complete example game code. I get the perf, interop/libs and portability of C/C++ but with Go's developer experience (well the build system involves C/C++ of course but I have something set up there that I now just use).
For the client I use a simple go -> c++ compiler I wrote and compile to wasm from that actually, on my side projects. It had zero overhead interfacing to / calls to C/C++ (including generics<->templates) since it's just generating that. Example web game made with that: https://github.com/nikki93/raylib-5k
I think I've seen wasmtime before. If I needed to interface to any C/C++ things on the server I would probably just write in C/C++ (or Gx) yeah.
Personally I'm not super familiar with its benefits if any on the server and would actually not use it on the server myself and just build binaries directly, probably using Go. But I've seen some references to Wasm on the serverside for something similar to containerization or loading plugins. It does seem less obvious to me than the client side.
What makes you say Wasm is a server side runtime / imply that it's meant to be one?
GC implementations over Wasm definitely have perf issues. eg. Go compiled to Wasm has these with its GC. The main way when compiling to Wasm now is to not use a GC, which is reasonable depending on the application in question. It's quite reasonable for games or graphics applications.
The extent of Wasm's availability on browsers is quite big right now. Both iOS and Android and all major desktop browsers. That sort of reach is what I meant by the term "the browser" used generically. Different from being an extension to one or a few browsers or something like that.
One of the things that killed Flash was not being supported on iOS while Wasm runs fine (and actually pretty well!) on iOS which is pretty good for reach.
I think you missed the part about it running in the browser? But yes, some previous technologies are similar to some new technologies, that's not really an insight at this point. Especially not about bytecode interpreters which seems like a standard practice.