You can absolutely parse HTML with regex, so long as the document is finite in length. Every finite language is regular, hence can be parsed with regex's.
> following a highly publicised mass-murder of children which was (entirely wrongly) being blamed on asylum seekers.
The mass-murder was a consequence of the asylum system. Given what is publicly known, parents of Axel Rudakubana were overwhelmingly likely to have been asylum seekers.
> The 2D map analogy was a nice stepping stone for building intuition but now we need to cast it aside, because embeddings operate in hundreds or thousands of dimensions. It’s impossible for us lowly 3-dimensional creatures to visualize what “distance” looks like in 1000 dimensions. Also, we don’t know what each dimension represents, hence the section heading “Very weird multi-dimensional space”.5 One dimension might represent something close to color. The king - man + woman ≈ queen anecdote suggests that these models contain a dimension with some notion of gender. And so on. Well Dude, we just don’t know.
nit. This suggests that the model contains a direction with some notion of gender, not a dimension. Direction and dimension appear to be inextricably linked by definition, but with some handwavy maths, you find that the number of nearly orthogonal dimensions within n dimensional space is exponential with regards to n. This helps explain why spaces on the order of 1k dimensions can "fit" billions of concepts.
A crude way:
To watermark:
First establish a keyed DRBG.
For every nth token prediction:
read a bit from the DRBG for every possible token to label them red/black.
before selecting the next token, set the logit for black tokens to -Inf, this ensures a red token will be selected.
To detect:
Establish the same DRBG.
Tokenize, for each nth token, determine the red set of tokens in that position.
If you only see red tokens in lots of positions, then you can be confident the content is watermarked with your key.
This would probably take a bit of fiddling to work well, but would be pretty much undetectable. Conceptually it's forcing the LLM to use a "flagged" synonym at key positions. A more sophisticated version of a shiboleth.
In practice you might chose to instead watermark all tokens, less heavy handedly (nudge logits, rather than override), and use highly robust error correcting codes.
Is that actually happening? I use o1 a lot for brainstorming etc, but I don't think I could build a program of any significant size without understanding how to program myself so to guide o1 and stitch it's outputs together. It's also not at all uncommon for me to have to do stuff completely myself, it doesn't have great insights into deep problems for instance.