broses

This gave me an idea: we can take a mixture of tokenizations with learned weights, just like taking a mixture of experts with learned weights. BLT is optimized for compression, but an approach like this could be optimized directly for model performance, and really learn to skim.

Concretely: we learn a medium sized model that takes a partial tokenization and outputs a probability distribution over the endpoints of the next token (say we let the token lengths range from 1 to 64 bytes, the model outputs 64 logits). Then we do a beam search to find the, say, 4 most likely tokenizations. Then we run the transformer on all four tokenizations, and we take the expected value of the loss to be the final loss.

If we train this on prompt-response pairs, so that it only has to learn what to say and doesn't have to predict the context, then it could learn to skim boring stuff by patching it into ~64 byte tokens. Or more if we want.

And ofc we'd use a short context byte level transformer to encode/decode tokens to vectors. Idk this idea is kinda half baked.

broses

작년discuss

Kinda cute. Just linking this somewhere so it will come up in Google.

broses

2년 전discuss

That's so cool! Is there a way to talk on mobile? Maybe we could add a talk button in the bottom right. I might add this to my website!

broses

2년 전discuss

I made this web app to work on my intonation listening, so I can get better at singing. I imagine it would also be useful for violinists and the like, or just anyone who wants to make their relative pitch more precise. The way it works is it plays a random five note melody (from the 12ET C major scale) with one note out of tune, and you have to figure out which note and how many cents out of tune it is.

What's cool about this game is that most of it was written by Claude.ai. I had a working, nearly complete prototype in two minutes without writing any code. Really mind-blowing stuff. I did spend a few hours polishing it though.

The other cool thing is that the simple algorithm I came up with to generate random melodies for the game sometimes generates surprisingly good melodies. I have to take credit for that part so that people know the AI didn't come up with it.

And here's a similar game with a similar story: https://aaron.kriegman.net/pitch

broses

2년 전discuss

SEEKING WORK | Boston | remote okay

I'm looking for some work this month (July) before I start my PhD (math at UCLA). I may also be open to part time work after I start. Here's some highlights from my resume:

Aaron Kriegman

aaronkplus2 @ gmail . com

github.com/akriegman

___Experience___

Symbolic Capital Partners -- Software Developer Intern

    Designed Ethereum trading infrastructure, using Rust with Tokio and PostgreSQL.

DreamsAI -- Software Developer Intern

___Projects___

Liquid Go -- WebRTC, WASM, Rust, Websockets, React -- liquidgo.xyz

    Real time multiplayer browser game based on the board game Go.

Metrica -- Unity, HLSL -- youtu.be/MtrgYVnKpGE

    A video game set in spherical space, i.e. the surface of a four dimensional ball.

broses

3년 전discuss

The problem with Tau is that it's already used for a lot of other things. I like to use π with a line through it for 2π ("pi cross"). But that's less likely to catch on since Tau for 2π is already well known.

broses

3년 전discuss

People have proposed introducing a symbol for 2π before, most often τ. I like to go a step further and introduce a symbol for 2πi. I use pi with a dot above it, pronounced "pi dot". Pi dot can be defined as the period of the exponential function (which can be defined in terms of its Taylor series). Then 2π is pi dot / i, and π is pi dot / 2i. Of π, 2π, and 2πi, 2πi is probably the most natural, even though it's imaginary. I suppose that depends on the type of math you're doing though.

On a similar note, when doing quantum physics, I like to introduce h dot, which is i × h bar. There are tons of formulas where you either get i × h bar or -i / h bar, but these are just h dot and 1 / h dot, so this removes a little sign confusion and saves a little handwriting.

People will argue that real constants are more natural, but maybe they're not. Maybe radians are naturally imaginary, so if h bar is meant to have dimensions of energy time per radian, then it's better to use the imaginary h dot.

broses

4년 전discuss

Awesome, I can't wait to try it (if it works on Linux)! Previously I used `npx http-server` for this, but I'd rather use something dependency free.

broses

4년 전discuss

What os are you using? I found it extremely glitchy on windows, but pretty good on Linux. (It still crashes sometimes when I leave it open for a long time on Linux, but still the best handwriting app I've used on windows or Linux.)

broses

4년 전discuss

I'm studying math at Cambridge University right now, and a lot of resources for our courses are available for free online. I do think that learning on your own is just not the same as learning for a degree, but if you have the motivation then you may find it useful to follow along with parts of our course.

Dexter Chua was a Cambridge math student who started in 2014, TeXed all his notes and shared them on his website [0]. More people have followed in his footsteps, and you can find most of them by googling `Cambridge math notes site:srcf.net`. Many professors also put notes up on their websites, which can also be found by googling `Cambridge math notes`. I find that for many courses the notes are just as good as lectures, even notes written by students, and sometimes they're even better. They're certainly a lot faster.

You'll probably want to start with our first year courses, which are designed to take bright highschoolers and teach them how to think like mathematicians. If you're interested in the math of machine learning, you'll probably want to look at our courses on linear algebra, probability, optimization, and statistics.

[0] https://dec41.user.srcf.net/notes/

broses

4년 전discuss

You should also check out http://aosabook.org/en/index.html, they've collected information on the architecture of a wide range of open source programs.

broses

4년 전discuss

Usually when I can't be bothered to remind myself the syntax for find, my go to these days is `echo **/*pattern*`. Of course, this is mainly just for small searches.

broses

4년 전discuss

A while ago I was trying to find a way to make my terminal scroll back up after a command executed, so that if the output was long I could read it from the top without having to scroll up manually. There are ways to get yours shell to print something after the command executes, so I just needed to find an ansi escape sequence that would scroll up. Unfortunately I didn't see any sequences that do this. Anyone have any ideas?

broses

4년 전discuss

It's probably a good idea to implement this with an existing language so that you don't end up making your own programming language. For example you could just make the whole document a JavaScript template string.

broses

4년 전discuss

That reminds me of a function I have in my environment called `dg` that does `cd` then `ls`. The name `dg` is short for dig, since the name `dig` was taken.

I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.

Submissions

[untitled]

Decibels Are Useful

[untitled]

I made an ear training app using ChatGPT

Run – a solution to a problem that every build system solves differently

Run

Show HN: Liquid Go – A Game Combining Liquid Wars and Go

comments