The Stick of Jan Sloot (2004)(spronck.net)
spronck.net
The Stick of Jan Sloot (2004)
https://www.spronck.net/sloot.html
16 comments
"The Sloot Digital Coding System was an alleged data sharing technique that its inventor claimed could store a complete digital movie file in 8 kilobytes of data — violating Shannon's source coding theorem by many orders of magnitude. The alleged technique was developed in 1995 by Romke Jan Bernhard Sloot (27 August 1945, Groningen – 11 July 1999,[1] Nieuwegein), a Dutch electronics engineer.[2] In 1999, just days before the conclusion of a contract to sell his invention, Sloot died suddenly of a heart attack. The source code was never recovered, and the technique and claim have never been reproduced or verified. "
From wikipedia
From wikipedia
Fascinating story though, possible or not. And there were some demos for investors, including iirc Sony Entertainment, which would have been hard to fake completely. So at worst, this guy was a decent magician..
There is a book on it with lots of details..
https://www.goodreads.com/book/show/3316995-de-broncode
There is a book on it with lots of details..
https://www.goodreads.com/book/show/3316995-de-broncode
It doesn't take much magic to convince certain pointy haired bosses.
I once had an idea for a file transfer system based on digits of pi that sounds the same as this idea.
You'd give everybody DVDs of digits of pi (or they could calculate them themselves) , and then transfer files faster by just sending them the offset into pi.
At the time I thought it could work with a big enough bank of digits of PI on both sides. If transfer was expensive, and calculating digits was cheap then you could give everyone an infinite supply of digits of pi and have a nearly infinite compression system.
I discovered that often the offset into pi is much larger than the data you are sending. Turns out it's an expensive way to sent things.
Also, it turns out that this area was already well understood. There are no free lunches with entropy.
But it was a fun idea to kick around.
You'd give everybody DVDs of digits of pi (or they could calculate them themselves) , and then transfer files faster by just sending them the offset into pi.
At the time I thought it could work with a big enough bank of digits of PI on both sides. If transfer was expensive, and calculating digits was cheap then you could give everyone an infinite supply of digits of pi and have a nearly infinite compression system.
I discovered that often the offset into pi is much larger than the data you are sending. Turns out it's an expensive way to sent things.
Also, it turns out that this area was already well understood. There are no free lunches with entropy.
But it was a fun idea to kick around.
It "exists" at https://github.com/philipl/pifs
Another potential problem with that approach is that it is possible that not all finite files appear in the digits of pi. It is believed that all appear but it hans't been proven.
Neural network encoding seems like the closest thing to that idea that's actually practical.
Yes 100% true.
You first have to prove that pi contains all possible finite substrings, which I believe is still an open problem.
Neat.
Although a flat memory space / full file approach would be nice, I wonder if a paging scheme would be possible?
Where you would only need all possible finite strings of length N, where N would ideally be the max page size of RAM, but could be less if the substring could not be found within a reasonable offset...
Ignoring compression speed, I suspect it'd have a practical limitation where using a maximum offset would mean that some strings wouldn't be as big as the cell needed to describe them. That's usually the gotcha in schemes like this...
In order to entertain this, I think all possible 12 byte strings (64bit offset + 32bit size) would need to be found within whatever a reasonable amount of storage is.
I suppose the computational challenge is "How big an offset is required to have all possible strings of length N?"
Although a flat memory space / full file approach would be nice, I wonder if a paging scheme would be possible?
Where you would only need all possible finite strings of length N, where N would ideally be the max page size of RAM, but could be less if the substring could not be found within a reasonable offset...
Ignoring compression speed, I suspect it'd have a practical limitation where using a maximum offset would mean that some strings wouldn't be as big as the cell needed to describe them. That's usually the gotcha in schemes like this...
In order to entertain this, I think all possible 12 byte strings (64bit offset + 32bit size) would need to be found within whatever a reasonable amount of storage is.
I suppose the computational challenge is "How big an offset is required to have all possible strings of length N?"
[deleted]
could just count in stead of pi, the offset would be the same number as the value at the offset.
Glad to hear I'm not the only person who tried this. :)
A little OT: do the information theory based proofs of limits on how effective compression can be also prove that backwards time travel is impossible?
If backwards time travel were possible then your "compression algorithm" could simply be deleting the file. To recover the file go back in time to before it was deleted and make a copy.
With this you can "compress" giant files down to just a short description of a place and time where the file was on your computer.
If backwards time travel were possible then your "compression algorithm" could simply be deleting the file. To recover the file go back in time to before it was deleted and make a copy.
With this you can "compress" giant files down to just a short description of a place and time where the file was on your computer.
Interesting idea, but you're not getting any entropy for free by going back in time. If you wanted to map every possible file of a set length to a specific time and location where it's stored, you'd still need, in general, the same number of bits as you would for the file itself, because you'd have to describe that many distinct time/location pairs.
Backwards time travel is perfectly possible, but perhaps not in our actual reality, but in a simulation of our universe, which would get you the data you want.
[deleted]
Weird, I once read about Roel Pieper, completely randomly, in a trade magazine for Unix that I think had very few issues. He had just become head of Unix Systems Laboratories. I don't know why that article stuck with me for so long. I didn't know he was also a professor, I thought he was just an exec. Turns out he was professor of business, not computer science, though his original university education was in CS.
Yes, it's unclear how much theory he knew. Looks like next to nothing.
Aside, I wonder if "Pied Piper" from the Silicon Valley series is a reference to him. Probably not but fun to entertain.
Aside, I wonder if "Pied Piper" from the Silicon Valley series is a reference to him. Probably not but fun to entertain.
Yeah, reading about this that all started to make a lot more sense as a deep joke.
I also always wondered about that when I watched the show!
Related. Others?
Was This Lost Computer Code Worth Billions? (Jan Sloot Digital Coding) (2020) [video] - https://news.ycombinator.com/item?id=36499676 - June 2023 (2 comments)
The Stick of Jan Sloot (2004) - https://news.ycombinator.com/item?id=29623524 - Dec 2021 (22 comments)
Ask HN: What was the secret that Jan Sloot took with him to the grave? - https://news.ycombinator.com/item?id=13443135 - Jan 2017 (4 comments)
The Stick of Jan Sloot (2004) - https://news.ycombinator.com/item?id=8699058 - Dec 2014 (18 comments)
Was This Lost Computer Code Worth Billions? (Jan Sloot Digital Coding) (2020) [video] - https://news.ycombinator.com/item?id=36499676 - June 2023 (2 comments)
The Stick of Jan Sloot (2004) - https://news.ycombinator.com/item?id=29623524 - Dec 2021 (22 comments)
Ask HN: What was the secret that Jan Sloot took with him to the grave? - https://news.ycombinator.com/item?id=13443135 - Jan 2017 (4 comments)
The Stick of Jan Sloot (2004) - https://news.ycombinator.com/item?id=8699058 - Dec 2014 (18 comments)
Hi I just see this via Twitter.
My name is Tom and I have found JS SDCS inner works and OS in 2021/22. I’m an ‘old school’ abstract artist and got through it by image research. Also my dad is a screenwriter so the tv was present in my upbringing. I have my personal story but it’s not about that. Under NDA I’m willing to share my sources and proof. Two people have validated my findings. One is too busy but has his own way by modern algorithms, one is taking it back to the past from step one. I need help and have a certain sense of urgency, btw I’m not afraid to be under attack. I cannot code, math, engineer or program properly. I do know how art, nature and stories of people work quite well. Thank you for your consideration and seriousness. I do need help to proof it with working samples. It’s one YouTube link away basically : ) and if you get it, it can be quite simple. Ty. Tom, Amsterdam, the Netherlands
> The whole problem of Pieper's way of thinking (and of every other person who believes that Sloot actually could compress movies with a factor of two million) is that he believes that the key does not need to contain every detail of a movie. Unfortunately, it does.
I wonder if one was scammed and bought a Blu-ray with everything, only to find out they got novels rather than movies. After the initial disappointment they might realize well written novels are better than movies and never watch another.
How about we get GPT to turn good movies into great books? Like a large inverse prompt engineering problem.
Could we then even feed that text as input to make a better, though arbitrarily different movie?
I wonder if one was scammed and bought a Blu-ray with everything, only to find out they got novels rather than movies. After the initial disappointment they might realize well written novels are better than movies and never watch another.
How about we get GPT to turn good movies into great books? Like a large inverse prompt engineering problem.
Could we then even feed that text as input to make a better, though arbitrarily different movie?
> How about we get GPT to turn good movies into great books? Like a large inverse prompt engineering problem.
GPT has trouble making great sentences, much less great books from a data type completely different from its training. GPT, at the core, is a "likely next word" generator. No great literature came as a process of "likely next word" in a vacuum. Likely next word from complex experiences of a decade and plot designed from those complex experiences, sure. But not the algorithm that GPT is.
> Could we then even feed that text as input to make a better, though arbitrarily different movie?
Nothing about a feedback loop of mediocrity describes a practical way to improve quality. This concept is as sound as the alien encoding stick and the sloot encoding scheme itself.
GPT has trouble making great sentences, much less great books from a data type completely different from its training. GPT, at the core, is a "likely next word" generator. No great literature came as a process of "likely next word" in a vacuum. Likely next word from complex experiences of a decade and plot designed from those complex experiences, sure. But not the algorithm that GPT is.
> Could we then even feed that text as input to make a better, though arbitrarily different movie?
Nothing about a feedback loop of mediocrity describes a practical way to improve quality. This concept is as sound as the alien encoding stick and the sloot encoding scheme itself.
What should be doable:
Deconstruct a movie into a set of 3D models, textures, voices, parameters for those 3D models' movement, procedural generation of background objects like trees, grass, sea, sand, weather conditions, etc, etc. Like the data that forms the content of modern, near-photorealistic games.
A game engine-like rendering system would then render a scene, tweak parameters until rendered scene / frames match the original movie closely, and work through the rest of the movie the same way.
When done, one could produce derivations of such movie by changing actors' voices, have them move differently or speak in another language, swap out buildings or other objects, reduce polygon count or resolution of textures, have a cornfield show a little taller stalks, put the sun in a different angle, etc, etc.
Of course this is way beyond current compute capabilities. Not to mention software frameworks to do this job. But in theory this should work. For a 2min trailer it would probably be pointless. But for a 2..3h movie, maybe not.
And yes, of course this would be lossy 'compression'. Just more high-level than current video compression schemes.
For an audio analogy: compare mp3 compression with MIDI + quality sound banks for every instrument under the sun + parameters like how hard a piano key was struck, etc. Vary such high-level parameters until rendered output matches the original music.
Deconstruct a movie into a set of 3D models, textures, voices, parameters for those 3D models' movement, procedural generation of background objects like trees, grass, sea, sand, weather conditions, etc, etc. Like the data that forms the content of modern, near-photorealistic games.
A game engine-like rendering system would then render a scene, tweak parameters until rendered scene / frames match the original movie closely, and work through the rest of the movie the same way.
When done, one could produce derivations of such movie by changing actors' voices, have them move differently or speak in another language, swap out buildings or other objects, reduce polygon count or resolution of textures, have a cornfield show a little taller stalks, put the sun in a different angle, etc, etc.
Of course this is way beyond current compute capabilities. Not to mention software frameworks to do this job. But in theory this should work. For a 2min trailer it would probably be pointless. But for a 2..3h movie, maybe not.
And yes, of course this would be lossy 'compression'. Just more high-level than current video compression schemes.
For an audio analogy: compare mp3 compression with MIDI + quality sound banks for every instrument under the sun + parameters like how hard a piano key was struck, etc. Vary such high-level parameters until rendered output matches the original music.
What is the limiting factor though? I could make a bad sketch and get a description of that used to render a better picture right? We currently don't optimize for this as a goal so could get better, no?
And how would you train a fitness function to recognize book goodness? It's inherently subjective.
GPT can make great sentences, it's just not the default operating mode. LLMs are capable of amazing things if you can get them into the right section of the latent space.
If the laws and initial state of the universe can be expressed in few enough bits, then all you need to store is a title and a date. Run a simulation of the universe, fast forward to the date the movie was released, use an AI to find a theater playing it, and digitize the tape. It may take longer to run than the current age of the universe, but eventually you’ll get it.
I've thought about this a lot over the years. What you're asking is, can we accurately emulate our own universe inside our own universe at greater than 100% speed? If so, it would allow us to build a type of space-time machine where we could visit any location at any time.
I don't think it is totally inconceivable. It is definitely possible that there are hacks such as compression that would allow a simulator to run at above 100% speed. It also might not need to be 100% accurate -- we might find that running below that threshold still produces an output that is practically indistinguishable from the real events, yet allows it to run.
I don't think it is totally inconceivable. It is definitely possible that there are hacks such as compression that would allow a simulator to run at above 100% speed. It also might not need to be 100% accurate -- we might find that running below that threshold still produces an output that is practically indistinguishable from the real events, yet allows it to run.
> I don't think it is totally inconceivable
You could create a kind of "reverse" grandfather paradox by preventing future events predicted by the simulation from happening. Perhaps it could run fine up to the point that it has to simulate itself, then it would slow down to the point of becoming useless.
You could create a kind of "reverse" grandfather paradox by preventing future events predicted by the simulation from happening. Perhaps it could run fine up to the point that it has to simulate itself, then it would slow down to the point of becoming useless.
Basically the plot of the wonderful TV series "Devs". Well worth the time investment.
Thank you, I never knew this is what Devs was about. I'm now watching the third episode already!
Quantum mechanics makes this impossible. This would require to know the position and speed of each particle precisely and that is in violation of the Heisenberg principle.
You're figuring that the universe is non-deterministic. If so, yes, this would be a more difficult problem.
If the universe is based on some solid deterministic rules with a fixed seed value, then it would work.
Quantum events appear non-deterministic, but that might just be because we don't have all the answers right now.
If the universe is based on some solid deterministic rules with a fixed seed value, then it would work.
Quantum events appear non-deterministic, but that might just be because we don't have all the answers right now.
JS made a copy system, which behaves metaphorically exactly like a seed.
Yeah, but perhaps we're living in a multiverse and you'd have to encode which universe the movie was released in, which may require more bits than the entire movie :)
However, interestingly, by scanning other universes you may find similar movies with slightly different plot turns.
However, interestingly, by scanning other universes you may find similar movies with slightly different plot turns.
Offhand, this brings to mind the Vocoder. A reasonable facsimile of fluent speech could be be produced just from the bandwidth of a skilled typist operating a keyboard.
What's encoded is not the audio - even the best audio codecs need on the order of 4kbits/sec to encode legible speech - but the actual semantics on how voice is produced.
Suppose you want to do this for movies. 8 kilobytes doesn't sound like enough, though the script of the movie could easily be compressed to that. But it's posible to imagine a system into which all the skill of the cinematographer, director, script writer, actors, etc. are built, with from 8 kilobytes of instructions could create, perhaps not the original movie, but something comparable.
Is this doable? Does some random crank have the remotest chance of pulling this off with the technology of the day? No. But that's likely the line of reasoning employed.
What's encoded is not the audio - even the best audio codecs need on the order of 4kbits/sec to encode legible speech - but the actual semantics on how voice is produced.
Suppose you want to do this for movies. 8 kilobytes doesn't sound like enough, though the script of the movie could easily be compressed to that. But it's posible to imagine a system into which all the skill of the cinematographer, director, script writer, actors, etc. are built, with from 8 kilobytes of instructions could create, perhaps not the original movie, but something comparable.
Is this doable? Does some random crank have the remotest chance of pulling this off with the technology of the day? No. But that's likely the line of reasoning employed.
Inventors are a different animal. One doesn't start with a reasonable theory but one works from the other end, one quite unreasonably starts with what would be wonderful to have. One doesn't test a theory but one ponders how the seemingly impossible might be accomplished which is a problem that can be broken down into more reasonable things until it's impossible component is defined properly so that one may make an often futile attempt to ask the question differently. With few exceptions there are no results. Those not skilled in the art consider that part cheating. This is logical because their line of reasoning seeks to further validate that what they already know, it thus involves an effort to prove something can't be done, or at least not by someone like that, or at least not by that person, or at least not by that method etc etc
The cheating in this context would be to study the thing the file represents rather than the representation.
A dumb example would be to make a 10 hour movie from a single image that doesn't move. There is no reason for the file to be larger than the original jpg.
> I can not only get Orson Welles' Citizen Kane. I can get Citizen Kane in colour!
https://www.youtube.com/watch?v=JI5qy9Zoh_0
To argue that this was not the method Sloot used is missing the point. The question is: How to do it, not how to imitate someone else.
In his demo Sloot was playing 16 full movies simultaneously on a 1995 laptop at any speed. A high end computer had 32 MB memory, 133 MHz cpu, PCI video cards had 4 MB ram, 66 MHz, 560 MB HDD
If it was not what he said it was why didn't he just sell what he had? Without the extraordinary claims the demo already requires cartoon physics. He drives the truck into the match box, making an U turn inside doesn't at all seem necessary???
The cheating in this context would be to study the thing the file represents rather than the representation.
A dumb example would be to make a 10 hour movie from a single image that doesn't move. There is no reason for the file to be larger than the original jpg.
> I can not only get Orson Welles' Citizen Kane. I can get Citizen Kane in colour!
https://www.youtube.com/watch?v=JI5qy9Zoh_0
To argue that this was not the method Sloot used is missing the point. The question is: How to do it, not how to imitate someone else.
In his demo Sloot was playing 16 full movies simultaneously on a 1995 laptop at any speed. A high end computer had 32 MB memory, 133 MHz cpu, PCI video cards had 4 MB ram, 66 MHz, 560 MB HDD
If it was not what he said it was why didn't he just sell what he had? Without the extraordinary claims the demo already requires cartoon physics. He drives the truck into the match box, making an U turn inside doesn't at all seem necessary???
Where you’re spot on is that is basically works on the principle of a cartoon. TVs are no computers. They send different signals.
At least one of the ideas alluded to in this article is reminiscent of an actual audio "compression" algorithm. SAOL aka "MPEG-4 Structured Audio"
https://sound.media.mit.edu/resources/mpeg4/sa-tools.html
It was a growing idea in the mid-2000s but AFAIK it has gone absolutely nowhere. Essentially, instead of somehow encoding the audio, you encode a description of how to generate the audio.
https://sound.media.mit.edu/resources/mpeg4/sa-tools.html
It was a growing idea in the mid-2000s but AFAIK it has gone absolutely nowhere. Essentially, instead of somehow encoding the audio, you encode a description of how to generate the audio.
Encoding a description of how to generate the audio was a growing idea in the 1990s and 1980s...
https://www.youtube.com/watch?v=p5kMukDpGlc
https://modarchive.org/
https://news.ycombinator.com/item?id=33273107
https://www.youtube.com/watch?v=p5kMukDpGlc
https://modarchive.org/
https://news.ycombinator.com/item?id=33273107
Isn't this trivially false, way before atoms, Turing machines, and any other drive-by namedrops I missed?
Given the notch = (A+B) / 10
You can only recover A + B. You can't recover A or B individually.
Given the notch = (A+B) / 10
You can only recover A + B. You can't recover A or B individually.
I read “attaches all these numbers to each other” to mean concatenation, not addition. Presumably you’d want to encode a “next book” token too.
This kind of compression is known as arithmetic coding. It does work.
The standard theoretical technique for encoding a list of numbers as a single number is Gödel encoding. It can be applied as many times as you like—e.g. to encode a list of lists of numbers, or a list of lists of lists of numbers. https://en.wikipedia.org/wiki/G%C3%B6del_numbering
Sure, but Gödel encoding is pretty much purely a theoretical exercise. I'm not sure anyone anywhere has ever practically manipulated Gödel-encoded expressions in a useful way. His original scheme also has the problem that prime factorization is rather computationally challenging--it is after all the basis of the RSA cryptosystem.
Whereas Arithmetic encoding is actually practical, extensively used, and a direct analogue to the stick.
Whereas Arithmetic encoding is actually practical, extensively used, and a direct analogue to the stick.
I'm aware of arithmetic encoding and it is definitely the most compelling example of encoding arbitrary data with a single number. On the other hand, there is a lot more to arithmetic coding than the ability to encode lists of numbers—all the considerations involving the context of each symbol, which are essential to the process of compression. I just felt that it might be helpful to give an example which didn't implicate all that complex compression apparatus.
I remember the weird paranoia news that suddenly seemed to pop up everywhere only days after his death. People told me he was murdered by the big shot movie execs because of his amazing algorithms. It all seemed so strangely exciting. I wish i knew the "exceptional claims need exceptional proof" sentence back then.
(2004)
From my (rather limited) interactions with him, I'd say he was far from genius, I feel he just got sucked in and had no way out of his network of lies without losing face.