p3n1s · HackerTrans

p3n1s

no profile record

comments

há 3 anosdiscuss

> QR Code does have the advantage that it's easier to read by imaging software than UPC-A or EAN-13, due to the targeting patterns in the corners. That could increase the speed of scanning items.

The UPC and EAN symbols have huge targeting patterns, the entire symbol is effectively a targeting pattern. That single dimension massive redundancy makes finding and aligning them very easy. When you have a 2D code you need robust dedicated targeting patterns, but locating thick parallel lines in an image is easier. The massive price paid is information density, not read speed. In the 90s there were CMOS imager barcode scanners built on now antiquated embedded PowerPC that could scan most linear barcodes in milliseconds.

But this "interview", yeah it's shite. QR codes are "barcodes" in the meaning that is important here: extremely cheap, conspicuous, accurate machine readable labels. The barcode isn't being replaced at all in this regard.

há 3 anosdiscuss

Curiosity did get the better of me and you seem to be spamming every fucking forum so WTH. The longest time it took for anything was waiting for the file to download. The rest of this was about 5 minutes of effort. For the record I do not know the MIDI spec, but it must be one of the most easy to google and well documented things out there, and it's pretty simple likely because it's nearly 40 years old and had to run on potatoes.

The file you could have reverse engineered is not enough of a challenge for an interview question in a low level/systems field and yet you didn't even attempt to do that part. The file hierarchy is absolute offsets and sizes in 4 byte little endian quantities. It took less than 30 seconds to figure that out. How? Find MThd string, what offset is it at, search for that offset as a value. Notice that it is adjacent to a null terminated string with a name ending in .mid and a small quantity. Is that small quantity added to the original offset the start of the next MThd, yes? Done. Anyone with the shittiest hex editor and Ctrl-F can do this in a minute. Rebuilding the hierarchy is quite simple since the absolute file offset to the entries is adjacent to every directory name. These midinfo things are interesting they also have some sidecar data. Their content might also be something novel to reverse, that might be worth bragging about.

> I spent almost a week (!) reverse engineering their absurd proprietary format using a hex editor and the MIDI spec.

Since this whole affair seems to have just boiled down to naively iterating for the 4 byte MIDI file magic and then examining the MIDI file metadata you didn't even need all the bluster of breaking out the hex editor and a calculator... Bam https://github.com/jimm/midilib, done (the actual get the MIDI data part can be done in 1 line with a string split). It's too bad ChatGPT didn't suggest that. That should extract the 3 pieces of metadata you attempted to store in the directories.

Sidebar, nothing here stands out as absurd. It just looks like the obvious solution some working stiff would put together to bundle some data up. They don't obfuscate or do anything that stands out as fucky. Since it's read only its not like not using sqlite is some cardinal sin and they probably gave it all of 3 seconds of thought it deserved.

If you were using this as a learning exercise, fine, but then go back and check your work because your tempos and key signatures are off. eg the tempos that you categorize as 104 have an actual encoded tempo of 571429 µs/beat, or 1.7499 bps or 104.9999 BPM, ie it's what humans would call 105 BPM. Not the end of the world but this is pretty rookie floating point mistake. And I'm pretty sure you bungled the key signature because at least ones that say Abm are Fm. Why is that... ah you have ignored relative keys. For instance since the meta event FF 59 02 FC 01 has a sf value of 0xFC which is 2's complement -4 that's 4 flats. If this were a major key that would be Ab, but it's a minor key so it's Fm. Oh no, even simple keys are fucked. FF (1 flat, or F major is coming out as Cb, 7 flats), seems like a bungling of 2's complement. My music education culminated in 2nd chair trombone in the 8th grade and everything I know about the MIDI spec I got from the top 3 hits of google, so caveat emptor.

https://en.wikipedia.org/wiki/Key_signature https://www.music.mcgill.ca/~ich/classes/mumt306/StandardMID...

The only reason I knew it was wrong was because I checked the files with mido (I know you claim ruby is superior, but I seem to be doing ok with dull old python like a rube).

Also in the file I count 640 MIDI headers and I only got 639 files out of your thingy so you might want to revisit that part I mentioned about bugs.

It might actually be interesting if you reconstructed the hierarchy in the metadata. That is not hard, it is all absolute offsets, sizes and null terminated ASCII strings.

But what you did seems to have written a (possibly buggy) loop that calls basic standard library functions in ruby, and attempt to poorly parse/convert a few MIDI meta-events. It's not reverse engineering if you have a nearly perfect spec in front of you, you completely skipped the (simple) reversing part and bungled the easy part. And this task shouldn't take a few days nor hours for someone experienced with basic scripting. https://adventofcode.com/2023

> the entire process of iterating through the binary blob, pulling out the MIDI header/track chunks, and then creating entries with smart naming in a zip archive is done entirely in-memory. For the non-programmers, this is otherwise known as "hard" or "showing off".

We have very different definitions of hard. Unless you were writing a shell script doing it any other way is bananas. These are like a few hundred bytes a piece.

Your technical chops are discordant with the arrogance and incivility you've displayed towards others on this forum. (And name dropping Carmack, that's bold)

eg: "Serious question: are you wickedly combative in all discussions, or just when you get the cold feeling that perhaps you might not know as much as you think you do?" in response to someone who simply had a differing opinion to this: "you keep implying HTMX is a good choice, when there are vastly more powerful solutions in play."

"I assume that the person who downvoted my comment is a bootcamp grad. Good luck with your future endeavors."

I see a common denominator here.

há 3 anosdiscuss

[dead]

há 3 anosdiscuss

It looks like you got lucky and this proprietary format is nothing more than standard MIDI file concatenated together with perhaps some additional data that you are able to ignore +/- some header patch. Frankly this barely qualifies as reverse engineering, at least it represents some trivial case, I mean I'm happy it was easy, but reverse engineering just rarely ever works out so straightforward.

And I would expect someone with competence in scripting language of choice to pop out that script which is a loop and file IO in a few minutes, not hours (assuming it is even correct). And if they have a basic experience working with binary files should know how to google the necessary info about MIDI in seconds.

However looking at the transcript I am also confused because it says (correctly): MIDI files typically start with the header "MThd" followed by the header length, format type, number of tracks, and division. It goes on: "Once a MIDI section is found, we'll extract it according to the MIDI file structure". OK. But the script does NOT do that it reads 4 bytes starting from offset 8 as a 32-bit big endian "length" which is not "according to the MIDI file structure". The standard format is 2 bytes for a format specifier (AKA type) (0, 1, or 2), and then 2 bytes for the number of tracks.

ie, this is wrong in some way:

    # Read the MIDI header and length (14 bytes in total: 'MThd' + 4 bytes for header length + 6 bytes header data)
    midi_chunk = io.read(14)

    # Extracting the length of the MIDI data from the header
    midi_data_length = midi_chunk[8..11].unpack('N').first

So either the proprietary format you're dealing with actually does have a variation on the header of the embedded MIDI file. If that's the case, I would have to deduct points from ChatGPT because I would expect a competent developer to comment/document this fact, no where in the transcript is this stated.

The other possibility I can see is that if your file is a bunch of standard Type 1 MIDI files, the unpack/parse is going to read that as 65536 + some small amount and will extract files that are all around that size. Since the next step is to look for another MThd magic it will just gleefully resync (I assume these are small segments), but you will end up skipping a whole bunch of files and they will be unceremoniously tacked onto others (which will just be ignored in many players).

So what did it end up being? If it was the second case, I would also be suspicious that a first crack LLM follow-up "fix" isn't subtly wrong and prone to false splits.

On further thought, how could it be the first case? If it were the outputted files are not standard MIDI. So something is fucked here. Either you have something totally broken or you have further follow-up and we have to believe it is not subtly broken.

"There was a whole lot of reading the MIDI spec, searching for strings in a hex viewer and calculating values in a hex to decimal calculator."

One pearl I would lend in relation to this: use your REPL, that is a productivity accelerator.

I am also sincerely interested in examples of LLMs reverse engineering something with compression or encryption or some checksum, or like some actual complicated structure that has to be teased out (this is something humans do all the time), maybe something that is most easily solved by cracking open the compiled parser, I'm not saying they can't do it, but plainly put this example is too trivial to be interesting and frankly barely qualifies as reverse engineering at least insofar as some sort of RE Turing Test analogue.

----

If the format works the way I think it does (and this is based on nothing more than general experience and this thread, so give me a break), the only robust way to deal with this is to either figure out where in the proprietary data some type of length field is, and clearly ChatGPT was not going in that direction, nor do I believe it would be able to divine that information from a file upload. Or to use this slightly wonkier method but actually read every MIDI chunk header, since standard MIDI has no total file size length encoded in it. The loop should be: look for MThd, read the NEXT 4 bytes for the length, skip, read and write out chunks (ie 4 byte magic followed by 4 byte length), split when chunk type not seen (that's what makes this a bit fragile, but its probably good enough). If you just look for MThd, you'll split if the MIDI data has an 'MThd' in it.

há 3 anosdiscuss

> It feels a bit like the error correction exists mainly so you can embed fancy logos in your QR code

Absolutely not the case. QR codes, like most 2D matrix codes were designed for logistics and supply chain, their leap into the consumer mobile phone arena was an interesting one, but not the original design intent at all (QR was developed at Denso, the automotive parts and logisitics arm of Toyota). The purpose for the error correction is so that the symbol can still be read on a label even if part of it is misprinted or obliterated, it works very well in practice. Industrial scanning software is mature technology, and more sophisticated than you imply (besides, segmenting and aligning barcodes, any barcode, in greyscale images is pretty simple). Position markers are easy to reconstruct and damage can be tolerated.

há 3 anosdiscuss

> nearly every camera in public and industrial use supports color

This is essentially false. Plenty of industrial cameras are black and white because of their speed and light collecting advantages. This matters for high speed identification. Going to color would not be an upgrade.

"missed opportunity"

Limited information density just isn't nearly as big an issue as you seem to believe.

Colors work great until they don't, for non emissive sources they depend on ambient lighting. A black and white code can be read with under virtually any color lighting (it doesn't even have to be in the visible spectrum). And if you've seen anyone futzing getting a read of an eticket off an emissive device such as a screen, that is just much worse once color is brought in.

Most auto ID applications have limited keys and identifiers encoded in the symbol. Just as color cameras are ubiquitous, the same goes for network connectivity, you don't need to store massive amounts of data in the symbol. If color symbols were really that much of a win they would have been adopted in industry. Attempts by groups that don't understand the subject have been made over the years, this is not new:

https://en.wikipedia.org/wiki/High_Capacity_Color_Barcode

"Yes, it requires different printers"

This is quite an understatement. There's no cheap upgrade path from monochrome thermal printing technology (or laser engraving), that's why they have never been replaced all these years.

Color imagers were readily available long before 2005. Color printing has not changed much at all in the past 20 years. If color symbols made sense they would have been adopted a long time ago. They are almost always the wrong approach, too many problems for too little payoff, that's why they are uncommon.

há 3 anosdiscuss

Sure, but GP stated wrt traffic patterns, "This is independent of whether the approach is IFR or VFR." which is not the case. For IFR, as was the case here, the LH was looking to execute an IFR approach, not ever enter the aerodrome traffic circuit.

há 3 anosdiscuss

Aerodrome traffic circuit is the ICAO term for traffic pattern, which is it what it is called in the US (it may be in the glossary but the point is that manual uses "traffic pattern" repeatedly). And this "traffic circuit" you refer to is almost always a VFR thing, as IFR approaches use specific charted procedures that generally do not end with a traffic circuit.

https://skybrary.aero/articles/aerodrome-traffic-circuit

há 3 anosdiscuss

A significant performance degradation due to normal use of the instruction (FSRM) not otherwise documented is a correctness problem. Especially considering that the workaround is to avoid using the CPU feature in many cases. People pay for this CPU feature now they need kernel tooling to warn them when they fallback to some slower workaround because of an alignment issue way up the stack.

há 3 anosdiscuss

Not obvious. Seems like if it can be corrected with microcode just have people use updated microcode rather than litter the kernel with fixes that are effectively patchable software problems.

The accepted fix would not be trivial to anyone not already experienced with the kernel. But more important, it obviously isn’t obvious what is the right way to enable the workaround. The best way is to probably measure at boot time, otherwise how do you know which models and steppings are affected.

há 3 anosdiscuss

> They always knew how to do it, for decades

Unless we're limiting to the purely theoretical this is quite the hyperbole and in that case applies equally well to the 56k modem.

Size and heat are also factors. The process nodes did not exist in the 70s and early 80s to produce the integrated circuits necessary. VLSI technically enabled DSL.

The idea that somehow the difference between 1960s integration and 1990s is "purely economic" is pretty silly. If only they threw trillions of dollars more into research in the 60s would we have had the Intel Pentium then rather than 30 years later, maybe, but that is a useless line of reasoning. Someone has to actually do the "science".

You need to actually design and produce a suitable line card equivalent for every subscriber that will support DSL speeds without taking up the space of an airport for it and the cooling. That didn't exist for decades before DSL was rolled out (which started barely a decade after it was patented).

> DSL technology was perfected before the 56K modem was even invented.

Considering that DSL standards continued to evolve after the 56K modem was obsolete this is also hyperbole.

The other thing not yet mentioned which is pretty fundamental is that DSL eschews the 3.3khz bandwidth of POTS.

há 3 anosdiscuss

56k was not relevant for point to point communicating between arbitrary points on the network though. 56kbps dial-up worked in an asymmetric fashion with a fully digitally connected modem on one end (eg the ISP). Station to station analog modem standards capped out at 33.6k.

> DSL isn't really fundamentally different from a 56k

There's a great deal of additional integrated digital signal processing to make DSL work that was too costly for consumer modems at the time. I wouldn't agree with this assessment. And other than operating on phone lines for last mile it is completely different on all the layers and uses completely different equipment.

> 56k hit a wall because circuit switching has fundamental disadvantages. Packet switching allowed us to get past that.

There were circuit switched networks far faster than 56k, so not sure this had much to do with any inherent limitation.

Packet switching has obvious advantages for network utilization and was a necessary evolution but I don't think it is relevant to any 56k limit or the revolution in DSP that make multi-megabit over copper pairs possible.

há 3 anosdiscuss

"Fast-forward to 1962. Ma Bell begins using T-1s to connect its switching centers. T-1s are a digital communications link, not analog. The entire phone network goes digital. In a modern phone network, the only analog communication is between an individual house's phone line and the phone company's box."

This is misleading. The phone network used analog links (typically microwave relay) for long distance almost exclusively for many years (into the 80s) after the introduction of T1. T1 itself is not terribly well suited to very long distance links though and can only be pressed into it with repeaters every mile or so.

há 3 anosdiscuss

[flagged]

há 3 anosdiscuss

Virtualbox is GPL licensed. Only the extension packs are non GPL. In any case virtualbox competes with VMWare player/Workstation/fusion which we all should know is not their core business.

há 3 anosdiscuss

It's really nothing at all like that. TTYs are the core interface of all terminals in *nix including the one you would use in any X11, Wayland or other graphical environment, those specific incarnations using something called psuedo-ttys but its all the same stuff in the kernel.

What you really mean are plain text mode consoles for which most UNIX likes further extend this to allowing multiple "virtual" consoles to share one display. And MacOS does have text consoles (see video_console.c in the darwin source), they are just not usually used and very hard to get to especially in Apple Silicon era macs (older macs can easily be booted in single user mode with Cmd+S)

But this is still not at all analogous since it's still the same OS underneath. Windows 95 family was a virtual machine manager launched from DOS so booting into DOS means not loading it at all. The issue for compatibility wasn't so much the graphical shell part as the VMM (virtual machine manager).

há 3 anosdiscuss

No, what they are referring to: you could actually just exit Windows entirely, or just skip booting into it and what you would be left at is a plain real mode DOS environment.

From your own reference: To end-users, MS-DOS appears as an underlying component of Windows 95. For example, it is possible to prevent the loading of the graphical user interface and boot the system into a real-mode MS-DOS environment.

You can even try this in the linked VM: Shutdown and Restart in MS-DOS mode. That is a real mode DOS without any Windows running. This was often necessary for games and other heavyweight protected mode DOS applications like CAD/CAM at the time that would not tolerate running in the VM86 environment.

há 3 anosdiscuss

>> The Win32 environment is essentially a protected-mode DOS application"

> 95 and prior were literally running on top of DOS.

Not sure how you're gathering that the GP is contradicting this. Windows 95, 98 and Me are all architecturally the same: a protected-mode DOS app.

Running DOS from within the Windows shell is not the same as exiting or skipping the boot of the Windows graphical shell, and is running in VM86 mode (which as the GP points out existed in Windows/386 / Windows 3.0 Enhanced 386 mode) See Virtual 8086 mode for some more overview.