Spider-Man (Neversoft) Decompilation Project Progress Checkpoint – May 2024(krystalgamer.github.io)
krystalgamer.github.io
Spider-Man (Neversoft) Decompilation Project Progress Checkpoint – May 2024
https://krystalgamer.github.io/spidey-decomp-status-may/
10 comments
If you haven't tried it, I highly recommend the Dreamcast version. It has the graphics and voice-acting of the PC version with the tight analog controls of the N64 version.
I thought Ultimate Spider-Man was a severe downgrade over Spider-Man 2. The mechanics for swinging just felt a lot more automatic and were less satisfying overall. Spider-Man 2 had, in my opinion, nearly perfect swinging mechanics and did a great job feeling truly open. Its biggest problem was that the actual missions weren't terribly fun, particularly the boss fights were clunky and unfun.
Still, it's fun enough to just swing around the city that I can just ignore the missions and just play around.
Spider-Man 2000 was still fun. I haven't played the Dreamcast version, but I played the hell out of the PS1 version, unlocking nearly everything in the process.
Still, it's fun enough to just swing around the city that I can just ignore the missions and just play around.
Spider-Man 2000 was still fun. I haven't played the Dreamcast version, but I played the hell out of the PS1 version, unlocking nearly everything in the process.
That's the thing--Spider-Man 2 was rather flawed overall. The main story missions were meh, and the side missions got tedious after awhile (how many people in this damn city are hanging from a ledge at any given moment?). It was much like GTA3, where it was almost a chore to make myself finish the thing, and I'd normally just fire it up to swing around town aimlessly. I'd probably only rate it an 8/10 overall (at the time). It just had the advantage of absolutely nailing the feeling of web-slinging in a way that no game ever did or arguably did again until Spider-Man (2018).
USM was definitely a downgrade on the freedom/webslinging front, but pretty much everything else was better, and it was a tighter experience. I guess it comes down to whether you're the type who prefers the game with the better overall package, or the game with one killer feature that it does really well and you can't find anywhere else.
USM was definitely a downgrade on the freedom/webslinging front, but pretty much everything else was better, and it was a tighter experience. I guess it comes down to whether you're the type who prefers the game with the better overall package, or the game with one killer feature that it does really well and you can't find anywhere else.
Yeah, that's all fair.
I recently tried playing the Spider-Man 3 game on the PS2, and while I thought the web slinging was better than Ultimate Spider-Man, it still wasn't as tight as Spider-Man 2. I'm not actually 100% sure why it's worse, it's the same company that made it (Treyarch), so presumably they'd have access to the same game engines and could effectively just redo Spider-Man 2 but refine the missions to make them more streamlined and less tedious. They did actually somewhat fix the latter part there, the missions are a lot less boring, but the web slinging just isn't as fun.
I wish someone would make a mod/romhack of Spider-Man 2 with unique missions that are actually fun.
I recently tried playing the Spider-Man 3 game on the PS2, and while I thought the web slinging was better than Ultimate Spider-Man, it still wasn't as tight as Spider-Man 2. I'm not actually 100% sure why it's worse, it's the same company that made it (Treyarch), so presumably they'd have access to the same game engines and could effectively just redo Spider-Man 2 but refine the missions to make them more streamlined and less tedious. They did actually somewhat fix the latter part there, the missions are a lot less boring, but the web slinging just isn't as fun.
I wish someone would make a mod/romhack of Spider-Man 2 with unique missions that are actually fun.
I've been working on a similar project (LEGO Island decompilation). We've developed an extensive set of annotations and corresponding tools that facilitate matching the assembly/binary:
https://github.com/isledecomp/isle/tree/master/tools
We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well.
https://github.com/isledecomp/isle/tree/master/tools
We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well.
> We've been considering creating a separate project/repository for the tools since they might be interesting for other projects such as yours as well.
Happy to use anything that makes my life easier! I myself also started to prototype some tools - https://github.com/krystalgamer/spidey-decomp/tree/master/to... - but didn't get much far. I see there's quite a bit of overlap so I might integrate them on my workflow
Curious to know about `patch_c2.py` it mentions a bugged warning. Is that patch just to remove the warning or something deeper?
Happy to use anything that makes my life easier! I myself also started to prototype some tools - https://github.com/krystalgamer/spidey-decomp/tree/master/to... - but didn't get much far. I see there's quite a bit of overlap so I might integrate them on my workflow
Curious to know about `patch_c2.py` it mentions a bugged warning. Is that patch just to remove the warning or something deeper?
It just removes the warning, which is impossible to get rid of otherwise. This bug has persisted all the way into MSVC 6.0:
https://web.archive.org/web/20000519112833/https://support.m...
Since the warning is relatively useless we opted to fix C2.EXE so that it is never emitted. It generates a lot of noise during compilation otherwise (if you are using templates).
https://web.archive.org/web/20000519112833/https://support.m...
Since the warning is relatively useless we opted to fix C2.EXE so that it is never emitted. It generates a lot of noise during compilation otherwise (if you are using templates).
I loved that game growing up. Now the brick-by-brick song is stuck in my head.
It really captured the open ended nonsense feeling of sitting down and playing with legos.
It really captured the open ended nonsense feeling of sitting down and playing with legos.
As someone who is very interested in decompiling old Windows executables, this is very relevant to my interests. Thanks for the link!
Cool to see krystalgamer here on HN. I found them a few years ago through the (then super tiny) SM2000 modding scene on youtube. It was my first 3D game, and I still have dreams about the graphics. Their work on the resource unpacker utility allowed me to make custom skins of my own characters for the game, which I screenshotted and used for a comic. It's weird: I often think and dream in the visual language of that game.
Anywayyyyy, will be following this development more closely now. Would be really cool to eventually see some type of level editor for this game, or maybe a multiplayer server (akin to this Bomb Rush Cyberfunk mod: https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopC... )
Anywayyyyy, will be following this development more closely now. Would be really cool to eventually see some type of level editor for this game, or maybe a multiplayer server (akin to this Bomb Rush Cyberfunk mod: https://thunderstore.io/c/bomb-rush-cyberfunk/p/NotNet/SlopC... )
Thanks so much for the kind words! Did you end up publishing your skins?
> Would be really cool to eventually see some type of level editor for this game
So there's already a tool by Zedek that can export the models to blender (not sure about re-importing). I heard that PSXPrev by rickomax can work with the game files but never tried myself. As for level editor, there was this guy that came into the community and made a proper level unpacker + packer, it was primitive but worked much better than anything that I've ever done. Sadly he disappeared but his tool still floats around from what I remember.
> maybe a multiplayer server
I get asked about this a lot :D Multiplayer will be a challenge in the way it handles events, there's a lot of hardcoded stuff. Not saying it's impossible, but it'd require a big re-write of the engine.
> Would be really cool to eventually see some type of level editor for this game
So there's already a tool by Zedek that can export the models to blender (not sure about re-importing). I heard that PSXPrev by rickomax can work with the game files but never tried myself. As for level editor, there was this guy that came into the community and made a proper level unpacker + packer, it was primitive but worked much better than anything that I've ever done. Sadly he disappeared but his tool still floats around from what I remember.
> maybe a multiplayer server
I get asked about this a lot :D Multiplayer will be a challenge in the way it handles events, there's a lot of hardcoded stuff. Not saying it's impossible, but it'd require a big re-write of the engine.
I can’t imagine decompiling a project that was likely written by at least a team of a dozen developers over the course of 2-3 yrs to be completed by a single person.
Looking at the commit history, I see only contributions by the author “krystalgamer”. Wild.
Rare to find a person with such resolve and enjoyment. Wish this person good luck!
Looking at the commit history, I see only contributions by the author “krystalgamer”. Wild.
Rare to find a person with such resolve and enjoyment. Wish this person good luck!
Appreciate your kind words!
I've been working on my own reverse-engineering/decompilation project (Tenchu: Stealth Assassins) and I've created a Ghidra extension that can export a program selection as a working, relocatable object file [1].
I've had some really good results on x86 since writing an analyzer for an architecture where relocation spots target 4-byte immediate fields inside of instructions is fairly easy. Unfortunately, the PlayStation uses a MIPS processor and writing an analyzer for split HI16/LO16 relocations is proving to be a devilishly tricky problem. I got it to a point where it works well enough on MIPS most of the time, but there's always a new weird edge case hidden inside a function thousands of instructions long where it breaks down...
[1] https://github.com/boricj/ghidra-delinker-extension
I've had some really good results on x86 since writing an analyzer for an architecture where relocation spots target 4-byte immediate fields inside of instructions is fairly easy. Unfortunately, the PlayStation uses a MIPS processor and writing an analyzer for split HI16/LO16 relocations is proving to be a devilishly tricky problem. I got it to a point where it works well enough on MIPS most of the time, but there's always a new weird edge case hidden inside a function thousands of instructions long where it breaks down...
[1] https://github.com/boricj/ghidra-delinker-extension
Seems like a neater alternative than the linker scripts used by most projects.
> Creating libraries, by extracting parts of a program and reusing them in another context;
This seems like a big win when creating (un-)packers for custom formats. For this game after isolating the routine that decompresses the textures it was so annoying to not mess up the logic - https://github.com/krystalgamer/spidey-tools/blob/master/psx...
> Creating libraries, by extracting parts of a program and reusing them in another context;
This seems like a big win when creating (un-)packers for custom formats. For this game after isolating the routine that decompresses the textures it was so annoying to not mess up the logic - https://github.com/krystalgamer/spidey-tools/blob/master/psx...
“ As described in this post, decompilation project is tedious and laborious. Therefore it's easy to lose motivation and leave it in backburner for a long-time. The solution I have found is consistency, everyday try to work on it a bit as all effort will compound.”
I’m current self studying calculus. This is exactly the situation I am falling into. I now spend an extra hour and a half or so at work in the lunch room (my workplace is 24x7) and go through the calculus book. I’ve been doing this for 6 weeks or so, and though my progress may seem slow to many (I just got through limits) I understand the material properly.
(In terms of 6 weeks, I did the prerequisites chapters first which took some time, then I had a holiday, and I spent almost a week and a half on the Delta-Epsion definition of a limit - and I do all the problems in the book and there are a lot)
I’m current self studying calculus. This is exactly the situation I am falling into. I now spend an extra hour and a half or so at work in the lunch room (my workplace is 24x7) and go through the calculus book. I’ve been doing this for 6 weeks or so, and though my progress may seem slow to many (I just got through limits) I understand the material properly.
(In terms of 6 weeks, I did the prerequisites chapters first which took some time, then I had a holiday, and I spent almost a week and a half on the Delta-Epsion definition of a limit - and I do all the problems in the book and there are a lot)
Awesome project. The PS1 version of this game is one of my favorites from growing up and also the reason my home server's hostname was "eelnats" for many years.
I've been doing a bit of research on and off for the past few years on decompilation and it's definitely challenging to decide how close you want to go to matching. If you can get the exact compiler and exact compilation settings, it's totally feasible to do matching decompilation, and if you're able to make this somehow incremental such that you can incrementally work up to 100% matching over time, it seems like a really good approach, but it requires a lot of groundwork and understanding how the compiler and linker really work. In the process of matching compilation of functions on a binary I was analyzing that was compiled with Visual Studio 2003, I realized that very subtle differences can cause e.g. different register allocation, even in an old compiler with dramatically less sophisticated optimization passes.
Anyway, I guess this tangent is really unrelated, but I think more people should be embarking on decompilation projects. It's very fun, and it's uniquely rewarding if you manage to get some non-trivial decompilation of code to work properly.
I had one odd use case for decompiling that was actually, as far as I know, completely licit: WebView2Loader. Microsoft distributed the WebView2 SDK as 3-BSD so that you could integrate it into your applications without worrying about licensing, but the glue logic that actually interacts with the WebView2 installation and instantiates the COM objects is closed source. But... since it is closed-source 3-BSD, without a EULA... we can reverse engineer it. It being a relatively small shim, I did just that[1]. This was an easy exercise armed with an interactive disassembler, and since it was relatively simple and very small I didn't need to bother with matching anything: I just roughly replicated the behavior instead. The use case for this was allowing people to make WebView2 bindings that didn't have any external dependencies; the OpenWebView2Loader code was ported to Pascal and Go by others, making it possible to have pure bindings that don't require any C code or external DLLs and can directly talk to the WebView2 installation. There's now a static copy of the WebView2Loader with the SDK, which obviates some of the use of this, but this is still a nice approach for Go where you can entirely avoid CGo or messing with weird object format conversion. (It's way better than my original approach for WebView2 in Go, which is to emulate the Windows linker to link and execute an entirely in-memory copy of the WebView2Loader DLL using a lot of unsafe code. That also works, but it is much more bug prone and frankly horrifying.)
[1]: https://github.com/jchv/OpenWebView2Loader
Anyway, I guess this tangent is really unrelated, but I think more people should be embarking on decompilation projects. It's very fun, and it's uniquely rewarding if you manage to get some non-trivial decompilation of code to work properly.
I had one odd use case for decompiling that was actually, as far as I know, completely licit: WebView2Loader. Microsoft distributed the WebView2 SDK as 3-BSD so that you could integrate it into your applications without worrying about licensing, but the glue logic that actually interacts with the WebView2 installation and instantiates the COM objects is closed source. But... since it is closed-source 3-BSD, without a EULA... we can reverse engineer it. It being a relatively small shim, I did just that[1]. This was an easy exercise armed with an interactive disassembler, and since it was relatively simple and very small I didn't need to bother with matching anything: I just roughly replicated the behavior instead. The use case for this was allowing people to make WebView2 bindings that didn't have any external dependencies; the OpenWebView2Loader code was ported to Pascal and Go by others, making it possible to have pure bindings that don't require any C code or external DLLs and can directly talk to the WebView2 installation. There's now a static copy of the WebView2Loader with the SDK, which obviates some of the use of this, but this is still a nice approach for Go where you can entirely avoid CGo or messing with weird object format conversion. (It's way better than my original approach for WebView2 in Go, which is to emulate the Windows linker to link and execute an entirely in-memory copy of the WebView2Loader DLL using a lot of unsafe code. That also works, but it is much more bug prone and frankly horrifying.)
[1]: https://github.com/jchv/OpenWebView2Loader
Perfect decompilation definitely has its advantages, but that's something I simply don't have it in me to pull off. Tracking down derelict toolchains and SDKs, endlessly tweaking compiler options and source code to get that thousand instruction long function to match perfectly is not for everybody.
The trouble with decompilation projects is that there's hardly any tooling or literature available on the topic [1], so people usually end up developing custom tooling and methodology on their own to solve their issues. For example, personally I went down the path of delinking programs back into object files in my own project. While I find it quite nifty, it also isolates me on my own desert island and I know of other decompilation projects with totally different approaches in a similar situation.
Decompilation projects can be quite intellectually rewarding, but they are essentially R&D projects in a barely explored field. Wander off the thin strip of beaten path and you're basically on your own in an endless primordial jungle.
[1] At least until decomp.me came along, but that's geared for perfect decompilation only. Any other approach and you're still on your own.
The trouble with decompilation projects is that there's hardly any tooling or literature available on the topic [1], so people usually end up developing custom tooling and methodology on their own to solve their issues. For example, personally I went down the path of delinking programs back into object files in my own project. While I find it quite nifty, it also isolates me on my own desert island and I know of other decompilation projects with totally different approaches in a similar situation.
Decompilation projects can be quite intellectually rewarding, but they are essentially R&D projects in a barely explored field. Wander off the thin strip of beaten path and you're basically on your own in an endless primordial jungle.
[1] At least until decomp.me came along, but that's geared for perfect decompilation only. Any other approach and you're still on your own.
>For example, personally I went down the path of delinking programs back into object files in my own project.
Wow, that's exactly the approach I wanted to take. However, as you've said, there's basically no tools for this, so it's a bit demoralizing.
Wow, that's exactly the approach I wanted to take. However, as you've said, there's basically no tools for this, so it's a bit demoralizing.
> However, as you've said, there's basically no tools for this, so it's a bit demoralizing.
I wrote a Ghidra extension to do that, you can check it out there: https://github.com/boricj/ghidra-delinker-extension
I wrote a Ghidra extension to do that, you can check it out there: https://github.com/boricj/ghidra-delinker-extension
Nice! I'm mainly touching Win32 stuff so I suppose it would need quite a lot of adapting (and a COFF implementation) but this is very intriguing, I'll need to dive into this.
Depending on what you want to do, most of the pieces might already be there.
I've designed my Ghidra extension so that the object file exporters are decoupled from the relocation analyzers through a generic relocation data model. I've already implemented a relocation synthesizer for 32-bit x86, so writing a COFF object file format exporter should be a fairly self-contained project that doesn't impact the rest of the extension.
A while ago somebody stumbled upon my stuff and asked some questions related to delinking C++ code on Windows, you might find some answers there: https://github.com/boricj/ghidra-delinker-extension/issues/4
Don't hesitate to contact me directly if you want to discuss this stuff more in-depth.
I've designed my Ghidra extension so that the object file exporters are decoupled from the relocation analyzers through a generic relocation data model. I've already implemented a relocation synthesizer for 32-bit x86, so writing a COFF object file format exporter should be a fairly self-contained project that doesn't impact the rest of the extension.
A while ago somebody stumbled upon my stuff and asked some questions related to delinking C++ code on Windows, you might find some answers there: https://github.com/boricj/ghidra-delinker-extension/issues/4
Don't hesitate to contact me directly if you want to discuss this stuff more in-depth.
Thanks for extending the offer! I am a bit busy but I really appreciate all of the information. I've got plenty of new things to dig into now, I'll have to seriously set aside some time to take another look, this could seriously help me with some of my reversing projects. I'm willing to put in tons of manual effort to bridge the gap, and it feels like if I could leverage this it could act as quite a force multiplier.
Which decompiler?
Man, makes me sad just how much time and effort is wasted on decompiling those old games.
All of this effort could’ve been spent on implementing features and fixing bugs by super dedicated fans of the games. Instead, there’s a constant fight with code and bloodsucking lawyers of greedy companies who don’t give a single shit about those games.
Most of those aren’t even sold anymore. Destined to rot, because of petty reasons.
All of this effort could’ve been spent on implementing features and fixing bugs by super dedicated fans of the games. Instead, there’s a constant fight with code and bloodsucking lawyers of greedy companies who don’t give a single shit about those games.
Most of those aren’t even sold anymore. Destined to rot, because of petty reasons.
If you check my repository of mods for the game - https://github.com/krystalgamer/spidey-tools
You can see bug fixes and new features. There's just a moment where bigger features become unfeasible to be done this way. It's the same dedicated fans that implement new features and fix bugs that then move into decompilation projects.
You can see bug fixes and new features. There's just a moment where bigger features become unfeasible to be done this way. It's the same dedicated fans that implement new features and fix bugs that then move into decompilation projects.
I meant that all the time and energy you're using on decompilation could've been used more productively if original source code was made readily available by the companies that produced the games.
The journey is it's own reward, my son.
I disagree.
There will be times where a game source code has been lost in the void... and then you get these godsend people who are able to reverse the game and recreate it.
We need these talented people.
There will be times where a game source code has been lost in the void... and then you get these godsend people who are able to reverse the game and recreate it.
We need these talented people.
I meant that companies should release source code for 20+ old games, especially if they’re not sold or actively developed anymore.
[deleted]
I would argue it would hold the crown for best Spider-Man game for some time until the flawed masterpiece Spider-Man 2 (2004) gave us truly amazing web-slinging physics in a sandbox environment, or the less-free-but-tighter-overall Ultimate Spider-Man in 2005.