Late to the party here but I didn't see until now.
Looking at the layout I'm very confident this controller is fully analog, with just a few gates of digital doing control stuff. There's only a few dozen to maaaaybe low hundreds of gates of digital in the whole die, not nearly enough to be any kind of MCU.
And there's a whole bunch of really large resistors and other analog stuff.
I do see a structure that might be some kind of ROM for piecewise linear calibration curves, but it's too obscured by wiring on top for me to draw any definitive conclusions without delayering the chip.
Yeah it's always tricky to decide how much detail to include in these posts. If I had gone on a whole explanation of the full 7 series GTX clocking architecture the post would easily have been another few pages in length.
Those addresses all make sense, as 0x4000 - 4fff appears to be where the 8051 has its RAM mapped (all of the peek/poke addresses used for accessing serdes fields are on the high end)
I avoid vendor toolchains and BSPs just because of how buggy they are.
From my perspective, it's much better to reproduce a bug with a 20-line C or assembler file that compiles with upstream gcc, completely ruling all of their custom stuff out as the root cause.
Just tell me what the silicon does when I poke this register and I'll work around it.
Personally I've standardized on just three STM32 parts:
* L031 for throwaway-cheap stuff where I'm never going to do field firmware updates (so no need to burn flash on a proper bootloader) and just need to toggle some GPIOs or something
* L431 for most "small" stuff; I use these heavily on my large/complex designs as PMICs to control power rail and reset sequencing. They come in packages ranging from QFN-32 to 100-ball 0.5mm BGA which gives a nice range of IO densities.
* H735 for the main processor in a complex design (the kinds of thing most people would throw embedded Linux at). I frequently pair these with an FPGA to do the heavy datapath lifting while the H735 runs the control plane of the system.
I don't think I've ever used a ST part without reporting a bunch of datasheet errors.
I haven't been bit by an undocumented silicon bug, but I step on documented STM32H7 bugs on a pretty regular basis and there are some poor design decisions around the OCTOSPI (in addition to bugs) that make me avoid it in almost every situation.
But at least they document (mostly correctly) the registers to talk to their crypto accelerator unlike the Renesas and NXP parts I looked at as potential replacements, both of which needed an NDA to get any info about the registers (although they did supply obfuscated or blob driver layers IIRC).
This is why I used a VSC PHY. After they bought Microsemi (and Vitesse as a division of Microsemi) it looked like the only viable option to get a QSGMII PHY since all the other players were much worse.
When I first started the project in 2012-13, Vitesse was just as NDA-happy and I ruled them out. The original roadmap called for a 24-port switch with 24 individual TI DP83867 SGMII PHYs on three 8-port line cards.
Yes, Vitesse had been on my "naughty list" of companies that were permanently banned from getting a design win from me because of refusing to share any docs or sell parts at distributors or other engineer-hostile practices popular with the likes of Marvell and Broadcom.
After MCHP bought them and opened up (what I thought was) the full datasheet I gave them a second chance. Seems they still held some back.
No idea, the high power consumption and latency of >1000baseT were such that I've never had any interest in anything newer in baseT land.
At home I run 10/100/1000baseT, 10Gbase-SR, 40Gbase-SR4, and am just beginning to prepare for 25Gbase-SR and 100Gbase-SR4 deployments (I have some NICs and optics fielded but no switching yet).
I've worked with 100baseT1 and 1000baseT1 for automotive projects at work and am familiar with the line coding, and have written a protocol decode for 100baseT1 among others, but I don't use them in my own projects.
When I last looked into it, which was admittedly several years ago, 10GbaseT PHYs were unobtainium in the "I want to buy just one, without an NDA" space. Your best option for putting 10Gbase-T on a DIY design is to put a SFP+ cage on there and slap in a baseT SFP+ module that somebody else designed.
The board-level architecture is going to be super simple as big FPGA designs go:
* XCKU5P in the middle
* 12x GTYs routed to 4x Samtec ARF6 connector for the line cards
* 2x GTYs routed to 2x SFP28 uplinks
* RGMII to back panel management PHY
* Parallel SRAM bus to STM32H735 management processor
* A bunch of Murata MYMGK modules for power conversion off the 12V rail
* STM32L431 in QFN48 or more likely BGA100 depending on IO requirements as a PMIC and to manage reset sequencing etc
This will be fully FPGA based, no separate switch ASIC, and I want to do all of the hardware design. I'm not sure there is much I can learn from somebody else's FPGA switch design at the board level - it's basically just going to be a bunch of transceivers hooked up to SFPs and some power distribution. All the magic happens inside the FPGA.
You're going to at least want to AC couple through some capacitors to avoid problems with common mode / RX bias issues between the different PHYs. But if it's on-mobo you don't need to have as much of a fault voltage rating compared to standard Ethernet which IIRC specs 1 kV isolation on the transformers.
For context of how lightweight this is, an -O3 release build of my entire firmware on the management processor right now (including the sshd, hardware drivers, TCP/IP stack, the CLI itself, all of the code to query the supported set of sensors, etc) uses 109 kB of flash and 84 kB of SRAM. The -Og debug build is smaller at 86 kB flash usage.
It compiles in five seconds from a clean build tree on my workstation.
Sure, this isn't as feature-rich as OpenSSH or even Dropbear and is missing a lot of the fancy features you get on Linux, but it's tiny and fast. Good luck getting buildroot or something to give you a 100 kB kernel+userspace image that builds in five seconds.
And it's fast: "time -p ssh testbed show ver" returns in 70 ms on a debug build. That's faster than some x86 Debian + OpenSSH machines I've benchmarked against. And I'm on a 500 MHz single-core Cortex-M7.
The code is at https://github.com/azonenberg/staticnet but I've intentionally avoided over-publicizing it since it hasn't had any kind of third party security review. As of now it's functional enough I'm willing to deploy it on a lab network but wouldn't trust it open to an untrusted network.
I work in embedded security and have tried to avoid any of the most gross footguns, deliberately simplifying the implementation as much as possible both to optimize for deeply embedded applications with double-digit kB flash/RAM footprint, and to minimize attack surface. The only supported cipher suite is ssh-ed25519 + aes128-gcm, and I didn't implement either of those (I use the DJB reference implementation or my line-by-line FPGA port of it for the 25519, and the hardware AES + RNG on the STM32).
But I've never done a formal code review myself (not that I'd trust one done by me as the author of the offending code, but it'd be a good starting point to find low-hanging fruit before I waste somebody else's time), fuzzed it, etc.
I have a bunch of additional features I want to add (notably, IPv6 support in the TCP/IP stack is very incomplete and not yet usable) and then am going to try to get at least some friends/coworkers to bang on it more.
The overall system will be powered by 48V DC on a 6-pin Molex Mini-Fit Jr connector, then stepped down to 12V by a 48->12V intermediate bus converter I've previously designed and characterized (https://serd.es/2024/10/15/Intermediate-bus-converter.html)
The 24-port line card has a 15W TDP overall (6.75W datasheet worst-case power consumption for each of the two 12-port PHYs, plus 1.5W added for power supply conversion losses and such). With my current bench setup (11/12 links up on PHY 0 and 2/12 on PHY 1, all linked up at gigabit but with PHY 1 not talking to the FPGA yet), they're happy but warm - 63.8 and 49.5C die temperature respectively, pulling a combined 7.173W (not counting power conversion losses) according to the internal sensors. PCB temperature is 37.2 to 49C at various measurement points.
The entire setup including the FPGA board, IBC, one of the two planned line cards, and some other glue components that won't be in the final switch is pulling 18W and change right now although the FPGA power consumption will go up as I build out more logic. I don't have measurements of just the FPGA's power consumption currently (I could put a current clamp on the 12V cable from the IBC I guess) but the PDU board does have I2C sensors that will measure consumption of the logic board and each line card separately; I just haven't written the firmware to read them yet. I also don't have FPGA temperature readings in the current firmware although last time I checked them via JTAG I had plenty of margin.
As of right now everything is happy with low-profile heatsinks (Wakefield-Vette 960-27-12-D-AB-0 on both the PHYs and FPGA) and passively cooled just sitting on my bench with no fans. I can go taller if increased power consumption or worse airflow in the future dictates, they're nowhere near the point of hitting the top of a 1U chassis.
My plan for the final system is to have air intakes somewhere on the sides and/or the front around the RJ45s and one or more fans exhausting out the back, details TBD based on more design and testing once I have better projections of the overall thermal load.
Very roughly my overall thermal/power budget is 15W per line card = 30W combined, plus no more than 20W for the logic board (probably quite a bit less) with all ports lit up and passing packets, for a total of 50W plus whatever the losses in the IBC are (it's roughly 94% efficient at this load based on previous testing). This is comfortably below the 72W output limit of the IBC, and should be very reasonable to reject from a 1U chassis with fairly basic air cooling, especially since it's not all coming from a single point load like a single-socket server.
I've wargamed how I'd backdoor an FPGA even given the ability to make a completely new mask set from a fork of the original CAD files, and it's really difficult.
You'd either have to add an enormous amount of logic or have extremely detailed a priori knowledge of exactly what someone was going to use it for (down to what functions each pin was being used for). Making it meet the original factory performance and timing specs would be immensely difficult.
The best idea I had was that you could add some logic inside the transceiver IP that would understand common networking line codes and then bridge packets with certain magic header values over to an ICAP or something, so that you could enable an unauthenticated partial reconfig over IP channel.
But when you have lots of different line codes out there and don't know the bus width or configuration the user is going to have now suddenly you have to implement half a dozen different PCSes inside your fork of the GTY IP without changing the layout enough to fail timing or change the bump map enough to be visible to someone looking at the substrate or...
Small stuff like adding bypasses to bitstream encryption I could see, but nothing that would be a major risk to something like this.
There are no blobs I'm aware of anywhere that are actually running in the system.
The STM32s have a small boot "ROM" burned into a write-protected region of flash but I have it jumpered so I boot from main user flash, not the bootloader.
I did a quick silicon overview of them (just out of curiosity... they came new from Digikey so I have no reason to believe they're fishy).
The 12-port PHYs are a fused-down version of a switch chip so there is a MIPS core on there, but to the best of my knowledge in the switch SKU it doesn't actually run (e.g. its RAM bus is NC and the IO power rail is grounded).
The FPGA is completely programmed from the bitstream I control. Tampering with a random resold FPGA to add some kind of backdoor is extraordinarily difficult and unlikely, it's the kind of thing you would see done as a targeted attack rather than just dumping FPGAs into the secondary market and hoping that a juicy intelligence target buys them rather than some guy tinkering around with open source projects.
And, if I ever get the slightest hint that there is something fishy about silicon I bought from a sketchy overseas source, I'm gonna take it into the lab at work and go to town. I do semiconductor reverse engineering at my day job and am the absolute last person anyone should try to sneak backdoored chips past. It's going to end up getting dissected inside a SEM and turn into an awesome talk at REcon or something.
While it would be slightly annoying to have my side projects disrupted, having a living breathing nation-state silicon backdoor show up in my lab would be a dream come true. I'd be ordering as many more chips as I could from the same seller and instrumenting it in every way possible, practicing on non-backdoored chips to make absolutely certain I didn't damage any of the spicy ones while studying the altered circuits, etc.
This is not going to be cheap. That was never the goal.
In low volume when you combine several custom multilayer boards, custom powder-coated sheet metal work, etc even if you allow for the practically-free recycled FPGA this is going to probably end up being a $5-10K project. The last custom one-off 1U I did from ProtoCase was like $800ish just for the enclosure, and that was before the recent tariff hikes.
Looking at the layout I'm very confident this controller is fully analog, with just a few gates of digital doing control stuff. There's only a few dozen to maaaaybe low hundreds of gates of digital in the whole die, not nearly enough to be any kind of MCU.
And there's a whole bunch of really large resistors and other analog stuff.
I do see a structure that might be some kind of ROM for piecewise linear calibration curves, but it's too obscured by wiring on top for me to draw any definitive conclusions without delayering the chip.