eGPU: A 750 MHz Class Soft GPGPU for FPGA(arxiv.org)
arxiv.org
eGPU: A 750 MHz Class Soft GPGPU for FPGA
https://arxiv.org/abs/2307.08378
10 comments
> The mind blowing part of all of this is the fact that they were able to close timing at 771MHz. That is insanely fast for an FPGA. For perspective, most modern FPGAs run their designs at around 300MHz
I could not find the datasheet of the exact FPGA they used (did not spend much time) but Google says that Intel Agilex FPGAs (which is what they used) rated at 600MHz+ are available.
I'm not saying that they did something easy but they obviously used a fast FPGA.
I could not find the datasheet of the exact FPGA they used (did not spend much time) but Google says that Intel Agilex FPGAs (which is what they used) rated at 600MHz+ are available.
I'm not saying that they did something easy but they obviously used a fast FPGA.
> The mind blowing part of all of this is the fact that they were able to close timing at 771MHz
It's true but I mean this is Intel in-house research right? If they can't get absolute peak fmax on their own parts that would be a really bad look right? Plus these stratix parts have hard FP blocks (not just DSPs) so they're basically mostly scheduling stuff rather building the whole datapath. But admittedly I haven't read the paper...
>Full Disclosure, I work for an FPGA company
I currently do too (as an intern, maybe even the same one as you) and I haven't looked very hard but I'm sure we have similar fmax achieving projects (maybe even GPUs since we're fighting hard to compete with Nvidia...).
It's true but I mean this is Intel in-house research right? If they can't get absolute peak fmax on their own parts that would be a really bad look right? Plus these stratix parts have hard FP blocks (not just DSPs) so they're basically mostly scheduling stuff rather building the whole datapath. But admittedly I haven't read the paper...
>Full Disclosure, I work for an FPGA company
I currently do too (as an intern, maybe even the same one as you) and I haven't looked very hard but I'm sure we have similar fmax achieving projects (maybe even GPUs since we're fighting hard to compete with Nvidia...).
I think it's more that the compile was unconstrained, so no floorplanning or handholding synthesis anything (aside from the usual bullshit to e.g. make sure DSPs map correctly). Which is pretty good; yeah, the architecture is also good and you'd expect that at minimum from a team like this, but they also didn't have to beat the tool into submission either which seems nice.
IME, until fairly recently at least, Quartus loved to get fiddly on even the most basic designs when using Stratix class parts where you wanted to rely on, say, hyper-registers or retiming. Like a thing that should clearly be pipelined just... doesn't, and falls over from a gust of wind. It seems to have gotten quite a bit better in the last few versions, though, so seeing an unconstrained compile go this high is pretty good I think. Too bad they don't seem to have released their code, though.
IME, until fairly recently at least, Quartus loved to get fiddly on even the most basic designs when using Stratix class parts where you wanted to rely on, say, hyper-registers or retiming. Like a thing that should clearly be pipelined just... doesn't, and falls over from a gust of wind. It seems to have gotten quite a bit better in the last few versions, though, so seeing an unconstrained compile go this high is pretty good I think. Too bad they don't seem to have released their code, though.
> I think it's more that the compile was unconstrained, so no floorplanning or handholding synthesis anything
i mean they don't say anything about how long they let it run for - constraints are to narrow the search space for human-time-scale benefit right? i'm sure if i let vivado run (i'm not an intel person) until heat death of the universe it would also give me some fantastic fmax numbers too.
> Too bad they don't seem to have released their code, though.
and they never will. i don't know what it is about this kind of research (or how you can even get away with this) but it's almost unheard of for the commercial groups to release their code/rtl/tcl scripts. i have emailed around before and i just continually get shined on - "oh we're planning to release but it has to go through legal". and this is from my capacity as a phd student ie not direct competitor. and it's funny to me because if you're selling these boards/part/fpgas aren't you strongly incenvitized to throw as much sample code over the fence as possible so that people buy your hardware in order to use your code?
i mean they don't say anything about how long they let it run for - constraints are to narrow the search space for human-time-scale benefit right? i'm sure if i let vivado run (i'm not an intel person) until heat death of the universe it would also give me some fantastic fmax numbers too.
> Too bad they don't seem to have released their code, though.
and they never will. i don't know what it is about this kind of research (or how you can even get away with this) but it's almost unheard of for the commercial groups to release their code/rtl/tcl scripts. i have emailed around before and i just continually get shined on - "oh we're planning to release but it has to go through legal". and this is from my capacity as a phd student ie not direct competitor. and it's funny to me because if you're selling these boards/part/fpgas aren't you strongly incenvitized to throw as much sample code over the fence as possible so that people buy your hardware in order to use your code?
>if you're selling these boards/part/fpgas aren't you strongly incenvitized to throw as much sample code over the fence as possible so that people buy your hardware in order to use your code?
I wish... this is a problem for general embedded things. We just recently bought a cubesat bus and the only way to interface our hardware with it is to send CAN bus frames to it. We asked them for an SDK and they don't have it. Instead they sent us a bunch of ICDs with screenshots of readthedocs snippets containing the frame structures.
The unfortunate answer to that question is that unfortunately we don't make purchasing decisions based on whether sample code exists, or detailed documentation quality for niche embedded hardware. Maybe because everyone is not giving those things.
I wish... this is a problem for general embedded things. We just recently bought a cubesat bus and the only way to interface our hardware with it is to send CAN bus frames to it. We asked them for an SDK and they don't have it. Instead they sent us a bunch of ICDs with screenshots of readthedocs snippets containing the frame structures.
The unfortunate answer to that question is that unfortunately we don't make purchasing decisions based on whether sample code exists, or detailed documentation quality for niche embedded hardware. Maybe because everyone is not giving those things.
For a GPU circuit, it basically comes down to the number of hardware multipliers on the FPGA, does it not?
I remember synthesizing a 16-bit Wallace tree in a lab exercise back in college. I think that single multiplier used up 70% of my LUTs.
You only will get massive amounts of hardware parallel multipliers if the underlying circuit has a ton of hardware multipliers (Like Xilinx's VLIW SIMD AI chips)
-------
At all computer sizes, a GPU probably will have more multiply circuits than an equivalent cost FPGA, with exception of maybe those AI chips from Xilinx (where the individual cores are basically presynthesized with hardcoded ISA).
Ex: at under 500mW power usage you probably will prefer some ARM NEON SIMD or TI DSP / VLIW. At cell phone levels you'd prefer a cell phone GPU, and at desktop/server levels you'd prefer a desktop GPU.
I remember synthesizing a 16-bit Wallace tree in a lab exercise back in college. I think that single multiplier used up 70% of my LUTs.
You only will get massive amounts of hardware parallel multipliers if the underlying circuit has a ton of hardware multipliers (Like Xilinx's VLIW SIMD AI chips)
-------
At all computer sizes, a GPU probably will have more multiply circuits than an equivalent cost FPGA, with exception of maybe those AI chips from Xilinx (where the individual cores are basically presynthesized with hardcoded ISA).
Ex: at under 500mW power usage you probably will prefer some ARM NEON SIMD or TI DSP / VLIW. At cell phone levels you'd prefer a cell phone GPU, and at desktop/server levels you'd prefer a desktop GPU.
> At all computer sizes, a GPU probably will have more multiply circuits than an equivalent cost FPGA
Very likely yes, but FPGAs often have hundreds to thousands of hardware multipliers, as part of the DSP blocks. Here for example newer AMD FPGAs: https://eu.mouser.com/datasheet/2/903/ds890_ultrascale_overv...
Very likely yes, but FPGAs often have hundreds to thousands of hardware multipliers, as part of the DSP blocks. Here for example newer AMD FPGAs: https://eu.mouser.com/datasheet/2/903/ds890_ultrascale_overv...
I wish people would stop quoting marketing material as some kind representation of what they know.
You're giving completely the wrong impression about dsp slices - it is absolutely not 1 dsp slice per FP operator at any precision that you would want to do floating point arithmetic. It's definitely at least 2 plus a whole bunch of LUTs (~500) for FP16 with 4 stages or something like that. And if you want faster (fewer stages) then you need more slices. On alveo u280, which is an ultrascale part, I have never been able to effectively utilize more than ~4000 dsp slices (out of 9024) for 5,4 mults and that cost basically 99% of clbs in SLR1 and SLR2.
And even then, disconnected FPUs are completely meaningless without a datapath implementing eg matmul and boy oh boy do you have no clue what you're in for there.
Takeaway: it's pointless to compare raw specsheet numbers when everything comes down to datapath.
You're giving completely the wrong impression about dsp slices - it is absolutely not 1 dsp slice per FP operator at any precision that you would want to do floating point arithmetic. It's definitely at least 2 plus a whole bunch of LUTs (~500) for FP16 with 4 stages or something like that. And if you want faster (fewer stages) then you need more slices. On alveo u280, which is an ultrascale part, I have never been able to effectively utilize more than ~4000 dsp slices (out of 9024) for 5,4 mults and that cost basically 99% of clbs in SLR1 and SLR2.
And even then, disconnected FPUs are completely meaningless without a datapath implementing eg matmul and boy oh boy do you have no clue what you're in for there.
Takeaway: it's pointless to compare raw specsheet numbers when everything comes down to datapath.
> everything comes down to datapath.
Ain't that the truth.
However: Xilinx's DSP58 blocks (Versal devices), and older Intel DSPs, do integrate floating point into the DSP tiles - which does narrow the gap between the datasheet's DSP count and the number of floating-point operations achievable per clock.
Ain't that the truth.
However: Xilinx's DSP58 blocks (Versal devices), and older Intel DSPs, do integrate floating point into the DSP tiles - which does narrow the gap between the datasheet's DSP count and the number of floating-point operations achievable per clock.
> Xilinx's DSP58 blocks (Versal devices)
this is correct (i am currently working on designs that target these parts) but those things are more expensive than comparably sized/equipped nvidia gpus so (in the context of this discussion) it's moot.
this is correct (i am currently working on designs that target these parts) but those things are more expensive than comparably sized/equipped nvidia gpus so (in the context of this discussion) it's moot.
Ballas mentions in a neighboring reply that the eGPU from the paper is using an Agilex 7: https://ark.intel.com/content/www/us/en/ark/products/217645/...
Yes but as you can see by reading, this comment chain starts with someone linking to a xilinx product
The Intel Agilex (that was used in this paper) DSP blocks natively support single and half precision floating point.
And my 6-year-old Consumer GPU (Vega64) has 4096 32-bit multipliers at 1500 MHz clock. (And I bought it a year or two late, so it was only $400). With 8GB HBM2 RAM at 512GBps (yes, Gigabytes per second) throughput. You absolutely are not getting anything close to a consumer GPU in terms of performance or cost on any other platform.
---------
FPGAs advantage is that systolic array arrangement. I'm pretty sure GPUs will win in a "hardware multiplier" war, but it might be harder to get all the data into a typical GPU
There's a fair number of programs that will never utilize a GPU with any level of efficiency. That's where FPGAs come in, utilization will be better because of LUTs and Routers and all that magic.
But if people are looking for a TFLOP fight with raw numbers of multipliers? My bet is on the GPU and SIMD-processing in general. Those architectures are just crazy.
--------------
BTW: That top end DSP slice is documented here: https://docs.xilinx.com/v/u/en-US/ug579-ultrascale-dsp
That's a 27-bit x 18-bit multiplier per dsp.
In contrast, a 32-bit floating point operation is 24-bit x 24-bit multiplier plus a bit of extra stuff for the exponent bits. I'm pretty sure that the 4096-shader Vega64 actually had 32-bit multipliers but I'm not 100% sure on that.
But the float 24-bit x 24-bit multiplier would need more than one dsp to replicate, maybe two of them. AMD's newest consumer 7900 xt GPUs are 6144-shader... but those shaders can execute _TWO_ multiplies per clock tick for a total of 12288 hardware multipliers operating at 2500 MHz clock.
---------
FPGAs advantage is that systolic array arrangement. I'm pretty sure GPUs will win in a "hardware multiplier" war, but it might be harder to get all the data into a typical GPU
There's a fair number of programs that will never utilize a GPU with any level of efficiency. That's where FPGAs come in, utilization will be better because of LUTs and Routers and all that magic.
But if people are looking for a TFLOP fight with raw numbers of multipliers? My bet is on the GPU and SIMD-processing in general. Those architectures are just crazy.
--------------
BTW: That top end DSP slice is documented here: https://docs.xilinx.com/v/u/en-US/ug579-ultrascale-dsp
That's a 27-bit x 18-bit multiplier per dsp.
In contrast, a 32-bit floating point operation is 24-bit x 24-bit multiplier plus a bit of extra stuff for the exponent bits. I'm pretty sure that the 4096-shader Vega64 actually had 32-bit multipliers but I'm not 100% sure on that.
But the float 24-bit x 24-bit multiplier would need more than one dsp to replicate, maybe two of them. AMD's newest consumer 7900 xt GPUs are 6144-shader... but those shaders can execute _TWO_ multiplies per clock tick for a total of 12288 hardware multipliers operating at 2500 MHz clock.
The FPGAs with enough multipliers to be competitive against an actual GPU are going to be quite a bit more expensive than a GPU aren't they?
How much would that FPGA cost?
One group at Georgia Tech in our building has also been working on open source GPU designs that can also target FPGAs and interoperate with RISCV. They have several publications on the work they have built up. Thought I might share since it’s not referenced in the submission paper.
https://vortex.cc.gatech.edu/
https://vortex.cc.gatech.edu/
They still haven't published the source code for their Skybox project, I wonder why. Unless I missed it in their repository? https://github.com/vortexgpgpu
Not sure, my best advice would be to just email the authors if you are interested. Seems like they presented to ASPLOS so maybe the students disappeared over the summer for internships or vacation.
The "G" in "GPU" or the second "G" in "GPGPU" stands for "graphics". The paper describes a compute-oriented processor that doesn't appear to contain any gfx-specific hardware.
For instance, in conventional GPUs, there is dedicated hardware for rasterizing the pixels covered by a triangle; there is dedicated hardware for doing texture map calculations and sample filtering. The instruction set disclosed in the paper doesn't contain anything to help accelerate those tasks. With enough code I'm sure one could get it to draw something, but it will be notably inefficient while doing so.
For instance, in conventional GPUs, there is dedicated hardware for rasterizing the pixels covered by a triangle; there is dedicated hardware for doing texture map calculations and sample filtering. The instruction set disclosed in the paper doesn't contain anything to help accelerate those tasks. With enough code I'm sure one could get it to draw something, but it will be notably inefficient while doing so.
Also, the 'e' in eGPU usually means that the card is housed externally to the rest of the computer - in its own enclosure.
They don't actually explain anywhere in their paper what 'eGPU' means. Is it 'emulated'? I thought circuits implemented in FPGAs are not considered emulation?
They don't actually explain anywhere in their paper what 'eGPU' means. Is it 'emulated'? I thought circuits implemented in FPGAs are not considered emulation?
> This paper describes the architecture and implementation of
a high performance embedded GPU
I'm tinking the e means embedded
I'm tinking the e means embedded
Also discussed here: https://old.reddit.com/r/FPGA/comments/15fnb6u/egpu_a_750_mh...
Uh, non-native question: what is the word "class" doing in the title?
Is a hyphen missing, so it should be "750 MHz-class"? I searched the linked page but the word only appears in the title, sans hyphen.
Is a hyphen missing, so it should be "750 MHz-class"? I searched the linked page but the word only appears in the title, sans hyphen.
Yes, the title is saying that the "soft GPU" being executed on the FPGA is running like (in the same "class" as) a 750 MHz GPU.
This page isn't loading for me. Mirrored at: https://archive.is/jewT4
Peaked my interest, but then:
- No (price or non-price) comparison/benchmark with any physical GPU of any sort;
- Seems to only be able to do FFTs;
- To be truly useful (or "General" - hey, they do mention _G_PGPU), it would have to be able to run OpenCL code;
- And on that note, what's the point, since (many) FPGAs can run OpenCL code directly?
Is there something I'm missing?
- No (price or non-price) comparison/benchmark with any physical GPU of any sort;
- Seems to only be able to do FFTs;
- To be truly useful (or "General" - hey, they do mention _G_PGPU), it would have to be able to run OpenCL code;
- And on that note, what's the point, since (many) FPGAs can run OpenCL code directly?
Is there something I'm missing?
[deleted]
Wonder it this could help to alleviate the momentary shortage of GPUs on the market.
10 year old entry level GPUs have 100 750Mhz cores
'Cores' are really overstated in GPUs. CUDA cores are really SIMD lanes and if you counted it the same way as a CPU does, you'd get somewhere in the dozens of cores range even for modern GPUs.
If you want to count "cores" in a GPU, the best equivalency is streaming multiprocessors.
While it's true that they're in the dozens, strictly speaking, we've got GPUs with well over 100 now (the RTX 4090, for instance) and 64+ is a pretty common configuration.
While it's true that they're in the dozens, strictly speaking, we've got GPUs with well over 100 now (the RTX 4090, for instance) and 64+ is a pretty common configuration.
That seems backwards to me. Sure, a GPU core is less general, but in terms of concurrent execution, memory bandwidth, and FLOPS I would expect hundreds to thousands of cores for all new GPU offerings. Apple’s double-digit GPU core counts for instance sound extremely understated.
If you have a truck that can carry 64 boxes, then according to Nvidia marketing each box would be considered its own core even though the boxes all go the same way.
Meanwhile if you had 64 passengers cars, then each of them would be able to drive a different route at the same time and if you counted the seats you would actually end up with 320 "Nvidia cores".
Meanwhile if you had 64 passengers cars, then each of them would be able to drive a different route at the same time and if you counted the seats you would actually end up with 320 "Nvidia cores".
It's not. The best comparison is the SM count for Nvidia hardware, or the wavefront count for AMD hardware. So a 4070 has 46 cores as you'd count them on a CPU.
A proper method is counting ALUs instead of vague "cores".
From the abstract: "The eGPU architecture is a streaming multiprocessor (SM) machine with 512 threads. Each SM contains 16 scalar processors (SP)."
[deleted]
If it's on an FPGA then it doesn't really compete with GPUs you can buy from just about any perspective other than openness.
I don't think this will be momentary. Reality is that there have been shortages of GPUs for a long time now and demand isn't going down. People are signing 3 year contracts with lambda now.
The mind blowing part of all of this is the fact that they were able to close timing at 771MHz. That is insanely fast for an FPGA. For perspective, most modern FPGAs run their designs at around 300MHz* While most of the heavy lifting in this design use hardened components like DSPs and FPUs, it's still very impressive to see!
What I didn't see talked about much was how memory is loaded in and out of the processor. I'm curious to see what the memory bandwidth numbers look like as well as the resource utilization of the higher level routing.
*For most hardware designs that aren't things like CPUs and GPUs, you don't always need a super high clock speed. You have a lot more flexibility to compute in space rather than in time (think more threads running slower.) The pros and cons of such tradeoffs are a bit of a complicated topic, but should at least be noted.