The ex-Alibaba KU3P boards are available for under $200, although you do need a separate JTAG cable with flying leads to load designs on them. I recommend starting with that. I'm building the next-generation version of Corundum on those boards, at least initially. I will certainly be adding support for Alveos and such in the near future. As far as adding functionality, it depends on what you want to do, but in general you should expect to have to do a fair amount of tinkering with the design.
This is correct, and the result of those streams has been released as corundum-proto here: https://github.com/fpganinja/taxi/tree/master/src/cndm_proto . Note that this simplified design is intended for educational purposes only, the "production" variants will be much more capable (corundum-micro, corundum-lite, and corundum-ng).
I'm using some very cheap (~$200) ex-Alibaba FPGA boards for the initial development of the next-generation version. They have 8 lanes of PCIe gen 3 and two SFP28 capable of operation at 25 Gbps. Honestly that's pretty hard to beat. Support for other boards will be added in the near future. The new library will also include a new IP stack called Zircon, which supports UDP, among other things.
The GPU uses https://github.com/alexforencich/verilog-pcie + the Xilinx PCIe hard IP core. When using the device-independent DMA engine, that library supports both Xilinx and Intel FPGAs.
Ah yeah, that's probably true. According to some back of the envelope math, it seems like the distribution should be more like 5%, 1%, 1%, 3%, 50%, 39%, ignoring the first and last size bins.
I think the author may have made a mistake in some of the math. The frame size distribution plots are likely based on the number of frames, not the amount of data contained in said frames. The 1500 byte and other large frames should therefore account for the lion's share of the actual data transferred. Correcting this error will totally change the final two graphs.
Well, we're planning on porting Corundum to the NetFPGA SUME hardware at some point in the near future. Should be relatively straightforward as the PCIe interface on the Virtex 7 is the same as on the Ultrascale parts.
NetFPGA does have a NIC reference design, but AFAIK it's just the Xilinx XDMA core connected to a Xilinx 10G MAC. No accessible transmit scheduler, no offloading of any kind, etc. Just about as spartan as you can get, and it's built from completely closed components so you can't really make many modifications to it.
For what we're doing, we can't use any existing commercial NICs or smart NICs because they can't provide the precision we need in terms of controlling transmit timing. We don't care about EBPF, P4, etc. We care about PTP synchronized packet transmission with microsecond precision.
Crosstalk is better than 30 dB, and double pass loss between ports is 5-8 dB. The switch is basically cycling through three or four different interconnection patterns that are defined by looped back fiber connections, so the signal has to pass through the switch twice.
Stradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet.
Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets.
The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.