FuryGpu – Custom PCIe FPGA GPU

I was about to recommend the Parallax Propeller (the first one that's available in DIP format), but arguably, that one is way more complex to program for (and also significantly more powerful, and at that point you might as well look into an ESP32 and that is "just use an MCU" :))

And yeah, video output is a significant issue because of the required bandwidth for digital outputs (unless you're okay with composite or VGA outputs, I guess they can still be done with readily available chips?). The recent Commander X16 settled for an FPGA for this.

I've been looking into graphics on MCUs and was disappointed to learn that the little "NeoChrom" GPU they're putting on newer STM32 parts is completely undocumented. Historically they have been good about not putting black boxes in their chips, but I guess it's probably an IP block they've licensed from a third party.

True, can't think of much else this popular.

He started posting videos again recently with some regularity after a lull. Audience is in the low hundreds of thousands. I assume fewer than 100k actually finish videos and fewer still do anything with it.

Hobby electronics seems surprisingly small in this era.

Registers can be worked around by using the stack and/or memory. Division could always be implemented as a simple function. It's part of the fun of working at that level.

Regarding graphics, initially output serial. Abstract the problem away until you are ready to deal with it. If you sneak up on an Arduino and make it scream, you can make it into a very basic VGA graphics card [1]. Even easier is ESP32 to VGA (also gives keyboard and mouse) [2].

[1] https://www.instructables.com/Arduino-Basic-PC-With-VGA-Outp...

[2] https://www.aliexpress.us/item/1005006222846299.html

Funny enough, that's exactly where this project started. After I built his 8 bit breadboard computer, I started looking into what might be involved in making something a bit more interesting. Can't do a whole lot of high-speed anything with discrete logic gates, so I figured learning what I could do with an FPGA would be far more interesting.

Googling the Xilinx Zynq UltraScale+ it seems kinda expensive.

Of course plenty of hobbies let people spend thousands (or more) so there's nothing wrong with that if you've got the money. But is it the end target for your project? Or do you have ambitions to go beyond that?

How much it depends on hard IP blocks ? I mean, can it be ported to FPGAs of other vendors, like Lattice ECP5 ? Did you implement PCIe in HDL or used vendor specific IP block ? Please, provide some resource utilization statistics. Thanks.

In the post about the texture unit, that ROM table for mip level address offsets seems to use quite a bit of space. Have you considered making the mip base addresses a part of the texture spec instead?

I have seen semi-regular updates from you on discord and it is awesome to see how far this project has come (and also a bit frustrating to see how relatively little progress I have made on my FPGA projects in the same time!). I was hoping you'd do a writeup, can't wait!

How good would a Ryzen with 32 cores be if it did just graphics?

If you are going to that effort, you might also want a decent resolution. Say we aim for one megapixel (720p) and 30 frames per second, then we have to calculate 27.7 megapixel per second. If you get your FPGA to run at 500 MHz, that gives you 18 clock cycles per pixel. So you would probably want something like 100 cores keeping in mind that we also have to run vertex shaders. We also need quick access to a sizable amount of memory and I am not sure if one can get away with integer respectively fixed point arithmetics or whether floating point arithemtics is pretty much necessary. Another complication that I would expect is that it is probably much easier to build a long execution pipeline if you are implementing a fixed function pipeline as compared to a programmable processor. Things like out-of-order execution are probably best off-loaded to the compiler in order to keep the design simpler and more compact.

So my guess is that it would be quite challenging to implement a modern GPU in an affordable FPGA if you want more than a proof of concept.

It all depends on your definition of "open", of course. As far as I know there is no open-source toolchain for any remotely-recent FPGA, so you're still stick with proprietary (paid?) tooling to actually modify it. You're pretty much out of luck if you need more than an iCE40 UP5k.

There's no open hardware GPU to speak of. Depending on license (can't find information?), this could be the first, and a starting point for more.

There's this which is about the same kind of GPU

https://github.com/asicguy/gplgpu

Depends what you mean by "GPU".

https://github.com/schlae/graphics-gremlin is an MDA/CGA compatible adapter

https://github.com/OmarMongy/VGA is a VGA core

https://github.com/archlabo/Frix is a whole IBM PC compatible SoC, including a VGA.

Ticket2Ride Number9 is a fixed function GPU from the late 90s that was completely open sourced under GPL

There's also Nyuzi which is more GPGPU focused https://github.com/jbush001/NyuziProcessor, but the author also experimented with having it do 3D graphics.

How so?

Very briefly, until someone makes an ASIC that does the same thing and FPGAs are relegated to niche use-cases once again.

FPGAs only make long-term sense in applications that are so low-volume that it's not worth spinning an ASIC for them.

Four-bit floats are not as useful as Nvidia would have you believe. Like structured sparsity it's mainly a trick to make newer-gen cards look faster in the absence of an improvement in the underlying tech. If you're using it for NN inference you have to carefully tune the weights to get good accuracy and it offers nothing over fixed-point.

How? NPUs are going to be included in every PC in 2025. The only differentiators will be how much SRAM and memory bandwidth you have or whether you use processing in memory or not. AMD is already shipping APUs with 16 TOPS or 4 TFLOPS (bfloat16) and that is more than enough for inference considering the limited memory bandwidth. Strix Halo will have around 12 TFLOPS (bfloat16) and four memory channels.

llama.cpp already supports 4 bit quantization. They unpack the quantization back to bfloat16 at runtime for better accuracy. The best use case for an FPGA I have seen so far was to pair it with SK Hynix's AI GDDR and even that could be replaced by an even cheaper inference chip specializing in multi board communication and as many memory channels as possible.

Entire landscape of open graphic chips?

Not every GPU should be used to train or infer so-called AI.

Please, stop, we need some hardware to put images on the screens.

you see they have ARM SoCs rather than Xylinx FGPAs

It's a mixed chip: FPGA and traditional SoC glued together. This mean you don't have a softcore MCU taking up precious FPGA resources just to do some basic management tasks.

Xilinx doesn't mention the exact FPGA p/n used in the Kria SoMs. However according to their public specs they appear to match [1] the ZU3EG-UBVA530-2L and ZU5EV-SFVC784-2L devices, with the latter being the only one featuring PCIe support.

Designing and bringing-up the FPGA board as described in the blog post is already a high bar to clear. I hope the author will at some point publish schematics and sources.

[1] https://docs.amd.com/v/u/en-US/zynq-ultrascale-plus-product-...

You're in luck! https://imgur.com/a/BE0h9cZ

As mentioned in the rest of this thread, the Kria SoMs are FPGA fabric with hardened ARM cores running the show. Beyond just being what was available (for oh so cheap, the Kria devboards are like $350!), these devices also include things like hardened DisplayPort IP attached to the ARM cores allowing me to offload things like video output and audio to the firmware. A previous version of this project was running on a Zynq 7020, for which I needed to write my own HDMI stuff that, while not super complicated, takes up a fair amount of logic and also gets way more complex if it needs to be configurable.

It's worth mentioning that it's easy enough to find absurdly cheap (~$20) early-generation dev boards for Zynq FPGAs with embedded ARM cores on Aliexpress, shucked from obsolete Bitcoin miners [1]. Interfaces include SD, Ethernet, 3 banks of GPIO.

[1] https://github.com/xjtuecho/EBAZ4205

Ages ago I bought TinyFPGA, which is like £40 and I was able to synthesize RISC-V cpu on it. It was fun.

Not in the grand scheme of things: you can get fpga dev boards for $50 that are already useable for this type of thing (you can go even lower, but those aren't really useable for "CPU like" operation and are closer to "a whole lot of logic gates in a single chip"). Of course the "industry grade" solutions pack significantly more of a punch, but they can also be had for <$500.

In general, yes. However, the Kria series are amazingly good deals for what you get - a quite powerful Zynq US+ part and a dev board for like $350.

The FPGA he is using has native pcie so usually all you get on this front is an interface to a vendor proprietary ip block. The state of open interfaces in FPGA land is abysmal. I think the best I’ve seen fully open source is a gigabit MAC

It uses https://github.com/alexforencich/verilog-pcie on top of the Xilinx PCIe hard IP core, which provides everything below the transaction layer.

Next blog post will be covering exactly that! Probably going to do a multi-part series - first one will be the PCB schematic/layout, then the FPGA interfaces and testing, followed by Windows drivers.

Maybe a little bit too low-powered for you, but: https://www.matrixorbital.com/ftdi-eve

What about MXM GPUs that used to be found in gaming laptops? I know the standard is very niche and thus expensive ($400 for a 3080M used on ebay) but it does exists and you could convert them to PCI-E and thus m.2

There's at least one m.2 GPU based on the Silicon Motion SM750 controller made by Asrock Rack. Similar products exist for mPCIe form factor.

Performance is nowhere near a modern iGPU, because an iGPU has access to all of the system memory and caches and power budget, and a simple m.2 device has node of that. Even low-end PCIe GPUs (single slot, half-length/half-height) struggle to outperform better iGPUs and really only make sense when you have to use them for basic display functionality.

You probably know this already, but for anyone else curious about going down that road: For this type of use, it's definitely worth it to constrain yourself to FPGAs with dedicated high-bandwidth transceivers. A "basic" 1080p RGB signal at 60hz requires some high-frequency signal processing that's really hard to contend with in pure FPGA-land.

It's not a "product" that will be "sold" or has intention of being "successful" in a commercial sense.

I'm in the process of actually trying to work out what would be feasible performance-wise if I were to spent the considerable effort to add the features required for base D3D support. It's not looking good, unfortunately. Beyond just "shaders", there are a significant amount of other requirements that even just the OS's window manager needs to function at all. It's all built up on 20+ years of evolving tech and for the normal players in this space (AMD, Nvidia, Intel, Imagination, etc.) it's always been an iterative process.