Tiny GPU: A minimal GPU implementation in Verilog

But how is that different from a CPU?

Is that pronounced gee pee you, or Gip Pee You?

That would ignore applications like crypto mining, which I'm guessing is still a biggie.

What is it exactly that sets these units apart from CPUs? Something to do with the parallel nature of the hardware?

I have seen the term NPU used in reference to neural network accelerators a lot. I think AMD, Intel and Qualcomm all use this term for their AI accelerators. I think Apple call their AI accelerators neural engines, but I've definitely heard others refer to these as NPUs even though that's not their official name.

I'll be honest I've never heard the AIA acronym used in this way. It seems all acronyms for all processors need to end in PU, for better or for worse.

The observation that graphic hardware and general purpose CPUs converge and diverge is not new: http://cva.stanford.edu/classes/cs99s/papers/myer-sutherland... .

But as you observe, we are stuck in a local optimum where GPUs are optimized for throughput and CPUs for latency sensitive work.

If it processes graphics, I think it counts, even if it has no output.

That's not a good definition, since a CPU or a DSP would count as a GPU. Both have been used for such purpose in the past.

There's still use for GPUs even if they're not outputting anything.

The issue is not their existence, it about calling them GPUs when they have no graphics functionality.

It's the shader core of a GPU. There are no graphics specific pipelines, eg: vertex processing, culling, rasterizer, color buffer, depth buffer, etc. That's like saying a CPU is also a GPU if it runs graphics in software.

Depends heavily on what system it is supposed to provide acceleration for.

If it is a MCU based on a simple ARM Cortex M0, M0+, M3 or RISC-V RV3I, then you could use an iCE40 or similar FPGA to provide a big acceleration by just using the DSPs and the big SPRAM.

Basically add the custom compute operations and space that doesn't exist in the MCU, operations that would take several, many instructions to do in SW. Also, just by offloading to the FPGA AI 'co-processor' frees up the MCU to do other things.

The kernel operations in the Tiny GPU project is actually really good examples of things you could efficiently implement in an iCE40UP FPGA device, resulting in substantial acceleration. And using EBRs (block RAM) and/or the SPRAM for block queues would make a nice interface to the MCU.

One could also implement a RISC-V core in the FPGA, thus having a single chip with a low latency interface to the AI accelerator. You could even implement the AI acceleator as a set of custom instructions. There are so many possible solutions!

An ice40UP-5K FPGA will set you back 9 EUR in single quantity.

This concept of course scales up to performance and cost levels you talk about. With many possible steps in between.

Something that appears to be hardly known is that the transformer architecture needs to become more compute bound. Inventing a machine learning architecture which is FLOPs heavy instead of bandwidth heavy would be a good start.

It could be as simple as using a CNN instead of a V matrix. Yes, this makes the architecture less efficient, but it also makes it easier for an accelerator to speed it up, since CNNs tend to be compute bound.

Love the Harris and Harris book!

I might add these:

- Computer Architecture, Fifth Edition: A Quantitative Approach - https://dl.acm.org/doi/book/10.5555/1999263

- Computer Organization and Design RISC-V Edition: The Hardware Software Interface - https://dl.acm.org/doi/10.5555/3153875

both by Patterson and Hennessy

Edit: And if you want to get into CPU design and can get a grip on "Advanced Computer Architecture: Parallelism, Scalability, Programmability" by Kai Hwang, then i'd recommend that too. It's super old and probably some things are made differently in newer CPUs, but it's exceptionally good to learn the fundamentals. Very well written. But I think it's hard to find a good (physical) copy.

My cheap little dell laptop is the most solid machine I have in my house and I haven't seen it crash yet and I half suspect it's because it's the only one with Intel gpu only in it :) . My tower machine can go a few days without crashing but inevitably it will whether it's 3 days or 1 week. Nvidia card. It's not often enough to really worry about but I've been thinking about switching to an AMD card, if I can find a second hand one reasonably priced.

The AMD documentation consists mostly of very laconic descriptions of the (hundreds or thousands of) registers and of their bit fields.

There is almost no explanation about how they are intended to be used and about the detailed microarchitecture of their GPUs. For that, the best remains to read the source code of their Linux drivers, though even that is not always as informative as it could be, as some of the C code may have been automatically generated from some other form used internally by AMD.

The Intel documentation is much more complete.

Nevertheless, AMD has promised recently in several occasions that they will publish in the near future additional GPU documentation and additional open-source parts of their GPU drivers, so hopefully there will be a time when the quality of their documentation will match again that of Intel, like it did until around 2000.

syncthreads synchronizes threads within a threadgroup and not across all threads.

Which experienced CUDA programmers do anyways!

No, that effectively syncs all warps in a thread group. This implementation isn't doing any synchronization, it's independently doing PC/decode for different instructions, and just assuming they won't diverge. That's... a baffling combination of decisions; why do independent PC/decode if they're not to diverge? It reads as a very basic lack of ability to understand the core fundamental value of a GPU. And this isn't a secret GPU architecture thing. Here's a slide deck from 2009 going over the actual high-level architecture of a GPU. Notice how fetch/decode are shared between threads.

https://engineering.purdue.edu/~smidkiff/ece563/slides/GPU.p...