Fundamentals of GPU Architecture

The first thing we have to think about is what is the space of GPUs, will GPUs replace CPUs, etc. There’s a couple things we need to think about. Back in 2005, we started to hit the power wall, which means that we can’t increase the voltage and have a hard time increasing the clock frequency as a result, making transistors smaller and trying to point clock frequency to the edge. 

CPUs exploit instruction level parallelism, picking out instructions for reorder buffer, data parallelism (maybe no dependencies there) or task level parallelism with completely different tasks we need to do. We have no more cores during every processor generation. 

How do we overcome this? A lot of this came down to vector hardware, doing things in parallel, with extensions and other hardware in a majority of parks now that specifically tries to tackle and assist in this area. GPUs are massively parallel processor. There’s a couple things though. It’s not just about finding parallelism, a lot of it comes down to minimizing data movement. If we have large numbers of data, we have finite cache sizes and finite memory, so we need all the data to be at the right place at the right time. 

We need to balance efficiency with the need to increase flexibility. Machine learning is extremely popular, not going away anby time soon so that a lot of research is “how can we improve machine learning performance?” We don’t want specialized hardware for everything, and this is where GPUs come in. GPUs are Turing complete programming model, so they can run arbitrary applications. 

GPUs are an order of magnitude more efficient than CPUs. This doesn't mean every task though. There’s a lot of things that have been going on with GPU development, architecture level, how to make things go faster. Another key thing is about the programming model. Parallel programming is incredibly difficult at times. It’s getting better despite the major gaps there. Right now, where you’re trying to efficiently create GPU code a lot of the times you have to explicitly know what type of GPU you are going to run on if you want to see the best performance. 

Fundamentally you can think of a CPU as a racecar and you can think of a GPU as a bus. CPU is fast but can hold fewer people, and this is the other way for the GPU. CPU is better for control flex while the GPU is better for tons of things that all run in parallel. They are really more seen as a coprocessors than a replacement for the CPU. 

Something that is fundamentally more sound is interfacing IO devices. From a CPU it makes sense going from a thread. When we are on a GPU we are not thinking about things in terms of a single thread, but large collections of threads. We don’t want 1,000 threads opening a file but there has been a considerable amount of work helping to directly do IO from the GPU. 

GPU is better at interfacing with I/O devices. Let’s read into a file from a GPU. It makes sense for the CPU but for the GPU we really don’t think about things in terms of a single thread but in terms of a large collection of threads. 

What’s the typical design of a CPU/GPU system? There are 2 main types of systems. A lot of times we have a discrete GPU or we can also have an integrated CPU/GPU.  The CPU and GPU can be kind of built in together with memory on the side. 

CPU DRAM is optimized for low latency access and GPU DRAM is utilized for high throughput. LPDDR is optimized for low power devices. CPU organizes movement of AD Data. We can transfer data from GPU memory to CPU memory leveraging virtual memory support, called unified memory. 

Cache coherence (problem is sharing) determines who has access to what data at a specific time, what happens when somebody wants to write something or somebody wants to read something. 

After the management of memory, we need to tell the GPU to do something so this is done by the driver, and as a result we need to specify which code should be run by the GPU. We want to optimize/speed up the inner loop/kernel and the kernel is the small portion of the program we want to exploit the parallelism of. The kernel is a computer program at the core of a computer's operating system and generally has complete control over everything in the system.

We need to say where all the data is being allocated in the GPU. We need to have the driver translate the information and place it into the location accessible to the GPU where the GPU can look for it. These are streaming multiprocessors or compute units. Each GPU executes a SIMT execution (single instruction, multiple threads). Each core on the GPU can typically run on the order of 1,000 threads that can be assigned and the threads can be communicated via scratchpad memory and synchronized using fast barrier applications. 

General layout is having a couple SIMT cores assigned to a specific memory partition which is assigned to a specific part of memory. To sustain competition throughput we need to do high memory bandwidth. In GPUs the parallel systems are provided and by having multiple memory channels we have high level memory. GPUs will have L1 cache, L2 cache, memory channel, and the backing memory. 

In terms of performance, when we are talking about CPUs we need to do a fair comparison against state-of-the-art. Superscalar out-of-order CPUS on highly parallel workloads. GPUs don’t have things like branch prediction, out of order execution, but they are much more simple but the number of ALUs GPU has is an extreme increase over what a traditional CPU will have. There is a balance, with regions where multi-core CPUs are better and a region where massively multithreaded GPUs are better. 

If we have something predict-dense, there’s multiple ALUs. On CPUs we’ll increase the threads until we can’t really match the cache anymore. As a result, there needs to be a tradeoff since parallelism with a platform isn’t necessarily a good thing. 

We don’t want to access very large memory structures that consume a ton of energy. GPUs used to be rendering for animation. Eventually we moved onto 2D and 3D acceleration, and then started to get some programmability to the GPU for the vertex shaders and the pixel shaders. People were like “what if I don’t really care about graphics that much?”. So people found interesting ways mapping matrix data into textures and applying shaders. 

We started to free up the memory system and then also enabled caching read-write data in certain architectures, so then we reached the modern architecture. We have dynamic parallelism, things like launching from the GPU itself, etc. Then of course one of the more recent introductions is Turing architecture, TensorFlow, machine learning acceleration.