Data Science PT 8: Computing Architectures

 

Let's now turn our attention to computational architectures. The random access machine model is somewhat simple, and some modern architectures offer significant speedups via parallelization. As a result, we need to be aware of these opportunities. 

A low end embedded processor has a clock speed of several MHz with the memory less than 1MB. There are math operations that require many clock cycles. A typical CPU has a clock speed of 2-5 GHz, hundreds of GB of memory, with many billion transistors and fast math operations. Some systems/chips support multiple processors. General Purpose GPUs are initially designed for graphics processing. It's highly parallelizable and can support thousands of cores. It's much faster is constrained (not fully parallel). Another example of Multi-Processors and cloud computing clusters.

Here is a pyramid. The top of the pyramid is the small amount of fast, expensive memory. The bottom of the pyramid is a large amount of slower but cheaper memory. We will search for top of the memory, where most of the data is usually cached, then spill into the lower levels. The CPU is a really fast register, and the L1, L2, and L3 are also registers. Caches store several MB of data with a dozen clocks. Disk/cloud is permanent memory, which is slow. We want hardware support for adjacent memory. 90% of the time we almost right away find what we seek.

There are several dozen registers on the CPU with same-clock access. There are around a dozen clocks for the caches, which is several MB in size. The main memory has around 100GB and takes several clocks to access. Permanent Memory is in TB (disk/cloud) and is very slow. GPUs have significant (GBs) on-chip memory. Each core has small fast local memory. GPU chips have significantly slower memory. As a solution, we want hardware support for adjacent memory access with high bandwidth or parallelism. Which brings me to parallel processing. 

We want a Parallel Random Access Machine, in contrast to a singular (Serial) Random Access Machine.

In concurrent read, Concurrent read, the PRAM algorithm reads concurrently (simultaneously) from same location, while in the exclusive read we never will simultaneously read from the same location. This is the same for writes. The 4 types of PRAM machines are: 

EREW – exclusive read exclusive write 

CREW – concurrent read exclusive write 

ERCW – exclusive read concurrent write 

CRCW – concurrent read concurrent write.

Concurrency is more complicated and thus slower compared to exclusivity. Fast memory is expensive. So we have a small amount of fast memory and large amount of slow memory.