HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees

People are now running deep learning algorithms on GPU clusters. Some clusters get worse queuing delay than other cluster. Tenants use quota the number of GPUs to reserve resources, where deep learning uses GPUs that are mainly unknown. 

HiveD shares a GPU cluster safely so such that would never happen by design. Each tenant reserves through Virtual Private Cluster, defined in terms of multi-level cell structures. The Hive-D can subsequently deploy schedulers to achieve respective design goals. 

Unfortunately there are sharing anomalies in 3 state-of-the-art schedulers with a long delay. HiveD can eliminate sharing anomaly in these schedulers, and separate them so each scheduler can perform their tasks separately and simultaneously. 

An organization can train deep learning models in a GPU cluster, where each tenant reserves

1. GPUs

2. CPU and memory. 

Training jobs would linger slightly longer for GPUs because the reservation mechanism is based on a quota, denoted in the number of GPUs. Resource reservation is heavily based on the number of GPUs. 

HiveD guarantees safe sharing by completely eliminating sharing anomalies. HiveD provides a new VC (virtual private cluster), which is called a cell, representing affinities that GPUs can satisfy. It focuses mainly on a resource reservation mechanism, and develops a buddy cell allocation algorithm to bind cells from a virtual cluster to a physical cluster. This advocates dynamic cell binding over static cell binding.

This dynamically creates and releases the binding of cells to GPUs while proving safety sharing despite unpredictable workloads. It can also be naturally extended to support low-priority jobs to scavenge unused cells and improve overall utilization. 

This is evaluated in 96 experiments, and there is a sharing anomalies on each state-of-the-art deep learning schedulers, and HiveD gets rid of all the sharing anomalies and queueing delay. HIVED is open source and integrated into OpenPAI, a Kubertnetes platform, and deployed on many clusters. This paper proposed a new abstraction to model VCs and have a buddy cell allocation algorithm, and has extensive evaluations on real cluster and through stimulation. 

This a joint work with colleagues with the university of Hong Kong and Microsoft.

Today, organizations are building large GPU clusters for training deep learning models. These clusters are built by multiple tenants and shared by each tenant, but we need to allocate resources for each tenant and prevent conflicts, with some tenants needing to reserve more resources than others.

Beyond number of GPUs, Deep learning clusters need a Good GPU affinity. State of the art is to get the quota and use some affinity mechanism. But in this, some tenants make worse performance on a shared cluster vs private cluster. Due to switching on jobs, there is way worse performance in shared clusters in comparison with private clusters. On shared cluster, a job has to run slower, because the GPU affinity is split into more clusters. 

Global defragmentation requires complex multi-objective optimization problems. Spreading the job has higher fragmentation and have the higher affinity of other tenants. 

HiveD implements sharing safety algorithms so that the conflicts in sharing goals would be virtually impossible. It separates the concern of sharing safety from other scheduling goals. 

Dynamic binding, or buddy allocation reduce preemptions and hardware, handling faulty hardware, and it is handled by buddy allocation algorithm.

For each cell, track k cell, and allocate k cell if any. Otherwise, we split a free K + 1 cell, to split into free-level K cell. Level k + 1 is larger than level K.  

If cells are no longer used, we will release them and merge them in an opposite manner. The Buddy Cell allocation guarantees to satisfy requests and thus creating dynamic bindings on the fly. 

HiveD has 2 cell views, one for high-priority cells and the other for low-priority jobs. We use dynamic binding as low-priority jobs. Marked cells for high-priority cells cannot be used by low-priority cell. We reduce cells as dirty in low priority cell, and the algorithm try to allocate high priority not to low-priority. However, we can allocate/change/preempt low priority jobs when possible.

HIVED comes with many features in Kubernetes. Stuff has been deployed at Microsoft, serving a lot of heterogeneous GPUs and details in the paper.

This is evaluated in 96 GPU clusters. There are 3 state-of-the-art deep learning.

There are clusters with quota, private cluster, and HIVE-D structures, and made possible by the affinity structure in the VC. The HiveD has a way more efficient queueing delay in comparison to the other clusters. HiveD eliminates excessive delay by low priority cluster. This shows HIVED shows the best of the private clusters and the shared clusters. 

We also did a 2 month trade of a 2,000 GPU cluster, and the results are much more significant. HiveD eliminates all of the sharing anomaly, 8,000 minutes. 2 large tenants only 37% of quota leave cluster, which leaves certain tenants for queuing delay, which means it is vital to guarantee safety. We want to reduce preemption (interruption) and avoid allocating dirty cells. We can pack cells more decently in order to reduce fragmentation. We fit the GPU number distribution of tenants job, and Multi-level fragment reduces the fragmentation times by over 20%.

To conclude, HiveD addresses the problem by defining a sharing safety (easily appreciated by tenants), proposing a new resource abstraction for defining structures. Buddy cell proves safety and support for low-priority jobs. 2-layer architecture also incorporates scheduling goals while simultaneously guaranteeing sharing safety.