#MEMORY MASTER WORKSHO FULL#

The following math is trying various topologies to fit into 80GB gpus Overall we know that DP is the fastest, then PP, then TP - but for PP to be efficient we need a big bs. in the current 1.3B ml setup, the 1st and last gpus use all of DRAM, but the rest of gpus use only 1/2 DRAM - and TLOPs are ~21 which is very underutilized. Most likely the embedding layer now will need to be partitioned together with the transformer blocks to do a good balancing of resources.

#MEMORY MASTER WORKSHO FULL#

Possible config:Ī single replica needs to fit 96 gpus and then we can do DP=4 to a full 384 gpusĮxtrapolating from the current 104B setup we can have: TP=4/PP=24 80GB + 150K vocab size (which is different from the 50k vocab in 104B - 3x bigger embed matrix plus bigger hidden size. Initially we will have only 144 gpus and then around mid-Feb we should have the rest.

We can plan to use 384 gpus out of 416 as 4 nodes of 8 gpus need to remain reserved for when some nodes happen to be down.

18GB (w/o embed) (2GB activations memory).

22GB (w/ embed) (4GB activations memory).

Memory master worksho