I doubt the RAM is added up. I think that’s only a feature reserved for their NV... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Bancakes 8 months ago \| parent \| context \| favorite \| on: All You Need Is 4x 4090 GPUs to Train Your Own Mod... I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.

minimaxir 8 months ago | [–]

It depends on how the parallelism is implemented, e.g. distributed data parallel (DDP) to synchronize gradients: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

It's a rabbit hole I stay away from for pragmatic reasons.

whimsicalism 8 months ago | [–]

yeah essentially this

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact