At 1byte/param that's 1.6GB (f8), at 2 bytes (f16) that's 2.3GB -- but there's o... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		mjburgess 8 months ago \| parent \| context \| favorite \| on: How large are large language models? At 1byte/param that's 1.6GB (f8), at 2 bytes (f16) that's 2.3GB -- but there's other space costs beyond loading the parameters for the GPU. So a rule of thumb is ~4x parameter count. So round up, 2B -> 2*4 = 8GB VRAM

1718627440 8 months ago [–]

That sounds about the size of a modern browser (aka. any Electron et al. application)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact