full 671b iinm is slightly below 1400GB memory required /gg

NVlink can, like using nvlink on two rtx3090, you get 48GB shared vram.

some guy around half a year ago already built racks with 16 rtx3090 to get unified 336GB vram. But I dunno how he set it up with what operating system etc.

But it can also mean some smaller companies can setup own server using local run deepseek r1 system for own employees for sensitive data liao

This post has been edited by terradrive: Jan 28 2025, 05:14 PM

maybe that is using app version, since it routes the question back to deepseek servers in china

Ok I have locally run deepseek R1 32b Q4 now, this is the response:

lower parameter probably more dumb lol

political topics might not be that accurate since it is affected by the training data

for example it is fairly light on the details on gamergate while showing more bias on "toxicity and harassment in gaming culture" which is untrue. toxicity is minor issue on gamergate, I can pull up the real event on inner circle on people preserving the evidences of gamergate that was always being deleted by the woke ppl (it's more about journalism corruption)

This post has been edited by terradrive: Jan 28 2025, 07:46 PM

full bit 671b requires around 1400GB memory size of either the system RAM (if running cpu AI) or GPU VRAM.

Not rich enough to have that monster pc

This post has been edited by terradrive: Jan 29 2025, 06:30 AM

the competitor for o1

but of course doesn't have image generation and video generation capability under this deepseek r1.

if you wanna pay $200 dollars a month for unlimited o1 use lol

plus the ability to run deepseek r1 offline solves privacy issues which is very important for some users

It depends on memory size. I can only answer for GPU part. If you have 24GB VRAM gpus like rtx3090, rtx4090, you can go for quantized 4 bit deepseek-r1:32b. If you have 12 or 16GB gpu then you can try 14b. 8b for 8GB gpus. But this is for the distilled version quantized 4 bit, not the full bit. full bit uses way more memory.

It can also be run on CPU only, I didn't read much about that but I saw ppl running 70b using 9950X with 64GB system memory but the response is slow. Another built a amd dual epyc system with 768GB memory that managed to run the full bit 671b with 6-8 tokens/s with total system cost at his place around 6000usd.