How many GB is your VRAM? Windows also uses some VRAM, if 12GB then not enough. When not enough VRAM then it will offload some to CPU and causes the slowdown.

12GB vram, i know...
Qwen3 with 14b and 9.3GB uses my 100% GPU, for a comparison





This post has been edited by c2tony: May 26 2025, 10:09 PM

donno about you guys.

i was enthusiastic about open models earlier this year with DeepSeek in Jan and the following months with other open models being released also.

however, since last month and this month with Google Gemini 2.5 released, don't think I would want to go back using open models since Gemini+DeepMind is getting extremely good at almost all things and none of the open models that can run with RTX3090 can come close to it.

after sometime, paying the 20usd per month is more productive for me to get things done than using open models.

This post has been edited by ipohps3: May 26 2025, 09:56 PM

Yeah, you paid. That's the whole point! "one-off" or "batch" processing are best when you pay. I wouldn't pay $20 for my use case.

Gemini is a closed system, you don’t get to tweak it, audit its training data, or run it locally.
For some users, this trade-off is worth it.
For others, it’s not. Not to mention there's a limit for API can't be use at offline

There you go, not enough VRAM.
Why your gemma3:12b-it-qat is 12GB? I see ollama page it is only 8.9GB

Gemini indeed has got a lot better, also ChatGPT. For me just using it for fun, I didn't pay for the more capable model. Maybe that's why I feel the free model is still less capable than open source model. Question such as this Gemini 2.5 Pro still got it wrong

yeah. sometimes it get the basic wrong. i tried on ChatGPT seems can get it right. but anyway i don't use it for this trivial stuff. i mainly use the YouTube video analysis, deep research, audio overview podcast, and canvas features for coding and research on new topics purposes. main thing is its large 1M context window which no one can support it locally at home even if you have open model that support 1M context window.

This post has been edited by ipohps3: May 27 2025, 01:28 AM

For suggesting new ideas or perspective LLM is useful but when it's analysis and research I find LLM missed what is important for me and in the end I still have to do the analysis and research by myself.

Mac Studio with 512GB RAM can handle 1M context window or more, but the price
Maybe the upcoming Intel GPU for AI will solve the memory bottleneck issue.

That's why I say after update, a 12b model of 8.9GB becomes 12GB when you actually runs it.

Gemini respond: It gets de-quantized or processed in a higher precision during runtime. The architecture, the specific runtime precision of activation and KV cache, and the optimizations applied by the inference framework
Gemma 3's multimodal nature and potentially different KV cache handling seem to be key contributors to its higher observed runtime memory usage compared to Qwen 2 14B models of similar quantization.

this mean time for a higher vram gpu or upgrade to npu

I see what you mean now. gemma3:latest is 3.3GB but when runs is using 6.0GB
Maybe ollama added more functions and that also keep increasing the memory usage.
I saw this, just having more context length will increase a lot of memory usage
https://github.com/open-webui/open-webui/discussions/8303

I think because with v0.7.0 new engine, ollama added support for multimodal and this increase the memory usage significantly


I checked few model size vs actual loaded
phi4:latest 9.1GB become 10GB
deepseek-r1:14b 9.0GB become 10GB
MiMo-7B-RL-GGUF:Q8_0 8.1GB become 9.6GB
gemma3:12b 8.1GB become 11GB
gemma3:latest 3.3Gb become 6.0GB
llama3.2:latest 2.0Gb become 4.0GB
granite3.2-vision:2b-fp16 6.0Gb become 8.8GB

Model that support vision the size increase a lot more than model without vision

Ollama's vision now after update is a lot better than few months ago

Using gemma3:12b there is some wrong data but a lot better than previously


Qwen 2.5 recently also update for it's vision model and it is more accurate than gemma3 even though only at 7b vs 12b


Even with bigger picture that have more data it still can get most thing right

Geez... You must using a lot of OCR?

They gets better and better with a lot of added rules and regulations, uncensored wild west are disappearing

anyone tried Gemma 3n 4B ?

Yup, mainly for hand written on such table format. Other app OCR such as Snipping Tool or OneNote can't correctly recognize all the text or it can't preserve the table format. Been using ChatGPT before this but after few photos already reached daily caps for free user. I then use Gemini but it is not as accurate as ChatGPT. Now can rely on ollama to do OCR.

Pros and cons. But the penalty on memory usage is hardest to swallow. Hopefully in future they can reduce back the memory usage.

This post has been edited by xxboxx: May 30 2025, 10:32 PM

what platform did you use on mobile or edge side? I installed PocketPal with llama-3.2-1b-instruct only, mobile have lots of distraction

Gemma3:4B?

I tried it, less capable than 12B model

I think he mean this:

Oh no wonder I didn't saw it, I only checked at ollama website and there haven't got yet

download https://github.com/google-ai-edge/gallery/releases/tag/1.0.3 and tried Gemma-3n-E4B-it-int4 at my phone today.
My Honor Magic 6 pro turn into hand warmer , 3.51 tokens/s
Lower if multitasking and I don't have the patient so i just close it

there's a youtuber talking about it
https://youtu.be/Vb8L5mtjLDo?si=fxp9nddnJ8zsuO08

anyone tried the DeepSeek R1 0528 Qwen distilled version?

how is it?