Phi3 updated to Phi3.5 and get it right this time

do anyone know how to close that thinking stuff on deepseek or qwen3? ollama webui of course

P/S: https://huggingface.co/jedisct1/MiMo-7B-RL-...f?download=true

can play with xiaomi ai

This post has been edited by c2tony: Apr 30 2025, 10:04 PM

yes, but i don't want it to show that arrow! it takes more times to show the steps, whether you click it or not

did not manage to try MiMo yet, i don't know how to load gguf

thanks for the command
I tried it also

Do you familiar with the thought experiment the ship of Theseus?
In the field of identify metaphysics?
If those removed planks are restored and reassembled, free of the rot, is that the ship of Theseus?

the third question it "think" about 7min

Neither is the true ship or both are the true ship?
- it's still thinking...

llama3.2:3b-instruct-fp16 after 2+ min answered: In the word "benzodiazepines", the letter "e" appears three times.
while smollm2:1.7b-instruct-fp16 gave me TypeError: NetworkError when attempting to fetch

XiaoMi's MiMo LLM are relative new.
Afterall they're all LLMs using the same "highway": pattern recognizing. If AI starting to understand then we might need to concern about their concise awakening

Lately gemma3 update for 12b are annoying, they distribute it to my cpu & gpu, just won't run at 100% GPU anymore.

12GB vram, i know...
Qwen3 with 14b and 9.3GB uses my 100% GPU, for a comparison





This post has been edited by c2tony: May 26 2025, 10:09 PM

Yeah, you paid. That's the whole point! "one-off" or "batch" processing are best when you pay. I wouldn't pay $20 for my use case.

Gemini is a closed system, you don’t get to tweak it, audit its training data, or run it locally.
For some users, this trade-off is worth it.
For others, it’s not. Not to mention there's a limit for API can't be use at offline

That's why I say after update, a 12b model of 8.9GB becomes 12GB when you actually runs it.

Gemini respond: It gets de-quantized or processed in a higher precision during runtime. The architecture, the specific runtime precision of activation and KV cache, and the optimizations applied by the inference framework
Gemma 3's multimodal nature and potentially different KV cache handling seem to be key contributors to its higher observed runtime memory usage compared to Qwen 2 14B models of similar quantization.

this mean time for a higher vram gpu or upgrade to npu

Geez... You must using a lot of OCR?

They gets better and better with a lot of added rules and regulations, uncensored wild west are disappearing

what platform did you use on mobile or edge side? I installed PocketPal with llama-3.2-1b-instruct only, mobile have lots of distraction

I think he mean this:

download https://github.com/google-ai-edge/gallery/releases/tag/1.0.3 and tried Gemma-3n-E4B-it-int4 at my phone today.
My Honor Magic 6 pro turn into hand warmer , 3.51 tokens/s
Lower if multitasking and I don't have the patient so i just close it

there's a youtuber talking about it
https://youtu.be/Vb8L5mtjLDo?si=fxp9nddnJ8zsuO08

It can't answer took more than 5min and still thinking so I stopped it.

Instead of compressing the photo into smaller jpeg, we tell the artist to paint with fewer color instead.

it = instruction tuned not that the model are fluent in Italian language 😁

quantization:
Convertion of finished painting to a desired jpeg compression

qat (Quantization-Aware Training):
Qat is like instead of compressing the photo into smaller jpeg, we tell the artist to paint with fewer color instead

hmm...... is that why Gemma3 occupy so much more memory but it's not that slow

btw

IT-QAT refers to instruction-tuned Quantization-Aware Training (QAT) models, specifically in the Gemma 3 series. These models are optimized using QAT to maintain high quality while significantly reducing memory requirements, making them more efficient for deployment on consumer-grade GPUs.

For example:
- Gemma 3 27B IT-QAT → Reduced from 54GB to 14.1GB
- Gemma 3 12B IT-QAT → Reduced from 24GB to 6.6GB
- Gemma 3 4B IT-QAT → Reduced from 8GB to 2.6GB
- Gemma 3 1B IT-QAT → Reduced from 2GB to 0.5GB

These models are designed to preserve similar quality as half-precision models (BF16) while using less memory, making them ideal for running locally on devices with limited resources.

This post has been edited by c2tony: Jun 7 2025, 08:50 AM

This is too extreme! I don't do much with AI nowadays other than satisfying my curiosity, so perplexity.ai , gemini and copilot are more than enough at phone.

ps: scanning every receipt and let AI do the accounting looks like a great use of AI

ikr
Intel had been ignorant about their processors
hopefully they won't make the same mistakes with GPUs this time

there's no easy route for running AI locally, let's hope for Intel Arc GPU

sometimes i just feeling the rush to get those old 2080 modified 22gb from china, but i chicken out

here's something interesting I found, AI processors with loads of ram use for larger models

https://youtu.be/B7GDr-VFuEo?si=mK-jvQuXkHwmptel

for the price, it's better value, only change processor motherboard and ram still better than single gpu card with the same price
It's relative new processor, only saw the intel core ultra.
Didn't saw anyone selling the amd AI processor yet, but you can get am5 8600G and 8700G for the same function

8700G = 16 TOPS
ryzen ai max+ 395 = 55 TOPS
RTX3060 12GB = 100 TOPS
Apple Mac Studio M4 Max = 38 TOPS

They all can run.

BTW, 55 TOPS may sound like more AI power than 38 TOPS,
the way Apple handles data and optimizes usage can deliver equivalent or faster AI execution
Even if your PC has 128GB of RAM, your GPU might be capped by its 24GB VRAM when loading a large AI model
With Apple’s unified memory, you might comfortably run a llama4:16x17b entirely in GPU addressable space if you have 96GB of ram.