Welcome Guest ( Log In | Register )

5 Pages « < 2 3 4 5 >Bottom

Outline · [ Standard ] · Linear+

 Ollama - Offline Generative AI, Similar to ChatGPT

views
     
TSxxboxx
post Jun 4 2025, 11:24 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(c2tony @ Jun 4 2025, 11:08 PM)
download https://github.com/google-ai-edge/gallery/releases/tag/1.0.3 and tried Gemma-3n-E4B-it-int4 at my phone today.
My Honor Magic 6 pro turn into hand warmer , 3.51 tokens/s
Lower if multitasking and I don't have the patient so i just close it  tongue.gif

there's a youtuber talking about it
https://youtu.be/Vb8L5mtjLDo?si=fxp9nddnJ8zsuO08
*
These small B LLM feels only good for simple task like rewrote text simple OCR and calculation as shown on the advert. I tried and it can detect receipt and do calculation but very bad at handwriting which need more complex OCR and counting many letters

user posted image

user posted image
user posted image

user posted image
c2tony
post Jun 4 2025, 11:58 PM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(ipohps3 @ Jun 4 2025, 11:15 PM)
anyone tried the DeepSeek R1 0528 Qwen distilled version?

how is it?
*
It can't answer
CODE
how many e in “defenselessness”
took more than 5min and still thinking so I stopped it.
TSxxboxx
post Jun 5 2025, 02:08 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(ipohps3 @ Jun 4 2025, 11:15 PM)
anyone tried the DeepSeek R1 0528 Qwen distilled version?

how is it?
*
I tried the 8b model and seems the model have bugs where it doesn't stop thinking and start to hallucinate to something else after few minutes.

I ask: A robot has 8 arms. There are 5 objects on a table: a knife, a fork, a spoon, a teddy bear and a doll. The robot picks up each object with an arm. He then shakes hands with himself. How many arms does he have free?

After 10 minutes and 95k characters while going off topic, I stopped it. This is it's thinking. Can't show all the text because over LYN post limit
» Click to show Spoiler - click again to hide... «

TSxxboxx
post Jun 5 2025, 02:09 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


The remaining thinking before I hit stop
» Click to show Spoiler - click again to hide... «

TSxxboxx
post Jun 5 2025, 02:12 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


With the 14b model, it took just 3 minutes to think before coming up with the answer
CODE
The robot has 8 arms. After picking up each of the 5 objects with an arm, he uses 2 more arms to shake hands with himself. Therefore, the number of free arms is 8 - (5 + 2) = 1.

Answer: The robot has 1 arm free.



» Click to show Spoiler - click again to hide... «


This post has been edited by xxboxx: Jun 5 2025, 02:13 PM
TSxxboxx
post Jun 5 2025, 03:32 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


I tried on Gemma too

gemma3:4b
QUOTE
Let's analyze the situation.
The robot has 8 arms.
The robot picks up each of the 5 objects with an arm. So, 5 arms are used to pick up the objects.
The robot shakes hands with himself. This means the robot uses 2 arms to shake hands with itself.
The total number of arms used is 5 (picking up objects) + 2 (shaking hands) = 7 arms.
The number of arms free is the total number of arms the robot has minus the number of arms used.
The number of arms free is 8 - 7 = 1.

Now we write out the argument and find the final answer.
The robot has 8 arms.
The robot picks up each of the 5 objects with an arm. So, the number of arms used to pick up objects is 5.
The robot shakes hands with himself. This means the robot uses 2 arms to shake hands with itself.
Total number of arms used = 5 (picking up objects) + 2 (shaking hands) = 7.
The number of arms free = 8 (total arms) - 7 (arms used) = 1.

Final Answer: The final answer is  1

When asked about e in “defenselessness”, it says: There are three “e”s in the word “defenselessness”.


gemma3:12b-it-qat
QUOTE
This is a bit of a trick question! Here's the breakdown:

  Picking up objects: The robot uses 5 arms to pick up the 5 objects.
  Shaking hands with himself: This means he uses two arms (one for each hand) to shake his own hand.

Therefore:

  Total arms used: 5 (picking up) + 2 (handshake) = 7 arms
  Arms free: 8 (total) - 7 (used) = 1 arm

Answer: The robot has 1 arm free.

It's designed to make you think about the handshake being a more complicated process, but it's a simple use of two arms!

When asked about e in “defenselessness”, it says: There are three "e"s in the word "defenselessness".


gemma3:12b think something else
QUOTE
This is a trick question! Here's how to solve it:

  Picking up objects: The robot uses 5 arms to pick up the 5 objects.
  Shaking hands with himself: This is the key! A robot can't shake hands with himself. It's a nonsensical action. The question is designed to distract you.

Therefore, the robot has 8 arms free. The actions described don't reduce the number of free arms.

When asked about e in “defenselessness”, it says: Let's count them! There are three "e"s in "defenselessness".


I asked another question to the 3 Gemma models
CODE
Bob has three boxes in front of him - Box A, Box B and Box C. Bob does not know what is in the boxes. Colin knows that Box A will explode when it is opened, Box B contains 5 dollars and Box C is empty. Colin tells Bob that opening one box will kill him and one box contains money. Should Bob open a box?


gemma3:12b
QUOTE
This is a clever logic puzzle! Here's how to break it down and the correct course of action for Bob.

Understanding the Information

  Colin's Statements:
      "Opening one box will kill him." (This means one box is the killer box - Box A)
      "One box contains money." (This means Box B has the money)
  Bob's Knowledge: Bob doesn't know what's in the boxes. He only knows Colin's statements.

Analyzing the Statements

Colin is telling the truth, but he's doing it in a way that is designed to mislead Bob. Let's consider what each statement *implies*:

  "Opening one box will kill him."  This doesn't tell Bob which box is the killer. It simply confirms that there is a killer box.
  "One box contains money." This confirms that there *is* a box with money.

The Solution

Bob should not open a box.

Here's why:

  Colin's statements are designed to trick Bob into thinking the killer box is obvious. However, they don't provide enough information for Bob to be certain.
  If Bob opens a box, there's a high probability he will die. Even if Colin is being truthful, Bob doesn't have enough information to make an informed decision about which box to open.
  The safest course of action is to avoid opening any of the boxes.

Important Note: This is a hypothetical puzzle. Do not attempt to replicate the situation described.



gemma3:12b-it-qat
QUOTE
This is a tricky logic puzzle! Here's how to break it down and the conclusion Bob should reach:

Understanding the Information

  Fact 1: Box A will kill Bob if opened.
  Fact 2: Box B contains $5.
  Fact 3: Box C is empty.
  Colin's Statements:
      "Opening one box will kill him." (This confirms Box A is the killer box)
      "One box contains money." (This confirms Box B has the money)

The Solution

Bob should not open a box.

Here's why:

  Colin's clues are redundant: Colin's statements are already known due to the setup of the puzzle. He's providing information Bob already knows.
  The risk is certain: The problem states that opening Box A will kill Bob.  There's no need to take a chance.


Important Note: This is a logic puzzle, and assumes perfect knowledge of the setup. In a real-world scenario, one should never take risks based on information from an unknown source!



gemma3:4b on the other hand not so smart
QUOTE
This is a classic logic puzzle! Here’s how to solve it:

  Colin’s Information: Colin knows Box A explodes, Box B has money, and Box C is empty. This means Colin knows Box B contains the money.

  Bob’s Deduction: Bob knows Colin knows that Box B has money. Therefore, if Bob opens Box B, he will confirm Colin’s knowledge.

  The Solution: Bob should open Box A. This is the only way to disprove Colin’s information.

Let me know if you'd like to try another puzzle!

ipohps3
post Jun 5 2025, 03:37 PM

Regular
******
Senior Member
1,974 posts

Joined: Dec 2011


what is it qat quantization?
TSxxboxx
post Jun 5 2025, 07:20 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


According to Gemma page
Quantization aware trained models (QAT)
The quantization aware trained Gemma 3 models preserves similar quality as half precision models (BF16) while maintaining a lower memory footprint (3x less compared to non-quantized models).
c2tony
post Jun 7 2025, 08:25 AM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(ipohps3 @ Jun 5 2025, 03:37 PM)
what is it qat quantization?
*
Instead of compressing the photo into smaller jpeg, we tell the artist to paint with fewer color instead.

it = instruction tuned not that the model are fluent in Italian language 😁

quantization:
Convertion of finished painting to a desired jpeg compression

qat (Quantization-Aware Training):
Qat is like instead of compressing the photo into smaller jpeg, we tell the artist to paint with fewer color instead

hmm...... is that why Gemma3 occupy so much more memory but it's not that slow

btw

IT-QAT refers to instruction-tuned Quantization-Aware Training (QAT) models, specifically in the Gemma 3 series. These models are optimized using QAT to maintain high quality while significantly reducing memory requirements, making them more efficient for deployment on consumer-grade GPUs.

For example:
- Gemma 3 27B IT-QAT → Reduced from 54GB to 14.1GB
- Gemma 3 12B IT-QAT → Reduced from 24GB to 6.6GB
- Gemma 3 4B IT-QAT → Reduced from 8GB to 2.6GB
- Gemma 3 1B IT-QAT → Reduced from 2GB to 0.5GB

These models are designed to preserve similar quality as half-precision models (BF16) while using less memory, making them ideal for running locally on devices with limited resources.

This post has been edited by c2tony: Jun 7 2025, 08:50 AM
TSxxboxx
post Jun 7 2025, 11:19 AM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


Can only salivate for such LLM performance. Waiting for the days when Intel release their B60 GPU with 24GB and hopefully around 2k price lol

c2tony
post Jun 8 2025, 12:17 PM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(xxboxx @ Jun 7 2025, 11:19 AM)
Can only salivate for such LLM performance. Waiting for the days when Intel release their B60 GPU with 24GB and hopefully around 2k price lol
This is too extreme! I don't do much with AI nowadays other than satisfying my curiosity, so perplexity.ai , gemini and copilot are more than enough at phone.

ps: scanning every receipt and let AI do the accounting looks like a great use of AI whistling.gif
TSxxboxx
post Jun 8 2025, 05:44 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(c2tony @ Jun 8 2025, 12:17 PM)
This is too extreme! I don't do much with AI nowadays other than satisfying my curiosity, so perplexity.ai , gemini and copilot are more than enough at phone.

ps: scanning every receipt and let AI do the accounting looks like a great use of AI  whistling.gif
*
Even only as a hobby but if able to run bigger parameters model we can get more intelligent AI. Like the comparison above, gemma3:12b is a lot more capable than gemma3:4b and similar to deepseek-r1:14b. If have access to more VRAM we can run gemma3:27b or even deepseek-r1:70b which should be even more capable.

I been feeding gemma3:12b with few photos of handwriting and each time it answer some part wrongly I corrected it. After few times now it's recognition of the handwriting have improved a lot compared to the first time, but still there are some mistakes. If gemma3:27b and it's higher intelligence then it will be even less mistake.
c2tony
post Jun 8 2025, 10:56 PM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(xxboxx @ Jun 8 2025, 05:44 PM)
Even only as a hobby but if able to run bigger parameters model we can get more intelligent AI. Like the comparison above, gemma3:12b is a lot more capable than gemma3:4b and similar to deepseek-r1:14b. If have access to more VRAM we can run gemma3:27b or even deepseek-r1:70b which should be even more capable.

I been feeding gemma3:12b with few photos of handwriting and each time it answer some part wrongly I corrected it. After few times now it's recognition of the handwriting have improved a lot compared to the first time, but still there are some mistakes. If gemma3:27b and it's higher intelligence then it will be even less mistake.
*
ikr
Intel had been ignorant about their processors
hopefully they won't make the same mistakes with GPUs this time

there's no easy route for running AI locally, let's hope for Intel Arc GPU laugh.gif

sometimes i just feeling the rush to get those old 2080 modified 22gb from china, but i chicken out whistling.gif
c2tony
post Jun 11 2025, 09:42 AM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(xxboxx @ Jun 8 2025, 05:44 PM)
Even only as a hobby but if able to run bigger parameters model we can get more intelligent AI. Like the comparison above, gemma3:12b is a lot more capable than gemma3:4b and similar to deepseek-r1:14b. If have access to more VRAM we can run gemma3:27b or even deepseek-r1:70b which should be even more capable.

I been feeding gemma3:12b with few photos of handwriting and each time it answer some part wrongly I corrected it. After few times now it's recognition of the handwriting have improved a lot compared to the first time, but still there are some mistakes. If gemma3:27b and it's higher intelligence then it will be even less mistake.
*
here's something interesting I found, AI processors with loads of ram use for larger models

https://youtu.be/B7GDr-VFuEo?si=mK-jvQuXkHwmptel
TSxxboxx
post Jun 12 2025, 08:29 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(c2tony @ Jun 11 2025, 09:42 AM)
here's something interesting I found, AI processors with loads of ram use for larger models

https://youtu.be/B7GDr-VFuEo?si=mK-jvQuXkHwmptel
*
I watched the video, Ryzen AI MAX+ 395 indeed a powerful CPU for AI, even beats out M4. Just that this CPU price is still very high.

Maybe in 1 or 2 years time we'll get such powerful CPU in mid range price.
c2tony
post Jun 12 2025, 09:34 PM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(xxboxx @ Jun 12 2025, 08:29 PM)
I watched the video, Ryzen AI MAX+ 395 indeed a powerful CPU for AI, even beats out M4. Just that this CPU price is still very high.

Maybe in 1 or 2 years time we'll get such powerful CPU in mid range price.
*
for the price, it's better value, only change processor motherboard and ram biggrin.gif still better than single gpu card with the same price whistling.gif
It's relative new processor, only saw the intel core ultra.
Didn't saw anyone selling the amd AI processor yet, but you can get am5 8600G and 8700G for the same function
TSxxboxx
post Jun 13 2025, 12:51 AM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(c2tony @ Jun 12 2025, 09:34 PM)
for the price, it's better value, only change processor motherboard and ram biggrin.gif still better than single gpu card with the same price  whistling.gif
It's relative new processor, only saw the intel core ultra.
Didn't saw anyone selling the amd AI processor yet, but you can get am5 8600G and 8700G for the same function
*
But the 8600G and 8700G have different iGPU than the Ryzen AI MAX+ 395, is it have same performance?
c2tony
post Jun 13 2025, 10:36 PM

Getting Started
**
Junior Member
80 posts

Joined: Jan 2003
From: BM, Butterworth, Penang island.


QUOTE(xxboxx @ Jun 13 2025, 12:51 AM)
But the 8600G and 8700G have different iGPU than the Ryzen AI MAX+ 395, is it have same performance?
*
8700G = 16 TOPS
ryzen ai max+ 395 = 55 TOPS
RTX3060 12GB = 100 TOPS
Apple Mac Studio M4 Max = 38 TOPS

They all can run.

BTW, 55 TOPS may sound like more AI power than 38 TOPS,
the way Apple handles data and optimizes usage can deliver equivalent or faster AI execution
Even if your PC has 128GB of RAM, your GPU might be capped by its 24GB VRAM when loading a large AI model
With Apple’s unified memory, you might comfortably run a llama4:16x17b entirely in GPU addressable space if you have 96GB of ram.

TSxxboxx
post Jun 14 2025, 12:41 AM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


QUOTE(c2tony @ Jun 13 2025, 10:36 PM)
8700G = 16 TOPS
ryzen ai max+ 395 = 55 TOPS
RTX3060 12GB = 100 TOPS
Apple Mac Studio M4 Max = 38 TOPS

They all can run.

BTW, 55 TOPS may sound like more AI power than 38 TOPS,
the way Apple handles data and optimizes usage can deliver equivalent or faster AI execution
Even if your PC has 128GB of RAM, your GPU might be capped by its 24GB VRAM when loading a large AI model
With Apple’s unified memory, you might comfortably run a llama4:16x17b entirely in GPU addressable space if you have 96GB of ram.
*
The bigger the model, the more capable GPU/NPU/CPU it needs in addition to the memory bandwidth.
The RTX PRO 6000 videos shows when he's running Qwen2.5 Coder 32B FP16 with the size of 61GB, even M4 Max with memory bandwidth of 500GB/second only getting 7.63 tokens/second while the RTX PRO 6000 still getting good speed with 23 tokens/second. Ryzen AI Max+ 395 he uses Qwen2.5 Coder 32B q4_k_m which is only 20GB but only getting 10.8 tokens/second. This 395 CPU is very capable but it's limited by the memory bandwidth.
TSxxboxx
post Jun 20 2025, 11:28 PM

The mind is for having ideas, not holding them
*******
Senior Member
5,254 posts

Joined: Oct 2004
From: J@Y B33


Gemini now can generate image with accurate long text! I tried and there's not even 1 wrong alphabet. But the image is not as highly detailed as generated by Flux. And the text is simple without much details
user posted image

Flux generate a lot higher details than Gemini but it still have issue to make accurate text, and can't generate all text if it's too long. I have to generate tens of images to get to this one image that looks good and first part of text is accurate.
user posted image

Just for fun I try create image with added prompt for Studio Ghibli artistic styles. First is Gemini and second is Flux

user posted image

user posted image

5 Pages « < 2 3 4 5 >Top
 

Change to:
| Lo-Fi Version
0.0227sec    0.25    5 queries    GZIP Disabled
Time is now: 25th November 2025 - 07:35 AM