stephen/138_llama_cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Working Version + long discuss with Gemini

Browse source
This commit is contained in:
stephen 2026-06-06 14:48:33 +10:00
parent ec2d246b65
commit 655ee2870d
3 changed files with 1209 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1083
doc/Gemma 4 12B： Local Multimodal LLM - Google Gemini (06_06_2026 14：18：24).html Normal file
View file

File diff suppressed because one or more lines are too long

134
doc/Notes.md
View file

@ -1,10 +1,46 @@
# 138_lama_cpp
- [1. 138\_lama\_cpp](#1-138_lama_cpp)
- [2. The test of llama.cpp](#2-the-test-of-llamacpp)
- [3. References](#3-references)
- [3.0.1. gemma reference](#301-gemma-reference)
- [3.0.2. unsloth Gemma 4 docs](#302-unsloth-gemma-4-docs)
- [3.0.3. hugging Face site](#303-hugging-face-site)
- [3.0.4. related google voice assistant](#304-related-google-voice-assistant)
- [4. Create the project](#4-create-the-project)
- [4.1. On the Forgejo Server](#41-on-the-forgejo-server)
- [4.2. clone development machine](#42-clone-development-machine)
- [5. Initial .gitignore](#5-initial--gitignore)
- [6. commit back to Forgejo server](#6-commit-back-to-forgejo-server)
- [7. create a source dir](#7-create-a-source-dir)
- [8. Build llama-server and llama-cli](#8-build-llama-server-and-llama-cli)
- [8.1. Attempting to get GGUF file and run](#81-attempting-to-get-gguf-file-and-run)
- [8.1.1. -This did not work there a few more tries but none of these worked, maybe because 8G file](#811--this-did-not-work-there-a-few-more-tries-but-none-of-these-worked-maybe-because-8g-file)
- [8.2. What happens next?](#82-what-happens-next)
- [8.3. Now install gguf file](#83-now-install-gguf-file)
- [8.3.1. Step 1: Download the File via curl](#831-step-1-download-the-file-via-curl)
- [8.3.2. Step 2: Launch the Server Region Locally](#832-step-2-launch-the-server-region-locally)
# The test of llama.cpp
# 1. 138_lama_cpp
# Create the project
# 2. The test of llama.cpp
## On the Forgejo Server
# 3. References
### 3.0.1. gemma reference
[gemma reference](https://ai.google.dev/gemma/docs)
### 3.0.2. unsloth Gemma 4 docs
[unsloth Gemma 4 docs](https://unsloth.ai/docs/models/gemma-4)
### 3.0.3. hugging Face site
[hugging Face site](https://huggingface.co/google)
### 3.0.4. related google voice assistant
[related google voice assistant](https://github.com/cris-m/voice-assistant)
# 4. Create the project
## 4.1. On the Forgejo Server
- sign in with
- stephen
@ -15,7 +51,7 @@ Create a blank repository
https://dev.oxnee.com./stephen/138_llama_cpp
## clone development machine
## 4.2. clone development machine
In a Mac terminal create the blank project
@ -73,12 +109,94 @@ origin https://dev.oxnee.com./stephen/138_llama_cpp.git (push)
```
I usually create a folder call doc and a folder below that for images
# Initial .gitignore
# 5. Initial .gitignore
```
./.git
.DS_Store
```
# commit back to Forgejo server
It shows 4 file commit them all
# 6. commit back to Forgejo server
It shows 4 file commit them all
# 7. create a source dir
In vs code terminal
```zsh
tephenlohning@Scotty 138_llama_cpp % mkdir src
stephenlohning@Scotty 138_llama_cpp % cd src
stephenlohning@Scotty src % git clone https://github.com/ggml-org/llama.cpp .
Cloning into '.'...
remote: Enumerating objects: 98051, done.
remote: Counting objects: 100% (204/204), done.
remote: Compressing objects: 100% (108/108), done.
remote: Total 98051 (delta 128), reused 100 (delta 96), pack-reused 97847 (from 3)
Receiving objects: 100% (98051/98051), 395.99 MiB | 22.23 MiB/s, done.
Resolving deltas: 100% (69995/69995), done.
stephenlohning@Scotty src % git remote -v
origin https://github.com/ggml-org/llama.cpp (fetch)
origin https://github.com/ggml-org/llama.cpp (push)
```
# 8. Build llama-server and llama-cli
My MacBook Pro already has a cpp compiler which was install with Xcode.
Step 1: Install CMake via Homebrew
You can run this right from where you are now to get the build tool your system is missing:
```bash
brew install cmake
```
Step 2:Jump Back into your llama.cpp Directory
Once brew finishes installing CMake, navigate back into the source repository directory where you cloned the code:
```bash
cd 138_llama_cpp/src
```
Step 3: Run the Modern Build Stream
Now that your terminal is in the correct directory and CMake is ready, initialize and compile the project by running these two steps in sequence:
```bash
# 1. Configure the build for your Apple Silicon Mac
cmake -B build
# 2. Compile the binaries using all your CPU cores
cmake --build build -j --target llama-cli llama-server
```
Once that hits 100%, your compiled binaries will be built and ready to go inside the build/bin folder.
Fantastic! The compilation log looks beautiful. CMake successfully detected your Apple Silicon layout (arm64), hooked into macOS’s native Accelerate framework, and fully compiled the Metal backend (libggml-metal.dylib). This means the model will run beautifully utilizing your Mac's integrated GPU and Unified Memory.
It even compiled the new built-in SvelteKit web front-end (llama-ui) for you.
## 8.1. Attempting to get GGUF file and run
### 8.1.1. -This did not work there a few more tries but none of these worked, maybe because 8G file
Now you can immediately spin up the model. Since llama.cpp has a built-in automated downloader, you don't need a separate tool anymore. Run this command from your current /src directory to download the Gemma 4 12B Q5_K_M model and start the server:
```bash
./build/bin/llama-server -hf ggml-org/gemma-4-12B-it-GGUF:gemma-4-12b-it-Q5_K_M.gguf --port 8080 -c 8192 --n-gpu-layers 99
```
## 8.2. What happens next?
The Download Begins: llama-server will start fetching the ~9.5 GB file directly from Hugging Face and show you a download progress bar right in the terminal window.
Metal Acceleration Boots: Once downloaded, it will initialize the context (-c 8192 sets an 8k token window) and offload all 99 layers (--n-gpu-layers 99) entirely onto your Mac's graphics pipeline.
The Web UI is Live: Open up your browser and head to http://localhost:8080. You'll be greeted by the natively built chat interface where you can interact with Gemma 4 locally.
## 8.3. Now install gguf file
**Ths worked**
### 8.3.1. Step 1: Download the File via curl
Stay right where you are in the src directory, and run this command. It downloads the file straight from Hugging Face's servers and saves it directly into your models directory:
```bash
curl -L -o models/gemma-4-12B-it-Q5_K_M.gguf "https://huggingface.co/bartowski/gemma-4-12B-it-GGUF/resolve/main/gemma-4-12B-it-Q5_K_M.gguf"
```
### 8.3.2. Step 2: Launch the Server Region Locally
Once the download finishes, use the local model flag (-m) instead of the buggy Hugging Face flag (-hf). This tells llama-server to read the file straight from your hard drive:
```Bash
./build/bin/llama-server -m models/gemma-4-12B-it-Q5_K_M.gguf --port 8080 -c 8192 --n-gpu-layers 99
```
This will bypass the network lookup code entirely, instantly load the model into your Mac's unified memory via Metal, and open up your port listener at http://localhost:8080.

BIN
doc/Notes.pdf
View file

Binary file not shown.