Notes
Introduction
I upgraded the Mac mini I use at home to a high-spec model. As a result I can now run local LLMs, so here are the setup steps and my impressions.
Environment
- Mac mini 2024
- Chip: Apple M4 Pro
- CPU: 14 cores
- GPU: 16 cores
- Memory: 64 GB
- SSD: 1 TB
- macOS: Squoia 15.1
What will I use it for?
Steps to run a local LLM
Install ollama
I manage my environment with Nix + Home Manager, so I import the following setting into my usual config.
If you are not using Nix + Home Manager, download and install the official installer.
If installed correctly, the ollama command is available in the terminal.
Download an LLM
First, download the LLM you want to use. You can search models on Ollama's website.
I will use Qwen2.5-Coder, which was introduced on Gigazine a few days before I wrote this article.
Run the following command to download the model.
Adjust the 32b after the colon based on your environment.
As I mention later, 32b consumed about 20 GB of memory on my machine.
Start the server
To run a local LLM, start a server with ollama.
You only need to execute this command.
If it succeeds, you will see logs like the following.
Run the local LLM
Now try running the downloaded LLM. Run the following command. The model name should match what you downloaded.
If it succeeds, you will see a prompt.
You can have a simple conversation with the LLM at this prompt.
Since this model is specialized for source code, let's ask it to write a simple program.
Maybe that was too easy. It even explained how to run it.
Next, let's ask it to write a program that outputs the first 100 Fibonacci numbers.
It seems to have written a correct Fibonacci program. Maybe that was too easy as well.
I could continue experimenting, but chatting in a terminal isn’t a great experience. In the next article, I will try using a local LLM through a different UI.
Conclusion
Installing a local LLM has become surprisingly easy. Thanks to Ollama, even if new models appear, I can probably run them with a single command.
If you want to try Japanese LLMs, it seems good to explore models that catch your interest in Japanese LLM list | LLM-jp.
I tried Llama-3-ELYZA-JP-8B-GGUF, but I felt Japanese LLMs sometimes fail to hold a conversation. This is a field that will keep evolving, so I plan to explore new models regularly.
