~/Rev_
← projects

#AI Lab

Local LLM inference on two GPUs: a GTX 1070 passed through on a PowerEdge, and an RTX 3070 workstation stack.

2026-04-29 → 2026-10-12

Two nodes

  • **ai-core-01 (T320):** the GTX 1070 is bound to vfio-pci on the hypervisor and passed through to a dedicated VM, for always-on local models in the lab.
  • Workstation (Zephyrus, RTX 3070, 40 GB): Docker with its data root on an external SSD, NVIDIA runtime in containers, and Ollama + Open WebUI via Compose.

Measured

DeepSeek-R1 14B ran at ~10.4 tokens/s, split roughly 71% GPU / 29% CPU because it exceeds 8 GB of VRAM. Qwen3 8B fits fully on the GPU. Lesson: on 8 GB cards, 4B–9B quantized models are the sweet spot.

Next

A LiteLLM gateway in front of both nodes, RAG over the CyberDen knowledge base, MCP servers, then AI-security experiments (prompt injection, tool abuse) against my own agents.

Sources: CyberDen Flip Breakdown (27 Apr 26), Local AI LLM Setup (27 Apr 26), Build Local AI Lab (12 Oct 26)

Time logged50m
Entries7
Active days1
Lanes fed3
Writeups2

Skills it built

Linux & systems Computer Science
5 · 50m
AI / ML engineering Artificial Intelligence / Machine Learning
1 · 0m
Hardware & circuits Beyond Computer Science
1 · 0m

Writeups

2026-10-12A GPU Local-AI Lab on an External NTFS SSD (Docker on a Sparse ext4 Image)Containers and models on an external drive without reformatting it: Docker's data root inside a sparse ext4 image, GPU in containers, measured tokens/sec.2026-05-03GTX 1070 Passthrough on a PowerEdge T320: A GPU Node for Local AIChoosing a card a tower server can power, validating it on the host, then binding it to VFIO and handing it to a dedicated inference VM.

Timeline

2026-10-12Linux & systems
Installed AI models · Homelab · 10m
notes
5. Installed AI models Model Disk size Status Qwen3 Abliterated 8B ~5 GB Installed and working DeepSeek-R1 Abliterated 14B ~9 GB Installed and working - Tested DeepSeek-R1 14B at approximately 10.37 tokens/second. - Observed approximately 71% GPU / 29% CPU offloading because the model exceeds available VRAM. - Identified 4B–9B quantized models as the preferred range for faster inference. - Discussed testing Qwen3/Qwen3.5 4B and 8B variants. No additional downloads confirmed.
2026-10-12Linux & systems
Ollama and Open WebUI · Homelab · 10m
notes
4. Ollama and Open WebUI - Deployed Ollama and Open WebUI through Docker Compose. - Configured Ollama model storage on the external SSD. - Configured persistent Open WebUI data storage inside the ext4 image. - Made Ollama available at 127.0.0.1:11434. - Made Open WebUI available at http://localhost:3000. - Successfully used local models with NVIDIA acceleration.
2026-10-12Linux & systems
Docker infrastructure · Homelab · 10m
notes
3. Docker infrastructure - Created a 100 GB sparse ext4 disk image at: /mnt/rev-ssd/ai-lab/docker-storage.ext4 - Mounted that image at /mnt/ai-docker. - Configured Docker's data root to /mnt/ai-docker/docker. - Configured containerd storage on the same ext4 filesystem. - Enabled NVIDIA GPU access inside Docker containers. - Successfully recovered the existing Docker storage after reboot without formatting or recreating anything. - Kept Docker and containerd disabled at boot to avoid unnecessary background resource usage.
2026-10-12Linux & systems
External SSD and storage architecture · Homelab · 10m
notes
2. External SSD and storage architecture - Established /mnt/rev-ssd as the intended external SSD mount point. - Created the AI lab under /mnt/rev-ssd/ai-lab/. - Organized directories for models, agents, RAG, datasets, training, evaluations, logs, backups and configuration. - Preserved the SSD's existing NTFS filesystem and data. - Confirmed ntfs-3g works for mounting the SSD; the attempted ntfs3 mount failed. - Still pending: Make /mnt/rev-ssd mount reliably across reboots without manual intervention or conflicts with GNOME automount.
2026-10-12Linux & systems
Hardware and environment · Homelab · 10m
notes
1. Hardware and environment - Laptop: ASUS ROG Zephyrus G15. - OS: Fedora Linux, Fish shell. - GPU: NVIDIA RTX 3070 Laptop GPU, 8 GB VRAM. - CPU: AMD Ryzen 9 5900HS. - RAM: Approximately 40 GB. - External storage: WD_BLACK SN850X 1 TB SSD. - Decided to run local AI inference using the laptop's GPU, with model storage on the external SSD.
2026-05-03AI / ML engineering
GPU passed through to ai-core-01; first local models on the GPU#CyberDen
notes
Writeup: W08
2026-04-29Hardware & circuits
GTX 1070 installed in the T320 and bound to vfio-pci#CyberDen
notes
Writeup: W08
AI Lab