DGX Spark: Running Production AI on a $3,000 Desktop

Offline inference, no API costs, no rate limits, no censorship filters. I tested whether a $3,000 desktop could replace recurring cloud LLM spend for real agent workloads.

The hardware

The DGX Spark is a desktop built around ARM silicon and 128GB unified memory, plus a Blackwell GPU. You do not need top-tier CPU or SSD performance for inference; what matters is memory bandwidth, and that is exactly where this machine is strong.

The stack

I chose this model size deliberately. At around 21GB in NVFP4, the 27B Qwen fits cleanly into 128GB. A 32B model pushes close to 30GB, which leaves too little headroom for KV cache. This machine benefits most from models that fill the memory without crowding it. The 27B option is close to ideal here.

Benchmarks

I tested with 128K context, not toy prompts. Across short question-and-answer tasks, throughput ranged from 34.9 to 57.0 tokens per second, with average time-to-first-token around 0.82 seconds. Speculative decoding acceptance sat at 64.8 percent, which is solid given the speed gain.

The speed boost came from DFlash, a small draft model needing only 3.2GB. That footprint means the draft model does not fight the main model for memory. Running both together is feasible, which is why the throughput gain is real rather than theoretical.

Hermes vLLM inference benchmarks on NVIDIA DGX
Hermes vLLM Inference performance on the DGX Spark

Sharing memory between services

The LLM and diffusion image workloads cannot share 128GB at once. I wrote swap scripts that stop one service and start the other in a single command. Hermes detects which capability the current task needs, then hands off to the right backend. The operator does not need to manage containers manually.

On the image side, ComfyUI plus Chroma1-HD generates 1024x1024 pictures in about seven seconds. That is fast enough to queue work without making users wait.

The cost case

Cloud LLM credits add up quickly at scale. Three months of active inference use on a major platform covers this machine. After that, the marginal cost is the power draw from the wall.

Beyond price, there is operational benefit. There are no rate limits. No request queues. No data leaving your network. No censorship models sitting between you and the model. If privacy, predictability, or uptime matters more than on-demand scale, this setup wins.

When this makes sense

I do not see this as a replacement for cloud AI. It is a specialist tool for workloads where control, cost and uptime are more important than elastic scale. Teams that need an always-on inference node for internal tools, automation pipelines or content production workflows are the main use case.

Sharing the setup

If you want the benchmark methodology or the swap scripts, ask. I am happy to share the config that produced these numbers.

Connect with me on LinkedIn.