Under the prompt.
Down to the metal.
The prompt box is the visible surface. The real system is model weights, quantization, memory bandwidth, accelerators, PCIe, networking, runtimes, identity, and the software that decides what the model is allowed to do.
$ inspect ai-stack application agent / tools runtime llama.cpp / vLLM model 70B Q4 memory VRAM + KV cache compute GPU / CPU fabric PCIe / Ethernet physical power / cooling status abstraction removed
Parameter count is only the beginning.
A model's size tells you roughly how much state exists. It does not tell you whether it will fit, how fast it will run, or how much context you can afford.
Raw weights
A 70B model at FP16 is roughly 140 GB of weights before runtime overhead.
Quantization
At an idealized four bits per parameter, the same 70B model falls near 35 GB before overhead.
Context
Long prompts and conversations consume additional memory through the KV cache.
VRAM is capacity.
Bandwidth is velocity.
Fitting the model is only step one. Autoregressive inference repeatedly moves model state through memory, so bandwidth can matter as much as raw compute.
70B FP16 70B × 2 bytes ≈ 140 GB 70B Q4 70B × 0.5 bytes ≈ 35 GB plus runtime overhead: metadata KV cache buffers batch state context
Old GPUs can still be useful.
Not because they are secretly fast. Because large amounts of cheap VRAM can unlock models that simply do not fit on smaller consumer cards.
Cheap capacity
Retired datacenter GPUs can offer unusually low acquisition cost per gigabyte of VRAM.
Power penalty
Older silicon often trades purchase price for heat, noise, and much higher energy use.
Platform matters
Slots, lane width, cooling, power delivery, and chassis airflow can become more important than the GPU itself.
100GbE only matters
when the workload earns it.
Single-node text inference moves very little network data. High-speed fabrics matter when the network becomes part of the compute path: model distribution, shared storage, multi-node inference, or distributed training.
single GPU inference 100GbE value: LOW shared model storage 100GbE value: MAYBE distributed training 100GbE value: HIGH rule: name the bottleneck before buying bandwidth.
The model should not be the trust boundary.
Tool-enabled AI becomes infrastructure. That means identity, validation, least privilege, segmentation, and auditability matter more than clever prompts.
Authenticate the requester
Know which person, service, or process is asking the AI system to act.
Validate tool calls outside the LLM
Natural language output is not authorization.
Log intent, action, and result
AI that touches real systems should be observable like any other privileged service.
“The abstraction is useful. The leaks are where the engineering starts.”
MagicTheLLM follows the entire path from token to transistor because cost, performance, reliability, and security all live below the prompt box.