How to Run Llama 3 on an RTX 4090: What Fits in 24GB and How Fast It Really Is

Key Takeaways

>

• Llama 3 on RTX 4090 runs the 8B models at 118–130 tokens per second at Q4_K_M, roughly 15 times faster than a person reads.
• Llama 3.1 8B occupies about 4.7GB of VRAM at Q4_K_M, 8.5–9GB at Q8_0 and 14–16GB at FP16, so 24GB leaves real room for long context.
• Llama 3.3 70B needs 39–42GB at Q4_K_M and therefore does not fit in 24GB; the two workarounds are CPU offloading (8–12 t/s) or 2-bit IQ2_XS fully in VRAM (up to 60 t/s, with measurable quality loss).
• One command gets you running: ollama run llama3.1:8b pulls the Q4_K_M build and places it on the GPU automatically.
• Local inference costs roughly $0.22–$0.25 per million output tokens against $2.50–$10.00 for frontier API models, which is why a 24GB card still pays back under heavy use.

Running llama 3 on rtx 4090 hardware produces two very different outcomes depending on parameter count: the 8B weights generate 118–130 tokens per second at 4-bit precision, while 70B weights at the same precision demand 39–42GB and consequently will not load into 24GB VRAM at all. That split is the whole story of this card. Capacity decides which models load and the 4090’s 1,008 GB/s memory bandwidth decides how fast they generate — raw compute barely registers in single-user token generation, because the bottleneck is reading weights out of memory, not multiplying them.

This guide is for anyone about to buy, borrow or repurpose a 4090 for local inference or fine-tuning. Every figure here is current as of October 2026 and covers the Llama 3, 3.1, 3.2 and 3.3 weight releases rather than Llama 4 and what comes next. Where a number depends on test conditions, those conditions are stated alongside it.

About the Author and How We Tested

About the author. Written by The Tech ABC editorial team, which covers GPUs, self-built PCs and local AI tooling as hardware features rather than as an industry beat. Our local-LLM coverage is built on a standing consumer GPU test bench and on hands-on use of the same runners readers install — Ollama, llama.cpp and LM Studio — rather than on vendor marketing decks.

iPhone 17 Camera AI Features: Smarter Image Processing

Transparency and methodology. The reference configuration for every figure below is a single RTX 4090 24GB on the current NVIDIA Game Ready/Studio driver branch as of October 2026, with current releases of llama.cpp and Ollama, DDR5 host memory, batch size 1 and a 4,096-token context unless a row states otherwise. The 1,008 GB/s bandwidth figure is NVIDIA’s published specification. Throughput numbers for the 8B Q4_K_M, 8B FP16 and 70B configurations are aggregated from llama.cpp benchmark threads, Ollama’s model library defaults and r/LocalLLaMA community aggregations; the Q8_0 and Llama 3.2 3B rows are community estimates rather than first-party bench results and are labelled as such. All throughput is reported as a range because batch size, context length and runner version each move the result.

What Actually Fits in 24GB of VRAM

Llama 3 on rtx 4090 fits any model up to 8B at full FP16 precision and 13B-class weights comfortably at 4-bit, but 70B weights at Q4_K_M require 39–42GB and therefore will not load. The table below is the lookup; the mechanism follows it.

Model	Quantization	VRAM needed	Fits in 24GB?
Llama 3.2 1B	Q4_K_M	~1GB	Yes, huge headroom
Llama 3.2 3B	Q4_K_M	~2GB	Yes, huge headroom
Llama 3.1 8B	Q4_K_M	~4.7GB	Yes
Llama 3.1 8B	Q8_0	8.5–9GB	Yes
Llama 3.1 8B	FP16	14–16GB	Yes, limited context headroom
Llama 3.3 70B	Q4_K_M	39–42GB	No, offload required
Llama 3.3 70B	IQ2_XS	~21.1GB	Yes, with quality loss

Llama 3.1 8B: 4.7GB at Q4_K_M, 14–16GB at FP16

Three precision tiers matter on this card. Q4_K_M lands at roughly 4.6–4.7GB, leaving about 19GB for the KV cache and other processes. Q8_0 takes 8.5–9GB and is effectively indistinguishable from the reference weights for chat. FP16 consumes 14–16GB, which leaves thin context headroom. Ollama pulls Q4_K_M by default because it cuts the weight footprint roughly 3.5x against FP16 at a small measured accuracy cost, which means the card spends its bandwidth generating tokens instead of shuffling weights.

Llama 3.2 1B and 3B: 1–2GB and full headroom

The lightweight Llama 3.2 weights occupy 1–2GB at 4-bit, leaving more than 22GB free. That headroom is the point. Because the model barely touches capacity, these weights are the practical choice for agent loops that hold several contexts at once, for draft models in speculative decoding, and for multimodal vision work running alongside a second resident model.

Llama 3.3 70B: 39–42GB at Q4_K_M, which means it will not fit

Be blunt about this one. An 80-layer 70B model at Q4_K_M needs 39–42GB against a hard 24GB ceiling, so the loader cannot place it. Roughly 45 of those 80 layers fit on the card; the remaining 35 land in system RAM. There are exactly two routes around the ceiling, and both carry a measured cost — the dedicated 70B section below prices each one.

KV cache math: why context length changes the answer

Weights are only the fixed cost. The KV cache is the variable one, and it grows with every token held in context, consuming VRAM linearly as the conversation or document lengthens. Consequently a model that loads cleanly at 4K context throws a CUDA out-of-memory error at 32K and refuses outright at 128K, even though the weights never changed size. KV cache quantization — storing the cache at 8-bit or 4-bit instead of 16-bit — buys back a large share of that headroom at a small accuracy cost, which is the single most useful lever for long-context work on a 24GB card.

Quantization Explained Without the Jargon

Quantization is the process of storing a model’s weights at lower numerical precision — 4 bits instead of 16 — which shrinks both the file on disk and the VRAM it occupies, at the cost of some accuracy. Nothing about the model’s architecture changes. Only the resolution of each individual number does.

Quant level	8B file size	Quality impact	Verdict
FP16	14–16GB	Reference baseline	Fine-tune export or research only
Q8_0	8.5–9GB	Near-lossless	Best quality per GB on a 4090
Q6_K	~6.6GB	Negligible	Good middle ground
Q4_K_M	~4.7GB	Small, rarely noticed in chat	Default, recommended
Q3_K_M	~4GB	Visible on reasoning tasks	Only to fit more context
IQ2_XS	~2.5GB (8B) / ~21.1GB (70B)	Noticeable degradation	70B-only workaround

The saving is also a speed win, and the reason is mechanical. Fewer bytes per weight means fewer bytes crossing the memory bus for every token generated. Because generation on the RTX 4090 is bandwidth-bound rather than compute-bound, a smaller model is a faster model — the 1,008 GB/s figure is a fixed budget, and quantization simply reduces how much of it each token spends.

Where it breaks is worth naming precisely. Q8_0 and Q6_K are functionally equivalent to FP16 for chat and summarization. Q4_K_M is the accepted community default because measured degradation stays small across standard evaluations. IQ2_XS is a different proposition entirely: a 2-bit model shows noticeable losses in instruction-following, long-chain reasoning and code generation, which is why the 21.1GB file size for a 70B model should never be quoted without that caveat attached. A 2022 arXiv paper on k-bit inference scaling laws found that accuracy degrades sharply below roughly 4 bits per parameter, which is exactly the cliff edge the IQ2 quants are sitting on.

How Fast It Really Is: Measured Throughput

Llama 3 on rtx 4090 generates 118–130 tokens per second with the 8B model at Q4_K_M, 50–72 tokens per second at FP16, and 8–12 tokens per second when a 70B model is split between GPU and system RAM. Every figure below carries its conditions.

Model and quant	Tokens/sec	Where weights live	Test conditions
Llama 3.1 8B Q4_K_M	118–130	All VRAM	4K context, batch 1, llama.cpp/Ollama
Llama 3.1 8B Q8_0	75–95 (community estimate)	All VRAM	4K context, batch 1
Llama 3.1 8B FP16	50–72	All VRAM	4K context, batch 1
Llama 3.2 3B Q4_K_M	180–220 (community estimate)	All VRAM	4K context, batch 1
Llama 3.3 70B Q4_K_M	8–12	~45 of 80 layers on GPU	DDR5 host RAM, 4K context
Llama 3.3 70B IQ2_XS	Up to 60	All VRAM	4K context, quality cost noted

8B at Q4_K_M: 118–130 tokens per second

Configuration first: Llama 3.1 8B at Q4_K_M, 4,096-token context, batch size 1, weights entirely resident in VRAM. At 118–130 t/s the model produces text roughly 15 times faster than a person reads it comfortably. The practical consequence is that latency stops being the constraint on this card and prompt design becomes the constraint — you will spend longer writing the instruction than the model spends answering it.

8B at FP16: 50–72 tokens per second

The drop is a direct bandwidth consequence, not a software regression. At FP16 the card must read 14–16GB of weights for every single token instead of 4.7GB, therefore throughput falls roughly in proportion to the footprint. Because Q8_0 is already near-lossless, the marginal quality gain from FP16 does not justify halving the speed for chat, drafting or summarization work.

70B hybrid offload: 8–12 tokens per second

With roughly 45 of 80 layers on the card and 35 in system memory, the CPU-resident layers are read at DDR5 bandwidth measured in tens of GB/s rather than 1,008 GB/s. Every generated token must traverse all 80 layers, which means the slowest segment of the pipeline sets the pace for the whole model. The result — 8–12 t/s — stays usable for interactive chat where you read as it writes, and is unusable for bulk generation or batch jobs.

70B at IQ2_XS fully in VRAM: up to 60 tokens per second

Compress the same 70B model to ~21.1GB and nothing spills to host memory, so the 1,008 GB/s bus serves every layer. Speed consequently jumps five to six times against hybrid offload, reaching up to 60 t/s. Read that number next to the quality warning above: you are buying throughput with instruction-following and reasoning accuracy, and on complex tasks that trade frequently goes the wrong way.

Step-by-Step: Getting Llama 3 Running Tonight

The fastest way to run llama 3 on rtx 4090 hardware is Ollama: install it, then run one command — ollama run llama3.1:8b — which pulls the 4.7GB Q4_K_M build and loads it onto the GPU automatically. The full sequence from a bare install takes well under 15 minutes on a reasonable connection.

From bare install to first token on an RTX 4090

This Is Why Photoshop Isn’t Your Only Option Anymore

1. Update to the current NVIDIA driver branch and reboot, because older branches mis-report available VRAM to llama.cpp.
2. Confirm the card is visible and note free VRAM with nvidia-smi.
3. Install Ollama for Windows, macOS or Linux — a single installer, with no separate CUDA Toolkit needed.
4. Pull and launch the model: ollama run llama3.1:8b.
5. Verify full GPU placement with ollama ps — the processor column should read 100% GPU.
6. Raise context length only after checking headroom, because the KV cache is what triggers out-of-memory errors.
7. Benchmark your own box with --verbose and record the tokens-per-second figure as your baseline.

Driver and CUDA prerequisites in late 2026

Stay on a current Game Ready or Studio driver branch; both expose the CUDA runtime that inference engines require. Ollama and LM Studio ship their own bundled CUDA libraries, therefore a manual CUDA Toolkit installation is unnecessary for inference alone. Install the toolkit only when you intend to fine-tune or to build llama.cpp from source with custom flags.

The one-command Ollama route

ollama run llama3.1:8b

First run prints pull progress for the ~4.7GB Q4_K_M blob, then drops you at an interactive prompt. Confirm placement immediately:

ollama ps

A 100% GPU reading in the processor column means nothing fell back to CPU. Set context length through OLLAMA_NUM_CTX or a Modelfile rather than guessing. For the lightweight option, ollama run llama3.2:3b loads in seconds and leaves over 20GB free.

llama.cpp for manual layer control with –n-gpu-layers

llama-server -m llama-3.1-8b-instruct-Q4_K_M.gguf -ngl 99 -c 8192 -fa

--n-gpu-layers (-ngl) is the single most important flag on a 24GB card because it decides how much of the model stays on the 1,008 GB/s bus instead of the host memory bus. -ngl 99 means “all layers” and is the correct setting for every model that fits in VRAM. -c sets context and -fa enables FlashAttention, which reduces attention memory growth as the conversation lengthens.

LM Studio for a GUI-first workflow

The honest LM Studio versus Ollama split: LM Studio wins for browsing available quants, reading visual VRAM estimates before you download and chatting without a terminal. Ollama wins for scripting, API serving and model management. Both sit on llama.cpp underneath, therefore throughput differences between them are small and come down to default flags.

Squeezing a 70B Model Onto One Card

There are exactly two ways to run a 70B Llama 3 model on a 24GB RTX 4090 — split the layers between GPU and system RAM, or compress the weights to 2 bits so they fit entirely in VRAM.

Dimension	Option A: hybrid offload (Q4_K_M)	Option B: IQ2_XS in VRAM
Fits entirely in 24GB	No, ~45 of 80 layers	Yes, ~21.1GB
Throughput	8–12 t/s	Up to 60 t/s
Output quality	Full Q4_K_M	Noticeably degraded
Also needs	64GB+ fast DDR5	Nothing extra
Best for	Reasoning, long answers	Speed-first experiments

Option A, hybrid offload. Set --n-gpu-layers to roughly 45 and let the remaining layers sit in system RAM. You keep full Q4_K_M output quality and accept 8–12 t/s. Host memory speed and channel count matter enormously here, because the CPU-resident layers are read at DDR5 bandwidth; a slow or single-channel configuration drags the entire pipeline below conversational usability, which is why this route demands dual-channel DDR5 and 64GB or more.

Option B, IQ2_XS at roughly 21.1GB. Everything stays in VRAM, nothing spills, and generation reaches up to 60 t/s. What you lose is specific rather than vague: instruction-following gets looser, multi-step reasoning breaks down more often, and generated code requires more correction.

Everything You Need to Know About the Nintendo Switch 2

Verdict. For reasoning-heavy work take Option A and accept 10 t/s, because a correct answer arriving at reading speed beats a fast wrong one. For bulk drafting where an 8B model would nearly do the job, skip 70B entirely — Llama 3.1 8B at Q8_0, running at an estimated 75–95 t/s with full quality, is the better trade on this card.

Fine-Tuning Llama 3.1 8B on a Single 4090

A single RTX 4090 completes a LoRA fine-tune of Llama 3.1 8B at roughly 18,000 training tokens per second, which puts a five-epoch run over 10,000 samples inside two hours. By late 2026 that run has become a standard acceptance benchmark for new dedicated GPU servers, which indicates how routine the workload has become.

It works because LoRA trains a small low-rank adapter instead of all eight billion weights, collapsing the optimizer-state memory that normally dominates training footprint. The method was introduced in a 2021 arXiv paper on Low-Rank Adaptation and extended to 4-bit base weights by the 2023 QLoRA paper. Layer on FlashAttention 2 and the 4090’s FP8-capable tensor cores, and the attention pass stays inside VRAM at useful sequence lengths. Reported throughput assumes a 4-bit quantized base model with modest batch size and sequence length — raise either and the figure falls.

Hard limits of fine-tuning on one 24GB card

• Full-parameter fine-tuning of 8B at FP16 exceeds 24GB — LoRA or QLoRA only.
• 70B fine-tuning is off the table at any adapter rank on a single card.
• No NVLink means multi-4090 training scales poorly, so rented enterprise GPUs win above 8B.
• Long sequence lengths consume VRAM faster than batch size does, therefore cut sequence length before cutting batch.

RTX 4090 vs RTX 5090 vs Everything Else

The RTX 4090’s 24GB and 1,008 GB/s still make it the strongest price-to-performance GPU for local LLM work in late 2026; the RTX 5090’s 32GB only changes the calculus if you must run 70B-class weights.

GPU	VRAM	Memory bandwidth	Largest Llama 3 it runs well
RTX 4090	24GB	1,008 GB/s	8B at FP16, or 70B at IQ2_XS
RTX 5090	32GB	~1,792 GB/s	70B at Q3_K_M with light offload
RTX 5080	16GB	~960 GB/s	8B at Q8_0, short context
RTX 4060 Ti 16GB	16GB	288 GB/s	8B at Q4, slow prefill
RTX 6000 Ada	48GB	960 GB/s	70B at Q4_K_M fully in VRAM

One principle governs this table. VRAM capacity sets which models load at all, memory bandwidth sets how fast they generate, and compute throughput barely moves single-user token generation because the pipeline is waiting on memory reads. Consequently these cards rank by gigabytes and gigabytes-per-second, not by gaming frame rates — which is why a 288 GB/s 4060 Ti with 16GB loads an 8B model but generates at a fraction of the 4090’s rate, and why the 48GB RTX 6000 Ada beats the faster-bandwidth 5090 for 70B work.

Gaming News and Reviews – The Tech ABC

Verdict. For 8B-and-below workflows, keep the 4090 or buy a used one — the 24GB ceiling is not the limiting factor. For 70B-at-Q4 ambitions, note that even the 5090’s 32GB still requires offload, so go dual-GPU or rent enterprise time instead; prospective 5090 buyers should also read our coverage of RTX 5090 power delivery issues before committing. For 16GB cards, 8B at Q4 runs fine but context headroom disappears quickly — see our RTX 5080 review and the RTX 5080 power consumption breakdown for the full picture.

The Cost Case for Running It Yourself

Generating tokens locally on an RTX 4090 costs roughly $0.22–$0.25 per million output tokens once hardware amortization and electricity are counted, against $2.50–$10.00 per million for frontier API models.

Option	Cost per 1M output tokens	What drives the cost
Llama 3.1 8B on RTX 4090	$0.22–$0.25	Electricity plus hardware amortization
Hosted open-weight 8B API	$0.10–$0.60	Provider margin, no hardware risk
Frontier closed model API	$2.50–$10.00	Model size and vendor pricing
Rented cloud 4090/L40S hour	Varies by provider	Hourly rate against your utilization

That local figure rests on three assumptions worth stating plainly: the card’s purchase price spread across a multi-year service life, US average residential electricity pricing, and the 4090’s sustained power draw under continuous inference rather than its idle or peak figures. The MIT Energy Initiative has documented how electricity pricing variation across regions moves data-centre and compute economics, and the same logic applies to a desk-side GPU — a reader on expensive power lands at the top of that range, a reader on cheap power below it.

ChatGPT Overload! When Does Daily Use Turn Into Addiction?

The break-even reasoning follows directly. At these rates a 4090 repays its purchase price against frontier API pricing somewhere in the low hundreds of millions of output tokens, which means local hardware wins decisively for heavy sustained usage and loses to a hosted open-weight API for occasional use. If you generate a few thousand tokens a week, rent. If you run agent loops or batch pipelines daily, buy — and read our broader tech trends and innovations coverage for how that calculus is shifting.

Troubleshooting the Three Failures You Will Hit

Quick reference: symptom to fix

• CUDA out of memory → reduce context length or quantize the KV cache
• 8B running at single-digit t/s → raise --n-gpu-layers and free VRAM held by other apps
• Slowdown over a long chat → enable FlashAttention (-fa) and trim conversation history
• Model loads but answers poorly → you pulled a 2-bit or 3-bit quant; move to Q4_K_M or higher

Failure 1 — CUDA out of memory on load or mid-conversation. The weights fit; the KV cache grew past the remaining headroom. That is precisely why the crash arrives after a long prompt rather than at startup, and why the same configuration worked yesterday on shorter inputs. Lower the context setting, enable KV cache quantization, or drop one quant level to free a gigabyte or two.

Failure 2 — generation crawls at single-digit tokens per second with an 8B model. Layers silently fell back to CPU, so the model is running at host memory bandwidth instead of 1,008 GB/s. Check ollama ps or the llama.cpp load log for the offloaded-layer count, then raise --n-gpu-layers. The underlying cause is usually another process holding VRAM — a browser with hardware acceleration, a running game, or a display-heavy creative application.

Failure 3 — speed collapses as the conversation lengthens. Attention cost grows with context, therefore every additional token of history slows the generation of the next one. Enable FlashAttention with -fa, trim the history you send back, or restart the session once a conversation runs long.

Frequently Asked Questions

Can an RTX 4090 run Llama 3 70B?

Not natively. A 70B model at Q4_K_M needs 39–42GB against the card’s 24GB ceiling. Two workarounds exist: split roughly 45 of 80 layers to system RAM for 8–12 tokens per second at full Q4_K_M quality, or compress to IQ2_XS at about 21.1GB for up to 60 tokens per second with noticeably degraded reasoning and instruction-following.

How many tokens per second does Llama 3.1 8B hit on a 4090?

118–130 tokens per second at Q4_K_M with weights fully resident in VRAM, 4K context and batch size 1. At FP16 the same model drops to 50–72 tokens per second, because the card must read 14–16GB of weights per token rather than 4.7GB and generation is bandwidth-bound on this hardware.

Is 24GB of VRAM still enough for local AI in 2026?

Yes, for 8B-class models and fine-tuning. 24GB loads Llama 3.1 8B at any precision including FP16, leaves headroom for long context, and supports LoRA fine-tuning at roughly 18,000 training tokens per second. It is not enough for 70B weights at usable quantization, which remains the main capability gap.

Which quantization should I use on a 4090?

Q4_K_M for general use, Q8_0 when quality matters most. Q4_K_M occupies about 4.7GB with small measured degradation and is what Ollama pulls by default. Q8_0 at 8.5–9GB is near-lossless and still fits easily. Avoid 2-bit and 3-bit quants on 8B models — the 4090 has no reason to make that trade.

Do I need to install the CUDA Toolkit to run Llama 3 locally?

No, not for inference. Ollama and LM Studio bundle their own CUDA libraries, so a current NVIDIA driver branch is the only prerequisite. A manual CUDA Toolkit installation becomes necessary when you fine-tune with PyTorch-based trainers or compile llama.cpp from source with custom build flags.

Limitations and What This Guide Does Not Cover

Several caveats apply to the numbers above. Throughput for llama 3 on rtx 4090 hardware is reported as ranges rather than single values because batch size, context length, runner version, driver branch and host CPU each move the result — a figure from a llama.cpp build six months old will not match a current one, and prefill speed on long prompts behaves differently from the decode speed quoted here. The Llama 3.1 8B Q8_0 and Llama 3.2 3B rows are community-aggregated estimates rather than first-party bench results, and should be treated as indicative. VRAM footprints are also approximate, because GGUF quantization variants from different publishers differ slightly in size for the same nominal quant level.

Alternative approaches exist that this guide does not price out. vLLM and TensorRT-LLM deliver substantially better throughput than llama.cpp under concurrent multi-user load, which matters if you are serving an application rather than chatting alone; they are more demanding to configure. Rented cloud GPUs remove capacity limits entirely and make sense for intermittent 70B work. Dual-GPU consumer builds are possible but constrained by the 4090’s lack of NVLink.

Finally, cost figures depend on local electricity prices, your purchase price and your actual utilization, and the per-token economics shift whenever API vendors reprice. Anyone making a purchase decision above a few thousand dollars — or deploying local inference in a business context with data-handling obligations — should validate the numbers against their own usage and seek appropriate professional advice on the compliance side.

Conclusion

Llama 3 on rtx 4090 hardware comes down to one number: 24GB. That ceiling admits every 8B and smaller model at any precision you would reasonably choose, and the card’s 1,008 GB/s bandwidth turns those weights into 118–130 tokens per second at Q4_K_M — fast enough that generation speed stops being the thing you think about. The same ceiling locks out 70B weights at Q4_K_M, and no amount of compute makes 39–42GB fit into 24.

So the decision is clean. If your work lives at 8B or below, the 4090 is a finished solution rather than a compromise: one Ollama command, full GPU placement, sub-two-hour LoRA fine-tunes and roughly $0.22–$0.25 per million output tokens. If you need 70B reasoning, accept hybrid offload at 8–12 tokens per second with fast DDR5 behind it, or move to a 48GB card or rented enterprise time — because 2-bit compression buys speed with exactly the reasoning quality you wanted the larger model for.

Run the benchmark on your own box, record the number, and use it as the baseline against which every quant and context change gets judged. Read more of our GPU and local-AI coverage to see how that baseline holds as the hardware moves.

References

• NVIDIA — GeForce RTX 4090 official specifications, source of the 24GB GDDR6X capacity and 1,008 GB/s memory bandwidth figures.
• Dettmers and Zettlemoyer — The Case for 4-bit Precision: k-bit Inference Scaling Laws, arXiv preprint, 2022. Accuracy-versus-bit-width relationship underpinning the quantization guidance.
• Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models, arXiv preprint, 2021. Method behind single-GPU fine-tuning.
• Dettmers et al. — QLoRA: Efficient Finetuning of Quantized LLMs, arXiv preprint, 2023. 4-bit base-model fine-tuning method.
• Dao — FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, arXiv preprint, 2023. Attention-memory optimization referenced in the fine-tuning and troubleshooting sections.
• ggerganov — llama.cpp project repository, reference implementation for GGUF quantization, --n-gpu-layers offloading and community benchmark threads.
• Ollama — Llama 3.1 model library page, source of the default Q4_K_M build and its ~4.7GB footprint.
• r/LocalLLaMA — community benchmark aggregations, source of aggregated consumer-GPU throughput ranges.
• MIT Energy Initiative — research on electricity pricing and compute energy costs, framing for the local cost-per-token assumptions.