Jordan Anderson

4-bit Mixture-of-Experts fine-tuning on small GPUs—Experts4bit primitive, streaming loader, expert offload

pjordanandrsn/experts4bit-qlora · PyPI (v0.2.0) · 2026

• transformers v5 stores each MoE layer’s experts as one fused 3-D parameter; bitsandbytes’ 4-bit walker only replaces nn.Linear modules, so load_in_4bit silently skips the experts—the overwhelming majority of a MoE’s weights (bitsandbytes#1849). The contract says “4-bit”; the implementation quietly leaves the experts in full precision.
• Built Experts4bit, the primitive that quantizes exactly that fused stack—as of v0.2.0 the 4-bit face of ExpertsNbit, which stores the same stack at selectable precision (nf4/fp4/int8/fp8/bf16/fp16) with a test-pinned reconstruction-error ordering, so the precision knob is a measured trade, not a vibe.
• Paired it with a streaming loader that never materializes the bf16 model in CPU or GPU RAM (verified under a 3 GB container cap), per-expert LoRA, and an expert-offload path that streams the frozen 4-bit experts from pinned CPU memory one layer at a time—so fused MoEs whose experts exceed VRAM still train, and the resulting adapters serve over the exact NF4 base they trained against, with no re-quantization shifting the error surface.
• Upstreaming both halves: the primitive to bitsandbytes (#1965, open) and the offload path as a self-contained axolotl integration (#3797, open), with a matched A/B—shared seed and data order, per-step telemetry—posted in the proposal thread.

Measured impact

Single RTX A2000 12 GB in a NAS’s PCIe 3.0 x8 slot; methodology and per-step logs in the repo.

• Fits Qwen3-30B-A3B QLoRA peaks at 7.16 GB and Gemma-4-26B-A4B at 8.47 GB—both OOM on 12 GB without offload; on OLMoE the matched A/B cuts peak VRAM 6.00 → 2.60 GB (−57%).
• Trains held-out eval improves 1.4813 → 1.0290 QLoRA-training over frozen NF4 experts; with offload on, the loss curve tracks offload-off step-for-step under a shared seed.
• Serves the same path decodes the 26B-class fine-tune in 6.2 GB—adapters over the identical NF4 base, train-to-serve.

• Package pypi.org/project/experts4bit-qlora —v0.2.0
• PR #1965 bitsandbytes-foundation/bitsandbytes #1965 —open
• PR #3797 axolotl-ai-cloud/axolotl #3797 —open

OpenVINO Intel GPU plugin—__local-pointer kernel compilation fixes + shared-header API extension

openvinotoolkit/openvino · 2026

• Worked through a systemic OpenCL compile failure across the GPU plugin’s LoRA, MoE, and fully-connected kernels on Intel Compute Runtime 23.x+—the __local-pointer overloads of intel_sub_group_block_read* had been dropped from the base cl_intel_subgroups extension and now require cl_intel_subgroup_local_block_io, which most current Intel discrete GPUs (Arc, Battlemage, Lunar Lake) don’t advertise. Anyone running LoRA or MoE inference on that hardware was hitting compile errors.
• Verified the substitution against the kernels’ sub-group attributes, SLM barrier ordering, and per-lane distribution semantics; tested end-to-end via a binary-patch of the shipped plugin (same-length replacement strings, bit-equivalent to a source rebuild) before any source PR.
• Designed the API extension to the shared sub_group_block_read.cl header—new __local-pointer overload family + companion macro + two-tier dispatch—migrated four kernels onto it across two PRs, surveyed the plugin for remaining call sites, and retrofitted the original PR’s kernel onto the shared API on reviewer request. Design approved by the subsystem architect and merge gatekeeper; validated on real Arc-class hardware via Intel’s DG2 internal CI.
• Followed up with a regression test that catches the bug class on any driver—the kernel calls the new _sub_group_block_read_slm* family directly, with the standard batch headers wired through the kernels_cache constructor (architect’s preferred pattern)—so pre-fix the symbols don’t exist in either dispatch branch and the build fails everywhere, while post-fix it compiles cleanly. The bug class can’t silently recur in CI.

Estimated impact

Modeled, not yet measured; vs CPU fallback under AUTO/HETERO; ranges span discrete Arc → low-power iGPU.

• Latency ~2–5× faster decode, up to ~10× faster TTFT on long prompts, by keeping inference on the GPU instead of falling back to CPU.
• Energy ~2–10× better energy-per-token (largest on low-power iGPUs like Lunar Lake).
• I/O in mixed-execution (HETERO) configs, removes the GPU↔CPU boundary tax (per-crossing sync + kernel-launch + layout-repack) that does no useful arithmetic, by keeping the LoRA/MoE/GEMV kernels resident on the GPU.

• PR #35661 openvinotoolkit/openvino #35661 —merged
• PR #35712 openvinotoolkit/openvino #35712 —merged
• PR #36017 openvinotoolkit/openvino #36017 —merged
• Profile github.com/pjordanandrsn

Friction by Other Means—how regulatory infrastructure preserved tariff effects after IEEPA fell

Essay · 2026 · ~7,600 words

The Trump II IEEPA tariff regime collapsed at the Supreme Court on February 20, 2026, but its practical effects largely did not. Tariff revenue, by Treasury’s own projection, will be “virtually unchanged” in 2026. Import friction—CBP audit pace, CF-29 issuance, first-sale challenges, transshipment enforcement, FDA entry-review centralization, de minimis closure, TTB queue lengthening—is at or above the IEEPA-era peak. Refunds are slow-walked through technical capacity constraints layered onto a narrow legal interpretation.

The strongest reading of the evidence is structural convergence with intentional ratification. Most of the infrastructure (CEEs, ACE, PREDICT, FSMA, FAA Act permitting, CBP audit authority) long predates the administration and has its own bureaucratic logic, but the timing, sequencing, public framing, and refund-mechanism design in 2025–2026 indicate the administration recognized the alignment and chose to operationalize it. The durable variable in modern American tariff policy is not the headline rate but the regulatory infrastructure that surrounds it.

• Essay writing/friction-by-other-means

The Structural Failure of America’s Defense Industrial Base — property, profit, and the production crisis

Essay · 2026 · ~4,200 words

The United States placed its war-fighting capacity inside publicly traded corporations whose legal obligations run to shareholders, not to readiness. In 2024, seven major defense primes returned 98 percent of free cash flow to investors. Surge capacity—idle factories, warm production lines, redundant suppliers—generates no shareholder return and gets cut. Operation Epic Fury exposed the result: 800 Patriot interceptors fired in five days against an annual production of 600; 168 Tomahawks fired in the first 100 hours against five-year purchases of 322; one carrier covering the Pacific while the fleet rebalances to the Persian Gulf.

The defense industrial base does not atrophy despite the market working correctly. It atrophies because the market is working correctly, toward an objective that has nothing to do with national security. The structural alternative is the Government-Owned, Contractor-Operated model—proven across 73 facilities in World War II and still operational for nuclear weapons production — which retains public ownership of productive capacity while leveraging private operational expertise.

• Essay writing/the-structural-failure-of-the-defense-industrial-base

The Great Recentralization—civilizational risk in the cloud-AI transition, 2026–2046

Essay · 2026 · ~9,000 words

The locus of meaningful computation has migrated from devices owned and controlled by users to a small set of hyperscale data centers operated by five American firms and one Taiwanese fabricator. Five threat vectors compound: economic concentration exceeding the railroad and oil trusts; surveillance infrastructure beyond what the Stasi could operate; supply-chain fragility centered on a single fab on a contested island; documented atrophy of cognitive capacity in populations that offload reasoning; and the disappearance of the off-grid-capable personal computer.

The cloud-AI transition is the enclosure of compute, of model weights, of the digital commons of personal data, and ultimately of cognition itself. The historical pattern is that enclosure, once accomplished, is not reversed by markets, only by politics. Existing antitrust and right-to-repair instruments regulate the symptoms of platform power while the underlying enclosure of compute proceeds unimpeded.

• Essay writing/the-great-recentralization