{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1800",
  "slug": "inference-moves-off-datacenter",
  "edition": "2026-11-19T12",
  "url": "https://hangingcontext.com/stories/2026-11-19/inference-moves-off-datacenter/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Inference is moving off the datacenter GPU onto phones, small chips and CPUs.",
  "headline": "Inference is moving off the datacenter GPU and onto the device",
  "dek": "A vision model that runs faster than a datacenter GPU, a simulator 135 times faster than real hardware, and Gemini on low-memory phones all point the same way.",
  "published_at": "2026-11-19T12:41:21Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "arxiv-ai-infra-inference-ops",
    "arxiv-frontier-methods-select",
    "arxiv-rag-search-knowledge",
    "google-gemini-app"
  ],
  "sources": [
    "arxiv-ai-infra-inference-ops",
    "arxiv-frontier-methods-select",
    "arxiv-rag-search-knowledge",
    "google-gemini-app"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "In a preprint, HETEROVLM decodes 1.3x and 4.0x faster than NVIDIA Tesla T4 and Jetson Orin Nano GPUs, and uses 51% of the energy per token of an RTX 3090 Ti.",
      "cites": [
        {
          "claim_ids": [
            "1795070757845643275",
            "1795070757856643276"
          ],
          "manifest_id": "1795070540999640951"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, MoMoVL cuts memory footprint by up to 15.44% in mixture-of-experts vision-language models compared with uniform-precision baselines.",
      "cites": [
        {
          "claim_ids": [
            "1795066467962609913"
          ],
          "manifest_id": "1795066178144606958"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, LEDGE simulates distributed LLM serving 135.3 times faster than physical inference, with 7.44% zero-shot latency error across three other model families.",
      "cites": [
        {
          "claim_ids": [
            "1795070757369643183",
            "1795070757389643186"
          ],
          "manifest_id": "1795070540999640946"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Google says the Gemini app is now available on eligible Android Go edition devices after the company re-architected it to run on hardware with less memory, and Transsion will preload it on upcoming devices.",
      "cites": [
        {
          "claim_ids": [
            "1795028291143168984",
            "1795028291144168985",
            "1795028291150168991"
          ],
          "manifest_id": "1795028034567166375"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "A small chip beats a big GPU",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795070757834643273",
                "1795070757845643275",
                "1795070757856643276"
              ],
              "manifest_id": "1795070540999640951"
            }
          ],
          "markdown": "The most surprising number in this week's infrastructure preprints comes from a small chip beating big ones. HETEROVLM is a system for running vision-language models on a Versal SoC, a chip that pairs AI engines with programmable logic, and it enables concurrent request scheduling and preemption by offloading compute-bound tasks to the AI Engine and memory-bound decode tasks to programmable logic. It achieves 1.3x and 4.0x faster single-request decoding than NVIDIA Tesla T4 and Jetson Orin Nano GPUs, respectively, and it uses 51% of the energy per token of an NVIDIA GeForce RTX 3090 Ti."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795070757834643273"
              ],
              "manifest_id": "1795070540999640951"
            }
          ],
          "markdown": "The trick is specialization rather than brute force. Splitting the work between the AI engines and the programmable logic means each part does what it is good at, and concurrent scheduling means one request does not block the next. The result is a system that serves vision-language models on the edge, where the power budget is measured in watts and the network may not be there at all."
        }
      ]
    },
    {
      "heading": "Shrinking the model to fit",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795066467962609913",
                "1795066467963609915"
              ],
              "manifest_id": "1795066178144606958"
            }
          ],
          "markdown": "MoMoVL, in a second preprint, shrinks the model itself for the same destination. It applies modality-aware adaptive quantization to mixture-of-experts vision-language models, and it reduces memory footprint by up to 15.44% compared with uniform-precision baselines. The insight is that visual and text tokens do not behave the same way under quantization: modality-specific characteristics of visual and text tokens impact expert quantization sensitivity in MoE-based VLMs, so treating them identically wastes precision where it is not needed and loses it where it is."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795076196813660185",
                "1795076196825660186",
                "1795076196807660184"
              ],
              "manifest_id": "1795075910574658143"
            }
          ],
          "markdown": "KatoANN, in a third preprint, does the same kind of work for retrieval, the step that finds the right documents before the model answers. It achieves high throughput and high recall simultaneously on CPU matrix engines, using a two-stage ranking process of high-byte ranking followed by full-distance refinement to reduce multiplication overhead. It also reduces memory footprint by avoiding full-precision vector copies. Retrieval on a CPU, at near-exact recall, means a whole class of applications no longer needs a GPU for the part of the pipeline that used to demand one."
        }
      ]
    },
    {
      "heading": "Simulating the edge before building it",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795070757369643183",
                "1795070757389643186",
                "1795070757404643188"
              ],
              "manifest_id": "1795070540999640946"
            }
          ],
          "markdown": "Designing for the edge is hard because the hardware is heterogeneous: different devices, different links, different contention. LEDGE, in a fourth preprint, is a simulator for distributed LLM serving on heterogeneous edge devices that lets engineers explore deployment strategies without running them on physical hardware. It achieves a 135.3 times aggregate speedup over physical inference, and performance models calibrated on one LLM family yield 7.44% zero-shot end-to-end latency MAPE across three other families. It constructs event graphs supporting non-uniform tensor and pipeline parallelism and simulates shared-link contention, which is the detail that makes or breaks a real deployment."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795070757369643183",
                "1795070757389643186"
              ],
              "manifest_id": "1795070540999640946"
            }
          ],
          "markdown": "A simulator that runs 135 times faster than the hardware it models changes what gets tried. An engineer can test a deployment plan in minutes that would take days to run for real, and the 7.44% error suggests the answers are close enough to act on. That is infrastructure for the edge in the literal sense: the tooling that lets small devices be treated as a serious serving target."
        }
      ]
    },
    {
      "heading": "The phone in your pocket",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795028291143168984",
                "1795028291144168985",
                "1795028291145168986",
                "1795028291150168991"
              ],
              "manifest_id": "1795028034567166375"
            }
          ],
          "markdown": "Google supplied the commercial proof point this week. The company says the Gemini app is now available on eligible Android Go edition devices, and that Google re-architected the Gemini app to run on hardware with less memory. Gemini Live supports more than 45 languages, and Transsion will preload the Gemini app on upcoming devices. Go edition phones are the low-memory devices sold in large numbers outside the flagship market, so this is not a niche: it is the mainstream phone, running the flagship assistant."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795028291146168987",
                "1795028291147168988",
                "1795028291149168990"
              ],
              "manifest_id": "1795028034567166375"
            }
          ],
          "markdown": "The features the company lists are the ones that work offline or on-device: users can use the camera to learn and translate notes or forms, Nano Banana allows users to create custom images, and users can access Gemini by pressing and holding the power button on supported phones. The through-line across all five pieces is the same. The model is no longer something you visit in the cloud. It is something that runs where you are, on hardware that costs a fraction of a datacenter GPU, and the engineering to make that work is arriving fast."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether HETEROVLM's speed advantage over the Tesla T4 and Jetson Orin Nano holds on workloads beyond single-request decoding.",
    "Whether MoMoVL's 15.44% memory saving preserves quality on vision tasks outside the ones it was tested on.",
    "How accurate LEDGE's 7.44% latency error stays when the edge devices and links differ more sharply than the families tested."
  ],
  "decision_card": {
    "not_for": [
      "Teams whose workloads need the largest models at full precision"
    ],
    "useful_for": [
      "Developers shipping AI features on phones and edge devices",
      "Teams deciding what can run locally instead of in the cloud"
    ],
    "compare_against": [
      "Running every request on a datacenter GPU",
      "Waiting for the next flagship phone chip"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-19T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1795070540999640951",
    "1795066178144606958",
    "1795070540999640946",
    "1795028034567166375",
    "1795075910574658143"
  ],
  "manifests": [
    {
      "title": "HETEROVLM: Edge Multi-Request Vision-Language Model Serving on a Heterogeneous Versal SoC with AI Engines and Programmable Logic",
      "source_url": "https://arxiv.org/pdf/2611.23878v1",
      "manifest_id": "1795070540999640951",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-19"
    },
    {
      "title": "MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models",
      "source_url": "https://arxiv.org/pdf/2611.23695v1",
      "manifest_id": "1795066178144606958",
      "source_name": "arxiv-frontier-methods-select",
      "published_date": "2026-11-19"
    },
    {
      "title": "LEDGE: An Efficient Simulator for Distributed LLM Serving on Heterogeneous Edge Devices",
      "source_url": "https://arxiv.org/pdf/2611.24085v1",
      "manifest_id": "1795070540999640946",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-19"
    },
    {
      "title": "The Gemini app is now available on lower-memory Android devices.",
      "source_url": "https://blog.google/innovation-and-ai/products/gemini-app/gemini-lower-memory-devices",
      "manifest_id": "1795028034567166375",
      "source_name": "google-gemini-app",
      "published_date": "2026-11-18"
    },
    {
      "title": "KatoANN: Enabling Near-Exact Vector Search at the Speed of Lossy Scans with CPU Matrix Engines",
      "source_url": "https://arxiv.org/pdf/2610.27508v3",
      "manifest_id": "1795075910574658143",
      "source_name": "arxiv-rag-search-knowledge",
      "published_date": "2026-11-19"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1795070540999640951",
      "1795066178144606958",
      "1795070540999640946",
      "1795028034567166375",
      "1795075910574658143"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1795070540999640951",
      "all": "for id in 1795070540999640951 1795066178144606958 1795070540999640946 1795028034567166375 1795075910574658143; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-19/inference-moves-off-datacenter/",
    "json": "https://hangingcontext.com/stories/2026-11-19/inference-moves-off-datacenter/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-19/inference-moves-off-datacenter/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-19T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-19/gemini-4-neon-genie-4-cadence/",
      "headline": "Google's fourth model in seven weeks arrives with a world model beside it",
      "slug": "gemini-4-neon-genie-4-cadence"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/",
      "headline": "Three inference providers disclosed prompt failures in the same week",
      "slug": "where-your-prompts-actually-go"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/",
      "headline": "Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop",
      "slug": "the-weeks-plumbing-papers"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/",
      "headline": "The agent platform is standardising: protocols, observability, governance",
      "slug": "the-agent-platform-standardizes"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-19/inference-moves-off-datacenter/"
  }
}
