# Local LLM Testing: A 2015 GPU vs 15 Cloud Models, Open and Closed

URL: https://nanosolutions.io/local-llm-testing-old-gpu
Date: 1791504000
Description: Three small local models on a 2015 GPU vs 15 cloud models: Claude, GPT, Gemini and six open ones. Same three jobs. Play every result.

---


<div class="post-tldr">
  <p class="post-tldr__label">The short version</p>
  <ul>
    <li>Same three jobs, 18 models: three compressed models on a 2015 graphics card, six open models and nine closed ones in the cloud.</li>
    <li>Every closed model drew a valid pelican first time, and five of the six open ones did. None of the local ones did.</li>
    <li>Open models are now close behind: GLM-5.3, Kimi K3, DeepSeek V3.2 and Qwen3 235B passed every check.</li>
    <li>Local still wins on one thing: nothing leaves the building. Play every result below.</li>
  </ul>
</div>

## Every model's pelican

The test [Simon Willison made famous](https://simonwillison.net/tags/pelican-riding-a-bicycle/): draw a pelican riding a bicycle, in SVG. These are the real files, unedited.

<div class="llmx-gallery not-prose">
  <figure>
    <div class="tile tile--broken">Never a valid drawing</div>
    <figcaption><b>Q2_0</b><span>Local, 2015 GPU</span></figcaption>
  </figure>
  <figure>
    <div class="tile tile--broken">Never a valid drawing</div>
    <figcaption><b>IQ2_XS</b><span>Local, 2015 GPU</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/coder/pelican-extra-turn.svg" alt="Coder IQ1_M's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Coder IQ1_M</b><span>Local, needed an extra turn</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/glm/pelican.svg" alt="GLM-5.3's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>GLM-5.3</b><span>Open model, Z.ai</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/kimi/pelican.svg" alt="Kimi K3's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Kimi K3</b><span>Open model, Moonshot AI</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/deepseek/pelican.svg" alt="DeepSeek V3.2's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>DeepSeek V3.2</b><span>Open model, DeepSeek</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/qwen235/pelican.svg" alt="Qwen3 235B's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Qwen3 235B</b><span>Open model, Alibaba</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/devstral/pelican.svg" alt="Devstral 2's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Devstral 2</b><span>Open model, Mistral</span></figcaption>
  </figure>
  <figure>
    <div class="tile tile--broken">Never a valid drawing</div>
    <figcaption><b>MiniMax M3</b><span>Open model, MiniMax</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/opus/pelican.svg" alt="Claude Opus 5.5's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Claude Opus 5.5</b><span>Closed model, Anthropic</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/sonnet/pelican.svg" alt="Claude Sonnet 5.5's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Claude Sonnet 5.5</b><span>Closed model, Anthropic</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/haiku/pelican.svg" alt="Claude Haiku 5.5's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Claude Haiku 5.5</b><span>Closed model, Anthropic</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/sol/pelican.svg" alt="GPT-6.1-Sol's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>GPT-6.1-Sol</b><span>Closed model, OpenAI</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/astra/pelican.svg" alt="GPT-6-Astra's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>GPT-6-Astra</b><span>Closed model, OpenAI</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/luna/pelican.svg" alt="GPT-6-Luna's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>GPT-6-Luna</b><span>Closed model, OpenAI</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/flash/pelican.svg" alt="Gemini 3.8 Flash's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Gemini 3.8 Flash</b><span>Closed model, Google</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/argon/pelican.svg" alt="Gemini 4 Argon's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Gemini 4 Argon</b><span>Closed model, Google</span></figcaption>
  </figure>
  <figure>
    <div class="tile"><img src="/demos/local-llm/muse/pelican.svg" alt="Muse Spark 1.3's pelican riding a bicycle" loading="lazy" width="400" height="300"></div>
    <figcaption><b>Muse Spark 1.3</b><span>Closed model, Muse</span></figcaption>
  </figure>
</div>

## Play the results yourself

Pick a job. The selected model's file runs right here; swap models to compare. The chart and scoreboard follow.

<div class="llmx not-prose" x-data="llmResults" data-src="/data/blog/local-llm-results.json">
  <div class="llmx__controls" x-cloak x-show="ready">
    <fieldset>
      <legend>Job</legend>
      <div class="llmx__opts"><label><input type="radio" name="llmx-task" value="game" x-model="task" checked> Game</label><label><input type="radio" name="llmx-task" value="dashboard" x-model="task"> Dashboard</label><label><input type="radio" name="llmx-task" value="pelican" x-model="task"> Pelican drawing</label></div>
    </fieldset>
    <fieldset>
      <legend>Measure</legend>
      <div class="llmx__opts"><label><input type="radio" name="llmx-metric" value="time" x-model="metric" checked> Time to finish</label><label><input type="radio" name="llmx-metric" value="tps" x-model="metric"> Speed</label><label><input type="radio" name="llmx-metric" value="tokens" x-model="metric"> Output length</label></div>
    </fieldset>
  </div>
  <section class="llmx__try" x-cloak x-show="ready" aria-labelledby="llmx-try-title">
    <p class="llmx__name" id="llmx-try-title">Try it yourself</p>
    <p class="llmx__meta">The models' real, unedited files. It starts by itself; pick another model to swap.</p>
    <div class="llmx__try-controls">
      <template x-for="g in groups" x-bind:key="g.id">
        <fieldset>
          <legend x-text="g.label"></legend>
          <div class="llmx__opts"><template x-for="m in group(g.id)" x-bind:key="m.id"><label><input type="radio" name="llmx-demo-model" x-bind:value="m.id" x-model="demoModel"> <span x-text="m.name"></span></label></template></div>
        </fieldset>
      </template>
      <fieldset x-show="demos().length > 1">
        <legend>Version</legend>
        <div class="llmx__opts"><template x-for="(v, i) in demos()" x-bind:key="v.src"><label><input type="radio" name="llmx-demo-version" x-bind:value="i" x-model.number="demoVersion"> <span x-text="v.label"></span></label></template></div>
      </fieldset>
    </div>
    <div class="llmx__stage" x-ref="stage" x-bind:class="task === 'game' && 'llmx__stage--game'" x-bind:style="stageStyle">
      <button type="button" class="llmx__play" x-show="!playing" x-on:click="play()" x-cloak><span aria-hidden="true">▶</span> <span x-text="playLabel"></span></button>
      <template x-if="playing && demo && demo.kind === 'html'">
        <iframe x-bind:src="demo.src" x-bind:title="playLabel" x-bind:style="frameStyle" x-bind:class="isGame && 'is-preview'" x-bind:tabindex="isGame ? -1 : 0" sandbox="allow-scripts" loading="lazy" referrerpolicy="no-referrer"></iframe>
      </template>
      <button type="button" class="llmx__open" x-show="playing && isGame" x-cloak x-on:click="openModal()"><span class="llmx__open-btn"><span aria-hidden="true">▶</span> <span x-text="playLabel"></span></span></button>
      <template x-if="playing && demo && demo.kind === 'svg'">
        <img x-bind:src="demo.src" x-bind:alt="modelName(demoModel) + &quot;'s pelican riding a bicycle, drawn as SVG by the model&quot;" class="llmx__svg">
      </template>
      <template x-if="playing && demo && demo.kind === 'broken'">
        <div class="llmx__broken">
          <p><strong>This never became a valid drawing.</strong> A browser shows nothing for it. Here is what the model wrote (its start and end):</p>
          <pre x-text="brokenText || 'The file could not be loaded.'"></pre>
        </div>
      </template>
    </div>
    <p class="llmx__meta" x-show="playing && demo && demo.kind === 'html'"><span x-show="isGame">Press Play to open the game in its own window: arrows or A/D to move, P to pause, ✕ to close.</span> <a x-bind:href="demo && demo.src" target="_blank" rel="noopener">Open in a new tab</a>.</p>
    <div class="llmx__modal" x-show="modal" x-cloak x-transition.opacity role="dialog" aria-modal="true" x-bind:aria-label="playLabel" x-on:click.self="closeModal()">
      <div class="llmx__modal-head">
        <p x-text="modelName(demoModel) + ' · Meteor Dodge'"></p>
        <button type="button" class="llmx__modal-close" x-on:click="closeModal()" aria-label="Close the game">✕</button>
      </div>
      <div class="llmx__modal-box" x-bind:style="modalBoxStyle">
        <template x-if="modal">
          <iframe x-ref="modalFrame" x-bind:src="demo && demo.src" x-bind:title="playLabel" x-bind:style="modalFrameStyle" sandbox="allow-scripts" referrerpolicy="no-referrer"></iframe>
        </template>
      </div>
      <p class="llmx__modal-hint">Arrows or A/D to move · P to pause · ✕ or tap outside to close</p>
    </div>
  </section>
  <figure class="llmx__chart" x-cloak x-show="ready">
    <p class="wac__title" aria-live="polite" x-text="caption"></p>
    <p class="llmx__hint" x-text="metricHint"></p>
    <template x-for="g in groups" x-bind:key="g.id">
      <div>
        <p class="llmx__group" x-text="g.label"></p>
        <template x-for="m in group(g.id)" x-bind:key="m.id">
          <div class="wac__row">
            <div class="wac__label"><span x-text="m.name"></span><b x-text="label(m.id)"></b></div>
            <div class="wac__track"><div class="wac__bar" x-bind:class="g.id !== 'local' && ('wac__bar--' + g.id)" x-bind:style="bar(m.id)"></div></div>
          </div>
        </template>
      </div>
    </template>
  </figure>
  <div class="llmx__board" x-cloak x-show="ready">
    <p class="llmx__name">Scoreboard: <span x-text="taskLabel"></span></p>
    <template x-for="g in groups" x-bind:key="g.id">
      <div>
        <p class="llmx__group" x-text="g.label"></p>
        <ul class="llmx__score">
          <template x-for="m in group(g.id)" x-bind:key="m.id">
            <li>
              <details>
                <summary>
                  <span class="llmx__score-name" x-text="m.name"></span>
                  <span class="llmx__score-pass" x-bind:class="passCount(m.id).pass === passCount(m.id).total ? 'is-all' : (passCount(m.id).pass === 0 ? 'is-none' : '')" x-text="passCount(m.id).pass + '/' + passCount(m.id).total"></span>
                </summary>
                <p x-text="note(m.id)"></p>
                <ul class="llmx__checks"><template x-for="c in checks(m.id)" x-bind:key="c[0]"><li><span class="llmx__mark" x-bind:class="c[1] ? 'llmx__mark--pass' : 'llmx__mark--fail'" x-text="c[1] ? '✓ Pass' : '✗ Fail'"></span><span x-text="c[0]"></span></li></template></ul>
              </details>
            </li>
          </template>
        </ul>
      </div>
    </template>
    <p class="llmx__meta">Local models got our detailed browser checks; hosted models got the automated ones (loads, starts, valid SVG). Tap a model for details.</p>
  </div>
  <details class="llmx__prompt">
    <summary>Show the exact prompt for this job</summary>
    <blockquote><p x-effect="ready && ($el.textContent = prompt)">Create a complete playable browser game called Meteor Dodge. Return ONLY one self-contained HTML document with inline CSS and JavaScript, no Markdown or explanation. No libraries, network calls or external assets. Use a canvas: a player spaceship moves with arrow keys or A/D, meteors fall, collisions cost one of 3 lives, surviving increases score, difficulty increases over time, and a game-over screen offers a working restart button. Include a Start button, pause/resume with P, visible score and lives, instructions, and touch left/right buttons. Use a polished dark space theme. Keep the code compact and complete, under 1700 tokens.</p><p x-show="promptNote" x-text="promptNote"></p></blockquote>
  </details>
</div>

## What we tested

- **Local:** a 2015 NVIDIA GTX TITAN X (12 GB) running three Qwen models squeezed to about 2 bits per weight, on our own CUDA inference engine.
- **Open models, hosted:** GLM-5.3, MiniMax M3, Kimi K3, DeepSeek V3.2, Qwen3 235B and Devstral 2, through opencode (our subscription, or Amazon Bedrock). These are the weights a business could run on its own bigger hardware.
- **Closed models, hosted:** Claude Opus, Sonnet and Haiku 5.5; GPT-6.1-Sol, GPT-6-Astra and GPT-6-Luna; Gemini 3.8 Flash and Gemini 4 Argon; Muse Spark 1.3. Each through a command-line tool, at low effort.
- **Same for everyone:** the exact prompts, one attempt, no code fixed by hand.

## What we found

- **Closed models are the safest bet.** All nine drew a valid pelican and all nine games start. Two games (Opus 5.5 and GPT-6-Luna) throw one JavaScript error but keep running.
- **Open models are close behind.** GLM-5.3, Kimi K3, DeepSeek V3.2 and Qwen3 235B passed every check. Devstral 2 had errors in its game and dashboard; MiniMax M3's game never starts and its pelican is broken.
- **The old card is slow but not useless.** It writes 14 to 29 tokens a second; hosted models managed 33 to 260 even counting tool overhead. IQ2_XS still built a dashboard that passed every check.
- **Valid isn't the same as good.** Every open pelican but one is valid SVG, but Qwen3 235B's is lying on the ground next to its bike and Devstral 2's is a tangle. Look at the output, not just the checks.
- **Bigger isn't always better.** Claude Sonnet 5.5 was the most consistent: 16 to 17 seconds a job, every check passed.

## What this means for you

1. **Open models are the middle path.** The best ones match closed models on these jobs, and you can host them yourself on a modern GPU.
2. **Test on your own work.** Ten real tasks with a pass/fail check beat any leaderboard.
3. **Run the output.** Two of the prettiest games had bugs only a browser found.

One run per job: a smoke test, not a ranking. Want to size hardware or test models on your own tasks? That's our [private AI and local LLM hosting](/services/managed-linux-infrastructure/private-ai).

<details class="llmx__prompt">
<summary>All the numbers in one table</summary>

<div class="overflow-x-auto my-6 contain-paint">

| Job | Model | First attempt (s) | Tokens/s | Output tokens | Cut off? | Browser checks passed |
|---|---|---:|---:|---:|---|---|
| Game | Q2_0 | 45.6 | 22.9 | 965 | No | 6 of 9 |
| Game | IQ2_XS | 90.5 | 17.6 | 1,530 | No | 5 of 7 |
| Game | Coder IQ1_M | 159.5 | 14.2 | 2,200 | Yes, finished in one extra turn | 3 of 6 |
| Dashboard | Q2_0 | 63.9 | 22.0 | 1,313 | No | 6 of 7 |
| Dashboard | IQ2_XS | 99.7 | 17.1 | 1,633 | No | 9 of 9 |
| Dashboard | Coder IQ1_M | 135.9 | 16.8 | 2,200 | Yes, finished in one extra turn | 1 of 6 |
| Pelican SVG | Q2_0 | 190.5 | 28.7* | 2,400 | Yes, still invalid after an extra turn | — |
| Pelican SVG | IQ2_XS | 246.1 | 25.6* | 2,400 | Yes, still invalid after an extra turn | — |
| Pelican SVG | Coder IQ1_M | 116.5 | 23.0* | 2,400 | Yes, valid after an extra turn | — |
| Game | Claude Opus 5.5 (cloud) | 28.3 | 80.0† | 2,263 | No | Loads, starts, 1 JS error |
| Dashboard | Claude Opus 5.5 (cloud) | 18.6 | 117.7† | 2,190 | No | Loads, no JS errors |
| Pelican SVG | Claude Opus 5.5 (cloud) | 20.0 | 84.2† | 1,685 | No | Valid SVG |
| Game | Claude Sonnet 5.5 (cloud) | 16.1 | 136.7† | 2,201 | No | Loads, starts, no JS errors |
| Dashboard | Claude Sonnet 5.5 (cloud) | 16.7 | 153.0† | 2,555 | No | Loads, no JS errors |
| Pelican SVG | Claude Sonnet 5.5 (cloud) | 15.6 | 120.3† | 1,877 | No | Valid SVG |
| Game | Claude Haiku 5.5 (cloud) | 32.5 | 241.4† | 7,846 | No | Loads, starts, no JS errors |
| Dashboard | Claude Haiku 5.5 (cloud) | 15.2 | 259.9† | 3,951 | No | Loads, no JS errors |
| Pelican SVG | Claude Haiku 5.5 (cloud) | 29.4 | 190.8† | 5,609 | No | Valid SVG |
| Game | GPT-6.1-Sol (cloud) | 53.8 | 35.7† | 1,918 | No | Loads, starts, no JS errors |
| Dashboard | GPT-6.1-Sol (cloud) | 47.1 | 39.6† | 1,867 | No | Loads, no JS errors |
| Pelican SVG | GPT-6.1-Sol (cloud) | 65.1 | 35.5† | 2,308 | No | Valid SVG |
| Game | GPT-6-Astra (cloud) | 44.0 | 39.7† | 1,747 | No | Loads, starts, no JS errors |
| Dashboard | GPT-6-Astra (cloud) | 44.0 | 43.1† | 1,898 | No | Loads, no JS errors |
| Pelican SVG | GPT-6-Astra (cloud) | 62.4 | 33.4† | 2,086 | No | Valid SVG |
| Game | GPT-6-Luna (cloud) | 18.4 | 93.5† | 1,721 | No | Loads, starts, 1 JS error |
| Dashboard | GPT-6-Luna (cloud) | 20.4 | 96.1† | 1,961 | No | Loads, no JS errors |
| Pelican SVG | GPT-6-Luna (cloud) | 18.7 | 84.2† | 1,574 | No | Valid SVG |
| Game | Gemini 3.8 Flash (cloud) | 23.0 | 138.4† | 3,183 | No | Loads, starts, no JS errors |
| Dashboard | Gemini 3.8 Flash (cloud) | 25.5 | 138.7† | 3,537 | No | Loads, no JS errors |
| Pelican SVG | Gemini 3.8 Flash (cloud) | 48.1 | 124.3† | 5,980 | No | Valid SVG |
| Game | Gemini 4 Argon (cloud) | 46.1 | 105.4† | 4,861 | No | Loads, starts, no JS errors |
| Dashboard | Gemini 4 Argon (cloud) | 39.2 | 103.1† | 4,043 | No | Loads, no JS errors |
| Pelican SVG | Gemini 4 Argon (cloud) | 79.0 | 94.9† | 7,500 | No | Valid SVG |
| Game | GLM-5.3 (open) | 35.2 | 57.8† | 2,033 | No | Loads, starts, no JS errors |
| Dashboard | GLM-5.3 (open) | 31.1 | 71.8† | 2,237 | No | Loads, no JS errors |
| Pelican SVG | GLM-5.3 (open) | 211.7 | 60.1† | 12,731 | No | Valid SVG |
| Game | MiniMax M3 (open) | 32.4 | 102.3† | 3,318 | No | Loads, never starts |
| Dashboard | MiniMax M3 (open) | 53.6 | 139.8† | 7,486 | No | Loads, no JS errors |
| Pelican SVG | MiniMax M3 (open) | 48.5 | 134.5† | 6,517 | No | Invalid |
| Game | Kimi K3 (open) | 27.7 | 82.1† | 2,274 | No | Loads, starts, no JS errors |
| Dashboard | Kimi K3 (open) | 91.8 | 55.5† | 5,096 | No | Loads, no JS errors |
| Pelican SVG | Kimi K3 (open) | 89.4 | 106.8† | 9,544 | No | Valid SVG |
| Game | DeepSeek V3.2 (open) | 90.1 | 47.4† | 4,270 | No | Loads, starts, no JS errors |
| Dashboard | DeepSeek V3.2 (open) | 87.8 | 48.5† | 4,257 | No | Loads, no JS errors |
| Pelican SVG | DeepSeek V3.2 (open) | 52.0 | 36.4† | 1,893 | No | Valid SVG |
| Game | Qwen3 235B (open) | 59.5 | 56.7† | 3,374 | No | Loads, starts, no JS errors |
| Dashboard | Qwen3 235B (open) | 43.8 | 57.0† | 2,493 | No | Loads, no JS errors |
| Pelican SVG | Qwen3 235B (open) | 36.3 | 50.4† | 1,829 | No | Valid SVG |
| Game | Devstral 2 (open) | 32.4 | 64.9† | 2,101 | No | Loads, starts, 1 JS error |
| Dashboard | Devstral 2 (open) | 66.3 | 71.8† | 4,758 | No | Loads, 1 JS error |
| Pelican SVG | Devstral 2 (open) | 32.1 | 54.9† | 1,765 | No | Valid SVG |
| Game | Muse Spark 1.3 (cloud) | 29.8 | 105.0† | 3,127 | No | Loads, starts, no JS errors |
| Dashboard | Muse Spark 1.3 (cloud) | 20.0 | 113.0† | 2,261 | No | Loads, no JS errors |
| Pelican SVG | Muse Spark 1.3 (cloud) | 12.1 | 184.8† | 2,232 | No | Valid SVG |

</div>

Hosted times are end-to-end through each command-line tool. †Hosted speed is effective speed: reported output (and reasoning) tokens divided by that end-to-end time. Hosted token counts include reasoning where the tool reports it. Local check counts aren't a score: each output got the checks that fitted what it built. Local pelican times can include loading the model. *Pelican speeds come from a separate streaming re-run on 9 October with the same prompt and settings; its drawings differed slightly but failed the same way.

</details>

---

**Photo credits.** Hero: two NVIDIA Titan X cards, [Back2Gaming on Unsplash](https://unsplash.com/photos/two-nvidia-titan-x-graphics-cards-side-by-side-hyiNFISCPm8), used under the [Unsplash License](https://unsplash.com/license).

**Sources.** Our own test runs on 8 and 9 October 2026. [Hugging Face GGUF docs](https://huggingface.co/docs/hub/gguf) for bits per weight. [GeForce 900 series](https://en.wikipedia.org/wiki/GeForce_900_series) for the card. [OAIC guidance on commercial AI products](https://www.oaic.gov.au/privacy/privacy-guidance-for-organisations-and-government-agencies/guidance-on-privacy-and-the-use-of-commercially-available-ai-products) on privacy.