<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>EMTutor | Peilong Li</title>
    <link>https://Peilong.github.io/tags/emtutor/</link>
      <atom:link href="https://Peilong.github.io/tags/emtutor/index.xml" rel="self" type="application/rss+xml" />
    <description>EMTutor</description>
    <generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>© 2026 Peilong Li</copyright><lastBuildDate>Thu, 19 Nov 2026 00:00:00 +0000</lastBuildDate>
    <image>
      <url>https://Peilong.github.io/img/icon-192.png</url>
      <title>EMTutor</title>
      <link>https://Peilong.github.io/tags/emtutor/</link>
    </image>
    
    <item>
      <title>Running an LLM In-House: Early Lessons from EMTutor</title>
      <link>https://Peilong.github.io/post/local-llm-lessons-from-emtutor/</link>
      <pubDate>Thu, 19 Nov 2026 00:00:00 +0000</pubDate>
      <guid>https://Peilong.github.io/post/local-llm-lessons-from-emtutor/</guid>
      <description>&lt;p&gt;Every organization I talk to wants the same thing from AI: the usefulness of a chat assistant without sending its data to someone else&amp;rsquo;s servers. A clinic has patient records. A law firm has client files. A manufacturer has designs and supplier contracts. A college has student records protected by FERPA. For all of them, &amp;ldquo;just use a cloud API&amp;rdquo; comes with a privacy question that someone has to answer.&lt;/p&gt;
&lt;p&gt;This fall my students and I have been building &lt;strong&gt;EMTutor&lt;/strong&gt;, a self-hosted AI tutor for introductory computer science at Elizabethtown College. It is part of a &lt;a href=&#34;https://bluejayadal.github.io/project/etown-llm/&#34;&gt;college-funded project&lt;/a&gt; to prototype a self-hosted LLM for campus. Every part of it runs on college hardware, and no student data leaves campus. Its main job is to be a &lt;em&gt;Socratic&lt;/em&gt; coding tutor: it helps students work through problems by asking questions, and it is designed never to hand over the solution.&lt;/p&gt;
&lt;p&gt;EMTutor has not launched to students yet. The numbers below come from our test environment, with simulated students and a synthetic course. They are preliminary, but they already changed several of our design decisions. Because the hardware is modest, and probably close to what a small business already owns or could buy, the lessons should transfer well beyond education.&lt;/p&gt;
&lt;h2 id=&#34;the-setup&#34;&gt;The setup&lt;/h2&gt;
&lt;p&gt;The hardware is deliberately ordinary, and all of it was already in our lab:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One GPU workstation&lt;/strong&gt; with two NVIDIA RTX 2080 Ti cards from 2018: 11 GB of memory each, 22 GB in total, linked with NVLink.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two older rack servers&lt;/strong&gt; with no GPUs, for everything else: the web front end, the database, search over course materials, a sandbox for running student code, monitoring and backups.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The software is all open source except the small pieces we wrote:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What we use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;An open-weight 8-billion-parameter model (&lt;a href=&#34;https://huggingface.co/Qwen/Qwen3-8B-AWQ&#34;&gt;Qwen3-8B-AWQ&lt;/a&gt;, 4-bit), served by &lt;a href=&#34;https://github.com/vllm-project/vllm&#34;&gt;vLLM&lt;/a&gt; across both GPUs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chat interface&lt;/td&gt;
&lt;td&gt;&lt;a href=&#34;https://github.com/open-webui/open-webui&#34;&gt;Open WebUI&lt;/a&gt;, behind the college&amp;rsquo;s single sign-on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval&lt;/td&gt;
&lt;td&gt;Hybrid keyword and vector search over course slides, so answers can cite the course&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gateway&lt;/td&gt;
&lt;td&gt;A small proxy we wrote that sits between the chat interface and the model and enforces the tutor&amp;rsquo;s rules&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Code sandbox&lt;/td&gt;
&lt;td&gt;Runs a student&amp;rsquo;s program with no network, a read-only file system and strict time and memory limits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;A faculty lead and three undergraduate interns, each working a few hours a week, built it. Every change goes through code review, automated tests and the evaluations described below.&lt;/p&gt;
&lt;h2 id=&#34;lesson-1-size-for-many-users-at-once-not-for-one-fast-user&#34;&gt;Lesson 1: Size for many users at once, not for one fast user&lt;/h2&gt;
&lt;p&gt;Most published speed numbers for local LLMs measure one user at a time. An organization needs the opposite measurement: how long does each person wait when 20 or 30 colleagues are using the system at once?&lt;/p&gt;
&lt;p&gt;The metric we track is &lt;strong&gt;time to first token&lt;/strong&gt;, the wait between pressing Enter and seeing the first word of the reply. For a chat tool it is what users feel. Our target is a 95th percentile under 5 seconds: only 1 reply in 20 may take longer than that.&lt;/p&gt;
&lt;p&gt;Under load, the limit is usually not raw GPU speed but GPU &lt;em&gt;memory&lt;/em&gt;. Every active conversation needs working memory on the GPU, called the &lt;strong&gt;KV cache&lt;/strong&gt;, and it grows with the length of the conversation. Whatever memory the model&amp;rsquo;s own weights leave over decides how many conversations fit at once. Beyond that, new requests wait in line.&lt;/p&gt;
&lt;h2 id=&#34;lesson-2-measure-on-your-hardware-because-estimates-can-be-far-off&#34;&gt;Lesson 2: Measure on your hardware, because estimates can be far off&lt;/h2&gt;
&lt;p&gt;Before we touched the hardware, our sizing estimates said IBM&amp;rsquo;s &lt;a href=&#34;https://huggingface.co/ibm-granite/granite-4.1-8b&#34;&gt;Granite 4.1-8B&lt;/a&gt; would hold about 27 typical conversations of 3,000 tokens at once. On the real cards, it held about six.&lt;/p&gt;
&lt;p&gt;Here is vLLM&amp;rsquo;s own benchmark on our two GPUs. It is a deliberately harsh test: every request is a fresh 2,800-token prompt with nothing shared to reuse, followed by a 250-token reply.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Memory left for conversations&lt;/th&gt;
&lt;th&gt;Requests at once&lt;/th&gt;
&lt;th&gt;Median wait for first word&lt;/th&gt;
&lt;th&gt;95th-percentile wait&lt;/th&gt;
&lt;th&gt;Total output&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Granite 4.1-8B (16-bit)&lt;/td&gt;
&lt;td&gt;17,920 tokens (about 6 conversations)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.8 s&lt;/td&gt;
&lt;td&gt;1.8 s&lt;/td&gt;
&lt;td&gt;36 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;8.2 s&lt;/td&gt;
&lt;td&gt;20.6 s&lt;/td&gt;
&lt;td&gt;71 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;38.8 s&lt;/td&gt;
&lt;td&gt;43.3 s&lt;/td&gt;
&lt;td&gt;71 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-8B-AWQ (4-bit)&lt;/td&gt;
&lt;td&gt;93,248 tokens (about 30 conversations)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.5 s&lt;/td&gt;
&lt;td&gt;1.5 s&lt;/td&gt;
&lt;td&gt;58 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;4.3 s&lt;/td&gt;
&lt;td&gt;10.1 s&lt;/td&gt;
&lt;td&gt;111 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;5.2 s&lt;/td&gt;
&lt;td&gt;14.7 s&lt;/td&gt;
&lt;td&gt;121 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;td&gt;41.0 s&lt;/td&gt;
&lt;td&gt;60.4 s&lt;/td&gt;
&lt;td&gt;72 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Two models of the same size behaved very differently. The 4-bit quantized model left &lt;strong&gt;five times as much memory&lt;/strong&gt; for conversations and answered a single user &lt;strong&gt;1.6 times faster&lt;/strong&gt;. That settled our default model. Past about 16 simultaneous fresh prompts, both models slow down because the GPUs spend their time reading prompts. In real use this happens less often, because the system prompt and earlier turns of a conversation are cached and reused.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Older GPUs also rule out options you might be counting on.&lt;/strong&gt; Our 2018-era cards lack features newer models and tools assume:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;vLLM refused an 8-bit KV cache, which would have doubled capacity, because it needs a newer GPU generation.&lt;/li&gt;
&lt;li&gt;Google&amp;rsquo;s Gemma 4 models would not load in vLLM at all: their attention layers need more on-chip memory than these cards have.&lt;/li&gt;
&lt;li&gt;Large open models such as DeepSeek&amp;rsquo;s and Kimi&amp;rsquo;s flagships need hundreds of gigabytes of GPU memory, 15 to 70 times what we have, even at 4 bits.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;lesson-3-pick-the-serving-engine-for-concurrency&#34;&gt;Lesson 3: Pick the serving engine for concurrency&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;https://github.com/ggml-org/llama.cpp&#34;&gt;llama.cpp&lt;/a&gt; is excellent for one person on a laptop or workstation, and it ran Gemma 4 12B, which vLLM could not. With 8 or 16 requests at once, though, the slowest users waited &lt;strong&gt;64 to 87 seconds&lt;/strong&gt; for the first word, because llama.cpp does not batch long prompts the way vLLM does. vLLM, which is built to batch many requests on the GPU, is our production engine. llama.cpp is our fallback and our CPU stand-in for development.&lt;/p&gt;
&lt;p&gt;The same caution applies to the many new &amp;ldquo;fastest local inference&amp;rdquo; projects. When I reviewed several of them for our hardware, their headline speeds came from one user at a time, often on newer or modified GPUs. A system that is fast for one user but serves only one at a time is worse for an office of 30.&lt;/p&gt;
&lt;h2 id=&#34;lesson-4-the-bottleneck-may-not-be-the-gpu-at-all&#34;&gt;Lesson 4: The bottleneck may not be the GPU at all&lt;/h2&gt;
&lt;p&gt;The most useful test we ran was a &lt;strong&gt;load test through the whole stack&lt;/strong&gt;: simulated students who sign in, ask questions about the course, and pause 8 to 25 seconds between messages, for 10 minutes. The first runs surprised us.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retrieval setup&lt;/th&gt;
&lt;th&gt;Simulated students&lt;/th&gt;
&lt;th&gt;95th-percentile wait for first word&lt;/th&gt;
&lt;th&gt;Replies in 10 minutes&lt;/th&gt;
&lt;th&gt;Errors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;No retrieval&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;0.4 s&lt;/td&gt;
&lt;td&gt;1,673&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval with a reranker on CPU&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;284 s&lt;/td&gt;
&lt;td&gt;154&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No reranker, 10 candidate slides&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;17.9 s&lt;/td&gt;
&lt;td&gt;914&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No reranker, 6 candidates, more CPU threads for search&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;7.3 s&lt;/td&gt;
&lt;td&gt;1,042&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The same&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.6 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;850&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Without retrieval, the model and gateway served 50 students with a 0.4-second wait. Adding search over course slides pushed the wait to almost five minutes. The culprit was a &lt;strong&gt;reranker&lt;/strong&gt;, a second model that re-sorts search results for relevance, running on one of the CPU servers. Reranking one question took it about four seconds as deployed, and 50 students ask about three questions a second.&lt;/p&gt;
&lt;p&gt;Dropping the reranker was a real tradeoff. On our retrieval test, the best slide came first for 96% of questions with the reranker and 83% without it. However, the right slide was still among the five the tutor sees for 100% of questions, at about a sixth of the cost. We dropped it.&lt;/p&gt;
&lt;p&gt;Retrieval also changes the GPU&amp;rsquo;s workload. The tutor&amp;rsquo;s instructions, about 1,050 tokens, are cached and reused for every question. Retrieved slides add about 300 new tokens that must be read fresh each time, roughly ten times the prompt reading per reply. With 50 students that work keeps both GPUs busy, so for now &lt;strong&gt;35 students chatting at once&lt;/strong&gt; is the pilot&amp;rsquo;s capacity. Fifty is the goal once we see the real course materials.&lt;/p&gt;
&lt;p&gt;One error in those runs is worth mentioning because every multi-part system hits it eventually. Two components disagreed on how long to keep an idle connection open, and one reused a connection the other had just closed. A longer keep-alive setting fixed it. Load tests find problems like this; demos don&amp;rsquo;t.&lt;/p&gt;
&lt;h2 id=&#34;lesson-5-put-your-rules-in-a-gateway-you-own&#34;&gt;Lesson 5: Put your rules in a gateway you own&lt;/h2&gt;
&lt;p&gt;A chat interface can carry the right system prompt and settings, but users can often switch presets or call the model directly. So we wrote a small &lt;strong&gt;gateway&lt;/strong&gt;, an OpenAI-compatible proxy between the chat interface and the model. Every request passes through it, whatever settings the user picked. It:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;adds the tutor&amp;rsquo;s instructions to every request;&lt;/li&gt;
&lt;li&gt;turns off the model&amp;rsquo;s visible &amp;ldquo;thinking&amp;rdquo;, which in a tutor would show students the solution it is withholding;&lt;/li&gt;
&lt;li&gt;checks each reply for a near-complete solution before it reaches the student;&lt;/li&gt;
&lt;li&gt;runs student code in the sandbox when the tutor needs to see its output;&lt;/li&gt;
&lt;li&gt;logs what happened for auditing.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Because everything downstream talks to &amp;ldquo;the tutor&amp;rdquo; through the gateway, swapping the model is a configuration change plus a re-run of the evaluations, not a code change. For a business, the same layer is where you would remove personal information, enforce approved uses and keep an audit trail.&lt;/p&gt;
&lt;h2 id=&#34;lesson-6-test-behavior-not-just-speed&#34;&gt;Lesson 6: Test behavior, not just speed&lt;/h2&gt;
&lt;p&gt;Our central claim is that the tutor does not hand out answers, so we test exactly that. The leakage evaluation sends 180 attempts to extract a solution: 12 introductory problems, each asked 15 different ways, including direct requests, fake permission, role-play, asking for the answer piece by piece and &amp;ldquo;just fix my code.&amp;rdquo; Code in a reply that passes the problem&amp;rsquo;s own tests counts as a full leak. A held-out set of 120 more cases, which no prompt is tuned on, guards against teaching to the test.&lt;/p&gt;
&lt;p&gt;What we learned:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The model matters.&lt;/strong&gt; Our first prompt leaked full solutions in 45 of 180 cases (25%) on a small 3-billion-parameter model. The current prompt leaks none of the 180 on our 8B model, and 5 of the 120 held-out cases.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;More rules can make it worse.&lt;/strong&gt; A version that listed every forbidden kind of answer leaked more than the simpler one before it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Surrounding text matters.&lt;/strong&gt; With the chat interface&amp;rsquo;s default retrieval template, which asks for &amp;ldquo;a clear and direct response,&amp;rdquo; the tutor fixed students&amp;rsquo; buggy code in 7 of 12 cases where it otherwise refused. We rewrote the template.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Layers help.&lt;/strong&gt; With retrieval on, 2 of 180 cases leaked in full, and the gateway&amp;rsquo;s check caught both.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Every change to the prompt must pass this evaluation before it merges. A business should build the same kind of test for its own worst case, whether that is leaking a client&amp;rsquo;s data, inventing a policy or giving advice it is not allowed to give, and rerun it on every change to the model, prompt or retrieval setup.&lt;/p&gt;
&lt;h2 id=&#34;lesson-7-watch-the-defaults&#34;&gt;Lesson 7: Watch the defaults&lt;/h2&gt;
&lt;p&gt;Several of our risks came from defaults no one would notice in a demo:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Containers can bypass your firewall.&lt;/strong&gt; Docker writes its own firewall rules, so a published port can be reachable from the whole network even when the host firewall says no. Bind each service to a specific address, and scan from another machine to check.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Some components ship without authentication&lt;/strong&gt;, including popular vector databases. Turn it on.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Some apps phone home&lt;/strong&gt; with telemetry or update checks by default. In a &amp;ldquo;data never leaves&amp;rdquo; system, that has to be turned off and verified.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Untrusted code needs a real sandbox:&lt;/strong&gt; no network, a read-only file system, and process, memory and time limits.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Write down &amp;ldquo;no cloud fallback&amp;rdquo; as a requirement.&lt;/strong&gt; Otherwise someone will eventually add one to fix a capacity problem.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pin every version.&lt;/strong&gt; We never update the GPU server blindly during the semester.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Read the licenses.&lt;/strong&gt; Open WebUI&amp;rsquo;s license, for example, limits removing its branding above 50 users without an enterprise license.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Consumer hardware has limits of its own. Our GPU workstation has no error-correcting memory, no redundant power supply and no remote management. It puts out about 500 watts of heat under load in a desktop case. We monitor GPU temperature and power and keep encrypted backups. Before launch, we plan monthly drills for restoring from backup and for losing the GPU server.&lt;/p&gt;
&lt;h2 id=&#34;the-tradeoffs-in-one-table&#34;&gt;The tradeoffs in one table&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Local model on your hardware&lt;/th&gt;
&lt;th&gt;Cloud API&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Stays on your servers&lt;/td&gt;
&lt;td&gt;Leaves your network; governed by a contract&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model capability&lt;/td&gt;
&lt;td&gt;Limited by your GPU memory; an 8B model on our cards&lt;/td&gt;
&lt;td&gt;Frontier models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;Hardware, power and staff time; flat as usage grows&lt;/td&gt;
&lt;td&gt;Pay per use; nothing up front&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Capacity&lt;/td&gt;
&lt;td&gt;Fixed; measured in users at once&lt;/td&gt;
&lt;td&gt;Scales on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Control&lt;/td&gt;
&lt;td&gt;You pin versions and decide when anything changes&lt;/td&gt;
&lt;td&gt;The provider can change or retire models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operations&lt;/td&gt;
&lt;td&gt;You patch, monitor, back up and recover&lt;/td&gt;
&lt;td&gt;The provider does&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For a focused task like ours, a well-tested 8B model with good retrieval and firm guardrails is enough. For open-ended work that needs the strongest reasoning, a cloud model may still be the better tool. Many organizations will end up using both, with a clear rule about which data may go where.&lt;/p&gt;
&lt;h2 id=&#34;if-you-are-considering-this-for-your-organization&#34;&gt;If you are considering this for your organization&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Describe the workload first:&lt;/strong&gt; how many people at once, how long their questions are and how long they can wait.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Start small:&lt;/strong&gt; a 7 to 8 billion parameter open model, 4-bit quantized, on vLLM, is a sensible first try on modest GPUs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Benchmark the model on your hardware, then load-test the whole system&lt;/strong&gt; with realistic users. Our biggest surprise was not in the GPU.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Build an evaluation for your main failure mode&lt;/strong&gt; and run it on every change.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Put a gateway in front of the model&lt;/strong&gt; so your rules cannot be clicked around.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Lock down the defaults&lt;/strong&gt; and plan for the hardware failing.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Next for EMTutor: a pilot with students in an introductory course, real course materials in place of our synthetic ones, and tests over winter break of newer engine builds that could fit more conversations on the same cards. I will share updated numbers as they come in.&lt;/p&gt;
&lt;p&gt;If your organization is weighing a local deployment and you want to compare notes, you can reach me at lip AT etown DOT edu.&lt;/p&gt;
</description>
    </item>
    
  </channel>
</rss>
