<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Writing | Peilong Li</title>
    <link>https://Peilong.github.io/post/</link>
      <atom:link href="https://Peilong.github.io/post/index.xml" rel="self" type="application/rss+xml" />
    <description>Writing</description>
    <generator>Source Themes Academic (https://sourcethemes.com/academic/)</generator><language>en-us</language><copyright>© 2026 Peilong Li</copyright><lastBuildDate>Thu, 19 Nov 2026 00:00:00 +0000</lastBuildDate>
    <image>
      <url>https://Peilong.github.io/img/icon-192.png</url>
      <title>Writing</title>
      <link>https://Peilong.github.io/post/</link>
    </image>
    
    <item>
      <title>Running an LLM In-House: Early Lessons from EMTutor</title>
      <link>https://Peilong.github.io/post/local-llm-lessons-from-emtutor/</link>
      <pubDate>Thu, 19 Nov 2026 00:00:00 +0000</pubDate>
      <guid>https://Peilong.github.io/post/local-llm-lessons-from-emtutor/</guid>
      <description>&lt;p&gt;Every organization I talk to wants the same thing from AI: the usefulness of a chat assistant without sending its data to someone else&amp;rsquo;s servers. A clinic has patient records. A law firm has client files. A manufacturer has designs and supplier contracts. A college has student records protected by FERPA. For all of them, &amp;ldquo;just use a cloud API&amp;rdquo; comes with a privacy question that someone has to answer.&lt;/p&gt;
&lt;p&gt;This fall my students and I have been building &lt;strong&gt;EMTutor&lt;/strong&gt;, a self-hosted AI tutor for introductory computer science at Elizabethtown College. It is part of a &lt;a href=&#34;https://bluejayadal.github.io/project/etown-llm/&#34;&gt;college-funded project&lt;/a&gt; to prototype a self-hosted LLM for campus. Every part of it runs on college hardware, and no student data leaves campus. Its main job is to be a &lt;em&gt;Socratic&lt;/em&gt; coding tutor: it helps students work through problems by asking questions, and it is designed never to hand over the solution.&lt;/p&gt;
&lt;p&gt;EMTutor has not launched to students yet. The numbers below come from our test environment, with simulated students and a synthetic course. They are preliminary, but they already changed several of our design decisions. Because the hardware is modest, and probably close to what a small business already owns or could buy, the lessons should transfer well beyond education.&lt;/p&gt;
&lt;h2 id=&#34;the-setup&#34;&gt;The setup&lt;/h2&gt;
&lt;p&gt;The hardware is deliberately ordinary, and all of it was already in our lab:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One GPU workstation&lt;/strong&gt; with two NVIDIA RTX 2080 Ti cards from 2018: 11 GB of memory each, 22 GB in total, linked with NVLink.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two older rack servers&lt;/strong&gt; with no GPUs, for everything else: the web front end, the database, search over course materials, a sandbox for running student code, monitoring and backups.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The software is all open source except the small pieces we wrote:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What we use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;An open-weight 8-billion-parameter model (&lt;a href=&#34;https://huggingface.co/Qwen/Qwen3-8B-AWQ&#34;&gt;Qwen3-8B-AWQ&lt;/a&gt;, 4-bit), served by &lt;a href=&#34;https://github.com/vllm-project/vllm&#34;&gt;vLLM&lt;/a&gt; across both GPUs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chat interface&lt;/td&gt;
&lt;td&gt;&lt;a href=&#34;https://github.com/open-webui/open-webui&#34;&gt;Open WebUI&lt;/a&gt;, behind the college&amp;rsquo;s single sign-on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval&lt;/td&gt;
&lt;td&gt;Hybrid keyword and vector search over course slides, so answers can cite the course&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gateway&lt;/td&gt;
&lt;td&gt;A small proxy we wrote that sits between the chat interface and the model and enforces the tutor&amp;rsquo;s rules&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Code sandbox&lt;/td&gt;
&lt;td&gt;Runs a student&amp;rsquo;s program with no network, a read-only file system and strict time and memory limits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;A faculty lead and three undergraduate interns, each working a few hours a week, built it. Every change goes through code review, automated tests and the evaluations described below.&lt;/p&gt;
&lt;h2 id=&#34;lesson-1-size-for-many-users-at-once-not-for-one-fast-user&#34;&gt;Lesson 1: Size for many users at once, not for one fast user&lt;/h2&gt;
&lt;p&gt;Most published speed numbers for local LLMs measure one user at a time. An organization needs the opposite measurement: how long does each person wait when 20 or 30 colleagues are using the system at once?&lt;/p&gt;
&lt;p&gt;The metric we track is &lt;strong&gt;time to first token&lt;/strong&gt;, the wait between pressing Enter and seeing the first word of the reply. For a chat tool it is what users feel. Our target is a 95th percentile under 5 seconds: only 1 reply in 20 may take longer than that.&lt;/p&gt;
&lt;p&gt;Under load, the limit is usually not raw GPU speed but GPU &lt;em&gt;memory&lt;/em&gt;. Every active conversation needs working memory on the GPU, called the &lt;strong&gt;KV cache&lt;/strong&gt;, and it grows with the length of the conversation. Whatever memory the model&amp;rsquo;s own weights leave over decides how many conversations fit at once. Beyond that, new requests wait in line.&lt;/p&gt;
&lt;h2 id=&#34;lesson-2-measure-on-your-hardware-because-estimates-can-be-far-off&#34;&gt;Lesson 2: Measure on your hardware, because estimates can be far off&lt;/h2&gt;
&lt;p&gt;Before we touched the hardware, our sizing estimates said IBM&amp;rsquo;s &lt;a href=&#34;https://huggingface.co/ibm-granite/granite-4.1-8b&#34;&gt;Granite 4.1-8B&lt;/a&gt; would hold about 27 typical conversations of 3,000 tokens at once. On the real cards, it held about six.&lt;/p&gt;
&lt;p&gt;Here is vLLM&amp;rsquo;s own benchmark on our two GPUs. It is a deliberately harsh test: every request is a fresh 2,800-token prompt with nothing shared to reuse, followed by a 250-token reply.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Memory left for conversations&lt;/th&gt;
&lt;th&gt;Requests at once&lt;/th&gt;
&lt;th&gt;Median wait for first word&lt;/th&gt;
&lt;th&gt;95th-percentile wait&lt;/th&gt;
&lt;th&gt;Total output&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Granite 4.1-8B (16-bit)&lt;/td&gt;
&lt;td&gt;17,920 tokens (about 6 conversations)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.8 s&lt;/td&gt;
&lt;td&gt;1.8 s&lt;/td&gt;
&lt;td&gt;36 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;8.2 s&lt;/td&gt;
&lt;td&gt;20.6 s&lt;/td&gt;
&lt;td&gt;71 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;38.8 s&lt;/td&gt;
&lt;td&gt;43.3 s&lt;/td&gt;
&lt;td&gt;71 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-8B-AWQ (4-bit)&lt;/td&gt;
&lt;td&gt;93,248 tokens (about 30 conversations)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.5 s&lt;/td&gt;
&lt;td&gt;1.5 s&lt;/td&gt;
&lt;td&gt;58 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;4.3 s&lt;/td&gt;
&lt;td&gt;10.1 s&lt;/td&gt;
&lt;td&gt;111 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;5.2 s&lt;/td&gt;
&lt;td&gt;14.7 s&lt;/td&gt;
&lt;td&gt;121 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;24&lt;/td&gt;
&lt;td&gt;41.0 s&lt;/td&gt;
&lt;td&gt;60.4 s&lt;/td&gt;
&lt;td&gt;72 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Two models of the same size behaved very differently. The 4-bit quantized model left &lt;strong&gt;five times as much memory&lt;/strong&gt; for conversations and answered a single user &lt;strong&gt;1.6 times faster&lt;/strong&gt;. That settled our default model. Past about 16 simultaneous fresh prompts, both models slow down because the GPUs spend their time reading prompts. In real use this happens less often, because the system prompt and earlier turns of a conversation are cached and reused.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Older GPUs also rule out options you might be counting on.&lt;/strong&gt; Our 2018-era cards lack features newer models and tools assume:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;vLLM refused an 8-bit KV cache, which would have doubled capacity, because it needs a newer GPU generation.&lt;/li&gt;
&lt;li&gt;Google&amp;rsquo;s Gemma 4 models would not load in vLLM at all: their attention layers need more on-chip memory than these cards have.&lt;/li&gt;
&lt;li&gt;Large open models such as DeepSeek&amp;rsquo;s and Kimi&amp;rsquo;s flagships need hundreds of gigabytes of GPU memory, 15 to 70 times what we have, even at 4 bits.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;lesson-3-pick-the-serving-engine-for-concurrency&#34;&gt;Lesson 3: Pick the serving engine for concurrency&lt;/h2&gt;
&lt;p&gt;&lt;a href=&#34;https://github.com/ggml-org/llama.cpp&#34;&gt;llama.cpp&lt;/a&gt; is excellent for one person on a laptop or workstation, and it ran Gemma 4 12B, which vLLM could not. With 8 or 16 requests at once, though, the slowest users waited &lt;strong&gt;64 to 87 seconds&lt;/strong&gt; for the first word, because llama.cpp does not batch long prompts the way vLLM does. vLLM, which is built to batch many requests on the GPU, is our production engine. llama.cpp is our fallback and our CPU stand-in for development.&lt;/p&gt;
&lt;p&gt;The same caution applies to the many new &amp;ldquo;fastest local inference&amp;rdquo; projects. When I reviewed several of them for our hardware, their headline speeds came from one user at a time, often on newer or modified GPUs. A system that is fast for one user but serves only one at a time is worse for an office of 30.&lt;/p&gt;
&lt;h2 id=&#34;lesson-4-the-bottleneck-may-not-be-the-gpu-at-all&#34;&gt;Lesson 4: The bottleneck may not be the GPU at all&lt;/h2&gt;
&lt;p&gt;The most useful test we ran was a &lt;strong&gt;load test through the whole stack&lt;/strong&gt;: simulated students who sign in, ask questions about the course, and pause 8 to 25 seconds between messages, for 10 minutes. The first runs surprised us.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retrieval setup&lt;/th&gt;
&lt;th&gt;Simulated students&lt;/th&gt;
&lt;th&gt;95th-percentile wait for first word&lt;/th&gt;
&lt;th&gt;Replies in 10 minutes&lt;/th&gt;
&lt;th&gt;Errors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;No retrieval&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;0.4 s&lt;/td&gt;
&lt;td&gt;1,673&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retrieval with a reranker on CPU&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;284 s&lt;/td&gt;
&lt;td&gt;154&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No reranker, 10 candidate slides&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;17.9 s&lt;/td&gt;
&lt;td&gt;914&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No reranker, 6 candidates, more CPU threads for search&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;7.3 s&lt;/td&gt;
&lt;td&gt;1,042&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The same&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.6 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;850&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Without retrieval, the model and gateway served 50 students with a 0.4-second wait. Adding search over course slides pushed the wait to almost five minutes. The culprit was a &lt;strong&gt;reranker&lt;/strong&gt;, a second model that re-sorts search results for relevance, running on one of the CPU servers. Reranking one question took it about four seconds as deployed, and 50 students ask about three questions a second.&lt;/p&gt;
&lt;p&gt;Dropping the reranker was a real tradeoff. On our retrieval test, the best slide came first for 96% of questions with the reranker and 83% without it. However, the right slide was still among the five the tutor sees for 100% of questions, at about a sixth of the cost. We dropped it.&lt;/p&gt;
&lt;p&gt;Retrieval also changes the GPU&amp;rsquo;s workload. The tutor&amp;rsquo;s instructions, about 1,050 tokens, are cached and reused for every question. Retrieved slides add about 300 new tokens that must be read fresh each time, roughly ten times the prompt reading per reply. With 50 students that work keeps both GPUs busy, so for now &lt;strong&gt;35 students chatting at once&lt;/strong&gt; is the pilot&amp;rsquo;s capacity. Fifty is the goal once we see the real course materials.&lt;/p&gt;
&lt;p&gt;One error in those runs is worth mentioning because every multi-part system hits it eventually. Two components disagreed on how long to keep an idle connection open, and one reused a connection the other had just closed. A longer keep-alive setting fixed it. Load tests find problems like this; demos don&amp;rsquo;t.&lt;/p&gt;
&lt;h2 id=&#34;lesson-5-put-your-rules-in-a-gateway-you-own&#34;&gt;Lesson 5: Put your rules in a gateway you own&lt;/h2&gt;
&lt;p&gt;A chat interface can carry the right system prompt and settings, but users can often switch presets or call the model directly. So we wrote a small &lt;strong&gt;gateway&lt;/strong&gt;, an OpenAI-compatible proxy between the chat interface and the model. Every request passes through it, whatever settings the user picked. It:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;adds the tutor&amp;rsquo;s instructions to every request;&lt;/li&gt;
&lt;li&gt;turns off the model&amp;rsquo;s visible &amp;ldquo;thinking&amp;rdquo;, which in a tutor would show students the solution it is withholding;&lt;/li&gt;
&lt;li&gt;checks each reply for a near-complete solution before it reaches the student;&lt;/li&gt;
&lt;li&gt;runs student code in the sandbox when the tutor needs to see its output;&lt;/li&gt;
&lt;li&gt;logs what happened for auditing.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Because everything downstream talks to &amp;ldquo;the tutor&amp;rdquo; through the gateway, swapping the model is a configuration change plus a re-run of the evaluations, not a code change. For a business, the same layer is where you would remove personal information, enforce approved uses and keep an audit trail.&lt;/p&gt;
&lt;h2 id=&#34;lesson-6-test-behavior-not-just-speed&#34;&gt;Lesson 6: Test behavior, not just speed&lt;/h2&gt;
&lt;p&gt;Our central claim is that the tutor does not hand out answers, so we test exactly that. The leakage evaluation sends 180 attempts to extract a solution: 12 introductory problems, each asked 15 different ways, including direct requests, fake permission, role-play, asking for the answer piece by piece and &amp;ldquo;just fix my code.&amp;rdquo; Code in a reply that passes the problem&amp;rsquo;s own tests counts as a full leak. A held-out set of 120 more cases, which no prompt is tuned on, guards against teaching to the test.&lt;/p&gt;
&lt;p&gt;What we learned:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The model matters.&lt;/strong&gt; Our first prompt leaked full solutions in 45 of 180 cases (25%) on a small 3-billion-parameter model. The current prompt leaks none of the 180 on our 8B model, and 5 of the 120 held-out cases.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;More rules can make it worse.&lt;/strong&gt; A version that listed every forbidden kind of answer leaked more than the simpler one before it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Surrounding text matters.&lt;/strong&gt; With the chat interface&amp;rsquo;s default retrieval template, which asks for &amp;ldquo;a clear and direct response,&amp;rdquo; the tutor fixed students&amp;rsquo; buggy code in 7 of 12 cases where it otherwise refused. We rewrote the template.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Layers help.&lt;/strong&gt; With retrieval on, 2 of 180 cases leaked in full, and the gateway&amp;rsquo;s check caught both.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Every change to the prompt must pass this evaluation before it merges. A business should build the same kind of test for its own worst case, whether that is leaking a client&amp;rsquo;s data, inventing a policy or giving advice it is not allowed to give, and rerun it on every change to the model, prompt or retrieval setup.&lt;/p&gt;
&lt;h2 id=&#34;lesson-7-watch-the-defaults&#34;&gt;Lesson 7: Watch the defaults&lt;/h2&gt;
&lt;p&gt;Several of our risks came from defaults no one would notice in a demo:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Containers can bypass your firewall.&lt;/strong&gt; Docker writes its own firewall rules, so a published port can be reachable from the whole network even when the host firewall says no. Bind each service to a specific address, and scan from another machine to check.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Some components ship without authentication&lt;/strong&gt;, including popular vector databases. Turn it on.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Some apps phone home&lt;/strong&gt; with telemetry or update checks by default. In a &amp;ldquo;data never leaves&amp;rdquo; system, that has to be turned off and verified.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Untrusted code needs a real sandbox:&lt;/strong&gt; no network, a read-only file system, and process, memory and time limits.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Write down &amp;ldquo;no cloud fallback&amp;rdquo; as a requirement.&lt;/strong&gt; Otherwise someone will eventually add one to fix a capacity problem.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pin every version.&lt;/strong&gt; We never update the GPU server blindly during the semester.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Read the licenses.&lt;/strong&gt; Open WebUI&amp;rsquo;s license, for example, limits removing its branding above 50 users without an enterprise license.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Consumer hardware has limits of its own. Our GPU workstation has no error-correcting memory, no redundant power supply and no remote management. It puts out about 500 watts of heat under load in a desktop case. We monitor GPU temperature and power and keep encrypted backups. Before launch, we plan monthly drills for restoring from backup and for losing the GPU server.&lt;/p&gt;
&lt;h2 id=&#34;the-tradeoffs-in-one-table&#34;&gt;The tradeoffs in one table&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Local model on your hardware&lt;/th&gt;
&lt;th&gt;Cloud API&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Stays on your servers&lt;/td&gt;
&lt;td&gt;Leaves your network; governed by a contract&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model capability&lt;/td&gt;
&lt;td&gt;Limited by your GPU memory; an 8B model on our cards&lt;/td&gt;
&lt;td&gt;Frontier models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;Hardware, power and staff time; flat as usage grows&lt;/td&gt;
&lt;td&gt;Pay per use; nothing up front&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Capacity&lt;/td&gt;
&lt;td&gt;Fixed; measured in users at once&lt;/td&gt;
&lt;td&gt;Scales on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Control&lt;/td&gt;
&lt;td&gt;You pin versions and decide when anything changes&lt;/td&gt;
&lt;td&gt;The provider can change or retire models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operations&lt;/td&gt;
&lt;td&gt;You patch, monitor, back up and recover&lt;/td&gt;
&lt;td&gt;The provider does&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For a focused task like ours, a well-tested 8B model with good retrieval and firm guardrails is enough. For open-ended work that needs the strongest reasoning, a cloud model may still be the better tool. Many organizations will end up using both, with a clear rule about which data may go where.&lt;/p&gt;
&lt;h2 id=&#34;if-you-are-considering-this-for-your-organization&#34;&gt;If you are considering this for your organization&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Describe the workload first:&lt;/strong&gt; how many people at once, how long their questions are and how long they can wait.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Start small:&lt;/strong&gt; a 7 to 8 billion parameter open model, 4-bit quantized, on vLLM, is a sensible first try on modest GPUs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Benchmark the model on your hardware, then load-test the whole system&lt;/strong&gt; with realistic users. Our biggest surprise was not in the GPU.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Build an evaluation for your main failure mode&lt;/strong&gt; and run it on every change.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Put a gateway in front of the model&lt;/strong&gt; so your rules cannot be clicked around.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Lock down the defaults&lt;/strong&gt; and plan for the hardware failing.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Next for EMTutor: a pilot with students in an introductory course, real course materials in place of our synthetic ones, and tests over winter break of newer engine builds that could fit more conversations on the same cards. I will share updated numbers as they come in.&lt;/p&gt;
&lt;p&gt;If your organization is weighing a local deployment and you want to compare notes, you can reach me at lip AT etown DOT edu.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Coding Agents in the Small-College CS Classroom</title>
      <link>https://Peilong.github.io/post/teaching-cs-with-coding-agents/</link>
      <pubDate>Wed, 18 Nov 2026 00:00:00 +0000</pubDate>
      <guid>https://Peilong.github.io/post/teaching-cs-with-coding-agents/</guid>
      <description>&lt;p&gt;A year ago, the question in CS department hallways was whether students should be allowed to use ChatGPT on homework. That question is already out of date. Tools like Claude Code and OpenAI&amp;rsquo;s Codex are not chatbots that hand you a snippet. They are &lt;em&gt;agents&lt;/em&gt;: they read a whole project, edit files, run the tests, see what broke and try again. Brown University&amp;rsquo;s new intro course design puts the difference well. A chat tool produces code &amp;ldquo;without an underlying validation and revision loop&amp;rdquo;; an agent has that loop built in.&lt;/p&gt;
&lt;p&gt;For a typical assignment in an intro or mid-level course, an agent can produce working code in minutes. So the question for those of us who teach computing is no longer &lt;em&gt;whether&lt;/em&gt; students will use these tools. It is what a computer science education should build when typing code is no longer the hard part.&lt;/p&gt;
&lt;p&gt;I teach computer science at Elizabethtown College, a small liberal arts college in Pennsylvania, and I run an undergraduate research lab. Below is my reading of the evidence so far, and why I think small colleges are in an unusually good position to get this right.&lt;/p&gt;
&lt;h2 id=&#34;what-the-evidence-says-so-far&#34;&gt;What the evidence says so far&lt;/h2&gt;
&lt;p&gt;The research is young, but a consistent picture is forming.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;AI widens the gap between students.&lt;/strong&gt; In &amp;ldquo;The Widening Gap,&amp;rdquo; Prather and colleagues watched novices solve a programming problem with generative AI tools across 21 lab sessions, using interviews and eye tracking. Students who were already doing well used the tools to speed up code they had planned to write anyway. Students who struggled found their existing difficulties compounded. Many thought they had done better than they had, and &amp;ldquo;finished with an illusion of competence.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Handing off the work can cost the learning.&lt;/strong&gt; In a randomized trial Anthropic published in January 2026, 52 mostly junior engineers learned an unfamiliar Python library, with or without AI help. The AI group averaged about 50% on the follow-up quiz, against 67% for those who coded by hand, and it was not significantly faster. The biggest gap was on debugging questions. How people used the AI mattered, though: participants who asked conceptual follow-up questions retained far more than those who let the AI write or fix the code for them.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Feeling faster is not the same as being faster.&lt;/strong&gt; In METR&amp;rsquo;s 2025 randomized study, experienced open-source developers working in their own large codebases took about 19% longer when they could use early-2025 AI tools. Afterward, they still believed the tools had sped them up by about 20%. If seasoned professionals misjudge their own productivity that badly, we should not expect students to judge their own learning accurately.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The entry-level job market is shifting.&lt;/strong&gt; Using ADP payroll data, Brynjolfsson, Chandar and Chen at Stanford found a 13% relative decline in employment for 22- to 25-year-olds in the most AI-exposed occupations. Early-career software developers were among the hardest hit, and the declines were concentrated where AI automates work rather than assisting with it. Their monthly dashboard still shows the pattern. Our graduates will be competing for jobs where &amp;ldquo;I can write the code&amp;rdquo; is no longer enough.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Educators have not figured it out either.&lt;/strong&gt; An ACM task force surveyed 763 computing educators from 49 countries in 2025. Their top concern was students&amp;rsquo; growing dependency on the technology (87%). Of those who answered the question, 68% had already changed how they assess students. The most common barrier to adapting was a lack of examples of best practices (48%). Nobody has a playbook yet, so small programs can experiment without waiting for one.&lt;/p&gt;
&lt;h2 id=&#34;why-a-small-college-is-well-placed&#34;&gt;Why a small college is well placed&lt;/h2&gt;
&lt;p&gt;Most of the published experiments come from large universities. Yet nearly every promising idea in them depends on something small colleges already do well.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Small classes make oral assessment practical.&lt;/strong&gt; The most direct response to AI-written code is to ask students to explain it. Researchers at Utah State made a 15-minute code review interview mandatory after every weekly assignment in a 96-student CS1 course. Students could use AI as much as they liked, as long as they could justify their code in the interview. Running it took eleven trained teaching assistants. At a college where classes are small, the professor can hold many of those conversations in person. We also know our students well enough to notice when a student&amp;rsquo;s confidence has drifted away from their understanding, which is exactly the illusion of competence the research describes.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The liberal arts already teach the skills agents demand.&lt;/strong&gt; Working well with an agent is mostly an exercise in writing and judgment: stating a problem precisely, breaking it into steps, reading someone else&amp;rsquo;s work critically and defending a decision. Those are the habits a liberal arts education claims to build in every major. A clear specification is a piece of writing. A code review is an argument with evidence.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ethics is not a separate course.&lt;/strong&gt; Agents raise questions students will face on the job: who is responsible for code nobody typed, when and how to disclose AI use, which data should never be pasted into a third-party tool. Small programs can work these questions into every course instead of leaving them to a single ethics requirement.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Undergraduate research can be more ambitious.&lt;/strong&gt; With agents, a small team of undergraduates can build systems that would have been out of reach a few years ago. That raises the bar for what students need to manage: specifications, branches, code review, tests and continuous integration. In my lab, &lt;a href=&#34;https://bluejayadal.github.io/&#34;&gt;ADAL&lt;/a&gt;, the &lt;a href=&#34;https://bluejayadal.github.io/#skills&#34;&gt;list of what we look for in student researchers&lt;/a&gt; now starts with agentic coding and verifying what agents write, ahead of fluency in any particular language.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The constraints are real.&lt;/strong&gt; Small colleges do not have rooms full of teaching assistants, every new form of assessment costs faculty time, and agent access costs money. Brown gave the students in its Agentic Studio course $20-a-month Claude plans by design, and the instructors note that those limits &amp;ldquo;curtailed some of what we asked them to do.&amp;rdquo; If access depends on what each student can afford, these tools will widen exactly the gaps small colleges work hard to close. Colleges should plan to provide access rather than leave it to individual students.&lt;/p&gt;
&lt;h2 id=&#34;how-i-think-we-should-teach-with-agents&#34;&gt;How I think we should teach with agents&lt;/h2&gt;
&lt;p&gt;These are the principles I am planning around. None is fully proven, but all are consistent with the evidence above.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Fundamentals by hand first, agents later.&lt;/strong&gt; Writing code by hand is how students build mental models of how programs work, and the evidence says novices pay the highest price for handing that work off. Introductory courses should keep substantial hand-written work and in-person exams. Brown&amp;rsquo;s pilot intro course allows only chat tools for the first two months and introduces agents in the last month. Upper-level courses and capstones are where agents belong, used fully.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Grade understanding, not just the artifact.&lt;/strong&gt; If an agent can produce the artifact, the artifact alone tells me little about what the student learned. Short code walkthroughs, in-class problems and &amp;ldquo;change this function while I watch&amp;rdquo; questions show what a student actually understands. In the Utah State course, exam scores held steady, even rising slightly (though not significantly), while the share of pasted code went up.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Make verification a core skill.&lt;/strong&gt; Treat everything an agent writes as a draft. Students should write tests before asking an agent for code, read every diff, and be able to explain why the result is correct. The Anthropic study found the largest losses in debugging, which is exactly the skill needed to catch an agent&amp;rsquo;s mistakes. Debugging deserves more class time, not less.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Turn specifications and design into writing assignments.&lt;/strong&gt; Have students write the specification, the design rationale and the code review, and grade them as seriously as the code. This plays directly to a liberal arts college&amp;rsquo;s strengths.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Go bigger, earlier.&lt;/strong&gt; Agents make it realistic for students to work on larger, longer-lived projects much sooner. Use that to teach what employers now look for in new graduates: working in an existing codebase, Git workflows, code review and testing. Brown&amp;rsquo;s students &amp;ldquo;found significant value in the peer-review and crit activities.&amp;rdquo;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Ask for transparency.&lt;/strong&gt; Brown&amp;rsquo;s Agentic Studio required reflection journals and chat transcripts. In the instructors&amp;rsquo; early observations, most students &amp;ldquo;developed their own personal boundaries for when to use agents and for how much.&amp;rdquo; Asking students to show how they worked, not just what they produced, turns reflection into a habit.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&#34;the-case-for-the-liberal-arts-gets-stronger&#34;&gt;The case for the liberal arts gets stronger&lt;/h2&gt;
&lt;p&gt;Brown&amp;rsquo;s students named the risk themselves. According to the instructors, &amp;ldquo;a majority of students were concerned about their lack of understanding of their code bases,&amp;rdquo; and most worried about the loss of learning if agents arrived in intro courses. That worry is healthy. The goal is not to shield students from these tools, or to hand the tools over without a plan. It is to graduate people who can decide what software should exist, direct agents to build it, and show that it works.&lt;/p&gt;
&lt;p&gt;That has always been the promise of a liberal arts education: think clearly, write precisely, weigh evidence and take responsibility for your work. As coding agents take over more of the typing, those skills are no longer the soft side of a computer science degree. They are the job.&lt;/p&gt;
&lt;p&gt;If you teach computing at a small college and are trying any of this, I would like to compare notes. You can reach me at lip AT etown DOT edu.&lt;/p&gt;
&lt;h2 id=&#34;sources&#34;&gt;Sources&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Prather et al., &lt;a href=&#34;https://arxiv.org/abs/2405.17739&#34;&gt;The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers&lt;/a&gt; (2024).&lt;/li&gt;
&lt;li&gt;Shen and Tamkin, &lt;a href=&#34;https://www.anthropic.com/research/AI-assistance-coding-skills&#34;&gt;How AI assistance impacts the formation of coding skills&lt;/a&gt;, Anthropic (January 2026).&lt;/li&gt;
&lt;li&gt;Becker et al., &lt;a href=&#34;https://arxiv.org/abs/2507.09089&#34;&gt;Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity&lt;/a&gt;, METR (2025).&lt;/li&gt;
&lt;li&gt;Brynjolfsson, Chandar and Chen, &lt;a href=&#34;https://digitaleconomy.stanford.edu/project/indicators/canaries/&#34;&gt;Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence&lt;/a&gt;, Stanford Digital Economy Lab, and the &lt;a href=&#34;https://digitaleconomy.stanford.edu/project/canaries-dashboard/&#34;&gt;monthly dashboard&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Gordon et al., &lt;a href=&#34;https://acm-education-genai-task-force.github.io/ACM_Taskforce_GenAI_Report_16Feb26.pdf&#34;&gt;ACM Task Force on Generative AI and Programming Assessment: Final Report&lt;/a&gt; (February 2026).&lt;/li&gt;
&lt;li&gt;Fowles et al., &lt;a href=&#34;https://arxiv.org/abs/2605.21374&#34;&gt;Combating Harms of Generative AI in CS1 with Code Review Interviews and a Flipped Classroom&lt;/a&gt; (preprint, May 2026).&lt;/li&gt;
&lt;li&gt;Brown University, &lt;a href=&#34;https://www.brown.edu/news/2026-06-11/agentic-studio-ai-programming&#34;&gt;Brown professors devise course to explore generative AI in computer science education&lt;/a&gt; (June 2026); &lt;a href=&#34;https://cs.brown.edu/courses/csci1970kf/agentic-spr-2026/visitors.html&#34;&gt;Agentic Studio notes for visitors&lt;/a&gt;; and &lt;a href=&#34;https://cs.brown.edu/courses/csci0111e/fall26/course-design.html&#34;&gt;CSCI 0111E course design&lt;/a&gt; (Fall 2026).&lt;/li&gt;
&lt;/ul&gt;
</description>
    </item>
    
  </channel>
</rss>
