Here’s a classroom idea that doubles as a reality check for anyone using large language models: a short quiz where students must pick between their own score or an LLM’s score—before seeing the questions. It’s a playful setup with a serious message about capabilities, limits, and ethical use.
- A dual-enrollment communications instructor plans a live demonstration on using LLMs ethically and effectively.
- Students get a 5-question general knowledge quiz (math, history, science, geography, language arts).
- Before seeing the quiz, each student chooses to keep either their own score or the LLM’s score.
- The quiz is designed so humans should find it easy, while a text-only LLM is likely to stumble.
- Goal: teach the danger of over-relying on LLMs and highlight what they genuinely do well for students.
Why this demo resonates with developers and educators
At AI Tech Inspire, this format stood out because it mirrors day-to-day engineering tradeoffs: when to trust automation and when to rely on direct observation or tools better suited for the job. LLMs like GPT advertise generality, but they are not oracles. Depending on configuration, they may lack browsing, calculators, local context, and up-to-date knowledge. Demonstrations like this make those boundaries visible—fast.
For developers, it’s a microcosm of product reality: the model’s raw reasoning is powerful, yet brittle. Without retrieval, a database, or a tool-use layer, an LLM can sound convincing and still be wrong. Engineers who’ve shipped systems with TensorFlow, PyTorch, or Hugging Face stacks know this well—capability comes from the entire pipeline, not just the model.
Designing “easy for humans, hard for LLMs” questions
To keep the demo fair and instructive, assume a text-only LLM with no internet, no calculator, and no code execution. Then lean into tasks where humans have a native advantage: real-world perception, recency, strict counting, and exact computation.
Sample prompts across five categories:
- History/current events: “Who currently holds [local office] as of this week?” or “Which team won last night’s [local game]?” These hinge on recency and local context an offline LLM won’t have.
- Language arts: “How many letter ‘y’s are in ‘yearly’?” or “Count the letter ‘e’ in this paragraph: ‘[paste paragraph here].’” Strict character counting is error-prone for LLMs, especially on longer strings.
- Math: “What day of the week will it be 10,000 days from today?” Without a calendar tool, the probability of error is high. Or try “Sum of the first 200 primes”—trivial with code, tricky in pure text.
- Science: “What is the approximate pH of freshly exposed distilled water at room temperature?” Humans familiar with CO₂ dissolution might answer near 5.6; LLMs often default to 7 without nuance.
- Geography: “Which is farther west: Reno or Los Angeles?” Questions that subvert intuition still trip models in zero-tool settings.
Bonus category—local perception:
- “What color is the exit sign in this room?”
- “What’s the third poster on the left wall promoting?”
- “What’s written on the classroom whiteboard right now?”
These are trivially easy for students but impossible for a text-only LLM. The lesson is simple: if you can see it or just read it yourself, you’re faster and more accurate than a general-purpose model.
Key takeaway: LLMs are strong at language and patterns, not omniscience, perception, or exact arithmetic without tools.
Why models fail here (and how engineers mitigate it)
Under the hood, these models predict tokens; they excel at patterns, not precise stateful computation. Without a tools layer, they can’t:
- Fetch fresh data (no browsing or RAG).
- See the room (no camera or multimodal input).
- Execute code or do exact math (no calculator or Python).
- Enforce deterministic operations like strict counting and sorting of long strings.
Engineering teams routinely pair models with retrieval, function calling, and verification. For example, a classroom assistant could combine GPT with retrieval from a syllabus, or pair a model with a calculator for robust math. Think of the model as the language layer in a system that also uses CUDA-accelerated inference, databases, and guardrails—much like image generation stacks around Stable Diffusion.
What LLMs do exceptionally well for students (ethically)
On the flip side, there are high-value, low-risk uses that consistently help learners—and save grading time.
- Explainers and refactors: Ask for two explanations of the same idea: one simple, one in formal academic style. Great for scaffolding.
- Outlining and planning: Turn a prompt into a project outline, a study plan, or a research roadmap. Humans fill in substance.
- Socratic tutoring: Prompt the model to ask guiding questions instead of giving answers:
Teach me via questions. Don’t reveal the answer unless I ask. Use hints. - Feedback on drafts: Evaluate clarity, coherence, tone, and structure against a supplied rubric. Avoid requesting factual citations from the model—use it to critique, not to fabricate sources.
- Style transformations: Convert notes to a formal memo, or a dense paragraph into bullet points. Use Ctrl + F to quickly spot suggested revisions.
- Practice dialogues: Role-play scenarios: interviews, debate prep, presentations. Students iterate without pressure.
- Regex and code hints: For CS classes, models shine at boilerplate, regex drafting, and error explanation—but verify outputs by running tests.
- Summarization of provided texts: Paste the passage; ask for a summary with key claims, assumptions, and counterarguments.
Ethics in one line: use LLMs to think better, not to pretend you didn’t need to think.
A concrete, classroom-ready runbook
Here’s a practical flow educators can replicate:
- 1) Set the stage: Explain that today’s LLM is text-only—no browsing, no calculator. Make the constraint visible.
- 2) Commitment device: Students write in pen whether they’ll keep their own score or the model’s.
- 3) Reveal the quiz: Use a mix of local-perception, recency, counting, and exact arithmetic items. Keep them genuinely easy for humans.
- 4) Run the model: Ask the LLM in one shot. No retries. No tools. Record its answers.
- 5) Debrief: Discuss each miss and why the system is limited. Then show where it would excel—e.g., revise a student paragraph live.
- 6) Ethics wrap: Set class rules: disclose AI use, never fabricate citations, verify facts, and use models as tutors or editors—not as authors of graded work.
That final step matters. Students should see that turning to an LLM for a quote, an outline, or a Socratic hint is ethical; submitting unverified, model-written work is not.
Sample quizzes you can copy-paste
Scenario A (no tools):
- History/current events: “Who currently holds [local office] as of this week?”
- Language arts: “How many ‘y’s are in ‘yearly’?”
- Math: “What day of the week will it be exactly 10,000 days from today?”
- Science: “Approximate pH of distilled water exposed to air at room temperature?”
- Geography: “Which is farther west: Reno or Los Angeles?”
Scenario B (local perception):
- “What’s written on the poster nearest the door?”
- “How many ceiling lights are on right now?”
- “What is today’s cafeteria main dish?”
- “What color is the teacher’s laptop case?”
- “What is the second item listed on the class agenda?”
For each, the LLM is likely to produce confident but incorrect answers. That confidence is the teachable moment.
For developers: turn this into a product insight
Engineers can read this as a template for capability scoping. If a feature depends on recency, perception, or exact computation, add retrieval, vision, or tools. If you need strict string operations, prefer code execution or deterministic functions. The classroom quiz reflects the same principle used in production systems: make failure modes visible, then design mitigations.
One more thought for the builders: consider a dual-mode assistant—freeform chat for brainstorming, and a locked-down, tool-backed mode for facts, math, and citations. The switch should be explicit to users. It’s the same mental model students learn in this demo.
Closing thought
LLMs are astonishing language partners, not all-seeing minds. A short quiz can prove that to students in minutes—and that clarity helps them use AI more effectively and ethically. Use models for patterns, drafts, feedback, and practice. Use humans (and tools) for perception, fresh facts, and exact answers. That division of labor is where learning—and great engineering—actually thrive.
Recommended Resources
As an Amazon Associate, I earn from qualifying purchases.
