Skip to content
Archived
AI/ML
Open Source
Mar 2024

Talker

An open source teaching assistant, built as a USC master's capstone, that drafts answers to course-forum questions from the lecture notes, with hybrid retrieval, a reranker, and an evaluation harness that scores its own retrieval.

BM25 + vector
Retrieval
6
Providers
LLM-as-judge
Evaluation
MS capstone
Context
Talker project visual

The constraint

Students wait on course forums for answers that are already somewhere in the lecture notes. A chatbot that answers from general knowledge is worse than waiting: it is confident about a course it has never seen.

The approach

A retrieval-augmented assistant over the course's own material. Questions run through hybrid retrieval (BM25 plus vector search in ChromaDB), a cross-encoder reranker and LLM query expansion, and the answer is drafted from what was retrieved. An optional Piazza bot closes the loop on the course forum itself.

Architecture
Hybrid retrieval over the course material, reranked before anything is generated.
High level
StudentForum questionRetrieverBM25 + ChromaDB vectorsSourcesLecture notes, syllabusGeneratorSix providers, one interfaceAnswerCitationshybrid searchgroundingrerankedfinalEvaluationLLM-as-judge harnessscored
Process
  1. 1
    Retrieval before generation

    Combined lexical and semantic search in one ensemble retriever, because course vocabulary is exactly the kind of term pure embeddings blur.

  2. 2
    Rerank the shortlist

    Added a cross-encoder pass over the retrieved passages so the context window holds the best few, not merely the first few.

  3. 3
    Measure the retriever

    Built a RAGAS-style evaluation harness so a change to chunking or retrieval comes with a score, not an impression.

  4. 4
    Meet students where they ask

    Wired an optional Piazza bot so drafted answers land on the course forum instead of in a separate app nobody opens.

Outcomes

  • Hybrid BM25 + vector retrieval, so exact course terms and paraphrased questions both find their passage
  • Six model providers behind one interface — OpenAI, Anthropic, Gemini, Cohere, Ollama and HuggingFace — with Ollama for a fully local setup, and token cost tracked per provider
  • A RAGAS-style LLM-as-judge harness that scores retrieval and answer quality instead of eyeballing it
Details

Built a retrieval-augmented teaching assistant as a USC MS capstone, with hybrid BM25 + vector retrieval over ChromaDB, a cross-encoder reranker, and six interchangeable model providers.

Added a RAGAS-style LLM-as-judge evaluation harness and an optional Piazza bot that drafts answers on the course forum.