Supervisor

Suitable for

Abstract

Supervised by Andrey Kravchenko andrey.kravchenko@cs.ox.ac.uk

Expected background of students and CS techniques that will be applied.

The student will know how to work or be able to quickly learn how to work with Pytorch, Hugging Face, IPython notebooks. They should also have a solid mathematical base.

Task-Tailored Schema Agents for Reliable LLM Automation

Design an agent that, given a task description and a small sample of target data, automatically proposes a fit-for-purpose schema and compiles a robust extraction/automation pipeline, then validates, executes, and self-improves it. The agent will (i) induce a schema (fields, types, constraints, relations) from task instructions and examples; (ii) compile this schema into a structured-output program with constrained decoding; and (iii) verify—and, when available, link—outputs against a reference ontology or database.  

To harden reliability, add an agentic refinement loop (Self-Refine; Reflexion): when coverage or validity drops, the agent critiques its outputs and edits the schema before re-running. Implement the workflow in DSPy, allowing the compiler to tune prompts, demonstrations, and tool ordering to the target metric (e.g., F1 × validity × latency). Evaluate across 2–3 domains (e.g., scientific IE, event extraction) using schema coverage/minimality, per-field F1, JSON/ontology validity, and robustness to small task shifts.

Outcomes: (1) an open-source agent that induces, enforces, and verifies task-specific schemas from a handful of examples; (2) measurable gains in structural validity and downstream F1 from combining constrained decoding with ontology-based verification; and (3) ablations, an error taxonomy, and a small reusable schema library plus tuned DSPy pipelines.