# LLM4KT: A Hierarchical Framework for Knowledge Tracing with Large Language Models

This repository contains the official implementation of **LLM4KT**, a novel hierarchical framework that leverages Large Language Models for Knowledge Tracing tasks.

## Overview

Knowledge tracing (KT) aims to model students' knowledge states and predict their future performance over time. While Large Language Models (LLMs) have shown promise in various fields, existing LLM-based KT methods demonstrate limited performance gains due to the bottleneck of processing lengthy text descriptions of students' interaction sequences. 

To address this challenge, we propose **LLM4KT**, a hierarchical framework with:
- An **Inter LLM** for semantic compression of textual interaction descriptions
- A **Tracer LLM** for temporal modeling of students' evolving knowledge states
- A novel **token reweighting mechanism** that dynamically identifies and preserves critical information during compression
- An **auxiliary question recall task** to enhance the model's memory of prior interactions

## Key Features

- **Hierarchical Architecture**: Decomposes semantic understanding and temporal modeling into two specialized LLM layers
- **Token Reweighting Mechanism**: Gradient-based dynamic token reweighting to preserve critical semantic information
- **Auxiliary Task Design**: Question recall task to explicitly enhance model memory
- **State-of-the-Art Performance**: Achieves new SOTA results on five benchmark datasets
- **Few-shot Robustness**: Maintains strong performance even with limited training data

## Model Architecture

<p align="center">
  <img src="paper/model4.png" alt="LLM4KT Framework Architecture" width="50%">
</p>

The figure illustrates the hierarchical architecture of LLM4KT:
- **(a)** Overall framework showing Inter LLM and Tracer LLM components
- **(b)** Query LLM and Recall LLM for task-specific compression
- **(c)** Different prompt templates for Inter LLM, Query LLM, and Recall LLM

## Quick Start

For reviewers who want to quickly run the model:

```bash
# 1. Clone the repository
git clone https://github.com/UltramanSleepless/IJCAI.git
cd IJCAI

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run training with default configuration (LLM4KT-4.6B)
export CUDA_VISIBLE_DEVICES=0
python main.py --plm qwen_id --embed_lm qwen_embed --model LLM4KT_pid --dataset assist2009_pid --lora yes

```

For a quick test with smaller models:
```bash
# Use BERT-base as Inter LLM (smaller memory footprint)
python main.py --plm qwen_id --embed_lm bert --model LLM4KT_pid --dataset assist2009_pid --lora yes
```

## Requirements

The code has been tested with:
- Python 3.10
- PyTorch 2.4.0
- CUDA 12.1
- NVIDIA A800 GPUs (80GB)

### Installation

1. Clone the repository:
```bash
git clone https://github.com/UltramanSleepless/IJCAI.git
cd IJCAI
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Download pre-trained language models:

The code will automatically download required models from Hugging Face on first run. Alternatively, you can manually download the models:

- **Qwen3-Embed-0.6B**: [Qwen/Qwen3-Embed-0.6B](https://modelscope.cn/models/Qwen/Qwen3-Embedding-0.6B)
- **Qwen3-Embed-4B**: [Qwen/Qwen3-Embed-4B](https://modelscope.cn/models/Qwen/Qwen3-Embedding-4B)
- **Qwen3-4B**: [Qwen/Qwen3-4B](https://modelscope.cn/models/Qwen/Qwen3-4B)
- **Qwen3-1.7B**: [Qwen/Qwen3-1.7B](https://modelscope.cn/models/Qwen/Qwen3-1.7B)
- **Qwen3-0.6B**: [Qwen/Qwen3-0.6B](https://modelscope.cn/models/Qwen/Qwen3-0.6B)
- **BERT-base**: [bert-base-uncased](https://huggingface.co/bert-base-uncased)
- **BERT-large**: [bert-large-uncased](https://huggingface.co/bert-large-uncased)

## Datasets

We evaluate LLM4KT on five widely-used benchmark datasets:
- **Assist09**: 3,644 students, 17,727 questions, 123 concepts, 281,890 interactions
- **Algebra05**: 571 students, 173,113 questions, 112 concepts, 607,014 interactions
- **Statics**: 333 students, 1,223 questions, 189,297 interactions
- **Ednet**: 5,000 students, 12,117 questions, 189 concepts, 676,276 interactions
- **Spanish**: 182 students, 409 questions, 221 concepts, 578,726 interactions

**Comparison with 13 Baseline Models:**
- **Non-LLM KT models (9)**: DKT, SAKT, AKT, ATKT, DTransformer, SimpleKT, FoLiBiKT, SparseKT, DisKT
- **LLM-based KT models (4)**: FT Ext, LOKT, CLST, LKT

### Data Preparation

1. Download the datasets from [https://base.ustc.edu.cn/data](https://base.ustc.edu.cn/data)

2. Place the downloaded datasets in the `DATASETS/` directory with the following structure:
```
DATASETS/
├── assist2009_pid/
├── algebra05_pid/
├── statics/
├── ednet_pid/
└── spanish_pid/
```



## License

Academic research use only.
