Self-CriTeach

Abstract

Large Language Models (LLMs) have recently shown strong promise for robotic task planning, particularly through automatic planning domain generation. However, planning domains are brittle under imperfect logical states and perception noise. Prior approaches largely treat generated planning domains as plan utilities, overlooking their potential as scalable sources of reasoning supervision and structured reward signals. At the same time, reasoning LLMs depend on chain-of-thought (CoT) supervision that is expensive to collect for robotic tasks, and reinforcement learning (RL) faces challenges on reward engineering.

We propose Self-CriTeach, an LLM self-teaching and self-critiquing framework in which an LLM autonomously generates symbolic planning domains that serve a dual role: (i) enabling large-scale generation of robotic planning problem–plan pairs, and (ii) providing structured reward functions. First, the self-written domains enable large-scale generation of symbolic task plans, which are automatically transformed into extended CoT trajectories for supervised fine-tuning. Second, the self-written domains are reused as structured reward functions, providing dense feedback for reinforcement learning without manual reward engineering. This unified training pipeline yields a planning-enhanced LLM with higher planning success rates, stronger cross-task generalization, reduced inference cost, and resistance to imperfect logical states.

Key Contributions

Method Overview

1. Automatic PDDL Domain Generation

The base LLM generates PDDL planning domains from task demonstrations, with iterative refinement through validation-repair-pruning loops to ensure domain correctness and compactness.

2. Symbolic-to-CoT Transformation

Symbolic plans are converted into chain-of-thought representations by eliciting natural language reasoning that includes:

• Plan explanation: Justification for each action selection
• State transition checking: Verification of preconditions and effects
• Alternative exploration: Reasoning about other applicable actions
• Failure backtracking: Analysis of infeasible branches

3. Supervised Fine-Tuning

The model is trained on tuples ⟨Problem, Plan, CoT⟩ using standard language modeling loss, learning to generate both action sequences and explanatory reasoning.

4. Reinforcement Learning with Structured Rewards

The self-generated planning domain provides fine-grained reward signals:

• Step-level rewards: R(X_t, a_t) = |X^goal ∩ X_t| / |X^goal|
• Constraint costs: Penalties for precondition violations and inconsistent states
• CPO optimization: Constrained policy optimization with explicit feasibility constraints

Experimental Results

Main Results: Model Performance

Per-Domain Performance (Qwen3-4B-P2E)

Key Findings

• Improved Success Rates: SCT-4B achieves 32.9% Pass@1, a 31% improvement over the baseline Qwen3-4B-Instruct (25.1%)
• Token Efficiency: 35% reduction in average token usage (21M vs 33M tokens) while maintaining higher accuracy
• Generalization: Strong performance on unseen domains (pack_machining_parts, prepare_experiment, reorganize_room)
• Legal Plan Generation: 93.0% overall legal plan generation rate across all domains
• Scalability: Training on 7,476 self-generated samples achieves significant improvements without human annotation

Code & Resources

Quick Start

# Install
pip install -r requirements.txt

# Load model
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("self-criteach/SCT-4B")
tokenizer = AutoTokenizer.from_pretrained("self-criteach/SCT-4B")

# Generate plan
problem = "Static predicates: [[box, 1], [box, 2], [table, 0], [robot, r]]..."
inputs = tokenizer(problem, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=2048)
plan = tokenizer.decode(outputs[0], skip_special_tokens=True)

Citation

If you find this work useful, please cite:

@inproceedings{huang2026selfcriteach,
  title={Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation},
  author={Huang, Jinbang and Li, Zhiyuan and Hu, Yuanzhao and Zhang, Zhanguang and Coates, Mark and Quan, Xingyue and Zhang, Yingxue},
  booktitle={ICLR 2026 Workshop on AI with Recursive Self-Improvement},
  year={2026},
  organization={Huawei Noah's Ark Lab}
}

Team