LLM Boost

Abstract

Large Language Models (LLMs) are emerging as promising tools for automated reinforcement learning (RL) reward design, owing to their robust capabilities in commonsense reasoning and code generation. By engaging in dialogues with RL agents, LLMs construct a Reward Observation Space (ROS) by selecting relevant environment states and defining their internal operations. However, existing frameworks have not effectively leveraged historical exploration data or manual task descriptions to iteratively evolve this space. In this paper, we propose a novel heuristic framework that enhances LLM-driven reward design by evolving the ROS through a table-based exploration caching mechanism and a text-code reconciliation strategy. Our framework introduces a state execution table, which tracks the historical usage and success rates of environment states, overcoming the Markovian constraint typically found in LLM dialogues and facilitating more effective exploration. Furthermore, we reconcile user-provided task descriptions with expert-defined success criteria using structured prompts, ensuring alignment in reward design objectives. Comprehensive evaluations on benchmark RL tasks demonstrate the effectiveness and stability of the proposed framework.

Our Rewards and Policies

In this demo, we visualize our unmodified best reward for each environment and the policy trained using this reward. Our environment suite spans 20 distinct tasks across the Bidexterous Manipulation (Dexterity) benchmark.

Dexterity

<b>ShadowHandBlockStack</b>, Our best reward:
[sep]
assets/reward_functions/BlockStack.txt

<b>ShadowHandBottleCap</b>, Our best reward:
[sep]
assets/reward_functions/BottleCap.txt

<b>ShadowHandCatchAbreast</b>, Our best reward:
[sep]
assets/reward_functions/CatchAbreast.txt

<b>ShadowHandCatchOver2Underarm</b>, Our best reward:
[sep]
assets/reward_functions/CatchOver2Underarm.txt

<b>ShadowHandCatchUnderarm</b>, Our best reward:
[sep]
assets/reward_functions/CatchUnderarm.txt

<b>ShadowHandDoorCloseInward</b>, Our best reward:
[sep]
assets/reward_functions/DoorCloseInward.txt

<b>ShadowHandDoorCloseOutward</b>, Our best reward:
[sep]
assets/reward_functions/DoorCloseOutward.txt

<b>ShadowHandDoorOpenInward</b>, Our best reward:
[sep]
assets/reward_functions/DoorOpenInward.txt

<b>ShadowHandDoorOpenOutward</b>, Our best reward:
[sep]
assets/reward_functions/DoorOpenOutward.txt

<b>ShadowHandGraspAndPlace</b>, Our best reward:
[sep]
assets/reward_functions/GraspAndPlace.txt

<b>ShadowHandKettle</b>, Our best reward:
[sep]
assets/reward_functions/Kettle.txt

<b>ShadowHandLiftUnderarm</b>, Our best reward:
[sep]
assets/reward_functions/LiftUnderarm.txt

<b>ShadowHandOver</b>, Our best reward:
[sep]
assets/reward_functions/Over.txt

<b>ShadowHandPen</b>, Our best reward:
[sep]
assets/reward_functions/Pen.txt

<b>ShadowHandPushBlock</b>, Our best reward:
[sep]
assets/reward_functions/PushBlock.txt

<b>ShadowHandReorientation</b>, Our best reward:
[sep]
assets/reward_functions/ReOrientation.txt

<b>ShadowHandScissors</b>, Our best reward:
[sep]
assets/reward_functions/Scissors.txt

<b>ShadowHandSwingCup</b>, Our best reward:
[sep]
assets/reward_functions/SwingCup.txt

<b>ShadowHandSwitch</b>, Our best reward:
[sep]
assets/reward_functions/Switch.txt

<b>ShadowHandTwoCatchUnderarm</b>, Our best reward:
[sep]
assets/reward_functions/TwoCatchUnderarm.txt

Select an image above:

The reward function from the selected task will be shown here.

BibTeX

@article{ma2024xxxxx,
    title   = {Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution},
    author  = {Zen Kit Heng and Zimeng Zhao and Tianhao Wu and Yuanfei Wang and Mingdong Wu and Yangang Wang and Hao Dong},
    year    = {2024},
    journal = {arXiv preprint arXiv: Comming Soon}
}