PIDM: Predictive Inverse Dynamic Models

What are Predictive Inverse Dynamic Models?

Imitation learning enables agents to learn complex behaviour from demonstrations, but in practice it often requires large datasets that are costly or impractical to collect. Our project studies how to make imitation learning significantly more data-efficient, enabling agents to learn effectively from limited demonstrations in complex, high-dimensional environments. We use real-world problems such as modern video games as testbeds to develop and study these methods under realistic conditions.

How it works

Most imitation learning approaches take a direct route: given a current situation, they ask “what action would an expert take here?” and try to predict these actions. Predictive inverse dynamics models (PIDMs), also known as world action models (WAMs), instead ask a goal-directed question, splitting the problem in two:

• Predict a desirable future — what should happen next?
• Act to get there — what action takes the agent from the current situation toward that future?

By first deciding where to head next and then choosing the action that goes there, PIDMs add a sense of direction that directly predicting actions can lack.

Why it helps: Expert behaviour is ambiguous, since the expert might take one of many reasonable actions in a situation depending on what they want to achieve. Standard approaches that map observations directly to actions still have to account for how the future will unfold in order to determine an appropriate action, but they do so implicitly. PIDMs make this reasoning explicit instead: by learning where to head next within any situation and grounding actions in these desired future situations, they leverage more information from each demonstration. Our theoretical analysis and empirical results show that this reduces ambiguity and lets agents learn effective strategies from fewer demonstrations.