Ph. D. Project
Title:
Physics-informed data-driven system identification for data-efficient reinforcement learning for control
Dates:
2023/10/01 - 2026/09/30
Student:
Supervisor(s):
Description:
Modeling and simulation of systems are still of critical importance for characterizing the underlying dynamics of time-varying nonlinear system. A recent
solution has been to exploit emerging methods from machine learning and deep learning.
To facilitate the fundamental needs of system identification of time varying nonlinear dynamics, physics-informed neural networks (PINN) has emerged as a
hybrid deep learning paradigm that imbues machine learning models with physical domain knowledge such as physical laws and constraints emanating from
basic Newtonian physics-based laws (Raissi et al. 2019). Within this paradigm, neural networks are trained to solve supervised learning tasks while
respecting any given laws of physics described by general nonlinear differential equations (Stiasny et al. 2021).The distinguishing feature of PINN is the
synergistic integration of data-driven methods that reflect system dynamics in real-time with the governing physics.
Additionally, Koopman operator based approach for identification of nonlinear dynamics has garnered an extensive attention in the last 5 years (Bevanda et
al. 2021). The Koopman operator allows for handling nonlinear systems through a globally linear representation. In general, the operator is infinite-
dimensional - necessitating finite approximations - for which there is no overarching framework. Also, Koopman operator theory has long-standing
connections to known system-theoretic and dynamical system notions that are not universally recognized. Although there are principled ways of learning
such finite approximations, they are in many instances overlooked in favor of, often ill-posed and unstructured methods. When developed in conjunction
with Deep neural networks, the so called Deep Koopman operators have shown remarkable results in identifying nonlinear dynamics over finite linear
approximations (Lusch et al. 2018; Mauroy and Goncalves 2019).
However, so far, the potential of combining PINNs with Deep Koopman operators for identification of nonlinear dynamical systems has not been explored
in a detailed manner. It is believed that leveraging Koopman operator-based approach with PINNs shall result in high-fidelity and reduced models that are
well aligned with generally known physics properties as well as "interpretable".
On the other hand, recent advancements in the domain of adaptive dynamic programming and Reinforcement learning (ADP-RL) have led to remarkable
results in optimal control design for non-linear systems in the absence of system knowledge (complete or partial) (Kiumarsi et al. 2017). RL is a mature field
with well-established mathematical grounds for optimal (sub-optimal) control of non-linear dynamical systems in continuous as well as discrete time where
in the optimal control synthesis is largely based upon iterative solution for non-linear Hamilton-Jacobi-bellman equation (HJB) using neural network-based
structure (Mu et al. 2016). However, while the existing approaches provide exceptional asymptotic performance, trial-and-error based reinforcement
learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. This hampers the training quality as well as leads to
semi-global performance. Moreover, there is no insight/explanation about the system model learnt by the RL agents in an implicit manner. Similar is the
case with the nonlinear control policy learnt within the RL framework i.e., there is very little or no understanding available about the dynamics of the learnt
control policy. These aforementioned factors oblige training with excessive data and training duration (episodes) that are necessarily long.
Very recent efforts have been made by representing the environment dynamics as a linear dynamical system in a high-dimensional space (Han et al. 2020),
wherein the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning (Shi
and Meng 2022).
While the first phase of the thesis will target development of PINN enabled Deep Koopman operator-based approaches for nonlinear system identification,
the second phase will investigate the appropriate integration and exploitation of such models within the deep reinforcement learning framework to render
the (sub)optimal control learning process data-efficient and interpretable. Overall, the goal will be to use recent data-driven identification methods and
Koopman theory to build adaptive, real-time models that can be integrated with reinforcement learning control frameworks. The field of application will be
robotics.
References
Bevanda, Petar, Stefan Sosnowski, and Sandra Hirche. 2021. "Koopman Operator Dynamical Models: Learning, Analysis and Control." Annual Reviews in
Control 52: 197-212.
Kiumarsi, Bahare, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L Lewis. 2017. "Optimal and Autonomous Control Using Reinforcement
Learning: A Survey." IEEE Transactions on Neural Networks and Learning Systems 29 (6): 2042-62.
Lusch, Bethany, J Nathan Kutz, and Steven L Brunton. 2018. "Deep Learning for Universal Linear Embeddings of Nonlinear Dynamics." Nature
Communications 9 (1): 1-10.
Mu, Chaoxu, Zhen Ni, Changyin Sun, and Haibo He. 2016. "Data-Driven Tracking Control with Adaptive Dynamic Programming for a Class of
Continuous-Time Nonlinear Systems." IEEE Transactions on Cybernetics 47 (6): 1460-70.
Raissi, Maziar, Paris Perdikaris, and George E Karniadakis. 2019. "Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward
and Inverse Problems Involving Nonlinear Partial Differential Equations." Journal of Computational Physics 378: 686-707.
Shi, Haojie, and Max Q-H Meng. 2022. "Deep Koopman Operator with Control for Nonlinear Systems." IEEE Robotics and Automation Letters 7 (3):
7700-7707.
Stiasny, Jochen, George S Misyris, and Spyros Chatzivasileiadis. 2021. "Physics-Informed Neural Networks for Non-Linear System Identification for Power
System Dynamics." In 2021 IEEE Madrid PowerTech, 1-6.
solution has been to exploit emerging methods from machine learning and deep learning.
To facilitate the fundamental needs of system identification of time varying nonlinear dynamics, physics-informed neural networks (PINN) has emerged as a
hybrid deep learning paradigm that imbues machine learning models with physical domain knowledge such as physical laws and constraints emanating from
basic Newtonian physics-based laws (Raissi et al. 2019). Within this paradigm, neural networks are trained to solve supervised learning tasks while
respecting any given laws of physics described by general nonlinear differential equations (Stiasny et al. 2021).The distinguishing feature of PINN is the
synergistic integration of data-driven methods that reflect system dynamics in real-time with the governing physics.
Additionally, Koopman operator based approach for identification of nonlinear dynamics has garnered an extensive attention in the last 5 years (Bevanda et
al. 2021). The Koopman operator allows for handling nonlinear systems through a globally linear representation. In general, the operator is infinite-
dimensional - necessitating finite approximations - for which there is no overarching framework. Also, Koopman operator theory has long-standing
connections to known system-theoretic and dynamical system notions that are not universally recognized. Although there are principled ways of learning
such finite approximations, they are in many instances overlooked in favor of, often ill-posed and unstructured methods. When developed in conjunction
with Deep neural networks, the so called Deep Koopman operators have shown remarkable results in identifying nonlinear dynamics over finite linear
approximations (Lusch et al. 2018; Mauroy and Goncalves 2019).
However, so far, the potential of combining PINNs with Deep Koopman operators for identification of nonlinear dynamical systems has not been explored
in a detailed manner. It is believed that leveraging Koopman operator-based approach with PINNs shall result in high-fidelity and reduced models that are
well aligned with generally known physics properties as well as "interpretable".
On the other hand, recent advancements in the domain of adaptive dynamic programming and Reinforcement learning (ADP-RL) have led to remarkable
results in optimal control design for non-linear systems in the absence of system knowledge (complete or partial) (Kiumarsi et al. 2017). RL is a mature field
with well-established mathematical grounds for optimal (sub-optimal) control of non-linear dynamical systems in continuous as well as discrete time where
in the optimal control synthesis is largely based upon iterative solution for non-linear Hamilton-Jacobi-bellman equation (HJB) using neural network-based
structure (Mu et al. 2016). However, while the existing approaches provide exceptional asymptotic performance, trial-and-error based reinforcement
learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. This hampers the training quality as well as leads to
semi-global performance. Moreover, there is no insight/explanation about the system model learnt by the RL agents in an implicit manner. Similar is the
case with the nonlinear control policy learnt within the RL framework i.e., there is very little or no understanding available about the dynamics of the learnt
control policy. These aforementioned factors oblige training with excessive data and training duration (episodes) that are necessarily long.
Very recent efforts have been made by representing the environment dynamics as a linear dynamical system in a high-dimensional space (Han et al. 2020),
wherein the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning (Shi
and Meng 2022).
While the first phase of the thesis will target development of PINN enabled Deep Koopman operator-based approaches for nonlinear system identification,
the second phase will investigate the appropriate integration and exploitation of such models within the deep reinforcement learning framework to render
the (sub)optimal control learning process data-efficient and interpretable. Overall, the goal will be to use recent data-driven identification methods and
Koopman theory to build adaptive, real-time models that can be integrated with reinforcement learning control frameworks. The field of application will be
robotics.
References
Bevanda, Petar, Stefan Sosnowski, and Sandra Hirche. 2021. "Koopman Operator Dynamical Models: Learning, Analysis and Control." Annual Reviews in
Control 52: 197-212.
Kiumarsi, Bahare, Kyriakos G Vamvoudakis, Hamidreza Modares, and Frank L Lewis. 2017. "Optimal and Autonomous Control Using Reinforcement
Learning: A Survey." IEEE Transactions on Neural Networks and Learning Systems 29 (6): 2042-62.
Lusch, Bethany, J Nathan Kutz, and Steven L Brunton. 2018. "Deep Learning for Universal Linear Embeddings of Nonlinear Dynamics." Nature
Communications 9 (1): 1-10.
Mu, Chaoxu, Zhen Ni, Changyin Sun, and Haibo He. 2016. "Data-Driven Tracking Control with Adaptive Dynamic Programming for a Class of
Continuous-Time Nonlinear Systems." IEEE Transactions on Cybernetics 47 (6): 1460-70.
Raissi, Maziar, Paris Perdikaris, and George E Karniadakis. 2019. "Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward
and Inverse Problems Involving Nonlinear Partial Differential Equations." Journal of Computational Physics 378: 686-707.
Shi, Haojie, and Max Q-H Meng. 2022. "Deep Koopman Operator with Control for Nonlinear Systems." IEEE Robotics and Automation Letters 7 (3):
7700-7707.
Stiasny, Jochen, George S Misyris, and Spyros Chatzivasileiadis. 2021. "Physics-Informed Neural Networks for Non-Linear System Identification for Power
System Dynamics." In 2021 IEEE Madrid PowerTech, 1-6.
Keywords:
System identification, physics-informed neural networks, deep learning, reinforcement learning
Department(s):
| Control Identification Diagnosis |
