LEARNING PARTICLE DYNAMICS FOR MANIPULATING RIGID BODIES, DEFORMABLE OBJECTS, AND FLUIDS

Transcription

1 LEARNING PARTICLE DYNAMICS FOR MANIPULATING RIGID BODIES, DEFORMABLE OBJECTS, AND FLUIDS Anonymous auhors Paper under double-blind review ABSTRACT Real-life conrol asks involve maer of various subsances rigid or sof bodies, liquid, gas each wih disinc physical behaviors. This poses challenges o radiional rigid-body physics engines. Paricle-based simulaors have been developed o model he dynamics of hese complex scenes; however, relying on approximaion echniques, heir simulaion ofen deviaes from real world physics, especially in he long erm. In his paper, we propose o learn a paricle-based simulaor for complex conrol asks. Combining learning wih paricle-based sysems brings in wo major benefis: firs, he learned simulaor, jus like oher paricle-based sysems, acs widely on objecs of differen maerials; second, he paricle-based represenaion poses srong inducive bias for learning: paricles of he same ype have he same dynamics wihin. This enables he model o quickly adap o new environmens of unknown dynamics wihin a few observaions. Using he learned simulaor, robos have achieved success in complex manipulaion asks, such as manipulaing fluids and deformable foam. The effeciveness of our mehod has also been demonsraed in he real world. Our sudy helps lay he foundaion for robo learning of dynamic scenes wih paricle-based represenaions. 1 INTRODUCTION Objecs have disinc dynamics. Under he same push, a rigid box will slide, modeling clay will deform, and a cup full of waer will fall wih waer spilling ou. The diverse behavior of differen objecs poses challenges o radiional rigid-body simulaors used in roboics (Todorov e al., 2012). Paricle-based simulaors aim o model he dynamics of hese complex scenes (Macklin e al., 2014); however, relying on approximaion echniques for he sake of percepual realism, heir simulaion ofen deviaes from real world physics, especially in he long erm. Developing generalizable and accurae forward dynamics models is of criical imporance for robo manipulaion of disinc real-life objecs. We propose o learn a differeniable, paricle-based simulaor for complex conrol asks, drawing inspiraion from recen developmen in differeniable physical engines (Baaglia e al., 2016; Chang e al., 2017). In roboics, he use of differeniable simulaors, ogeher wih coninuous and symbolic opimizaion algorihms, has enabled planning for increasingly complex whole body moions wih muli-conac and muli-objec ineracions (Toussain e al., 2018). Ye hese approaches have only ackled local ineracions of rigid bodies. We develop dynamic paricle ineracion neworks (DPI- Nes) for learning paricle dynamics, focusing on capuring he dynamic, hierarchical, and long-range ineracions of paricles (Figure 1a-c). DPI-Nes can hen be combined wih classic percepion and gradien-based conrol algorihms for robo manipulaion of deformable objecs (Figure 1d). Learning a paricle-based simulaor brings in wo major benefis. Firs, he learned simulaor, jus like oher paricle-based sysems, acs widely on objecs of differen saes. DPI-Nes have successfully capured he complex behaviors of deformable objecs, fluids, and rigid-bodies. Wih learned DPI- Nes, our robos have achieved success in manipulaion asks ha involve deformable objecs of complex physical properies, such as molding plasicine o a arge shape. Second, he paricle-based represenaion poses srong inducive bias for learning: paricles of he same ype have he same dynamics wihin. This enables he model o quickly adap o new environmens of unknown dynamics wihin a few observaions. Experimens sugges ha DPI-Nes Our projec page: hps://sies.google.com/view/learn-paricle-dynamics 1

2 Under review as a conference paper a ICLR 2019 (a) (b) Dynamic Ineracion Graph Local Propagaion (c) Muli-Sep Effec Propagaion Hierarchical Propagaion Gradien-Based Opimizaion (d) Loss Percepion Predicion Targe Figure 1: Learning paricle dynamics for conrol. (a) DPI-Nes learn paricle ineracion while dynamically building he ineracion graph over ime. (b) Build hierarchical graph for muli-scale effec propagaion. (c) Muli-sep message passing for handling insananeous force propagaion. (d) Percepion and conrol wih he learned model. Our sysem firs reconsrucs he paricle-based shape from visual observaion. I hen uses gradien-based rajecory opimizaion o search for he acions ha produce he mos desired oupu. quickly learn o adap o characerize a novel objec of unknown physical parameers by doing online sysem idenificaion. The adaped model also helps he robo o successfully manipulae objec in he real world. DPI-Nes combine hree key feaures for effecive paricle-based simulaion and conrol: muli-sep spaial propagaion, hierarchical paricle srucure, and dynamic ineracion graphs. In paricular, i employs dynamic ineracion graphs, buil on he fly hroughou manipulaion, o capure he meaningful ineracions among paricles of deformable objecs and fluids. The use of dynamic graphs allows neural models o focus on learning meaningful ineracions among paricles, and is crucial for obaining good simulaion accuracy and high success raes in manipulaion. As objecs deform when robos inerac wih hem, a fixed ineracion graph over paricles is insufficien for robo manipulaing non-rigid objecs. Experimens demonsrae ha DPI-Nes significanly ouperform ineracion neworks (Baaglia e al., 2016), HRN (Mrowca e al., 2018), and a few oher baselines. More imporanly, unlike previous paper ha focused purely on forward simulaion, we have applied our model o downsream conrol asks. Our DPI-Nes enable complex manipulaion asks for deformable objecs and fluids, and adaps o scenarios wih unknown physical parameers ha need o be idenified online. We have also performed real-world experimens o demonsrae our model s generalizaion abiliy. 2 R ELATED W ORK Differeniable physics simulaors. Researchers have developed many differeniable physics simulaors (Ehrhard e al., 2017; Degrave e al., 2016). In paricular, Baaglia e al. (2016) and Chang e al. (2017) have explored approximaing objec ineracions wih neural neworks. Li e al. (2018) proposed learning o propagae signals along he ineracion graph. These mehods mosly focus on modeling rigid body dynamics. Differeniable simulaors for deformable objecs have been less sudied. Recenly, Schenck & Fox (2018) proposed SPNes for differeniable simulaion of posiion-based fluids (Macklin & Mu ller, 2013). A concurren work from Mrowca e al. (2018) explored learning o approximae paricle dynamics of deformable shapes wih a hierarchical represenaion. Compared wih hese papers, our model uses dynamic graphs o learn paricle ineracion of various maerials (rigid bodies, deformable shapes, fluids) under complex behaviors. We also demonsraed how i can be used in conrol asks in boh simulaion and real world. Our approach is also complemenary o some recen work on learning o discover he ineracion graphs (van Seenkise e al., 2018; Kipf e al., 2018). Our model can also be naurally augmened 2

3 wih a percepion module o handle raw visual inpu, as suggesed by Waers e al. (2017); Wu e al. (2017); Fragkiadaki e al. (2016). Model-predicive conrol wih a differeniable simulaor. Many recen papers have sudied model-predicive conrol wih deep neworks (Lenz e al., 2015; Gu e al., 2016; Nagabandi e al., 2018; Farquhar e al., 2018; Srinivas e al., 2018). They ofen learn an absrac sae ransiion funcion, insead of an explici accoun of he environmen (Silver e al., 2017; Oh e al., 2017), and hen use he learned funcion o faciliae raining of a policy nework. A few recen papers have employed analyical, differeniable simulaors (de Avila Belbue-Peres e al., 2018; Schenck & Fox, 2018) for conrol problems, such as ool manipulaion and ool-use planning (Toussain e al., 2018). Our model builds on and exends hese approaches by learning a general physics simulaor ha akes raw objec observaions (e.g., posiions, velociies) of each paricle as inpu. We hen inegrae i ino classic rajecory opimizaion algorihms for conrol. Compared wih pure analyical simulaors, our learned simulaor can beer generalize o novel esing scenarios where objec and environmen parameers are unknown. A few papers have explored using ineracion neworks for planning and conrol. They ofen learn a policy based on ineracion neworks rollous (Racanière e al., 2017; Hamrick e al., 2017; Pascanu e al., 2017). In conras, our model learns a dynamics simulaor and direcly opimizes rajecories for coninuous conrol. Recenly, Sanchez-Gonzalez e al. (2018) have applied ineracion neworks for conrol, and Li e al. (2018) have furher exended ineracion nes o handle insance signal propagaion for conrolling muliple rigid bodies under parial observaions. Compared wih hem, our dynamic paricle ineracion nework simulae and conrol deformable, paricle-based objecs, using dynamic graphs o ackle scenes wih complex objec ineracions. 3 APPROACH 3.1 PRELIMINARIES We firs describe how ineracion neworks (Baaglia e al., 2016) represen he physical sysem; we hen exend hem for paricle-based dynamics. The ineracions wihin a physical sysem are represened as a direced graph, G = O, R, where verices O = {o i } represen objecs and edges R = {r k } represen relaions. Specifically, o i = x i, a o i, where x i = q i, q i is he sae of objec i, conaining is posiion q i and velociy q i. a o i denoes is aribues (e.g., mass, radius). For relaion, we have r k = u k, v k, a r k, 1 u k, v k O, where u k is he receiver, v k is he sender, and a r k is he ype and aribues of relaion k (e.g., collision, spring connecion). The goal is o build a learnable physical engine o capure he underlying physical ineracions using funcion approximaors φ. The learned model can hen be used o infer he sysem dynamics and predic he fuure from he curren ineracion graph as G +1 = φ(g ), where G denoes he scene sae a ime. Ineracion neworks. Baaglia e al. (2016) proposed ineracion neworks (IN), a generalpurpose, learnable physics engine ha performs objec- and relaion-cenric reasoning abou physics. INs define an objec funcion f O and a relaion funcion f R o model objecs and heir relaions in a composiional way. The fuure sae a ime + 1 is prediced as e k, = f R (o uk,, o vk,, a r k ) k=1... R, ô i,+1 = f O (o i,, k N i e k, ) i=1... O, where o i, = x i,, a o i denoes objec i a ime, u k and v k are he receiver and sender of relaion r k respecively, and N i denoes he relaions where objec i is he receiver. Propagaion neworks. A limiaion of INs is ha a every ime sep, i only considers local informaion in he graph G and canno handle insananeous propagaion of forces, which however is a common phenomenon in rigid-body dynamics. Li e al. (2018) proposed propagaion neworks o handle he insananeous propagaion of forces by doing muli-sep message passing. Specifically, hey firs employed he ideas on fas raining of RNNs (Lei & Zhang, 2017; Bradbury e al., 2017) o encode he shared informaion beforehand and reuse hem along he propagaion seps. The encoders for objecs are denoed as fo enc and he encoder for relaions as fr enc, where we denoe co i, = f O enc(o i,), c r k, = f R enc(o u k,, o vk,, a r k ). 3

4 A ime, denoe he propagaing influence from relaion k a propagaion sep l as e l k,, and he propagaing influence from objec i as h l i,. For sep 1 l L, propagaion can be described as Sep 0: h 0 i, = 0, i = 1... O, (1) Sep l = 1,..., L: Oupu: e l k, = f R (c r k,, h l 1 u k,, h l 1 v k,), k = 1... R, (2) h l i, = f O (c o i,, ), i = 1... O, (3) e l 1 k,, hl 1 i, k N i ô i,+1 = f oupu O (h L i,), i = 1... O, (4) where fo l denoes he objec propagaor a propagaion sep l and f R l denoes he relaion propagaor. 3.2 DYNAMIC PARTICLE INTERACTION NETWORKS Paricle-based sysem is widely used in physical simulaion due o is flexibiliy in modeling various ypes of objecs (Macklin e al., 2014). We exend exising sysems ha model objec-level ineracions o allow paricle-level deformaion. Consider objec se {o i }, where each objec o i = {o k i } k=1... o i is represened as a se of paricles. We now define he graph on he paricles and he rules for influence propagaion. Dynamic graph building. The verices of he graph are he union of paricles for all objecs O = {o k i } i=1... O,k=1... o i. The edges R beween hese verices are dynamically generaed over ime o ensure efficiency and effeciveness. The consrucion of he relaions is specific o environmen and ask, which we ll elaborae in Secion 4. A common choice is o consider he neighbors wihin a predefined disance. An alernaive is o build a saic, complee ineracion graph, bu i has wo major drawbacks. Firs, i is no efficien. In many common physical sysems, each paricle is only ineracing wih a limied se of oher paricles (e.g., hose wihin is neighborhood). Second, a saic ineracion graph implies a universal, coninuous neural funcion approximaor; however, many physical ineracions involve disconinuous funcions (e.g. conac). In conras, using dynamic graphs empowers he model o ackle such disconinuiy. Hierarchical modeling for long-range dependence. Propagaion neworks Li e al. (2018) require a large L o handle long-range dependence, which is boh inefficien and hard o rain. Hence, we add one level of hierarchy o efficienly propagae he long-range influence among paricles (Mrowca e al., 2018). For each objec ha requires modeling of he long-range dependence (e.g. rigid-body), we cluser he paricles ino several non-overlapping clusers. For each cluser, we add a new paricle as he cluser s roo. Specifically, for each objec o i ha requires hierarchical modeling, he corresponding roos are denoed as õ i = {õ k i } k=1... õ i, and he paricle se conaining all he roos is denoed as Õ = {õk i } i=1... O,k=1... õ i. We hen consruc an edge se R LeafToRoo ha conains direced edges from each paricle o is roo, and an edge se R RooToLeaf conaining direced edges from each roo o is leaf paricles. For each objec ha need hierarchical modeling, we add pairwise direced edges beween all is roos, and denoe his edge se as R RooToRoo. We employ a muli-sage propagaion paradigm: firs, propagaion among leaf nodes, φ LeafToLeaf ( O, R ); second, propagaion from leaf nodes o roo nodes, φ LeafToRoo ( O Õ, R LeafToRoo ); hird, propagaion beween roos, φ RooToRoo ( Õ, R RooToRoo ); fourh, propagaion from roo o leaf, φ RooToLeaf ( O Õ, R RooToLeaf ). The signals on he leaves are used o do he final predicion. Applying o objecs of various maerials. We define he ineracion graph and he propagaion rules on paricles for differen ypes of objecs as follows: Rigid bodies. All he paricles in a rigid body are globally coupled; hence for each rigid objec, we define a hierarchical model o propagae he effecs. Afer he muli-sage propagaion, we average he signals on he paricles o predic a rigid ransformaion (roaion and ranslaion) for he objec. The moion of each paricle is calculaed accordingly. For each paricle, we also include is offse o he cener-of-mass o help deermine he orque. Elasic/Plasic objecs. For elasically deforming paricles, only using he curren posiion and velociy as he sae is no sufficien, as i is no clear where he paricle will be 4

5 resored afer he deformaion. Hence, we include he paricle sae wih he resing posiion o indicae he place where he paricle should be resored. When coupled wih plasic deformaion, he resing posiion migh change during an ineracion. Thus, we also infer he moion of he resing posiion as a par of he sae predicion. We use hierarchical modeling for his caegory bu predic nex sae for each paricles individually. Fluids. For fluid simulaion, one has o enforce densiy and incompressibiliy, which can be effecively achieved by only considering a small neighborhood for each paricle (Macklin & Müller, 2013). Therefore, we do no need hierarchical modeling for fluids. We build edges dynamically, connecing a fluid paricle o is neighboring paricles. The srong inducive bias leveraged in he fluid paricles allows good performance even when esed on daa ouside raining disribuions. For he ineracion beween differen maerials, wo direced edges are generaed for any pairs of paricles ha are closer han a cerain disance. 3.3 CONTROL ON THE LEARNED DYNAMICS Model-based mehods offer many advanages when comparing wih heir model-free counerpars, such as generalizaion and sample efficiency. However, for cases where an accurae model is hard o specify or compuaionally prohibiive, a daa-driven approach ha learns o approximae he underlying dynamics becomes useful. Funcion approximaors such as neural neworks are naurally differeniable. We can rollou using he learned dynamics and opimize he conrol inpus by minimizing a loss beween he simulaed resuls and a arge configuraion. In cases where cerain physical parameers are unknown, we can perform online sysem idenificaion by minimizing he difference beween he model s predicion and he realiy. An ouline of our algorihm can be found in Secion A. Model predicive conrol using shooing mehods. Le s denoe G g as he goal and û 1:T be he conrol inpus, where T is he ime horizon. The conrol inpus are par of he ineracion graph, such as he velociies or he iniial posiions of a paricular se of paricles. We denoe he resuling rajecory afer applying û as G = {G i } i=1:t. The ask here is o deermine he conrol inpus as o minimize he disance beween he acual oucome and he specified goal L g (G, G g ). Our dynamic paricle ineracion nework does forward simulaion by aking he dynamics graph a ime as inpu, and produces he graph a nex ime sep, Ĝ+1 = Φ(G ), where Φ is implemened as DPI-Nes as described in he previous secion. Le s denoe he he hisory unil ime as Ḡ = {G i } i=1..., and he forward simulaion from ime sep as Ĝ = {Ĝi} i=+1...t. The loss L g (Ḡ Ĝ, G g) can be used o updae he conrol inpus by doing sochasic gradien descen (SGD). This is known as he shooing mehod in rajecory opimizaion (Tedrake, 2009). The learned model migh deviae from he realiy due o accumulaed predicion errors. We use Model-Predicive Conrol (MPC) (Camacho & Alba, 2013) o sabilize he rajecory by doing forward simulaion and updaing he conrol inpus a every ime sep o compensae he simulaion error. Online adapaion. In many real-world cases, wihou acually ineracing wih he environmen, inheren aribues such as mass, siffness, and viscosiy are no direcly observable. DPI-Nes can esimae hese aribues on he fly wih SGD updaes by minimizing he disance beween he prediced fuure saes and he acual fuure saes L s (Ĝ, G ). 4 EXPERIMENTS We evaluae our mehod on four differen environmens conaining differen ypes of objecs and ineracions. We will firs describe he environmens and show simulaion resuls. We hen presen how he learned dynamics helps o complee conrol asks in boh simulaion and he real world. 4.1 ENVIRONMENTS FluidFall (Figure 2a). Two drops of fluids are falling down, colliding, and merging. We vary he iniial posiion and viscosiy for raining and evaluaion. BoxBah (Figure 2b). A block of fluids are flushing a rigid cube. In his environmen, we have o model wo differen maerials and he ineracions beween hem. We randomize he iniial posiion of he fluids and he cube o es he model s generalizaion abiliy. 5

6 Mehods FuildFall BoxBah FluidShake RiceGrip IN (Baaglia e al., 2016) 2.74 ± 0.56 N/A N/A N/A HRN (Mrowca e al., 2018) 0.21 ± ± ± ± 0.11 DPI-Ne w/o hierarchy 0.15 ± ± ± ± 0.13 DPI-Ne 0.15 ± ± ± ± 0.07 Table 1: Quaniaive resuls on forward simulaion. MSE ( 10 2 ) beween he ground ruh and model rollous. The hyperparameers used in our model are fixed for all four environmens. FluidFall and FluidShake involve no hierarchy, so DPI-Ne performs he same as he varian wihou hierarchy. DPI-Ne significanly ouperforms HRN (Mrowca e al., 2018) in modeling fluids (BoxBah and FluidShake) due o he use of dynamic graphs. FluidShake (Figure 2c). We have a box of fluids and he box is moving horizonally, The speed of he box is randomly seleced a each ime sep. We vary he size of he box and volume of he fluids o es generalizaion. RiceGrip (Figure 2d). We manipulae an objec wih boh elasic and plasic deformaion (e.g., sicky rice). We use wo cuboids o mimic he fingers of a parallel gripper, where he gripper is iniialized a a random posiion and orienaion. During he simulaion of one grip, he fingers will move closer o each oher and hen resore o is original posiions. The model has o learn he ineracions beween he gripper and he sicky rice, as well as he ineracions wihin he rice iself. We use all four environmens in evaluaing our model s performance in simulaion. We use he rollou MSE as our meric. We furher use he laer wo for conrol, because hey involve fully acuaed exernal shapes ha can be used for objec manipulaion. In FluidShake, he conrol ask requires deermining he speed of he box a each ime sep, in order o make he fluid mach a arge configuraion wihin a ime window; in RiceGrip, he conrol ask corresponds o selec a sequence of grip configuraions (posiion, orienaion, closing disance) o manipulae he deformable objec as o mach a arge shape. The meric for performance in conrol is he Chamfer disance beween he manipulaion resuls and he arge configuraion. 4.2 PHYSICAL SIMULATION We presen implemenaion deails for dynamics learning in he four environmen and perform ablaion sudies o evaluae he effeciveness of he inroduced echniques. Implemenaion deails. For FluidFall, we dynamically build he ineracion graph by connecing each paricle o is neighbors wihin a cerain disance d. No hierarchical modeling is used. For BoxBah, we model he rigid cube as in Secion 3.2, using muli-sage hierarchical propagaion. Two direced edges will be consruced beween wo fluid paricles if he disance beween hem is smaller han d. Similarly, we also add wo direced edge beween rigid paricles and fluid paricles when heir disance is smaller han d. For FluidShake, we model fluid as in Secion 3.2. We also add five exernal paricles o represen he walls of he box. We add a direced edge from he wall paricle o he fluid paricle when hey are closer han d. The model is a single propagaion nework, where he edges are dynamically consruced over ime. For RiceGrip, we buld a hierarchical model for rice and use four propagaion neworks for muli-sage effec propagaion (Secion 3.2). The edges beween he rice paricles are dynamically generaed if wo paricles are closer han d. Similar o FluidShake, we add wo exernal paricles o represen he wo fingers and add an edge from he finger o he rice paricle if hey are closer han he disance d. As rice can deform boh elasically and plasically, we mainain a resing posiion ha helps he model resore a deformed paricle. The oupu for each paricle is a 6-dim vecor for he velociy of he curren observed posiion and he resing posiion. More raining deails for each environmen can be found in Secion D. Deails for daa generaion are in Secion C. Resuls. Qualiaive and quaniaive resuls are in Figure 2 and Table 1. We compare our mehod (DPI-Ne) wih hree baselines, Ineracion Neworks (Baaglia e al., 2016), HRN (Mrowca e al., 2018), and DPI-Ne wihou hierarchy. Noe ha we use he same se of hyperparameers in our model for all four esing environmens. Specifically, Ineracion Neworks (IN) consider a complee graph of he paricle sysem. Thus, i can only operae on small environmens such as FluidFall; i runs ou of memory (12GB) for he oher 6

7 GT HRN (Mrowca e al.) DPI Ne (a) FluidFall GT HRN (Mrowca e al.) DPI Ne (b) BoxBah GT HRN (Mrowca e al.) DPI Ne (c) FluidShake GT HRN (Mrowca e al.) DPI Ne (d) RiceGrip Figure 2: Qualiaive resuls on forward simulaion. We compare he ground ruh (GT) and he rollous from HRN (Mrowca e al., 2018) and our model (DPI-Ne) in four environmens (FluidFall, BoxBah, FluidShake, and RiceGrip). The simulaions from our DPI-Ne are significanly beer. We provide zoom-in views for a few frames o show deails. Please see our video for more empirical resuls. hree environmens. IN does no perform well, because is use of a complee graph makes raining difficul and inefficien, and because i ignores influence propagaion and long-range dependence. Wihou a dynamic graph, modeling fluids becomes hard, because he neighbors of a fluid paricle changes consanly. Table 1 shows ha for environmens ha involve fluids (BoxBah and FluidShake), he model wih a saic ineracion graph (Mrowca e al., 2018) does no perform well. Wihou 7

8 Number of Roos Propagaion Seps Neighbor Radius Separae Unified Moion Predicors (a) RiceGrip (b) BoxBah Figure 3: Ablaion sudies. We perform ablaion sudies o es our model s robusness o hyperparameers. The performance is evaluaed by he mean squared disance ( 10 2 ) beween he ground ruh and model rollous. (a) We vary he number of roos when building hierarchy, he propagaion sep L during message passing, and he size of he neighborhood d. (b) In BoxBah, DPI-Nes use separae moion predicors for fluids and rigid bodies. Here we compared wih a unified moion predicor. hierarchy, i is hard o capure long-range dependence, leading o performance drop in environmens ha involve hierarchical objec modeling (BoxBah and RiceGrip). Appendix B includes resuls on scenarios ouside he raining disribuion (e.g., more paricles). DPI-Ne performs well on hese ou-of-sample cases, successfully leveraging he inducive bias. Ablaion sudies. We also es our model s sensiiviy o hyperparameers. We consider hree of hem: he number of roos for building hierarchy, he number of propagaion seps L, and he size of he neighborhood d. We es hem in RiceGrip. As can be seen from he resuls shown in Figure 3a, DPI-Nes can beer capure he moion of he rice by using fewer roos, on which he informaion migh be easier o propagae. Longer propagaion seps do no necessarily lead o beer performance, as hey increases raining difficuly. Using larger neighborhood achieves beer resuls, bu makes compuaion slower. Using one TITAN Xp, each forward sep in RiceGrip akes 30ms for d = 0.04, 33ms for d = 0.08, and 40ms for d = We also perform experimens o jusify our use of differen moion predicors for objecs of differen saes. Figure 3b shows he resuls of our model vs. a unified dynamics predicor for all objecs in BoxBah. As here are only a few saes of ineres (solids, liquids, and sof bodies), and heir physical behaviors are drasically differen, i is no surprising ha DPI-Nes, wih sae-specific moion predicors, perform beer, and are equally efficien as he unified model (ime difference smaller han 3ms per forward sep). 4.3 CONTROL We leverage dynamic paricle ineracion nework for conrol asks in boh simulaion and real world. Because rajecory opimizaion using shooing mehod can easily suck o a local minimum, we firs randomly sample N sample conrol sequences, and selec he bes performing one according o he rollous of our learned model. We hen opimize i via shooing mehod using our model s gradiens. We also use online sysem idenificaion o furher improve he model s performance. Figure 4 and Figure 5 show qualiaive and quaniaive resuls, respecively. More deails of he conrol algorihm can be found in Secion E. FluidShake. We aim o conrol he speed of he box o mach he fluid paricles o a arge configuraion. We compare our mehod (RS+TO) wih random search over he learned model (wihou rajecory opimizaion - RS) and Model-free Deep Reinforcemen Learning (Acor-Criic mehod opimized wih PPO (Schulman e al., 2017) (RL). Figure 5a suggess ha our model-based conrol algorihm ouperforms boh baselines wih a large margin. Also RL is no sample-efficien, requiring more han 10 million ime seps o converge while ours requires 600K ime seps. RiceGrip. We aim o selec a sequence of gripping configuraions (posiion, orienaion, and closing disance) o mold he sicky rice o a arge shape. We also consider cases where he siffness of he rice is unknown and need o be idenified. Figure 5b shows ha our dynamic paricle ineracion nework wih sysem idenificaion performs he bes, and is much more efficien han RL (150K vs. 10M ime seps). RiceGrip in he real world. We generalize he learned model and conrol algorihm for RiceGrip o he real world. We firs reconsruc objec geomery using a deph camera mouned on our Kuka robo using TSDF volumeric fusion (Curless & Levoy, 1996). We hen randomly sampled N fill paricles wihin he objec mesh as he iniial configuraion for manipulaion. Figure 4c and Figure 5b 8

9 Trial #1 Resul Targe (a) Trial #2 Trial #1 (b) Trial #2 Trial #1 (c) Figure 4: Qualiaive resuls on conrol. (a) FluidShake - Manipulaing a box of fluids o mach a arge shape. The Resul and Targe indicae he fluid shape when viewed from he cuaway view. (b) RiceGrip - Gripping a deformable objec and molding i o a arge shape. (c) RiceGrip in Real World - Generalize he learned dynamics and he conrol algorihms o he real world by doing online adapaion. The las wo columns indicae he final shape viewed from he op FluidShake Known Physics Unknown Physics Real World (a) (b) RiceGrip Figure 5: Quaniaive resuls on conrol. We show he resuls on conrol (as evaluaed by he Chamfer disance ( 10 2 ) beween he manipulaed resul and he arge) for (a) FluidShake and (b) RiceGrip by comparing wih four baselines. RL: Model-free deep reinforcemen learning opimized wih PPO; RS: Random search he acions from he learned model and selec he bes one o execue; RS + TO: Trajecory opimizaion augmened wih model predicive conrol; RS + TO + ID: Online sysem idenificaion by esimaing uncerain physical parameers during run ime RL RS RS+TO RS+TO+ID shows ha, using DPI-Nes, he robo successfully adaps o he real world environmen of unknown physical parameers and manipulaes a deformable foam ino various arge shapes. The learned policy in RiceGrip does no generalize o he real world due o domain discrepancy, and oupus invalid gripping configuraions. 5 CONCLUSION We have demonsraed ha a learned paricle dynamics model can approximae he ineracion of diverse objecs, and can help o solve complex manipulaion asks of deformable objecs. Our sysem requires sandard open-source roboics and deep learning oolkis, and can be poenially deployed in household and manufacuring environmen. Robo learning of dynamic scenes wih paricle-based represenaions shows profound poenials due o he generalizabiliy and expressiveness of he represenaion. Our sudy helps lay he foundaion for i. 9

10 REFERENCES Peer W. Baaglia, Razvan Pascanu, Mahew Lai, Danilo Rezende, and Koray Kavukcuoglu. Ineracion neworks for learning abou objecs, relaions and physics. In NIPS, James Bradbury, Sephen Meriy, Caiming Xiong, and Richard Socher. Quasi-recurren neural neworks. In ICLR, Eduardo F Camacho and Carlos Bordons Alba. Model predicive conrol. Springer Science & Business Media, Michael B Chang, Tomer Ullman, Anonio Torralba, and Joshua B Tenenbaum. A composiional objec-based approach o learning physical dynamics. In ICLR, Brian Curless and Marc Levoy. A volumeric mehod for building complex models from range images. In SIGGRAPH, Filipe de Avila Belbue-Peres, Kevin A Smih, Kelsey Allen, Joshua B Tenenbaum, and J Zico Koler. End-o-end differeniable physics for learning and conrol. In Neural Informaion Processing Sysems, Jonas Degrave, Michiel Hermans, and Joni Dambre. A differeniable physics engine for deep learning in roboics. In ICLR Workshop, Sebasien Ehrhard, Aron Monszpar, Niloy Mira, and Andrea Vedaldi. Taking visual moion predicion o new heighfields. arxiv: , Gregory Farquhar, Tim Rockäschel, Maximilian Igl, and Shimon Whieson. Treeqn and areec: Differeniable ree planning for deep reinforcemen learning. In ICLR, Kaerina Fragkiadaki, Pulki Agrawal, Sergey Levine, and Jiendra Malik. Learning visual predicive models of physics for playing billiards. In ICLR, Shixiang Gu, Timohy Lillicrap, Ilya Suskever, and Sergey Levine. Coninuous deep q-learning wih model-based acceleraion. In ICML, Jessica B Hamrick, Andrew J Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peer W Baaglia. Meaconrol for adapive imaginaion-based opimizaion. In ICLR, Diederik P. Kingma and Jimmy Ba. Adam: A mehod for sochasic opimizaion. In ICLR, Thomas N Kipf, Ehan Feaya, Kuan-Chieh Wang, Max Welling, and Richard S Zemel. Neural relaional inference for ineracing sysems. arxiv: , Tao Lei and Yu Zhang. Training rnns as fas as cnns. arxiv preprin arxiv: , Ian Lenz, Ross A Knepper, and Ashuosh Saxena. Deepmpc: Learning deep laen feaures for model predicive conrol. In RSS, Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Anonio Torralba, and Russ Tedrake. Propagaion neworks for model-based conrol under parial observaion. arxiv preprin arxiv: , Miles Macklin and Mahias Müller. Posiion based fluids. ACM TOG, 32(4):104, Miles Macklin, Mahias Müller, Nuapong Chenanez, and Tae-Yong Kim. Unified paricle physics for real-ime applicaions. ACM TOG, 33(4):153, Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural represenaion for physics predicion. arxiv preprin arxiv: , Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural nework dynamics for model-based deep reinforcemen learning wih model-free fine-uning. In ICRA, Junhyuk Oh, Sainder Singh, and Honglak Lee. Value predicion nework. In NIPS,

11 Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebasien Racanière, David Reicher, Théophane Weber, Daan Wiersra, and Peer Baaglia. Learning model-based planning from scrach. arxiv: , Sébasien Racanière, Théophane Weber, David Reicher, Lars Buesing, Arhur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peer Baaglia, David Silver, and Daan Wiersra. Imaginaion-augmened agens for deep reinforcemen learning. In NIPS, Alvaro Sanchez-Gonzalez, Nicolas Heess, Jos Tobias Springenberg, Josh Merel, Marin Riedmiller, Raia Hadsell, and Peer Baaglia. Graph neworks as learnable physics engines for inference and conrol. In ICML, Connor Schenck and Dieer Fox. Spnes: Differeniable fluid dynamics for deep neural neworks. arxiv preprin arxiv: , John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opimizaion algorihms. arxiv: , David Silver, Hado van Hassel, Maeo Hessel, Tom Schaul, Arhur Guez, Tim Harley, Gabriel Dulac- Arnold, David Reicher, Neil Rabinowiz, Andre Barreo, and Thomas Degris. The predicron: End-o-end learning and planning. In ICML, Aravind Srinivas, Allan Jabri, Pieer Abbeel, Sergey Levine, and Chelsea Finn. Universal planning neworks. In ICML, Russ Tedrake. Underacuaed roboics: Learning, planning, and conrol for efficien and agile machines course noes for mi 6.832, Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based conrol. In IROS, pp IEEE, Marc Toussain, K Allen, K Smih, and J Tenenbaum. Differeniable physics and sable modes for ool-use and manipulaion planning. In RSS, Sjoerd van Seenkise, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relaional neural expecaion maximizaion: Unsupervised discovery of objecs and heir ineracions. In ICLR, Nicholas Waers, Andrea Tacchei, Theophane Weber, Razvan Pascanu, Peer Baaglia, and Daniel Zoran. Visual ineracion neworks. In NIPS, Jiajun Wu, Erika Lu, Pushmee Kohli, Bill Freeman, and Josh Tenenbaum. Learning o see physics via visual de-animaion. In NIPS,

12 A CONTROL ALGORITHM Algorihm 1 Conrol on Learned Dynamics a Time Sep Inpu: Learned forward dynamics model Φ Inpu: Prediced dynamics graph Ĝ Inpu: Curren dynamics graph G Inpu: Goal G g, curren esimaion of he aribues A Inpu: Curren conrol inpus û :T Inpu: Saes hisory Ḡ = {G i} i=1... Inpu: Forward simulaion ime N and ime horizon T Oupu: Conrols û :T, prediced nex ime sep Ĝ+1 Updae A by descending wih he gradiens A L s (Ĝ, G ) for i = 1,..., N do Forward simulaion using he curren graph Ĝ+1 Φ(G ) Make a buffer for soring he simulaion resuls G Ḡ Ĝ+1 for j = + 1,..., T 1 do Forward simulaion: Ĝj+1 Φ(Ĝj); G G Ĝj+1 end for Updae û :T by descending wih he gradiens û:t L g (G, G g ) end for Reurn û :T and Ĝ+1 Φ(G ) B GENERALIZATION ON EXTRAPOLATION We show our model s performance on fluids, rigid bodies, and deformable objecs wih a larger number of paricles han hey have in he raining se. Figure 6 shows qualiaive and quaniaive resuls. Our model scales up well o larger objecs. GT DPI Ne (a) Exrapolae Generalizaion on Fluids GT DPI Ne (b) Exrapolae Generalizaion on Rigid Bodies GT DPI Ne (b) Exrapolae Generalizaion on Deformable Objecs Figure 6: Exrapolae generalizaion on fluids, rigid bodies, and deformable objecs. The performance is evaluaed by he MSE ( 10 2 ) beween he ground ruh and rollous from DPI-Nes. The blue bars denoe he range of paricle numbers ha have been seen during raining, which indicae inerpolaion performance. The red bars indicae exrapolaion performance ha our model can generalize o cases conaining wo imes more paricles han cases i has been rained on. 12

13 C DATA GENERATION The daa is generaed using NVIDIA FleX. We have developed a Pyhon inerface for he ease of generaing and ineracing wih differen environmens. We will release he code upon publicaion. FluidFall. We generaed 3,000 rollous over 120 ime seps. The wo drops of fluids conain 64 and 125 paricles individually, where he iniial posiion of one of he drop in he 3 dimensional coordinaes is uniformly sampled beween (0.15, 0.55, 0.05) and (0.25, 0.7, 0.15), while he oher drop is uniformly sampled beween (0.15, 0.1, 0.05) and (0.25, 0.25, 0.15). BoxBah. We generaed 3,000 rollous over 150 ime seps. There are 960 fluid paricles and he rigid cube consis paricles ranging beween 27 and 150. The fluid paricle block is iniialized a (0, 0, 0), and he iniial posiion of he rigid cube is randomly iniialized beween (0.45, , 0.02) o (1.2, , 0.4). FluidShake. We generaed 2,000 rollous over 300 ime seps. The heigh of he box is 1.0 and he hickness of he wall is For he iniial fluid cuboid, he number of fluid paricles is uniformly sampled beween 10 and 12 in he x direcion, beween 15 and 20 in he y direcion, 3 in he z direcion. The box is fixed in he y and z direcion, and is moving freely in he x direcion. We randomly place he iniial x posiion beween -0.2 o 0.2. The sampling of he speed is implemened as v = v + rand( 0.15, 0.15) 0.1x, in order o encourage moion smoohness and moving back o origin, where speed v is iniialized as 0. RiceGrip. We generaed 5,000 rollous over 30 ime seps. We randomize he size of he iniial rice cuboid, where he lengh of he hree sides is uniformly sampled beween 8.0 and The maerial propery parameers clusersiffness is uniformly sampled beween 0.3 and 0.7, cluserplasicthreshold is uniformly sampled beween 1e-5 and 5e-4, and cluserplasiccreep is uniformly sampled beween 0.1 and 0.3. The posiion of he gripper is randomly sampled wihin a circle of radius 0.5. The orienaion of he gripper is always perpendicular o he line connecing he origin o he cener of he gripper and he close disance is uniformly sampled beween 0.7 o 1.0. Of all he generaed daa, 90% of he rollous are used for raining, and he res 10% are used for validaion. D TRAINING DETAILS The models are implemened in PyTorch, and are rained using Adam opimizer (Kingma & Ba (2015)) wih a learning rae of The number of paricles and relaions migh be differen a each ime sep, hence we use a bach size of 1, and we updae he weighs of he neworks once every 2 forward rounds. The neighborhood d is se as 0.08, and he propagaion sep L is se as 2 for all four environmens. For hierarchical modeling, i does no make sense o propagae more han one ime beween leaves and roos as hey are disjoin paricle ses, and each propagaion sage beween hem only involves one-way edges; hence φ LeafToLeaf uses L = 2. φ LeafToRoo uses L = 1. φ RooToRoo uses L = 2, and φ RooToLeaf uses L = 1. For all propagaion neworks used below, he objec encoder fo enc is an MLP wih wo hidden layers of size 200, and oupus a feaure map of size 200. The relaion encoder fr enc is an MLP wih hree hidden layers of size 300, and oupus a feaure map of size 200. The propagaor f O and f R are boh MLP wih one hidden layer of size 200, in which a residual connecion is used o beer propagae he effecs, and oupus a feaure map of size 200. The propagaors are shared wihin each sage of propagaion. The moion predicor f oupu O is an MLP wih wo hidden layers of size 200, and oupu he sae of required dimension. ReLU is used as he acivaion funcion. FluidFall. The model is rained for 13 epochs. The oupu of he model is he 3 dimensional velociy, which is muliplied by and added o he curren posiion o do rollous. BoxBah. In his environmen, four propagaion neworks are used due o he hierarchical modeling and he number of roos for he rigid cube is se as 8. We have wo separae moion predicor for 13

14 fluids and rigid body, where he fluid predicor oupu velociy for each fluid paricle, while he rigid predicor akes he mean of he signals over all is rigid paricles as inpu, and oupu a rigid ransformaion (roaion and ranslaion). The model is rained for 5 epochs. FluidShake. for 5 epochs. Only one propagaion nework is used in his environmen, and he model is rained RiceGrip. Four propagaion neworks are used due he hierarchical modeling, and he number of roos for he rice is se as 30. The model is rained for 20 epochs. E CONTROL DETAILS N sample is chosen as 20 for all hree cases, where we sample 20 random conrol sequences, and choose he bes performing one as evaluaed using our learned model. The evaluaion is based on he Chamfer disance beween he conrolling resul and he arge configuraion. FluidShake. In his environmen, he conrol sequence is he speed of he box along he x axis. The mehod o sample he candidae conrol sequence is he same as when generaing raining daa of his environmen. Afer seleced he bes performing conrol sequence, we firs use RMSprop opimizer o opimize he conrol inpus for 10 ieraions using a learning rae of We hen use model-predicive conrol o apply he conrol sequence o he FleX physics engine using Algorihm 1, where N is seleced as 3. RiceGrip. In his environmen, we need o come up wih a sequence of grip configuraions, where each grip conains posiions, orienaion, and closing disance. The mehod o sample he candidae conrol sequence is he same as when generaing raining daa of his environmen. Afer seleced he bes performing conrol sequence, we firs use RMSprop opimizer o opimize he conrol inpus for 20 ieraions using a learning rae of We hen use model-predicive conrol o apply he conrol sequence o he FleX physics engine using Algorihm 1, where N is seleced as 5. RiceGrip in Real World. In his environmen, we need o come up wih a sequence of grip configuraions, where each grip conains posiions, orienaion, and closing disance. The mehod o sample he candidae conrol sequence is he same as when generaing raining daa of RiceGrip, and N fill is chosen as 768. Differen from he previous case, he physical parameers are always unknown and has o be esimaed online. Afer seleced he bes performing conrol sequence, we firs use RMSprop opimizer o opimize he conrol inpus for 20 ieraions using a learning rae of We hen use model-predicive conrol o apply he conrol sequence o he real world using Algorihm 1, where N is seleced as