Coprocessor memory definition. Loic Pallardy / Arnaud Pouliquen

Transcription

1 Coprocessor memory definition Loic Pllrdy / Arnud Pouliquen

2 Objective 2 The gol of following slides is to sum up on-going discussion in OpenAP weekly bout Remoteproc/Rpmsg memory lloction. Following proposl is bsed on resource tble used s centrl element for informtion shring between the two processors. Different use cses re tken into ccount (in priority order) ster min OS is loding nd strting slve processor. ster lloctes resources nd initites communiction (current minly) Slve strts before ster min OS, ster lloctes resources nd initites IPC communiction Whtever boot order, ech processor lloctes its own resources nd is ble to initite communiction once link redy (peer to peer mode). For ech use cse emory resource cn be dynmiclly defined t runtime emory resource cn be fixed t code genertion tking into ccount project constrints (SoC mpping, security )

3 Current sitution 3 Remoteproc is in chrge of 2 lloctions Crveout: memory region dedicted to coprocessor ccessible by both processors Ech region declred in resource tble is llocted thnks to dm_lloc_coherent cll nd mnged in list by rproc Rsc tble updted by rproc for informtion shring with slve coprocessor (p only) Crveout region re used minly for firmwre nd log buffer Region is ccessible thnks to rproc_d_to_v interfce No fixed definition support by remoteproc frmework (mens if crveout resource defined in rsc tble, rproc try to llocte it) BT hook exit t pltform driver level to mnge pltform locl crveout list Vrings: buffers used to estblish communiction t virtio level The two vrings re dynmiclly llocted thnks to dm_lloc_coherent ccording to rsc tble definition (nb element, lignment ) Rsc tble is updted by rproc for informtion shring with slve coprocessor No fixed vring definition supported Nb of vring per vdev fixed to 2. Nb of vdev per rproc fixed to 1.

4 Current sitution 4 Rpmsg is in chrge of shred buffer lloction Only dynmic lloction supported bsed on dm_lloc_coherent Relies on rproc pltform device dm memory pool (grnd-fther) Doesn't updte ny informtion in resource tble s buffer link done t vring level Considers tht coprocessor hs complete memory ccess (s mster processor) (or tht memory ccess hs been grnt/defined by nother wy before)

5 Different memory regions configurtion 5 emory region: generic term to ddress crveout, vring nd rpmsg shred buffers Current implementtion : No constrint mster could perform dynmic lloction in system memory (externl RA in generl). ster hs the responsibility to provide ll needed informtion to slve to hndle memory mpping on its side se of specific internl SoC memory ster could perform dynmic lloction but in one internl memory using dedicted lloctor ster hs the responsibility to provide ll needed informtion to slve to hndle memory mpping on its side (vi rsc tble) se of fixed memory region All or prt of coprocessor memory regions re defined t product definition level to tke into ccount SoC, security, product requirements ster nd slve should rely on ddress defined in rsc tble to enble memory ccess ster must not chnge pre-defined vlue in rsc tble

6 Proposl overview 6 To cover ll memory configurtions for coprocessor firmwre, vrings nd buffers, following items should be covered: se crveout resource in rsc tble to define ll memory regions shred between mster nd slve Add support of fixed memory region in rproc crveout hndler Add pltform driver specific memory lloction support (covered by crveout region mngement) Add support of fixed memory region for vring lloction Provide dedicted DA memory pool to virtio_rpmsg device for buffer lloction Peer to peer use cse to be dded on the top

7 Crveout resource mngement 7 On mster side Need to llocte requested memory region Need to grnt CP ccess to this memory region Need to mp it on device memory domin if supported (s) On slve side Enble memory ccess on defined crveout region (P/ configurtion) At the end of mster nd slve remoteproc initiliztion, both re ble to ccess the different crveout regions All crveout resources should be processed before other resources Common reference between mster nd slve is physicl ddress.

8 How to define fixed memory region 8 Physicl mpping is the reference t project/soc level PA field in the different resource description is good cndidte to differentite the different type of memory (fixed or not). Rule could be the following one if PA == FW_RSC_ADDR_ANY then dynmic lloction ( = current implementtion) else memory fixed => just mp it In both cse, memory region should be dded in crveouts list A new flg is needed in struct rproc_mem_entry to identify how crveout ccess is mnged (DA lloction, mem_mp, other?). ybe priv field could be used?

9 How to rely on specific memory lloctor 9 Rely on pltform driver crveout registering for dedicted lloction mngement Free cllbck ssocited to ech crveout to keep remoteproc core generic

10 Vring lloction 10 Need to dd fixed vring loction support in vring lloction function. Differentition between fixed nd dynmic Vring could be done thnks to PA field (sme s crveout): if PA == FW_RSC_ADDR_ANY then dynmic lloction ( = current implementtion) else memory fixed => find mtch between fixed vring nd crveout list get ssocited VA ddress Need new helper function "rproc_p_to_v" to prse crveouts list with physicl ddress Need to updte rproc_free_vring function to only free vring if doesn't belong to crveout region Flg needed in struct rproc_vring?

11 Rproc client buffer lloction (virtio_rpmsg or virtio_console) 11 Virtio bsed clients buffers re dynmiclly lloction using dm_lloc_coherent function Device used for lloction is client grnd-fther device, i.e. pltform remoteproc driver No direct link between client nd remoteproc. Isoltion hs to be preserved. Possibility to ccess cfg extension field of struct fw_rsc_vdev of the rsc tble to get informtion (see virtio get nd set ops) BT should be used for custom/optionl configurtion, not for memory definition which is lredy present in crveout resource

12 Rproc client buffer lloction (virtio_rpmsg or virtio_console) 12 Proposl is to ssocite dedicted memory pool to virtio client for its lloctions Either ssign DA memory pool to virtio device t its cretion nd chnge ll client to rely on fther (virtio device) insted of grnd-fther (pltform rproc device) Or Introduce sub-device per vdev t remoteproc level which will be provided s prent of virtio device. No chnge t virtio client level. Assocition done thnks to dm_declre_coherent_memory function ster nd slve will hve ccess to buffers s ccess grnted during crevout resources hndling No need to exchnge more informtion bout shred buffer ddress nd size Need to define wy to get informtion bout shred buffer loction (nd ssocited memory region) Rely on crveout resource nme? Specific pltform driver helper function? New field in struct fw_rsc_vdev providing index of crveout resource to use?

13 Exmples 13

14 Initil resource tble from FW Full dynmic lloction ("current" Linux support) Resource tble fter Rproc initiliztion 14 Resource tble p updted d not updted Resource tble crveout.p = 0xFFFFFFFF.len= 0xyyyy VDEV Allocted by rproc Cover coprocessor code nd dt 1 crveout.p = "crveout_p".len= 0xyyyy VDEV TX Vring p = 0xFFFFFFFF d = 0xFFFFFFFF RX Vring p = 0xFFFFFFFF d = 0xFFFFFFFF 2 Allocted by rproc_lloc_vring No gurnty bout Vring continuity TX Vring p = 0xFFFFFFFF d = "V0_p" RX Vring p = 0xFFFFFFFF d = "V1_p" Buffer dynmiclly llocted by mster, but no informtion in resource tble DDR Crveout buf Vring0 Vring1 Buffers crveout_p V0_p V1_p d updted to physicl ddress p not used (not defined in OpenAP) need to be dded Fixed region Dynmic lloc

15 Sttic code/dt crveout definition Dynmic vring/rpmsg buffer lloction Initil resource tble from FW Resource tble crveout.p = crveout_p.len= 0xyyyy VDEV TX Vring p = 0xFFFFFFFF d = 0xFFFFFFFF RX Vring p = 0xFFFFFFFF d = 0xFFFFFFFF p not updted d updted for System (s) mngement Reserved memory for coprocessor Region0. emory region should hve be reserved ( section, DT ) 1 Allocted by rproc_lloc_vring No gurnty bout Vring continuity 2 Resource tble fter Rproc initiliztion Resource tble crveout.p = "crveout_p".len= 0xyyyy VDEV TX Vring p = 0xFFFFFFFF d = "V0_p" RX Vring p = 0xFFFFFFFF d = "V1_p" 15 1 DDR Crveout buf d updted p not used (not defined in OpenAP) crveout_p Buffer dynmiclly llocted by mster, but no informtion in resource tble. 2 2 Vring0 Vring1 Buffers V0_p V1_p Fixed region Dynmic lloc

16 Sttic code/dt crveout & vring definition Dynmic rpmsg buffer lloction Initil resource tble from FW Resource tble p not updted d updted for s Resource tble fter Rproc initiliztion Resource tble 16 crveout.p = crveout_p.len= 0xyyyy VDEV TX Vring p = "V0_p" d = 0xFFFFFFFF RX Vring p = "V1_p" d = 0xFFFFFFFF Reserved memory for coprocessor Region0. emory region should hve be reserved ( section, DT ) 1 Vring p lredy defined s prt of crveout region 2 crveout.p = "crveout_p".len= 0xyyyy VDEV TX Vring p = "V0_p" d = "V0_d" RX Vring p = V1_p d = "V1_d" DDR p needs to be defined in OpenAP vrings re prt of crveout so known by both processors Buffer dynmiclly llocted by mster, but no informtion in resource tble Crveout buf Vring1 Vring0 Buffers V1_p V0_p crveout_p p not updted d updted for s mngement Fixed region Dynmic lloc

17 Not pplicble if rmsg buffer get by nme Initil resource tble from FW Resource tble Full sttic definition With one Crevout Resource tble fter Rproc initiliztion Resource tble 17 crveout0.p ="crveout0_p".len= 0xyyyy 1 Reserved memory for coprocessor region0 crveout0.p ="crveout0_p".len= 0xyyyy VDEV VDEV TX Vring p = "V0_p" d = 0xFFFFFFFF 2 Vring p lredy defined s prt of crveout region TX Vring p = "V0_p" d = V0_d RX Vring p = "V1_p d = 0xFFFFFFFF RX Vring p = "V1_p d = V1_d DDR 1 Crveout 0 RPSG buffers re prt of one crveout so known by both processors 2 Vring1 Vring0 Buffers V1_p V0_p crveout0_p Fixed region Dynmic lloc

18 Initil resource tble from FW Full sttic definition With severl Crevouts Resource tble fter Rproc initiliztion 18 Resource tble crveout0.p ="crveout0_p".len= 0xyyyy Crveouts could be in DDR or in embedded RA Resource tble crveout0.p ="crveout0_p".len= 0xyyyy crveout1.p = "crveout1_p".len= 0xwwww Reserved memories for coprocessor Region 0, 1 & 2 1 crveout1.p = "crveout1_p".len= 0xwwww crveout2.p = "crveout2_p".len= 0xzzzz.nme="RPSG" VDEV TX Vring p = "V0_p" d = 0xFFFFFFFF RX Vring p = "V1_p d = 0xFFFFFFFF RPSG buffers in dedicted crveout nmed "RPSG" 1 2 V1_p V0_p Vring p lredy defined s prt of crveout 1 region 2 emory (DDR/SRA) Crveout 0 Crveout 1 Vring1 Vring0 Crveout 2 Buffers crveout0_p crveout1_p crveout2_p crveout2.p = "crveout2_p".len= 0xzzzz.nme="RPSG" VDEV TX Vring p = "V0_p" d = 0xFFFFFFFF RX Vring p = "V1_p d = 0xFFFFFFFF Fixed region Dynmic lloc

19 Not pplicble if rmsg buffer get by nme Need one crveout by RPSG buffer RX nd TX Initil resource tble from FW Full sttic definition With severl Crevouts Resource tble fter Rproc initiliztion 19 Resource tble crveout0.p ="crveout0_p".len= 0xyyyy Crveouts could be in DDR or in embedded RA Resource tble crveout0.p ="crveout0_p".len= 0xyyyy crveout1.p = "crveout1_p".len= 0xwwww Reserved memories for coprocessor Region 0, 1 & 2 1 crveout1.p = "crveout1_p".len= 0xwwww crveout2.p = "crveout2_p".len= 0xzzzz VDEV Vring p lredy defined s prt of crveout region 2 crveout2.p = "crveout2_p".len= 0xzzzz VDEV TX Vring p = "V0_p" d = 0xFFFFFFFF RX Vring p = "V1_p d = 0xFFFFFFFF RPSG buffers re split in 2: RX nd TX in this exmple 1 2 emory (DDR/SRA) Crveout 0 Crveout 1 Vring1 Buffers Crveout 2 Vring0 Buffers crveout0_p V1_p crveout1_p V0_p crveout2_p TX Vring p = "V0_p" d = 0xFFFFFFFF RX Vring p = "V1_p d = 0xFFFFFFFF Fixed region Dynmic lloc

20 emory ddress definition 20

21 32bit/32bit All options pping could be different from the different bus msters SoC memory mp 21 View from min processor 0xFFFFFFFF 0xFFFFFFFF View from slve processor i n C P v Physicl ddress is the reference to ccess 4GB memory mp from min CP. PA is decoded by bus p DDR Peripherls DDR (optionl) p Device ddress is the reference from slve processor. Will be decoded by bus to ccess peripherls d or s d / P v S l v e C P Peripherls Locl virtul ddress 0x SRA RO SRA 0x Optionl s to mp some memory chunks in slve processor ddress spce See Locl virtul ddress (optionl)

22 32bit/32bit Option1: coprocessor Without or s pping could be different from the different bus msters SoC memory mp 22 View from min processor 0xFFFFFFFF 0xFFFFFFFF View from slve processor i n C P v Physicl ddress is the reference to ccess 4GB memory mp from min CP. PA is decoded by bus p DDR Peripherls DDR (optionl) Device ddress is the reference from slve processor. Will be decoded by bus to ccess peripherls d S l v e C P Peripherls Locl virtul ddress 0x SRA RO SRA 0x Slve coprocessor hndles only physicl ddress (ned d from SoC point of view)

23 32bit/32bit Option1: coprocessor Without with s pping could be different from the different bus msters SoC memory mp 23 View from min processor 0xFFFFFFFF 0xFFFFFFFF View from slve processor i n C P v Locl virtul ddress Physicl ddress is the reference to ccess 4GB memory mp from min CP. PA is decoded by bus p 0x DDR Peripherls SRA RO DDR (optionl) Peripherls SRA p 0x Device ddress is the reference from slve processor. Will be decoded by bus to ccess peripherls s d Optionl s to mp some memory chunks in slve processor ddress spce S l v e C P Slve coprocessor hndles only physicl ddress (ned d from SoC point of view)

24 64bit min /32bit coprocessor All options from copressor pov pping is different s processors don't hve the sme ddress spce SoC memory mp 24 View from min processor View from slve processor i n C P v Physicl ddress is the reference to ccess the complete memory mp from min CP. PA (until 48bit) is decoded by bus p 0xFFFFFFFF DDR Peripherls DDR mirroring (optionl) 0xFFFFFFFF p Device ddress (32bit) is the reference from slve processor. Will be decoded by bus to ccess peripherls d or s d / P v S l v e C P Peripherls Locl virtul ddress 0x SRA RO SRA 0x Optionl s to mp some memory chunks in slve processor ddress spce Locl virtul ddress (optionl)

25 64bit min /32bit coprocessor Option1: coprocessor Without or s pping is different s processors don't hve the sme ddress spce SoC memory mp 25 View from min processor View from slve processor i n C P v Physicl ddress is the reference to ccess the complete memory mp from min CP. PA (until 48bit) is decoded by bus p 0xFFFFFFFF DDR Peripherls DDR mirroring (optionl) 0xFFFFFFFF Device ddress (32bit) is the reference from slve processor. Will be decoded by bus to ccess peripherls nd memories d / P v S l v e C P Peripherls Locl virtul ddress 0x SRA RO SRA 0x Locl virtul ddress (optionl) If no, DA is the reference

26 32bit min /64bit coprocessor Cortex considered s min processor In chrge of loding nd strting Cortex-A Appliction coprocessor. Linux slve rpmsg use cse pping is different s processors don't hve the sme ddress spce SoC memory mp 26 View from min processor View from slve processor i n C P v Physicl ddress is the reference to ccess the complete memory mp from min CP. PA (until 32bit) is decoded by bus p 0xFFFFFFFF DDR mirroring DDR Peripherls 0xFFFFFFFF Device ddress (48bit) is the reference from slve processor. Will be decoded by bus to ccess peripherls nd memories d / P v S l v e C P Peripherls Locl virtul ddress (if ny) SRA SRA Locl virtul ddress 0x RO 0x

27 Crveout hndling ccording to PA nd DA definition Vlue filed during firmwre genertion 27 se cses PA DA Actions Cse 1 Cse 2 = 0xFF FFFF (-1) = 0xFF FFFF (-1) = 0xFF FFFF (-1) in CP to llocte crveout buffer nd fill PA field. If s min cpu to progrm it to bind PA on DA nd fill DA field. If no s min CP should know how to convert PA in DA (prt of pltform driver) nd fill DA field = defined ddress Two different cses: - Crveout memory hs been reserved (pre-llocted) s this re is defined/fixed due to sme remote processor constrints. in CP to get PA vi pltform driver (crveout list). Access/lloction nd s configurtion (if ny) under pltform driver responsibility. - DA not found in crveout list, in CP need to llocte crveout buffer nd fill PA field. If s min cpu to progrm it to bind PA on specified DA Cse 3 = defined ddress = defined ddress Crveout memory should be reserved (pre-llocted nd registered in crveout list). in CP to enble its ccess If s min cpu to progrm it to bind PA on specified DA If no s nothing to do

28 Resource tble definition updte 28 To be future proof, ll resources should support versioning To support peer to peer mode, crveout resource owner should be defined. Crveout resource need to specify dedicted memory ttributes like cched/coherency Address should be 32bit nd 64bit pltform complint Sttus my be needed to indicte region is redy to ccess

29 Resource heder updte 29 /** * struct fw_rsc_hdr - firmwre resource entry heder resource type resource dt * * Every resource entry begins with 'struct fw_rsc_hdr' heder providing * The content of the entry itself will immeditely follow * this heder, nd it should be prsed ccording to the resource type. */ struct fw_rsc_hdr { u32 type; u8 dt[0]; } pcked; struct fw_rsc_hdr { union { All bytes of type field re not used tody, se union to insert versionning } u; u8 dt[0]; } pcked; u32 type; struct { u16 type; u8 ver; u8 ed; } s;

30 Crveout resource definition updte 30 Current fw_rsc_crveout struct is: struct fw_rsc_crveout { u32 d; u32 p; u32 len; u32 flgs; u32 reserved; u8 nme[32]; } pcked; 32bit ddress not complint with 64bit product flgs field is reserved to IO configurtion

31 Crveout resource definition updte 31 pdte proposl ccording to our discussion for fw_rsc_crveout struct : Is 32bit lenght enough in cse of 64bit coprocessor? Should we keep reserved s versionning exists now? struct fw_rsc_crveout { u64 d; u64 p; u32 len; u16 mem_flgs; u16 iommu_flgs; u8 owner_id; u8 sttus; u8 pd[2]; u32 reserved; u8 nme[32]; } pcked; DA field during firmwre genertion or during IO configurtion Need for peer to peer lloction

32 Vring resource definition updte 32 struct fw_rsc_vdev_vring { u64 d; u32 lign; u32 num; u32 notifyid; u64 p; } pcked; Different use cses: - Fixed during firmwre genertion, should belong to crveout re - Filled by remoteproc core during vring lloction; d = iommu configurtion or p trnsltion done pltform driver PA mybe needed by remote processor for DA trnsfer progrmming

33 Trce buffer updte 33 struct fw_rsc_trce { u64 d; u32 len; u32 reserved; u8 nme[32]; } pcked; Different use cses: - Fixed during firmwre genertion, should belong to crveout re - Filled by remoteproc core during vring lloction; d = iommu configurtion or p trnsltion done pltform driver

34 How to crete link between vdev nd RPsg buffer crveout 34 2 possibilities No informtion in resource tble: simply rely on crveout rsc nme to find crveout ssocited to RPsg buffer se n dditionl "resource index" to refer to crveout dedicted to RPsg buffer. Add crveout index in struct fw_rsc_vdev if only one memory pool is needed for RX nd TX buffers. Vlid in current configurtion in which mster lloctes ll buffers Add crveout index in struct fw_rsc_vdev_vring to ssocite memory pool with vring. Will be complint with peer to peer.

35 Remoteproc initiliztion 35 Rproc_core Rproc_drv probe New API to declre crveout int rproc_dd_crveout(struct rproc *rproc, struct rproc_mem_entry *crveout) With following chnge in struct rproc_mem_entry: struct rproc_mem_entry { void *v; dm_ddr_t dm; int len; u64 d; void *priv; void (*free)(struct rproc *rproc, struct rproc_mem_entry *mem); struct list_hed node; }; Could priv field be used? rproc_lloc rproc_dd_crveout rproc_dd rproc_hndle_crveout DT_prse For ech crveout region

36 Crveout hndling 36 DA = (- 1)? no yes Prse crveout list to find DA no DA found? Allocte region yes Configurtion lredy done by plt driver no s? yes Configure s with DA Register in Crveout list

37 Exmple for ST pltform B Need 2 crveouts 1 crveout with fixed ddress for code nd dt (optionl s pltform driver could register crveout bsed on DT definition) 1 crveout RPsg buffer loction, dynmiclly llocted by mster Resource tble DDR Resource tble Crveout.p = 0xFFFFFFFF.d = 0x len= 0x nme= fw Crveout.p = 0xFFFFFFFF.d = 0xFFFFFFFF.len= 0x80000.nme= RPSG RPSG vring1 vring0 RPSG_p Vring1_p Vring0_p Crveout.p= 0x d= 0x len= 0x nme= fw Crveout.p = RPSG_p.d = plt_drv_p2d (RPSG_p).len= 0x80000.nme= RPSG VDEV VDEV TX Vring.d= 0xFFFFFFFF TX Vring.d = plt_drv_p2d(vring1_p) RX Vring.p= 0xFFFFFFFF FW Bse ddress p= 0x RX Vring.d = plt_drv_p2d(vring0_p)

38 History 38 Dte Version Comment 12/07/17 V1 Cretion 10/08/17 V2 Add memory ddress definition 17/08/17 V3 Complete memory definition ccording to 10/08/17 OpenAP weekly Add fw_rsc_crveout struct evolution proposl 31/08/17 V4 Add resource evolution proposl Add B2260 resource tble exmple 24/09/17 V5 pdte resource tble structures 04/10/17 V6 pdte fter review with Wendy Add crveout hndler flowchrt