Truncate, replicate, sample

Size: px

Start display at page:

Download "Truncate, replicate, sample"

Patience Adams
5 years ago
Views:

1 Truncate, replicate, sample A probabilistic method of integerisation Robin Lovelace & Dimitris Ballas Sheffield Presented at the IMA, May 2012, Dublin Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 1 / 26

2 Outline 1 Introduction Background Integerisation: what and why? Existing approaches 2 TRS: A probabilistic approach Truncate Replicate Sample 3 Results Speed of calculation Population size Accuracy 4 Conclusion 5 Further work Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 2 / 26

3 Background [2] Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 3 / 26

4 Background Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 4 / 26

5 Spatial microsimulation Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 5 / 26

6 Non-integer weights Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 6 / 26

7 What is integerisation? It s going from this: To this: Table: IPF results ID Weight Table: Integerised results ID Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 7 / 26

8 Why integerise? Makes your analysis easier* Conceptual advantage: e.g. discrete car data Gini index for inequality Mean is available from IPF results The relationships between discrete workers and jobs Individuals useful for agent-based modelling and dynamic microsimulation Best of both worlds? [4] *Subject to debate Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 8 / 26

9 Simple rounding and inclusion thresholds Simple rounding includes individuals who have weights > 0.5 [1] The problem: population totals do not match Solution: add individuals up to a given threshold count Inclusion threshold Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 9 / 26

10 Performance of existing approaches Simulated 4000 Method Rounding Threshold Census Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 10 / 26

11 Performance of existing approaches 2 All microdata (n = 4933) Sampled (n = 2541) Weight Index Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 11 / 26 Index

12 Design criteria for better integerisation Simplicity of simple rounding Correct population size of threshold approach Representative of individuals with higher weights Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 12 / 26

13 Outline 1 Introduction Background Integerisation: what and why? Existing approaches 2 TRS: A probabilistic approach Truncate Replicate Sample 3 Results Speed of calculation Population size Accuracy 4 Conclusion 5 Further work Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 13 / 26

14 Truncate As simple as In R, performed by command trunc() Always leads to a population underestimate (useful) Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 14 / 26

15 Replicate Simply replicate the individuals whose truncated weight > 1 Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 15 / 26

16 Sample The important part Fills up partially empty zones with representative individuals Sample size = zone population population after truncation and replication Probability is determined by weight remainder (e.g ) Simple to code in R: sample(x, size =..., prob =...) Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 16 / 26

17 Integerisation in action All microdata (n = 4933) Sampled (n = 2541) Weight Index Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 17 / 26 Index

18 Results Rounding Threshold TRS 3000 Simulated 2000 Constraint Age/sex Distance Mode NS SEC Census Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 18 / 26

19 Speed of calculation New method is faster than threshold method But integerisation small portion of overall computational time 3 seconds for integerisation vs 5 minutes for IPF Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 19 / 26

20 Population size Table: Differences between Census and simulated populations. Metric Rounding Threshold TRS Mean Standard deviation Max Min Mean oversample -13% 0.3% 0.0% TRS method ensures correct population size Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 20 / 26

21 Table: Accuracy results for integerisation techniques [3]. Method Variables TAE SAE Errors > 5% Zm 2 IPF Age/sex 9 0% 0% 0 Distance % 14% 640 Mode % 6% 593 NS-SEC 0 0% 0% 0 All % 5% 1233 Rounding Age/sex % 81% 5247 Distance % 80% Mode % 82% 8896 NS-SEC % 76% 7758 All % 80% Threshold Age/sex % 49% 1074 Distance % 83% 8890 Mode % 68% 3678 NS-SEC % 56% 2132 All % 63% TRS Age/sex % 28% 309 Distance % 53% 1449 Mode % 50% 1035 NS-SEC % 28% 392 All % 39% 3184 Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 21 / 26

22 Allows intra-zone analysis Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 22 / 26

23 Key references Dimitris Ballas, Graham Clarke, Danny Dorling, Heather Eyre, Bethan Thomas, and David Rossiter. SimBritain: a spatial microsimulation approach to population dynamics. Population, Space and Place, 11(1):13 34, January Stan Openshaw. The modifiable areal unit problem. Geo Books Norwich UK, David Voas and Paul Williamson. Evaluating Goodness-of-Fit Measures for Synthetic Microdata. Geographical and Environmental Modelling, 5(2): , November P Williamson, Mark Birkin, and P H Rees. The estimation of population microdata by using data from small area statistics and samples of anonymised records. Environment and Planning A, 30(5): , Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 23 / 26

24 Integerisation in action Age Constraint Mode Constraint Simulated Variable type Age/sex Distance Mode NS SEC Distance Constraint NS SEC Constraint Simulated Census Census Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 24 / 26

25 Integerisation in action 0.6 Error (proportion of points beyond 5% of census value Constraint Age/sex Distance Mode N. cars NS SEC Iteration Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 25 / 26

26 Contact me! Try the model! Robin Lovelace & Dimitris Ballas (Sheffield) Truncate, replicate, sample Integerisation 26 / 26

Package raker. October 10, 2017

Package raker. October 10, 2017 Title Easy Spatial Microsimulation (Raking) in R Version 0.2.1 Date 2017-10-10 Package raker October 10, 2017 Functions for performing spatial microsimulation ('raking') in R. Depends R (>= 3.4.0) License