HOW TO TEST CROSS-DEVICE PRECISION & SCALE

Transcription

1 HOW TO TEST CROSS-DEVICE PRECISION & SCALE

2 Introduction A key consideration when implementing cross-device campaigns is how to strike the right balance between precision and scale. Your cross-device campaign will only be effective if it can identify numerous device matches (scale), with a level of statistically relevant precision. There are numerous ways to use crossdevice data; some of which require large scale, and some of which require precision. A precision and accuracy test will help you determine if your data is more aligned with one versus the other, and any trade-off you may need to make. It is also a way of testing the quality of a cross-device graph. Precision & Defined Precision is the number of predicted matches in the device map that are truly linked to the same individual. is the percentage of true matches that are predicted by the device map. For example, say you have a deterministic data set which tells you that a user has six different IDs across different devices, which we ll call A, B, C, D, E and F. If the device map tells you that A, B, D and G are linked, the precision of the device map is 75%. However the scale is only 50%, since only three of the six IDs are correctly identified (G is not a correct prediction).

3 Precision vs. There will always be a tradeoff between precision and scale. A probabilistic device map matches devices to individual consumers by analyzing signals that a device may belong to the same user. If a device map vendor wanted to achieve 100% precision, they would have to only include those matches where the signals were so clear that it was almost certain that two devices belong to the same person. To achieve this level of precision though, the device vendor would have to exclude all the matches that are below 100% certainty, which would make the results much smaller. Conversely, a marketer could aim for very high scale by including matches where the vendor was only 50% sure. In this case, precision would suffer. Precision Vs Recall Actual Device Matches High Precision Prediction High Recall Prediction Precision: Percentage of correct predicted matches. Most important for measurement Recall: Percentage of true matches predicted Most important for targeting How you plan to use the data will help you determine whether to focus on precision or scale or vice versa. For sequential messaging, precision is more important than scale. On the other hand, if you are trying to increase your scale across new screens, you may be more interested in the scale of the device map, at the cost of some accuracy..

4 01 Cross-Device Testing Here are example uses for cross-device data, and which metric will be most relevant for you accordingly: USE CASE DESCRIPTION PRECISION OR SCALE? advertising Target users across additional devices retargeting Capture a customer Audience extension Expand audience to reach additional screens Frequency capping Limit ad views across devices Precision/ Sequential messaging Run cross-device sequential campaigns Precision/ Content personalization Personalize content across a user s device Precision/ Offer personalization Provide offers based on a user s interests Precision/ analytics Understand a customer s journey across devices Precision attribution Understand when multiple devices are involved in a conversion Precision True unique reach Count people, not devices Precision 3

5 In order to achieve the best performance from a device map, there are some use cases where it is preferable to focus on precision at the expense of scale. In others, scale is the more important. Precision matters most for cross-device analytics and attribution use cases. Incorrect data in analytics use cases can lead to reduced effectiveness and decrease ROI. For analytics applications, precision is more important than covering the entire market - as a representative sample can provide the necessary intelligence to drive business decisions. For larger advertising campaigns with high volume, it makes sense to focus on reaching all of the devices associated with a particular customer, and hence, scale. The lack of 100% precision means that some devices will not belong to the intended audience, but the cost of a mistake is relatively small. Use the graph below to determine the ideal balance between precision and scale for your cross-device campaign Target Target Measure Device graph scale (Recall) campaign retargeting Audience extension Frequency capping Measure Customer journey insight Measure Conversion attribution Device graph performance (Precision) -- 4

6 02 A Guide To Cross-Device Terminology Before we go into the details of running a test, here are some terms you should know: Cookie Sync Linked Device Device Maps True Positive True Negative False Positive False Negative Probabilistic data Deterministic data A cookie sync is the way you share your cookie IDs with a cross-device partner to identify a user s browser. A prediction that devices are connected. A map of different devices, and the connections between them. A correct prediction that two devices are linked. ( True as in Correct and Positive as in prediction there is a link between devices ) A correct prediction that two devices are not linked ( True as in Correct and Negative as in prediction there is not a link between devices ) An incorrect prediction that two devices are linked ( False as in incorrect and Positive as in prediction that there is a link between devices ) An incorrect prediction that two devices are not linked ( False as in incorrect and Negative as in prediction that there is not a link between devices ) A probabilistic map of devices that uses different data signals to create links between devices based on anonymous data. A precision and scale test ascertains the strength and the scope of these signals. A map of IDs that uses login data. Deterministic data is often used to provide a statistically perfect set of data to measure probabilistic data against. 5

7 03 How To Run A Precision and Test Following is an example of how to work with a cross-device partner to test the precision and scale of their cross-device map. In this example, we re interested in scale but we also want to make sure that there s a relatively high level of precision because attribution is important to us. Step 1. The cross-device partner provides a cookie sync pixel. Step 2. Place the cookie sync pixel where your users interact with your campaign creative or your website. Step 3. After two weeks of syncing, the cross-device partner looks up your cookies to find all other cookies and device IDs to which they are linked. Step 4. The cross-device partner will give you access to the mapping files in a storage location. Step 5. You will then be able to compare the cross-device mappings to your deterministic set in order to compute the results. Step 6. The results are in! You are ready to calculate precision and scale. POSITISVE NEGATIVE TRUE FALSE Calculate precision using the following formula: Precision = TP / (TP + FP) Precision = (40 / )*100 = 80% Calculate scale using the following formula: = TP / (TP + FN) = 40 / ( )*100 = 57% 6

8 04 How To Run A Precision and Test 1. Avoid Duplicated Pairs: It s important to make sure that your deterministic truth set contains a matching pair only once. It s common to find truth sets that contain the same match listed twice, but in a different order. To prevent this, try to ensure that all of your matched pairs are listed in a consistent order. It doesn t matter which order is selected, as long as it is consistent across truth sets. The same, of course, is true of the map that is provided by your cross-device partner. 2. Use a Large Deterministic Set: It s important to use a large enough deterministic set to properly understand differences between two device maps. As we mentioned before, probabilistic graphs work by linking devices according to certain criteria. For example, only consider the two devices to be linked if a device belongs to the same household. As the size of your truth set increases, you have more chance of having data on more than one person in a household. There is therefore more likelihood of identifying a false positive. Therefore, when the truth set is small, precision will be artificially high. 3. Ensure That The Deterministic Set Is A Good Match For The General Population: It s important to ensure that your deterministic truth set is representative of the population at large. For example, a truth set of the data that was sourced from a very specific geographic location. If the deterministic truth set you use to measure a cross-device graph only matches users from Guam, for instance, it won t be a good representation of the general US population, as 99.96% of the US population do not live in Guam. 7

9 4. Use Clean Data Sets: As we said before, deterministic truth sets are susceptible to their own problems, particularly with dirty data. Some examples of dirty truth sets might be those which contain users who have a huge number of devices (perhaps someone whose job it is to review smartphones and mobile devices), or those where there are several devices (this could be a problem with devices that are typically shared, such as tablets). Now You re Ready To Run Your Own Test! If you are considering evaluating a cross-device map, Adbrain invites you to contact us to help you ensure that your measurements are accurate. We ve worked with many companies to help them run precision and scale tests. Contact us for more information.