Creating a customer event based data warehouse

Size: px

Start display at page:

Download "Creating a customer event based data warehouse"

Tamsin Hudson
5 years ago
Views:

Creating a customer event based data warehouse David Porter June 12th 2002 2000 Detica Limited; ALL RIGHTS RESERVED Copyright in the whole and every other part of this document belongs to Detica

1 Creating a customer event based data warehouse David Porter June 12th Detica Limited; ALL RIGHTS RESERVED Copyright in the whole and every other part of this document belongs to Detica Limited (the "Owner") and may not be used, sold, transferred, copied or reproduced in whole or in part in any manner or form or in or on any media to any person other than with the Owners prior written consent.

2 Creating a customer event based data warehouse 2! Creating a single view of customer " risks issues & advantages! Understanding customer events! Case study! Tips

3 Creating a single view of customer 3! In reality, we can establish multiple single views " Household Individuals grouped by address Difficult/low value tracking over time " Marketing/Analytic view Matching individuals based on limited data Powerful but obtuse " Data/fact view Matching individuals with perfect data Accurate - DPA compliant - but limited value " B2B Hierarchical and aggregated views of the businesses you do business with Limited agency data - add your own intelligence! Key is consistent data & managed approach

4 Why we want a single view 4! To make money " Improve customer satisfaction " Improve marketing effectiveness! To save money " Improve marketing & service efficiency Differentiated service " Optimise expenditure across product groups " Reduce risk Fraud Bad debt

5 Barriers to getting a single view 5! Getting Data! Data quality! Bogus data! Shelf life! Matching! Changing customer & changing business! Computational complexity

6 Common customer data problems 6! Poorly maintained! Ambiguous! Not consistent across all business processes! Time dependent! Bogus " Corrupt " Fraudulent " Profane " Mickey Mouse

7 The key to getting better data 7! Do not expect or assume honesty! Make it easy to provide! Explain data use " the benefits of giving it to you " how they can change it later! Get explicit permission to use their data " note that data without contact permission is also interesting " Do the people who opt out have any particular demographic! This should be thought of as ongoing process " more likely to get truth and more up to date

8 Check the sell-by date 8! Be wary about making decisions on demographic data that is greater than 1 year old.! Agency data often very short shelf life! Time-stamp all data you collect! Consider how your own circumstances have changed in last year. " Who is more interesting to the marketeer - the changers or the stayers?

9 Weaknesses in typical data warehouse designs 9! Focus on transactional change " Changes in the non-transactional and aggregate data over time are lost e.g. change of address! Organisation re-organisations create havoc " Adding and removing feeds is a major exercise! Event driven campaigns require hard coding - takes time to change! Legal rules complex to code and administer! De-duplication/Matching seen as simple coding issue

10 The challenge of matching 10 Address address Name NI number Date of birth!none of these data items guarantee an individuals identity but combinations can achieve a high degree of certainty Telephone Account number User ID!Real time capture is easier than offline matching

11 Add breadth to data 11! Be creative " Web analysis - what route do my customers take through my site. " Put web links to other sites that you think may interest your customers note what works " Build profiles of customer likes/dislikes " Think about offering products/rewards that match these customer interests! Supermarkets sell newspapers not for really for profit but for demographics and where to advertise

12 You are now at base camp! 12! A lot of companies do not get this far! Some get here without enough rigour! Many organisations simply stop here.! The key to the next level of benefits is to start looking at changes in the data.

13 Handling change 13! So now you have nice clean de-duplicated data! it s time to segment Gold Silver Prospect Bronze

14 Delta Segmentation 14! How many customer types do I have? Gold Silver Prospect Bronze

15 Delta Segmentation 15! Think about the movement between segments. This does not mean changing your segmentation!

16 Focus on the customers who change 16 I must make this customer feel special Is there a fraud going on here? Now may be a good time for a special offer What campaign worked here? I must instigate an outbound call campaign Good riddance? Is my best customer thinking of leaving? What did it cost to acquire this customer?

17 What is an Event? 17 # a personal event is usually categorised by some change in status for an individual e.g. birth, marriage, death, promotion, moving house # some events can be inferred from data - combinations of small changes to data are used to identify potential customer status changes e.g. knowing that someone has changed car, moved to a better area, may be of interest? # different data sources are better or worse at inferring customer events usually to do with the frequency of update # by identifying customers across brands, events can be shared e.g. it may take the AA up a year to find out someone has moved, whilst British Gas may even know in advance

18 Case Study: Centrica 18! Background " Very large complex consumer business, diversified by multiple acquisitions " Single view of customer needed to leverage advantages of being multi-brand! Solution " New type of data warehouse incorporating semantic database architecture and fuzzy matching technology " Warehouse efficiently handles 2Tb of information on 18m customers, fed from over 50 systems " Enables sophisticated multi-brand event marketing " Legal complexities handled implicitly by architecture " Implementation completed in 14 months

19 Case Study - differences 19! Delta Repository " only processed customer data that had changed in some way. " processing down from 4 weeks to a few hours. " Standard reports include new customers, leaving customers and data quality statistics.! We used some techniques from Semantic db theory " capturing every small change in the static data as an event record..! Shannons communication theory: Message is more significant if likelihood of change is small.

20 Technology independent modular architecture 20 Marketing Automation Data feeds Delta Management Historical Data Store Data-Mart Creation Ad-hoc Analysis Customer Matching Meta-data Management

21 Technology independent modular architecture 21 Protagona Data feeds Base SAS IW Macro SAS Datasets & SPDS SAS datasets & Oracle tables SAS Enterprise Miner Trillium SAS Warehouse Administrator

22 22 System Overview Source data flat file Storing data over time Flat file data is organised into a more efficient structure for historical data storage. Historical Data Store Data Marts Data marts can be created to fit business requirements. Protagona SAS Business Objects

23 23 Storing data over time - deltas Source data flat file Delta Creation To minimise data storage only changed records are passed through to the HDS. Historical Data Store Data Marts Protagona SAS Business Objects

24 24 Matching the customer Changed or new records (deltas) that have changes to name or address fields are passed to the CMA. Source data flat file Delta Creation Historical Data Store Electoral Roll CMA Post Office PAF Customer Matching Area Database Query The CMA queries the existing customer base to hunt for possible matches. Data Marts Protagona SAS Business Objects

25 25 Adding new feeds Source data flat file Delta Creation Electoral Roll Post Office PAF A distinct historical store is built for each feed. This facilitates parallel builds and allows for future separation if necessary. Historical Data Store CMA Customer Matching Area Database Queries Data Marts Protagona SAS Business Objects

26 26 Managing Complexity Source data flat file Delta Creation Electoral Roll Post Office PAF Historical Data Store CMA Customer Matching Area Database Query Data Marts Protagona SAS Business Objects METADATA management

27 27 Customer Hub & Operational CRM Systems Transaction systems Actions Customer Contact Channels Automated Call Centre Matching Queries CMA CHub Campaigns Operational marts Treatments & models

28 28 Matching the customer Modified name & address records returned Name Address... Name Address... Name Address... Name Address... Electoral Roll Customer Matching Area can be independently called by other, operational, systems! Matching Rulesets Flat file of Name & address records input Name Address... Name Address... Name Address... Name Address... CMA Post Office PAF The CMA generates a list of premise id s. List used to create a query on all the HDS to build list of people who live at same premise id. Premise ID Premise ID Premise ID Premise ID Candidate List Name Address... Name Address... Name Address... Name Address... Name Address... Name Address... Name Address... Fuzzy logic is used to establish the corresponding unique premise id from the PAF file. Candidate list is compared to incoming names and electoral roll - matches are identified Customer id s inherited.

29 Benefits 29 Function Benefit Beneficiary Customer centric accounting Customer memory Customer event detection Multiple single views Single customer view Flexible repository Consolidated repository Business-wide tracking of customer KPIs Can be used to diagnose causes of customer changes Identification of customer changes Different views for different business requirements Holistic picture of the customer Avoids redevelopment cost when requirements change Eliminate cost of managing multiple data stores Group Market, Sales & Service operations IT operations Value 50m 10m 1m

30 Top Tips 30! Data Quality must be first priority! Understand time in your data - especially change data! Innovate:- engineer the customer experience to collect and use data to mutual advantage! Design data warehouse to augment an event based campaign tool but keep it discrete from it! Events need to be acted upon in a timely manner. " Multi brand organisations often have different idea of timely! Integrate with operational systems and measure success

31 Data Loading 31 Raw Data Exceptions Date exceptions Records with null fields Duplicates Read in raw data, process exceptions (base SAS) Staging layer Dimension/ fact tables Statistics Exception statistics Load time statistics

32 Updating the HDS 32 Dimension/ fact tables Statistics & IW control IW Event marts SAS IW delta detection (N-1) Next Nikey table Historical data store Primary key table Dimension/fact tables Column change statistics Key tables Population statistics Open views Update statistics Timing statistics File size statistics

33 Customer Matching 33 Historical data store Dimension/fact tables Key tables Open views Anti-Match table Delta driven CMA output Allocate premise IDs Premise ID query DPA individual match DPA ER individual match

34 Anti-Match App DPA Enquiry App CMA Applications 34 Global tables Divergence App DPA Enquiry table Anti-Match table GURN x-reference Premise table CMA flag table CMA Flag App Temporary tables Flag table # state of the art matching at the heart of the Customer hub # all new or changed customer records are matched # produces the only up to date Centrica-wide view of individuals # processes in place to match regular, ad-hoc or agency list data with Centrica customer base # multiple rule sets to allow for different confidence levels of match for different purposes Divergence table DPA individual match DPA ER individual match Historical Data store Premise dimension (PIDS) HDS update (DPA ruleset) HDS update (Analysis ruleset) Customer dimension (GURN & single_compound) CMA stats DPA ER table

35 Raw Data 35 Exceptions Date exceptions Records with null fields Read in raw data, process exceptions (base SAS) Staging layer Statistics Exception statistics Load time statistics Duplicates Dimension/ fact tables Statistics & IW control IW Event marts SAS IW delta detection (N-1) Next Nikey table Anti-Match App DPA Enquiry App Historical data Store Dimension/fact tables Primary key table Column change statistics Global tables Key tables Population statistics DPA Enquiry table Open views Update statistics Anti-Match table Timing statistics GURN x-reference Premise table Delta driven CMA output File size statistics CMA flag table Protagona Allocate premise IDs Divergence App CMA Flag App Household Mart Premise ID query Temporary tables Flag table Individual Mart Divergence table Historical Data store Premise dimension (PIDS) DPA individual match HDS update (DPA ruleset) DPA ER individual match HDS update (Analysis ruleset) business mart business mart business mart Customer dimension (GURN & single_compound) CMA stats DPA ER table

DATABASE DEVELOPMENT (H4)

IMIS HIGHER DIPLOMA QUALIFICATIONS DATABASE DEVELOPMENT (H4) December 2017 10:00hrs 13:00hrs DURATION: 3 HOURS Candidates should answer ALL the questions in Part A and THREE of the five questions in Part