Outlier Detection With SQL And R. Kevin Feasel, Engineering Manager, ChannelAdvisor Moderated By: Satya Jayanty

Size: px

Start display at page:

Download "Outlier Detection With SQL And R. Kevin Feasel, Engineering Manager, ChannelAdvisor Moderated By: Satya Jayanty"

Imogen Fleming
5 years ago
Views:

1 Outlier Detection With SQL And R Kevin Feasel, Engineering Manager, ChannelAdvisor Moderated By: Satya Jayanty

2 Technical Assistance If you require assistance during the session, type your inquiry into the question pane on the right side. Maximize your screen with the zoom button on the top of the presentation window. Please fill in the short evaluation following the session. It will appear in your web browser.

3 Thank You to Our Sponsors Empower users with new insights through familiar tools while balancing the need for IT to monitor and manage user created content. Deliver access to all data types across structured and unstructured sources. Redgate Software makes ingeniously simple software used by 650,000 IT professionals who work with SQL Server,.NET, and Oracle. More than 100,000 companies use Redgate products, including 91% of the Fortune 100. Redgate s philosophy is to design highly usable, reliable tools which elegantly solve the problems that developers and DBAs face every day.

4 Join PASS PASS is a not-for-profit organization which offers year-round learning opportunities to data professionals. Access to online training and content Join Local Groups and Virtual Groups Get advance notice of member exclusives Enjoy discounted event rates MEMBERSHIP IS FREE, JOIN TODAY

5 Kevin Feasel Engineering Manager, Predictive Analytics Manager of a predictive analytics team at ChannelAdvisor in the Raleigh-Durham, NC area. Data Platform LinkedIn Curated SQL

6 Outlier Detection with SQL and R Kevin Feasel, Engineering Manager, Predictive Analytics

ChannelAdvisor in the Raleigh-Durham, NC area.

7 Kevin Feasel Engineering Manager, Predictive Analytics Manager of a predictive analytics team at ChannelAdvisor in the Raleigh-Durham, NC area. Data Platform LinkedIn Curated SQL

8 A Case Study Wake County transportation fraud

A Case Study Wake County transportation fraud An auto parts company, working in conjunction with several county employees, defrauded Wake County of at least $5 million over a multi-year period.

9 A Case Study Wake County transportation fraud An auto parts company, working in conjunction with several county employees, defrauded Wake County of at least $5 million over a multi-year period. Participants took advantage of lax vendor policy for small purchases and submitted fraudulent invoices, all under the cutoff for further scrutiny. Fraud went up to the top, to the head of the transportation department.

10 A Case Study Wake County transportation fraud The end result was several guilty pleas and jail time for participants.

11 A Case Study Wake County transportation fraud Wake County was able to claw back $5 million and Barnes Motor & Parts paid back $3 million, including a $2.5 million criminal fine.

12 Outlier Detection Know Your Data This is a quick overview of data samples used in this talk. Ideally, you want to know what the data elements represent and have as detailed an understanding of your data as possible. Patterns on their own are interesting, but are not indicative of malicious or irresponsible behavior.

13 Data Sets Raleigh police incidents Durham crime Wake County transactions FY Durham city budget FY North Carolina population by city 2013 My HOA s budgets

14 Know Your Data Understand the Data For your data sets, you should ideally have a detailed understanding of what data points represent and how valid the data is. Understand Inaccuracies For these open data sets, sometimes the data is purposefully incomplete or inaccurate. For example, police incident data does not include the acting officer or exact incident location, to protect all parties involved. Also, data pulled from Kaggle competitions and the like will have relevant features masked.

15 Outlier Detection Basic Analysis Our basic analysis will focus on a few important areas: Growth Analysis Gaps In Data Duplicates Cardinality Regression Analysis

16 Growth Analysis Growth analysis looks at changes in ratios over time. Our goal is to look for major changes which look to be far outside the norm. Demo upcoming! Typically, we will perform growth analysis against important additive measures: company growth, revenue, cost, number of customers, etc.

17 Gaps Sometimes, it's the lack of a value which is interesting. We want to look for sequences which should be complete but aren't--examples include receipts, purchase orders, check numbers, invoice numbers, etc. There are several T-SQL constructs which can help us find missing values, including the Gaps and Islands solutions and using a tally table. Warning: if you generate invoice numbers (for example) using a SQL Server identity integer, gaps may occur when transactions get rolled back. It may be better to use a surrogate key for storage but generate an invoice number only after the new transaction saves successfully. Demo upcoming!

18 Duplicates True Duplicates Mostly Duplicates Similarity Measures These are exact matches on relevant columns. For true duplicates, are there one-time codes or unenforced unique keys which show duplicates? Ex: internal tracking IDs. These are cases in which there are sometimes-unexpected differences. For "mostly" duplicates, are there combinations of fields which ought to be unique but aren't? Ex: both "Fundation" and "Foundation" showing up in company name. There are several measures for the similarity of two text phrases, including Levenshtein distance, soundex, and even pattern matching using LIKE. Depending upon the level of sophistication necessary, getting the similarity of two items based on text descriptors can take quite a bit of effort!

19 Duplicates: Wake County Tying back to the Wake County fraud case, there were 24 separate days in which the auto supplier submitted 50+ invoices under $2500 to the county. Here, the duplication is in vendor and date and is well outside the expected behavior for a vendor on the up-and-up. An important question to ask in your code base is, what combinations of measures look out of place? This requires a detailed understanding of your data.

20 Outlier Detection Regression Analysis Regression analysis can be extremely complicated, given issues like multicollinearity, serial correlation, and heteroskedasticity. Still, a basic ordinary least squares linear regression can tell us a lot.

21 Outlier Detection Cohort Analysis Looking at top-level aggregates or a single slice (customer, vendor, group) might help us find problems, and looking at a broad-based comparison of these slices may give us a clearer picture. A data set can be sliced many ways, so there could be a large number of relevant cohorts.

22 Box Plots Box plots are valuable for comparing cohorts. Each box plot will show, by group, the following data points: the mean, 25th percentile, 75th percentile, 1.5 * interquartile ranges, and any outliers.

23 Outlier Analysis Outliers Clusters Differences Plotting a trend and looking for outliers (defined either in a technical sense or in the general sense of "well outside the norm") can dig up strange behavior. Real data sets will always have outliers, though, so the existence of outliers is not in itself suspicious. Something interesting when analyzing a cohort is to look for clustering around certain values. Examples: expense reports and invoices. Going back to our Wake County example, there was a cluster around $2500, as that was the cutoff point after which a second person needed to sign off on a payment order. When comparing different members of a cohort, look at differences between groups and differences within a group. Differences between groups may indicate suspicious behavior if one member of the group acts well outside the norm, such as in the number of invoices submitted per day. Looking at differences within a group can help explain anomalous aggregate behavior.

24 DEMO TIME

25 Outlier Detection Time Series Analysis The easiest time series analysis is a count of items per day. Going back to the Wake County example, auditors could have found something suspicious with the Barnes company pretty easily: there were 24 separate days with at least 50 invoices, certainly a suspicious scenario. Other time series examples include the sums of measures by day or month, such as revenue, cost, number of orders, and amounts of refunds.

26 Time Series Analysis Balances Date Checks Holidays For semi-additive measures such as account balances, look at the state of the measure at fixed points in time, such as looking at account balances every day at 9 PM UTC. Look for "odd" date combinations: order date later than ship date? Adjustments after a journal closes? Look at behavior based on date: focus on day of week, day of month, weekdays vs weekends, mid-week versus early week vs late week. Sum and count measures grouped on these date slices. Holidays can lead to different behavior like more sales of higher amounts. They can also lead to less activity, like how when invoices are due on a holiday, they may be submitted the next working day. Be aware of floating holidays like Easter, as well as observed holidays like Christmas Day observed. It's easiest to store holidays in a date table to avoid trying to calculate these rules on the fly.

27 DEMO TIME

28 Outlier Detection Numeral Analysis Even without a detailed understanding of data elements, we can gain some insight simply from looking at the digits which make up these data points.

29 Round Numbers One high-level analysis of values is to get a count of "round numbers." We group into types based on the number of 0s at the end (ignoring cents): $5000 (type 3), $10,200 (type 2), $180 (type 1), $17,999 (type 0). Demo upcoming.

30 Benford s Law Benford's Law is an interesting phenomenon, where the first digit (or two digits, or three digits) of a set of numbers tends to follow a specific pattern. Demo upcoming.

Last Digit Analysis As we saw in the last demo, Benford's Law doesn't apply to the last digit in a sequence; instead, we assume those are uniform unless there is a reason to believe otherwise.

31 Last Digit Analysis As we saw in the last demo, Benford's Law doesn't apply to the last digit in a sequence; instead, we assume those are uniform unless there is a reason to believe otherwise. People are terrible at generating random numbers; the human mind tends to think 3, 7, and 8 feel more random than 0 or 5, so we can see certain digit patterns pop up when people fudge the data.

32 Wrapping Up This has been a quick survey of forensic accounting and outlier detection techniques. You can apply these techniques to a number of data sets, not only to find potential instances of fraud, but also to search for potential data integrity issues and even to gain more insight into your data. To learn more, go here: And for help, contact me.

33 QUESTIONS?

34 Coming up next! Securing SQL Server Processes with Certificates Robert Davis

35 THANK YOU FOR ATTENDING

Securing SQL Server Processes with Certificates. Robert, Davis, Database Engineer, BlueMountain Capital Management Moderated By: Ivan Sanders

Securing SQL Server Processes with Certificates Robert, Davis, Database Engineer, BlueMountain Capital Management Moderated By: Ivan Sanders Technical Assistance If you require assistance during the session,