DATA MINING II - 1DL460

Transcription

1 DATA MINING II - 1DL460 Sprig 2017 A secod course i data miig Kjell Orsbor Uppsala Database Laboratory Departmet of Iformatio Techology, Uppsala Uiversity, Uppsala, Swede

2 Aomaly Detectio (Ta, Steibach, Kumar ch. 10) Kjell Orsbor Departmet of Iformatio Techology Uppsala Uiversity, Uppsala, Swede

3 What are a aomaly or outlier? What are aomalies/outliers? Sigle or sets of data poits that are cosiderably differet tha the remaider of the data (i.e. ormal data) E.g. uusual credit card purchase, sports: Usai Bolt, Leo Messi, Outliers are differet from the oise data Noise is radom error or variace i a measured variable Noise should be removed before outlier detectio Outliers are iterestig: It violates the mechaism that geerates the ormal data Outlier detectio vs. ovelty detectio: early stage, outlier; but later merged ito the model Applicatios: Credit card fraud detectio, telecommuicatio fraud detectio, etwork itrusio detectio, fault detectio, customer segmetatio, medical aalysis

4 Aomaly/outlier detectio Variats of aomaly/outlier detectio problems: Give a database D, fid all the data poits x D with aomaly scores greater tha some threshold t Give a database D, fid all the data poits x D havig the top- largest aomaly scores f(x) Give a database D, cotaiig mostly ormal (but ulabeled) data poits, ad a test poit x, compute the aomaly score of x with respect to D

5 Types of outliers (I) Three kids: global, cotextual ad collective outliers Global outlier (or poit aomaly) Object is O g if it sigificatly deviates from the rest of the data set Ex. Auditig stock tradig trasactios Issue: Fid a appropriate measuremet of deviatio Cotextual outlier (or coditioal outlier, ote: special case is local outlier) Object is O c if it deviates sigificatly based o a selected cotext Ex. -20 o C i Uppsala: outlier? (depedig o summer or witer?) Attributes of data objects should be divided ito two groups Cotextual attributes: defies the cotext, e.g., time & locatio Behavioral attributes: characteristics of the object, used i outlier evaluatio, e.g., temperature Ca be viewed as a geeralizatio of local outliers whose desity sigificatly deviates from its local area. Issue: How to defie or formulate meaigful cotext? Global Outlier

6 Types of outliers (II) Collective Outliers A subset of data objects collectively deviate sigificatly from the whole data set, eve if the idividual data objects may ot be outliers Applicatios: : e.g., itrusio detectio: Collective outlier Whe a umber of computers keep sedig deial-of-service packages to each other Detectio of collective outliers Cosider ot oly behavior of idividual objects, but also that of groups of objects Need to have the backgroud kowledge o the relatioship amog data objects, such as a distace or similarity measure o objects. A data set may have multiple types of outliers Oe object may belog to more tha oe type of outlier

7 Challeges of outlier detectio Modelig ormal objects ad outliers properly Hard to eumerate all possible ormal behaviors i a applicatio The border betwee ormal ad outlier objects is ofte a gray area Applicatio-specific outlier detectio Choice of distace measure amog objects ad the model of relatioship amog objects are ofte applicatio-depedet E.g., cliic data: a small deviatio could be a outlier; while i marketig aalysis, larger fluctuatios Hadlig oise i outlier detectio Noise may distort the ormal objects ad blur the distictio betwee ormal objects ad outliers. It may help hide outliers ad reduce the effectiveess of outlier detectio

8 Challeges of outlier detectio cot Uderstadability Uderstad why these are outliers: Justificatio of the detectio Specify the degree of a outlier: the ulikelihood of the object beig geerated by a ormal mechaism How may outliers are there i the data? Whe method is usupervised Validatio ca be quite challegig (just like for clusterig) Outlier detectio ca be compared to fidig eedle i a haystack Workig assumptio: There are cosiderably more ormal observatios tha abormal observatios (outliers/aomalies) i the data

9 Ozoe depletio history: Importace of aomaly detectio I 1985 three researchers (Farma, Gardiar ad Shakli) were puzzled by data gathered by the British Atarctic Survey showig that ozoe levels for Atarctica had dropped 10% below ormal levels Why did the Nimbus 7 satellite, which had istrumets aboard for recordig ozoe levels, ot record similarly low ozoe cocetratios? The ozoe cocetratios recorded by the satellite were so low they were beig treated as outliers by a computer program ad discarded! Sources:

10 Aomaly detectio schemes Geeral steps Build a profile of the ormal behavior Profile ca be patters or summary statistics for the overall populatio Use the ormal profile to detect aomalies Aomalies are observatios whose characteristics differ sigificatly from the ormal profile Types of aomaly detectio schemes: Graphical & Statistical based Proximity based Desity based Clusterig based

11 Graphical approaches Boxplot (1-D), Scatter plot (2-D), Spi plot (3-D) Limitatios Time cosumig Subjective

12 Covex hull method Extreme poits are assumed to be outliers Use covex hull method to detect extreme values Data are assiged to layers of covex hulls that are peeled of to detect outliers What if the outlier occurs i the middle of the data?

13 Statistical approaches Assume a parametric model describig the distributio of the data (e.g., ormal distributio) Apply a statistical test that depeds o Data distributio Parameter of distributio (e.g., mea, variace) Number of expected outliers (cofidece limit)

14 The Grubbs test Detect outliers i uivariate data (i.e. data icludig oly oe attribute) assumig data sample comes from ormal distributio: The Grubb's test (also called maximum ormed residual test) Outlier coditio is defied as: G exp > G critical For each object x i a data set, compute its z-score (i.e. G exp ): z = x x where s is stadard deviatio ad x is the mea (also G exp ) s x is a outlier if: G exp = (also termed G critical ) where is the value take by a two-sided t-distributio at a sigificace level of α/(2n), ad N is the o of objects i the data set

15 Statistical-based likelihood approach Idetifyig outliers by calculatig the chage i likelihood whe movig a poit from oe distributio to aother i a mixture of 2 distributios. The overall probability distributio of the data: D = (1 λ) M + λ A, where λ is the expected fractio of outliers. M is a probability distributio estimated from data, usually Gaussia ca be based o ay modelig method (aïve Bayes, maximum etropy, etc) A is assumed to be a uiform distributio Likelihood ad log likelihood at time t: = = = = t i t t i t t i t t t i t t A x i A t M x i M t t A x i A A M x i M M N i i D t x P A x P M D LL x P x P x P D L ) ( log log ) ( log ) log(1 ) ( ) ( ) ( ) (1 ) ( ) ( 1 λ λ λ λ

16 Statistical-based likelihood approach Assume the data set D cotais samples from a mixture of two probability distributios: M (majority distributio, typically Gaussia) A (aomalous distributio, typically uiform) Geeral approach of algorithm 10.1 (Ta et al): Iitially, assume all the data poits belog to M Let LL t (D) be the log likelihood of D at time t For each poit x t that belogs to M, move it to A Let LL t+1 (D) be the ew log likelihood. Compute the differece, Δ = LL t (D) LL t+1 (D) If Δ > c (some threshold), the x t is declared as a aomaly ad moved permaetly from M to A

17 Statistical-based likelihood approach Algorithm 10.1 (Ta et al):

18 Limitatios of statistical approaches Most of the tests are for a sigle attribute I may cases, the data distributio may ot be kow For high dimesioal data, it may be difficult to estimate the true distributio

19 Proximity-based outlier detectio I proximity-based outlier detectio a object is a outlier if it is distat from most poits called distat-based outliers More geeral ad easily applied tha statistical approaches sice usually easier to defie proximity measure There are various ways to defie outliers: Data poits for which there are fewer tha p eighborig poits withi a distace D Data poits whose distace to the kth earest eighbor is greatest Ca be sesitive to value of k Data poits whose average distace to the k earest eighbors is greatest more robust tha oly distace to kth earest eighbor Compute the distace betwee every pair of data poits ca make it expesive, O(m 2 ) Grid-based methods ad idexig ca improve performace ad complexity Does ot hadle widely varyig desities well sice usig global thresholds

20 Nearest-eighbor based approach Example where the outlier score is give by the distace to its k-earest eighbor

21 Desity-based outlier: Desity-based outlier detectio Outliers are poits i regios of low desity The outlier score of a object is the iverse of the desity aroud the object. Iverse distace desity (iverse of averaged distace to the k-earest eighbours):, where N(x,k) is the set of k-earest eighbors of x, N(x,k) is the size of that set ad y is a earest eighbor. No of poits withi regio desity (DBSCAN): The desity aroud a object is equal to the o of objects that are withi a specified distace d of the object

22 Desity-based outlier detectio (the LOF approach) For each poit, compute the desity of its local eighborhood Compute the local outlier factor (LOF) of a sample p as the average of the ratios of the desity of sample p ad the desity of its earest eighbors Outliers are poits with largest LOF value p 2 p 1 I the Nearest-eighbor approach, p 2 is ot cosidered as outlier, while LOF approach fid both p 1 ad p 2 as outliers

23 Desity-based outlier detectio usig relative desity Average relative desity (ard) is e.g. give as the ratio of the desity of a poit x ad the average desity of its earest eighbors as follows: ard(x, k) = desity(x, k) y N (x,k) desity(y, k)/ N(x, k (Eq 10.7) A simplified versio of the LOF techique usig ard(x, k) is give by:

24 Example of relative desity (LOF) approach (usig k = 10)

25 Clusterig-based outlier detectio Clusterig-based outlier: a object is a cluster-based outlier if the object does ot strogly belog to ay cluster Basic idea: Cluster the data ito groups of differet desity Choose poits i small cluster as cadidate outliers Compute the distace betwee cadidate poits ad o-cadidate clusters. If cadidate poits are far from all other o-cadidate poits, they are outliers

26 Clusterig-based outlier example

27 Outliers i lower dimesioal projectio (a grid-based approach) I high-dimesioal space, data is sparse ad otio of proximity becomes meaigless Every poit is a almost equally good outlier from the perspective of proximity-based defiitios Lower-dimesioal projectio methods A poit is a outlier if i some lower dimesioal projectio, it is preset i a local regio of abormally low desity

28 Outliers i lower dimesioal projectio (a grid-based approach) Divide each attribute ito φ equal-depth itervals Each iterval cotais a fractio f = 1/φ of the records Cosider a k-dimesioal cube created by pickig grid rages from k differet dimesios If attributes are idepedet, we expect a regio to cotai a fractio f k of the records If there are N poits, we ca measure sparsity of a cube D icludig poits as by the sparsity coefficiet S: where expected fractio ad stadard deviatio of the poits i a k-dimesioal cube is Nf k Nf k (1 f k ) give by ad respectively. Negative sparsity idicates cube cotais smaller umber of poits tha expected Ref: Outlier Detectio for High Dimesioal Data, Charu C. Aggarwal ad Philip S. Yu, ACM SIGMOD 2001 May 21-24, Sata Barbara, Califoria, USA,

29 Example for sparsity coefficiet N=100, φ = 5, f = 1/5 = 0.2, N f 2 = 4 (expected fractio)