Survey of the Mathematics of Big Data Issues with Big Data, Mathematics to the Rescue Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) Math & Big Data Fall 2015 1 / 28
Introduction We survey some mathematical techniques used with Big Data. The goal here is to make you aware of these techniques rather than giving you detail about them. This will be done later on, at least for some of the techniques we survey. Philippe B. Laval (KSU) Math & Big Data Fall 2015 2 / 28
Outline 1 Issues with Big Data 2 How Can Mathematics Help Types of Data Mathematics to the Rescue! 3 Acquisition of Data Information Content Compressed Sensing 4 Data Analysis 5 Conclusion High Dimensional Data Imaging Data Philippe B. Laval (KSU) Math & Big Data Fall 2015 3 / 28
Issues with Big Data We live in a digital world, which generates a lot of data. New technologies produce enormous amount of data. We are gathering more data than ever, even from old technologies. Problem: Acquisition/storage, analysis and transmission of data. Total data generated > total storage. Increase in generation rate >> increase in communication rate. Analysis can be very complex. Problem: Data is noisy, unstructured and dynamic. Philippe B. Laval (KSU) Math & Big Data Fall 2015 4 / 28
How Can Mathematics Help? One answer is given in a video from a previous lecture, remembering that many of the skills mathematicians have are needed in programming. But also, mathematics......allows us to formalize both the data and the problem....provides a big "chest" of tools or methodologies....allows validation of the methodologies (proof of functionality). Philippe B. Laval (KSU) Math & Big Data Fall 2015 5 / 28
How Can Mathematics Help? - Types of Data Signals. Can be represented by a function f : R R Philippe B. Laval (KSU) Math & Big Data Fall 2015 6 / 28
How Can Mathematics Help? - Types of Data Signals. Can be represented by a function f : R R Once the signal is represented by f, we can: Enhance the signal. Philippe B. Laval (KSU) Math & Big Data Fall 2015 7 / 28
How Can Mathematics Help? - Types of Data Signals. Can be represented by a function f : R R Once the signal is represented by f, we can: Enhance the signal. Remove noise from the signal (PDE techniques, Fourier transform, wavelets, statistics) Philippe B. Laval (KSU) Math & Big Data Fall 2015 7 / 28
How Can Mathematics Help? - Types of Data Signals. Can be represented by a function f : R R Once the signal is represented by f, we can: Enhance the signal. Remove noise from the signal (PDE techniques, Fourier transform, wavelets, statistics) Extract features from the signal. Philippe B. Laval (KSU) Math & Big Data Fall 2015 7 / 28
How Can Mathematics Help? - Types of Data Signals. Can be represented by a function f : R R Once the signal is represented by f, we can: Enhance the signal. Remove noise from the signal (PDE techniques, Fourier transform, wavelets, statistics) Extract features from the signal. The next slide shows a clean signal (red), some noise (green) and a noisy signal. Philippe B. Laval (KSU) Math & Big Data Fall 2015 7 / 28
How Can Mathematics Help? - Types of Data Philippe B. Laval (KSU) Math & Big Data Fall 2015 8 / 28
How Can Mathematics Help? - Types of Data Images. Can be represented by a function f : R 2 R Philippe B. Laval (KSU) Math & Big Data Fall 2015 9 / 28
How Can Mathematics Help? - Types of Data Images. Can be represented by a function f : R 2 R Put a grid on the image (the finer the grid, the highest the resolution). Each cell of the grid contains a positive integer indicating a gray level (for gray images). That s the function f. For color images (RGB), each cell is a vector that contains 3 positive integers indicating the amount of Red, Green and Blue. As for signals, once we have f we can enhance the image, remove noise from it, extract features. Philippe B. Laval (KSU) Math & Big Data Fall 2015 10 / 28
Experiment: Clean Image and Image With 20% Noise Animated version Philippe B. Laval (KSU) Math & Big Data Fall 2015 11 / 28
Experiment: Clean Image and Image With 50% Noise Animated version Philippe B. Laval (KSU) Math & Big Data Fall 2015 12 / 28
How Can Mathematics Help? - Types of Data Data on manifolds. Can be represented by a function f : S 2 R Philippe B. Laval (KSU) Math & Big Data Fall 2015 13 / 28
Acquisition of Data As noted above, the main problem is the size of the data. However, this cannot be solved by increasing storage capacity. Possible solutions: Search for better ways to compress. Acquire less data. Philippe B. Laval (KSU) Math & Big Data Fall 2015 14 / 28
Acquisition of Data - Better Ways to Compress Data Compression saves storage but also decreases bandwidth. General ideas behind compression: Take advantage of redundant information. Only keep the most relevant information. Einstein: "Not everything that can be counted counts, and not everything that counts can be counted." Philippe B. Laval (KSU) Math & Big Data Fall 2015 15 / 28
Acquisition of Data - Better Ways to Compress The standard for data compression was JPEG. It is based on the discrete cosine transform (DCT) (see Fourier analysis, Fourier transform, fast Fourier transform (FFT) and the discrete fast Fourier transform (DFFT). It expresses the image in terms of building blocks, the functions cos [ π n ( i + 1 2 ) k ]. Definition The DCT is a linear, invertible function f : R n R n. It transforms the numbers (x 0, x 1,..., x n 1 ) into (X 0, X 1,..., X n 1 ) by the formula n 1 [ ( π X k = x i cos i + 1 ) ] k n 2 i=0 for k = 0, 1,..., n 1 Philippe B. Laval (KSU) Math & Big Data Fall 2015 16 / 28
Acquisition of Data - Better Ways to Compress The latest standard for data compression is JPEG2000. It is similar to JPEG, but it uses wavelets instead of the discrete cosine transform. The branch of mathematics which studies wavelets is called Harmonic Analysis. Like the DCT, wavelets decompose the image in terms of building blocks. But the building blocks are not limited to the cosine function. Both the DCT and wavelets use the divide and conquer method. Decompose an image in terms of its building blocks. Only keep the most relevant building blocks. Philippe B. Laval (KSU) Math & Big Data Fall 2015 17 / 28
Acquisition of Data - Better Ways to Compress Problem: The raw data still has to be stored first Solution: Compressed sensing (Donoho, Candès, Romberg, Tao (2006) Philippe B. Laval (KSU) Math & Big Data Fall 2015 18 / 28
Acquisition of Data - Compressed Sensing Main idea: Goal: Assumption: Only a small percentage of the data to be captured is really relevant (sparse data). Problem: We don t know which. Solution: Use random building blocks. This is pushing the sampling theorem beyond its limits! Goal: To capture a signal with as few points as possible. Thus the raw data will be already compressed. Requires a lot of linear algebra, solving sparse systems. Philippe B. Laval (KSU) Math & Big Data Fall 2015 19 / 28
Applications of Compressed Sensing Business Compression Astronomy Radar Biology Communications Imaging Science Geology Medicine Information theory Remote sensing Optics Philippe B. Laval (KSU) Math & Big Data Fall 2015 20 / 28
Data Analysis There are many techniques, statistical and mathematical which already exist. Problem: The analysis maybe be too complex for current computing power. Data can be noisy, unstructured and dynamic. However, these problems cannot be simply solved by more computing power. Possible solutions include: Graph Theory. TDA (Topological Data Analysis). Philippe B. Laval (KSU) Math & Big Data Fall 2015 21 / 28
Data Analysis - High-Dimensional Data Strategy: Detection of inner structure. Reduction of dimension Examples: A circle can be approximated by lines. A sphere can be approximated by flat surfaces. Philippe B. Laval (KSU) Math & Big Data Fall 2015 22 / 28
Data Analysis - High-Dimensional Data Philippe B. Laval (KSU) Math & Big Data Fall 2015 23 / 28
Data Analysis - High-Dimensional Data Question: How do we identify inner structure? By using topology. The link will open a website. Scroll down to play the video. Philippe B. Laval (KSU) Math & Big Data Fall 2015 24 / 28
Data Analysis - Imaging Data Tasks: Removing Noise. Pattern recognition. Reconstructing missing data.... Tools: Statistics. Fourier and harmonic analysis. Partial differential equations. Compressed sensing. Linear Algebra. Philippe B. Laval (KSU) Math & Big Data Fall 2015 25 / 28
Conclusion Big Data is here to stay. We need to tame it! The techniques presented are fairly diffi cult and require a lot knowledge in various branches of mathematics, statistics and science. Not everybody needs to be a mathematician. But chances are you will work with one. Knowing at least some mathematics will make the communication and the teamwork easier. Questions? Philippe B. Laval (KSU) Math & Big Data Fall 2015 26 / 28
Some Questions 1 Name some new technologies which generate data which were not mentioned during the talk. 2 Give examples of unstructured data not mentioned during the talk. 3 Why is removing noise from data (images as well as other data) important? 4 Give applications of pattern recognition in images. Philippe B. Laval (KSU) Math & Big Data Fall 2015 27 / 28
Online Articles to Read 1 The Mathematical Shape of Things to Come from Quanta Magazine. 2 The Real Secret to Unlocking Big Data? Math. 3 The New Shape of Big Data. 4 New Mathematical Method May Help Tame Big Data from Communications of the ACM. 5 Better Way to Make Sense of Big Data from Science Daily. Philippe B. Laval (KSU) Math & Big Data Fall 2015 28 / 28