The Mathematics of Big Data

Transcription

1 The Mathematics of Big Data Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Math & Big Data Fall / 10

2 Introduction We briefly present Big Data and the issues associated with Big Data. Philippe B. Laval (KSU) Math & Big Data Fall / 10

3 Outline 1 What is Big Data? 2 How Big is Big Data? 3 Issues with Big Data 4 How Can Mathematics Help Philippe B. Laval (KSU) Math & Big Data Fall / 10

4 What is Big Data? Definition Big Data refers to a data set that is so large and/or complex that it cannot be perceived, acquired, managed, and processed by traditional Information Technology (IT) and software/hardware tools within a tolerable time. What questions does this definition raise? Philippe B. Laval (KSU) Math & Big Data Fall / 10

5 What is Big Data? Definition Big Data refers to a data set that is so large and/or complex that it cannot be perceived, acquired, managed, and processed by traditional Information Technology (IT) and software/hardware tools within a tolerable time. What questions does this definition raise? Big data can be characterized by the 3 V s: Volume (great volume), Variety (different type of data, structured or unstructured), and Velocity (rapid generation). Philippe B. Laval (KSU) Math & Big Data Fall / 10

6 What is Big Data? Definition Big Data refers to a data set that is so large and/or complex that it cannot be perceived, acquired, managed, and processed by traditional Information Technology (IT) and software/hardware tools within a tolerable time. What questions does this definition raise? Big data can be characterized by the 3 V s: Volume (great volume), Variety (different type of data, structured or unstructured), and Velocity (rapid generation). Some people now add a 4th V: Value, it refers to the value that could be saved if Big Data techniques were creatively and effectively used to improve effi ciency and quality. Philippe B. Laval (KSU) Math & Big Data Fall / 10

7 What is Big Data? Big Data has given rise to several new and related technologies. 1 Cloud computing. 2 Internet of Things (IoT). 3 Data Centers. 4 Hadoop. Philippe B. Laval (KSU) Math & Big Data Fall / 10

8 How Big is Big Data? - SI Prefixes Prefix Unit Name Symbol SI Meaning kilo kilobyte kb or KB 10 3 mega megabyte MB 10 6 = ( 10 3) 2 giga gigabyte GB 10 9 = ( 10 3) 3 tera terabyte TB = ( 10 3) 4 peta petabyte PB = ( 10 3) 5 exa exabyte EB = ( 10 3) 6 zetta zettabyte ZB = ( 10 3) 7 yotta yottabyte YB = ( 10 3) 8 Philippe B. Laval (KSU) Math & Big Data Fall / 10

9 How Big is Big Data? We live in a digital world, which generates a lot of data. Below are similar but more recent figures from the July issue of Time. Every day, humanity tweets 500 million times. Every day, humanity shares 70 million photos on Instagram. Every day, humanity watches 4 billion videos on Facebook. Every minute, we upload 300 hours of new content on YouTube. A 2014 study by the market-research firm IDC estimated that the world of digital data would grow by a factor of 10 from 2013 to 2020, to 44 zettabytes. Philippe B. Laval (KSU) Math & Big Data Fall / 10

10 Issues with Big Data New technologies produce enormous amount of data. We are gathering more data than ever, even from old technologies. Problem: Acquisition/storage, analysis and transmission of data. Total data generated > total storage. Increase in generation rate >> increase in communication rate. Analysis can be very complex. Problem: Data is noisy, unstructured and dynamic. Philippe B. Laval (KSU) Math & Big Data Fall / 10

11 How Can Mathematics Help? One answer is given in this video, remembering that many of the skills mathematicians have are needed in programming. But also, mathematics......allows us to formalize both the data and the problem....provides a big "chest" of tools or methodologies....allows validation of the methodologies (proof of functionality). Philippe B. Laval (KSU) Math & Big Data Fall / 10

12 Conclusion In this class, we will study some of the mathematical techniques needed with Big Data. Our next lecture will be a survey of these mathematical techniques we plan to study in depth. See the problems at the end of my notes on Definitions and Overview of the Issues with Big Data. Philippe B. Laval (KSU) Math & Big Data Fall / 10