Introduction to Data Science Day 2

Transcription

1 Introduction to Data Science Day 2 Data Matters Summer workshop series in data science Sponsored by the Odum Institute, RENCI, and NCDS Thomas M. Carsey carsey@unc.edu

2 Examples of Data Science Google Flu Billion Prices Project Medicare Fraud medicare-fraud-strike-force-charges-90-individuals-forapproximately-260-million-in-false-billing Twitter trending Angie s List

3 More Examples (research) Agricultural crop insurance fraud Satellite data Diffusion of Policy innovation Plagiarism and networks networks and topic models Structure in Wikipedia Repeat offenders individual or group-based causes?

4 Big Data Computing MapReduce and Hadoop Designed to process large operations quickly Distributes the problem across multiple distribute servers The Map part filters and sorts data into bins or queues based on some share characteristic The Reduce part then executes some operation on each bin of data. Results are then reassembled It is like parallel processes, but distributed across servers rather than just processors Scalable and has a fault tolerance Hadoop is an open-source versoin

5 More on MapReduce Pig Software platform used for creating MapRedce programs used by Hadoop. Hive A date warehouse infrastructure built on top of Hadoop. Used to query, summarize, or analyzed data.

6 Database management SQL Structured Query Language A programming language designed for management of relational databases MySQL Open source implementation of an SQL-like system for management of relational databases (used by Wikipedia, Google, Facebook, Twitter, Flickr, YouTube) NoSQL (Not Only SQL) Used for databases where the data is in some form other than tabular relations like those used in relational databases Cassandra (Apache) Distributed database management with not single node of failure Scalable with no down time

7 Cloud Computing Standard client-server model where computing operations don t happen on the local (desktop) machine. What s new? Virtualization. You are not connecting to a specific server. Servers are virtual One server can run multiple virtual machines One virtual machine can use multiple servers This makes the machine scalable, moveable, configurable. Allows selling software, platforms, and even computing infrastructure as a service

8 Data Access and Research Transparency (DA-RT) Increased regulations from government to provide access to data funded by government dollars. Increased professional publication standards for sharing research data Increased pressure for transparency in research methods more generally, including data access Symposium in Political Science (PS: Political Science and Politics, Vol. 47, Issue 01.) jid=psc&volumeid=47&seriesid=0&issueid=01

9 Privacy and Ethics Several questions emerge from the explosion of digital data about us: Identity: What is the relationships between our offline and online identities? Privacy: Who should control access to online data? Ownership: Who owns data? Can ownership rights be transferred? What are the obligations of those who generate and use data? Reputation: How can we determine if data is trustworthy? How can we know if provider acquired data legitimately? There are entire industries on reputation management.

10 Ethical Claims Davis claims Big Data, like all technology, is ethically neutral However, business is a social/public enterprise, so you will encounter the values others hold. Big data magnifies this encounter. Google says between 15 and 50 billion webpages with 3.5 billion searches every day. YouTube has more than 1 billion unique visitors every month watching over 6 billion hours of video. 100 hours of video are uploaded every day 500 million Tweets every day billion Facebook users

11 Ethical Decision Points Inquiry: what are our values? Analysis: what are our current data handling practices? Do they align with our values? Articulation: explicit written expression of alignment and gaps between values and practices Action: specific plans and activities used to close those gaps.

12 Buying vs. Selling Company data policies: 34 of 50 Fortune-class companies said the would not sell personal data. No company said explicitly that the would sell personal data. No company made any explicit statement that they would not buy personal data 11 companies said explicitly that buying personal data was allowed. Why different standards for buying vs. selling personal data?

13 Privacy Physical Privacy Informational Privacy Organizational Privacy Privacy of Our communications Our behavior Our person Is digital communal living different than physical communal living? Who decides? Who regulates? Who Enforces?

14 Privacy as a Common Pool Resource? To govern our collective behavior: We need norms/rules that are agreed upon and known We need effective monitoring We need a mechanism to sanction violations The question is, do we need an external force or can we govern ourselves? (Ostrom s work)

15 Traps in Empirical Research Simpson s Paradox

16 Other Traps Aggregation effects more generally E.g. mis-report bias in voting Residential segregation Regression to the mean Special education programs Hawthorne effect Thanks for noticing Superstitious learning Midas touch vs. Damned if you do, damned if you don t.

17 How to Avoid Traps Theory Think critically Use simulations but that s another course!