How do I develop and found a search engine? Files and file search in the Internet from the perspective of FindFiles.net

Transcription

1 How do I develop and found a search engine? Files and file search in the Internet from the perspective of FindFiles.net Claudius Gros Institute for Theoretical Physics Goethe University Frankfurt, Germany 1

2 overview data in the Internet Mime types and statistics the file search engine FindFiles.net science with data files neuropsychological constraints to human data production 2

3 Internet Statistics 3

4 Internet hosts Hosts Domains Sites active Mio domains [source: Netcraft.com] 4

5 Internet users users in worldwide Asia Europe North American s in number of s sent 89% share of spam s (success rate: 1:12 Mio?) number of users [source: Pingdom.com] 5

6 Internet social media number of blogs number of tweets on Twitter Facebook acounts pieces of content: links, notes, images, number of activated apps (per day) [source: Pingdom.com] 6

7 social media images and videos streaming videos tubes watched per day on Youtube (one per Internet user) 35 hours of video uploaded (every minute) images, pictures photos hosted by Flickr 3000 photos uploaded (per minute) [source: Pingdom.com] 7

8 social media blogs & bookmarking blogs are everywhere 2010: blogs social is everthing Digg, Mister Wong, Delicious,... social shopping,

9 Internet rules of thumb movie per day per user (Youtube) 1 search query per day per every 2 users (Google) every Internet user uses 5 true s per day per Internet user 25% of Internet users use novel social media most domains are blogs 9

10 Internet Startups 10

11 slow beginnings for startups in the Internet Twitter linear scale exponential or linear growth? Apr daily tweets 11

12 growth is generically not exponential Google 30 revenues per year (billions, US-$) Google yearly revenues year linear scale 12

13 Internet: the winner takes all flow of attention in complex networks in-degree distribution p k heavy tails preferential attachment 13

14 in-degree distribution power law scale invariant 8 6 log(number of hosts) number of incomming links linear fit, slope -2.2: *x number of incomming links scaling constant for 20 years starting at one! [source: Findfiles.net] 14

15 limiting diverging in-degree distribution p k 1 k α, k k k2 α dk kα 2 α K c diverging mean in-degree lim k α 2 Internet: α limiting dominating tail» limiting winners take all «makes life difficult for small startups 15

16 the big two uphill fights a new Internet startup needs to... fight for attention fight for novelty traffic and quality heavy tail in-degree distribution makes it difficult to attract traffice extremly high service standards act as effective entry barriers 16

17 FindFiles.net a new file search engine 17

18 public data on the Internet 280 Million domains in data files per domain Internet Media type Mime type categorization of all file types attachments browser add-ons about 600 Mime types in use 18

19 Mime types major Mime categories together: 99% 33.2% application/ 2.9% audio/ 58.0% image/ 5.1% text/ 0.7% video/ Mime types examples application/pdf application/msword application/vnd.android.package-archive application/vnd.ms-powerpoint application/jar application/x-deb application/x-gzip audio/mpeg audio/midi chemical/x-pdb image/jpeg image/vnd.djvu text/xml model/vrml 19

20 FindFiles.net search engine for data files G. Kaczor & C. Gros 2011 supports all Mine types 20

21 FindFiles.net some stats daily queries 400 Mio data files 10 Million mp3 files apps for Symbian/Android smartphones... [source: FindFiles.net] 20 Mio host crawled 21

22 blogs, legal issues & financing blog & press coverage copyright & non-legal files files protected by copyright/licence are not indexed (nofollow) links to pirate files removed from index financing network Unibator banks are cautious most startups fail 22

23 Science with Data Files 23

24 the Wikipedia/DMOZ corpus all outgoing links of Wikipedia (all languages) DMOZ open directory project (all languages) 7.7 Mio hosts (domains) 252 Mio data files (FindFiles.net crawler) analysis of file size distribution tails & scaling behaviour 24

25 number of files per domain files per host vs. in-degree most files hosted on small domains 25

26 file size distribution number of files of given size 8 6 log(number of files) all Mime categories Mime category application/ Mime category audio/ Mime category image/ Mime category text/ Mime category video/ 10 B 100 B 1 K 10 K 100 K 1 M 10 M 100 M 1 G 10 G file size [Bytes] 252 Mio files in total 9 orders of magnitude 26

27 power-law scaling of image-size distribution compression gif: lossless; jpeg: lossy 6 log(number of files) all Mime categories Mime type image/jpeg linear fit, slope -2 linear fit, slope -4 Mime type image/gif linear fit, slope K 10 K 100 K 1 M 10 M 100 M 1 G file size [Bytes] kink at 4 Mbytes: amateur professional 27

28 lognormal multimedia size distribution all audio and video Mime types 4 log(number of files) all Mime categories Mime category video/ quadratic fit (lognormal distribution) Mime category audio/ quadratic fit (lognormal distribution) 1 K 10 K 100 K 1 M 10 M 100 M 1 G 10 G file size [Bytes] quadratic fit lognormal distribution 28

29 lognormal distribution vs. powerlaw scaling files-size distribution p(s) e [log(s) µ]2 /σ 2 s α not a Taylor-series correction log(p(s)) α log(s) β log 2 (s) images: α<0, β=0. audio/video: α>0, β>0 6 4 log(number of files) all Mime categories Mime type image/jpeg linear fit, slope -2 linear fit, slope -4 Mime type image/gif linear fit, slope log(number of files) all Mime categories Mime category video/ quadratic fit (lognormal distribution) Mime category audio/ quadratic fit (lognormal distribution) 1 K 10 K 100 K 1 M 10 M 100 M 1 G file size [Bytes] 1 K 10 K 100 K 1 M 10 M 100 M 1 G 10 G file size [Bytes] 29

30 one vs. two-dimensional cost functions economical cost functions for data production size storage costs production costs psychophysical cost functions for data production size (images) time needed to take an image is independent of resolution size and time (audio & video) time and resolution are psychophysical distinct variables 30

31 Weber-Fechner law neuopsychological cost functions are logarithmic in sensory stimulus intensity number of objects time perception music: tone pitch log(frequency) (octave) photometry: brightness log(intensity) (lumen) acoustics: sound level log(intensity) [decibel] information production: number of objects / time 31

32 information entropy Shannon information entropy p(s) log(p(s))ds p(s)ds=1 for a distribution function p(s) a measure for the information content Shannon coding theorem Mimimal amount of bytes needed to encode a transmission is given by the information entropy of the signal statistics 32

33 neuropsychological cost functions conditional entropy maximization δ [ p(s) log(p(s)) ds λ ] p(s)c(s) ds = 0 Shannon information entropy: cost function: file size distribution: p(s)log(p(s))ds c(s) p(s) maximal file size distributions exponential c(s) s physical p(s) e λc(s) power law c(s) log(s) 1-dim neuro lognormal c(s) log 2 (s) 2-dim neuro 33

34 physical vs. neuropsychological cost functions images physical exponential [not seen] 1-dim neuro power law [linear] log(number of files) all Mime categories Mime type image/jpeg linear fit, slope -2 linear fit, slope -4 Mime type image/gif linear fit, slope K 10 K 100 K 1 M 10 M 100 M 1 G file size [Bytes] audio/video physical exponential [not seen] 2-dim neuro lognormal [quadradic] log(number of files) all Mime categories Mime category video/ quadratic fit (lognormal distribution) Mime category audio/ quadratic fit (lognormal distribution) 1 K 10 K 100 K 1 M 10 M 100 M 1 G 10 G file size [Bytes] 34

35 global human data production basic assumptions information production as underlying driving force information entropy as a suitable measure law of large numbers average over production processes / producting agents compression/technology correspond to rescaling data production on a global level characterized by neuropsychological cost functions and not be economic constraints 35

36 the Internet & complex system theory complex system theory still an emergent field many models and paradigms yet to be formulated network theory / game theory / allocation problems macroecology / systems biology / cognitive systems theory... information entropy maximization human data production on a global level neuropsychological cost functions... 36

37 graduate level textbook Information theory and complexity Phase transitions and self-organized criticality Life at the edge of chaos and punctuated equilibrium Cognitive system theory and diffusive emotional control second edition