Leverage the power of SQL Analytical functions in Business Intelligence and Analytics. Viana Rumao, Asher Dmello

Transcription

1 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July Leverage the power of SQL Analytical functions in Business Intelligence and Analytics Viana Rumao, Asher Dmello Abstract It is said that the world s most powerful resource is no longer oil, but data. As rightly said by E.O Wilson, we are drowning in data, but striving for wisdom. Big Data, Analytics, Business Intelligence, Data Science require processing huge amounts of data in various and complex ways using a vast array of statistical methods and tools. Market increasingly needs manpower with both databases and data warehouses technological skills and statistical competencies to decipher the business patterns and trends hidden in the mountains of data[1] This paper presents an overview of SQL and its role in business intelligence and data analytics. The paper further focuses on the power of analytical functions in SQL. We experimentally prove the efficiency of the same using Spark-SQL in Python. Index Terms SQL, Analytical Functions, Business Intelligence, Analytics, Python, Spark, Data Science 1 INTRODUCTION In today's world, the revolution in technology and business and true methods that are prevalent at that time, whereas for has bought in the large data storage from phones, computers Cuzzocrea et al. (2011) Big Data refers to enormous amounts of and various other processes. This has indirectly made us store unstructured data produced by high-performance applications large amount of data as data used is required to analyse and falling in a wide and heterogeneous family of application scenarios: from scientific computing applications to social networks, from interpret decisions and statistics in the field of medicine and sports and various upcoming fields. People, businesses, and e-government applications to medical information systems, and so devices have all become data factories that are pumping out forth. incredible amounts of information to the web each day.[2][3]. Big volumes of data, but small analytics usually involves using The challenge to manage and store this data and get meaningful insights out of the same has given rise to Business Intelli- AVG, GROUP BY, HAVING functions and clauses) on large data- regular SQL queries (SELECT with MIN, MAX, SUM, COUNT, gence. We discuss the significance of the field of BI and how sets. All types of SQL (relational) databases, commercial (Oracle, Analytical functions in SQL play a pivotal role in the same. IBM DB2, Microsoft SQL Server) or open-source (PostgreSQL, MySQL) could be platforms/tools for this type of processing. Big 2 SIGNIFICANCE OF BUSINESS INTELLIGENCE AND ANALYTICS BI is a concept that usually involves the delivery and integration of relevant and useful business information in an organization. Companies use BI to detect significant events and identify/monitor business trends to adapt quickly to their changing environment and a scenario. In today's world, we use business intelligence for better decision making and to interpret, collect and analyse data. Business intelligence involves activities that include data mining, online analytical processing, querying and reporting. Business intelligence involves making accurate business decisions and correcting the mistakes made in the previous business decisions. Business analytics is a methodology or tool to make a sound commercial decision. It is used to analyse primary and secondary data. It is used as predictive analysis which is based on past predictive analytics. It is also used for constructing a business model or a project based upon the past models.[4][5] 3 SQL FOR DATA STORAGE, RETRIEVAL AND ANALYTICS According to Jacobs (2009), Big Data should be defined at any point in time as data whose size forces us to look beyond the tried data analytics on big volumes of data requires combining ETL (Extract-Transform-Load) tools with statistical packager. Big data analytics denote regression, data mining, machine learning and other types of more complex processing. Data could be extracted from various data sources using SQL queries and/or ETL tools. Complex analyses require packages such as SPSS, R, SAS, etc. and sometimes a good deal of coding.[1] An efficient but a not-sopopular technique to handle complex analysis is the use of analytical functions in SQL. In the next section, we unleash the power and efficiency of the same 4 ANALYTICAL FUNCTIONS IN SQL For an IT professional getting into data warehousing (DWH) or business intelligence, the need is to go beyond plain SQL and start digging into the analytical functions (also known as window functions, windowing functions, or OVER clauses). We demonstrate SQL Analytical functions over Spark framework using the pyspark library in Python on Jupyter notebook. Partition and Over Clause: The analytical function consists of a function with an over keyword and an optional list of columns for ordering. The ordering indicates criteria for function evaluation, not the final ordering of results. RANK() RANK is a simple analytical function that does not take any

2 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July column list arguments. It returns a number or a rank based on the ordering of the rows; the ordering is based on a defined condition. DENSE_RANK() DENSE RANK works like RANK in that it needs no additional arguments and it ranks items in descending or ascending order. The only difference is that DENSE RANK does not allow gaps between groups. LAG() LAG is the opposite of LEAD. We can even implement LAG using LEAD and vice versa. The difference is in the direction we look for the offset value. LEAD() LEAD returns an offset (incrementally increased) value of an argument column. The offset amount can be defined in the code; its default amount is 1. The new value is returned in the same row. Windowing The window concept is used extensively with analytical functions. A window is a collection of rows in which an analytical function is calculated. We tie window concepts to changes in numeric variables. Windows are typically defined by time intervals, like years or months. Windows can be specified as units of physical rows using the ROWS keyword, or they can be defined in logical rows using the RANGE keyword. Physical windows are specified by absolute values of rows we offset before and after the current row. Logical windows are specified by values, such as a number of days. Numeric variables are calculated in moving windows so that the variables can be compared across windows. Windows can be cumulative or sliding, as we will see in the coming examples. Examples of physical-row windows are: ROWS UNBOUNDED PRECEDING The window is the current row and every preceding row. ROWS 2 PRECEDING The window is the current row and 2 preceding physical rows. ROWS 2 FOLLOWING The window is the current row and 2 following physical rows. Please refer appendix 1 for examples. REFERENCES [1] Marin Fotache, Catalin Strimbei, SQL and Data Analysis. Some Implications for Data Analysts and Higher Education, Procedia Economics and Finance, Volume 20, 2015, [2] [3] [4] [5] [6] Dataset: [7] 5 THE POWER OF ANALYTICAL FUNCTIONS As a part of the experiment, we execute a query on 1000 rows employee dataset using analytical functions and the execute the same using joins. We see that the query with analytical functions took 4.35 seconds to execute, where joins the query executed in seconds. The analytical functions query was shorter and a one-step process. The joins query was a complex 2 step process. This explains the power of analytical functions in SQL. Please refer appendix 2 for implementation screenshots. 4 CONCLUSION AND FUTURE WORK The paper provides an overview of the field of BI and the role of SQL analytical functions. We prove their efficiency compare to normal SQL as well. We can further explore on how Analytical functions can revolutionize the field of BI.

3 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July Appendix 1

4 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July

5 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July

6 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July Appendix 2

7 International Journal of Scientific & Engineering Research Volume 9, Issue 7, July