2. Make an input file for Query Execution Steps for each Q1 and RQ respectively-- one step per line for simplicity.

Transcription

1 General Suggestion/Guide on Program (This is only for suggestion. You can change your own design as needed and you can assume your own for simplicity as long as it is reasonable to make it as assumption.) 1. You need to have a data structure for each major input output parameters such as Table, Query_Plan, etc - Table: Info on Table name, Table Size (for a new temp table, it needs to be estimated at the end of each execution step), Related Join or Predicate Selectivity, etc - Query_Plan: 1) All info about the current query processing cost: Query_Name, Has_Subquery, Is_Correlated, Query_Excution_Steps (or Join Order with Table Info), Join_Methods used for each join, Total_Query_Cost under this plan, etc 2) This Current_Query_Plan represents all info about the current factors that change total query processing cost and it will be your output to print out at the end of each iteration of your loop. 2. Make an input file for Query Execution Steps for each Q1 and RQ respectively-- one step per line for simplicity. 3. Assume that: - Query Rewrite is done and Q1 is rewritten to RQ as given in Lab3. - Query Execution Steps in Sequence to process for each Q1 and RQ are done and saved in each Query Tree before entering your Optimizer module. It is used as one of Input for your Optimizer. We are pretending each transformed Query Execution Tree for Q1 and RQ is in Query Execution Tree form as input, which is actually stored, in real, in each flat input file you manually made to make process simpler instead of in Query Execution Tree form. You will use those as one of input parameters for your Optimizer because each input file has all information about the sequence of processing of the query to calculate each total query processing cost with different factors in the program. 4. Write subroutine for each operator in Query Execution Steps to calculate each processing cost for the operator with given input (operands(tables), join method ) 5. You are not asked to implement those relational operators here. You need to calculate processing cost for those operator to get total query processing cost with given factors. Your program is not dealing with real table and real data. 6. Make a routine to permute three tables in a query to vary join order before calling your join operator to calculate current join cost to compare each different join cost and for each join order, try all possible join methods in the main loop to compute each total cost for the current query plan.

2 7. You will need all the comprehensive knowledge you learned in the lectures on Query Processing Cost and Optimization to get the correct best cost query plan for this project. For example: - In your program, Q1 should have a loop cost with subquery for each tuple of outer table with no other join orders and join methods to choose because of subquery correlation while RQ has more freedom to choose over cheaper join orders and join methods without loop because there is no subquery and correlation. 8. How to Estimate Size of Temp Table with Selectivity - Size of Temp table after applying a predicate_1 on single Table = size of Table * selectivity of predicate_1 - Size of Temp table after applying two predicates together on single Table = size of table * selectivity of predicate_1 * selectivity of predicate_2 - If no selectivity or any info provided, then you can assume the size of Temp table is size of Table at worst case. - Size of Temp table of Binary Join after applying one Join Condition = size of Left table * size of Right table * selectivity of Join Cond_1 - Size of Temp table of Binary Join after applying two Join Conditions together = size of Left table * size of Right table * selectivity of JoinCond_1 * selectivity of JoinCond_2 - If there is no Join Condition or any info provided on Join, then it is Cartesian Product of Left table and Right table = size of Left table * size of Right table 9. I was wondering how to integrate selectivity calculations into the code because they seem query-specific?? In real, selectivity is provided as table and column info then Optimizer uses as input for the conditions in the Where clause in a query. Here, you can assume whatever needed for selectivity input. One way is to store it in each related Table data structure. Then it can be used in join calculations for the table. 10. Cost of Hash Join - Cost of HJ with enough memory requirement for hash table for entire Left Table = 3*(size of Left table + size of Right table) - Memory requirement is root of M, so hash join with not enough memory needs to do processing twice if memory is < 33pages for this case. I corrected the hash join memory size to 20, 40 pages respectively for this.

3 - Cost of HJ with a half of memory requirement for hash table for entire Left Table would be two times of the cost of HJ with enough memory requirement because it would need Disk I/O twice for each scan because the hash table fits in main memory can be build only for a half of Left Table. So read the first half of Table, then build hash table for that to find matching, then write the result for the first half and read in the later half of Table to build hash table for matching. Therefore roughly it would need twice Disk I/O of Hash Join with enough memory requirement. = 2 * 3*(size of Left table + size of Right table) 11. One benefit of Sort Merge Join - The output of Sort Merge Join result is sorted on Join columns, so in case when the next execution step is Sort on the join columns (for Projection or Group By and/or Aggregation), then you can skip the Sorting step, so the cost of Sorting O(MlogM) can be saved Optimization! 12. Yes, processing costs for SMJ and HJM (with required memory for hash table) are about the same under the very naive ideal assumptions made here without taking data into consideration. However, they are different in real cases case by case. Mainly, a) Data Skew, a lot of duplicates in Outer and inner, then SMJ is getting very bad, M*N at worst case. HJM is getting bad because of collision reasoning. b) Output of Sort Merge is sorted on join columns, so if the next step is sorting related operation, then you can save sorting cost After SMJ. 13 Selectivity is for join operation with join conditions or Select operator for predicates with join. For Project, you can assume size of Temp after Project is O(size of Table) at worst case. At any case, every operators should take Table or Table information as one of input! 14. How do I find the cost of Group By operator? Group by (with Aggregation on the fly at the last scan) is Sorting cost. Both Sorting base and Hash based algorithms for Group By costs 3(M + N). However, when it is optimized as explained in class, it can be done in 2(M + N). 14. What should be the unit of the final processing cost? Do I have to convert Number of Disk I/Os to mins/sec? How do I do that? Yes, you have to convert total cost calculated in Disk I/O to Hour/Min/Sec at the end whenever print out the total cost for each current plan. We ignore Data Transfer cost here.

4 Disk Acess Time = Seek Time + Latency Average Seek time = 8 ms Average Latency = 4 ms QueryProcessing Cost = Disk I/O Cost = # of Disk I/O * Disk Access Time = # of Disk I/O * (8 ms + 4 ms) 15. In Lab3 specification, I already listed Query Execution Steps for Q1 and RQ1 to avoid variations and confusion. Semantics of each query is not important here for this lab. The queries are simply given to build its execution steps. As directed in Lab3, you can assume the given text files for the execution steps for Q1 and RQ1 are coming in as inputs to Optimizer routine to calculate a cost for each step then total of them would be a query processing cost. For simplicity, main factors to vary each query plan here are Join order and Join Method only. You can assume whatever makes it simpler for this lab because it will be way too complex if you consider all the realistic cases. 16. T1: Each tuple on the table is 20 bytes long therefore I can store 204 tuples on each page and tuples in each block. This table size is 1000 pages therefore the table is stored on 10 blocks. herefore the total number of tuples in this table T1 = 20,4000 tuples T2: Each tuple on the table is 40 bytes long therefore I can store 102 tuples on each page and tuples in each block. This table size is 500 pages therefore the table is stored on 5 blocks. therefore the total number of tuples = 5,1000 tuples 17. Calculation of each nested loop join would be: If Temp <== L TNL R on L.C1 = R.C2 with Selectivity(js) of join column L.C1 = R.C2: Cost of TNL = LPages + (#Tuples_Per_Page_In_L * LPages) * RPages Cost of PNL = LPages + ( LPages * RPages) Cost of BNL = LPages + (LPages / BlockSize) * RPages Estimated Size of join result Temp = js * L * R * LengthOfTupleTemp / 4096 Length Of Tuple of Temp = (Length of Tuple L + Length of Tuple R) Or roughly you can estimate as Estimated Size of join result Temp = js * (LPages * RPages)

5 Yes, if there is no join condition, then js = 1, which means Cartesian Product of L and R. 18. For Lab3, we assume that there is no index on any columns. So Index nested loop join is excluded here. However, in a real optimizer, it usually starts with checking if there is join index already built ahead, then index look up is first considered as an access path and then the cost with index look up is calculated and compared with all the other alternatives to pick the best plan. To understand how to calculate Index look up as an access path, look at the Lecture Notes on an example for the cost to calculate in the Index Nested Loop section in the class notes. We assume that Hash Index or B+ tree index is used and data is uniform and average, so there are average 2.5 duplicates (so 2.5 unclustered matching) for a column(fk) for each join condition. 19. > If TNL was the lowest cost, I would then move to the next step "Join Temp1 T3"...etc...adding the lowest costs as I go until >completion? Yes, it would be one way to find the lowest cost. TNL can not be the lowest, though. >"Join T1 T2", how would my processor know that I have another join coming up between that tempt table and table 3. For RQ1, you have three tables and all possible join orders to try. Maybe count first how many Join steps in Query Execution Steps for RQ1, then do procedure for permutation. You can assume whatever makes it simpler for this lab because it will be way too complex if you consider all the realistic cases. 20. You can assume that all the projection rates for each SELECT are the same, so you can ignore all the projection rate Changes in Lab3: In RQ, t1 join t3 in inline table, change its Selectivity to 1 % on T1.x1 = T3.x3 (To make it more realistic) 21. You can assume that all the projection rates for each SELECT are the same, so you can ignore all the projection rate Changes in Lab3: In RQ, t1 join t3 in inline table, change its Selectivity to 1 % on T1.x1 = T3.x3 (To make it more realistic) 22. Is this how we handle various join orders in RQ1 or am I missing any permutation here.??

6 Join t1 t3(using all types of joins) Join t3 t1(using all types of joins) select which of the above is cheaper. Call it cost A. Then calculate : Join t1 temp1 Join temp1 t1 select which of the above is cheaper. Call it cost B.Add A +B Then calculate : Join t2, temp2 Join temp2,t2 Select which of the above is cheaper.call it cost C.Add A +B +C A+B+C will be the least cost associated with joining the tables. That would be three different possible join order with 6 different join methods out of 6 different join orders. You can make that as one possible query plan and write the total cost as output at the end of loop. However, there are more join orders to try. For the first join is fixed as Temp1 <== t1 join t3 for inline table always first. Then you have 3 table to join: t1, t2, temp1 where (t1 join t2) join temp1 and two more different join plans by switching left table, right table for each join (t1 join temp1) join t2 and two more different join plans by switching left table, right table for each join (t2 join temp1) join t1 and two more different join plans by switching left table, right table for each join Therefore, There are 12 different join order (query plans). For each join, you can iterate 6 different join methods to get the minimum cost join methods. Or you can make 12 different join orders with 6 different join methods for each join, total 12 * (3 joins in each 12 join order)*6 different join methods, so total 12*3*6 different join plan (query plan) to write total cost for each at the end of loop will be more complete, but if that makes the program too complicated, then you can do your way to make it simpler by just trying 12 different join orders * 6 join methods. 22. This size of the tuple after we project has not been specified in Lab3. Can you plese let us know what we should assume it as? (Project temp3 in RQ1 and project temp2 in Q1) We assume all the projection rates are the same here for simplicity, which won't be any factor to compete. So we ignore the projection rate or assume it is 1 for all.

7 23. I would like to know if I can keep different files as input for each possible binary join order and proceed. How you make your input for your program is entirely up to you. You can assume whatever needed. 24. Question: The output I am getting for Q and RQ there is not much difference in the value. I calculated manually also i am getting this only. I am not getting output like yours. Do you think something is wrong in my program?? Or is it possible to get result like mine also as shown below. =============== BEST PLAN FOR Q ================ Join T1 T2 SMJ Join TEMP1 T3 PNJ Project TEMP2 GroupBy TEMP3 # Disk I/O: Processing Time(hr:min:sec):3428:7:12 =============== BEST PLAN FOR RQ ================ Join T1 T3 SMJ Join T2 TEMP1 SMJ Join T1 TEMP2 SMJ Project TEMP3 GroupBy TEMP4 # Disk I/O: Processing Time(hr:min:sec):3010:14:42 No, it is not a correct output. It means that your program has wrong logic to calculate total cost for Q1. For the join step temp and t3 in Q1, because of correlation, your program has to calculate cost for processing subquery (here it is scanning inner table t3) that has to be done for

8 each tuple of Temp1. You can't just use a cost of one join step with temp1 and t3 as a cost of the correlated join step temp with t3 for that step. 25. Do not compare your output in page size directly to the size in the example on the web. Input of those example programs are all different. The correct output means the correct query plan of the winner -- which query with which Join order and which join method. There is one specific order with one or two specific join methods for a certain query should be the winner if you put all the optimization logic in your calculation correctly. It is pretty obvious. The common mistake in your logic for Q1: Q1 has a correlated subquery, so subquery cost should be calculated for each tuple of outer table in the second join step with the subquery. You can not just count the second step as one join cost. Certainly Q1 can not be the best cost winner. 26. In Page Nested Join we do that Outer_Table + (Outer_Table * Inner Table) doesn't it mean that for each outer tuple we are scanning the inner table?? No, it is not! Outer table is in page size, not in # of tuples. You can use inner table in Page size for PNL. 27. What is the cost of Project?? Temp table size that Project is operating on : T with duplicate elimination then general cost is T + T + TlogT + T If sorted, T 28. In general the formula for group by is (M+M) + MlogM but if the previous step is sort merge join the cost is (M+M) (MlogM is not taken into account ). Is this right?? Similarly for Group by the cost is 3(M + N) but if the previous step is SMJ then it is 2(M+N). Is this right?? Yes, it is correct! 29. I was wondering how to integrate selectivity calculations into the code because they seem query-specific?? In real, selectivity is provided as table and column info then Optimizer uses as input for the conditions in the Where clause in a query. Here, you can assume whatever needed for

9 selectivity input. One way is to store it in each related Table data structure. Then it can be used in join calculations for the table. NEW: 30. What is selectivity of temp1 join t3 after getting temp1 t1 join t2? I see that. Although it can be inferred from the each table, I see it is not specifically mentioned anywhere. It is hard to make each join case with the inferred realistic selectivity., so it will be simpler if each selectivity is explicitly given. For Q1, Because of correlation, this would be so called correlated tuple join, which is, calculate scanning full t3 per the number of tuples in temp (<=t1 join t2), For the size estimation of the result of this correlation join, you need a selectivity on temp join t3. I see I only gave 15% selectivity for t1 join temp Use the same selectivity 15% for temp join t3 as well! Use 15% selectivity for t1 join t2 as well. For RQ, 1) temp 1 <= t1 Join t3 2) temp2 <= temp 1 join t1 ( for this 15% selectivity is given)