INSTITUTIONAL DATA USERS GROUP

Transcription

1 INSTITUTIONAL DATA USERS GROUP October 12 th 2016 October Meeting

2 IDUG Agenda MAUI Admissions Row Level Security Adam Pyatt, Eric Hudson MAUI Snapshot Changes Eric Hudson Campus Data Portal Denny Crall, Michael Alberhasky, Ed Hill ITS-AIS ITS-AIS ITS-AIS

3 MAUI Admissions Row Level Security Adam Pyatt Eric Hudson ITS-AIS Enrollment Management Technical Lead ITS-AIS Database Administrator

4 MAUI Admissions Row Level Security Overview Quick Room Survey How many filter Application data using college or program How may filter Student data ( tests, GPA, etc ) using college or program Remove Row Level Security from both the Prospect and Application tables Impact on Student and Test tables Why Performance Increase Transparency

5 MAUI Admissions Row Level Security Prospect Row Level Security 11 prospect users impacted around campus Data Impacted Mail Item History Referral Codes STUD_* TST_* Current Security Program Prospect Type Prospect Status

6 MAUI Admissions Row Level Security Prospect Row Level Security User has Prospect Access for all US_FRESHMEN select count(*) from MAUI.PRSP_PROSPECT p where p.purge_session_cd = and p.admn_prospect_type_en = 'US_FRESHMAN

7 MAUI Admissions Row Level Security Application Row Level Security Data Impacted Applications Scores Institution Attended Current Security College Program Sub Program Admission Decision

8 MAUI Admissions Row Level Security Application Row Level Security User has Applicant Access to all Admitted apps in College A select count(*) from MAUI.APPL_APPLICATION appl where appl.session_cd = and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from MAUI.APPL_PROGRAM_OF_STUDY apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_key = 'A' )

9 MAUI Admissions Row Level Security Implementation Plan Communication with impacted users of the warehouse Examples will be sent with the communication Removal of Row Level Security on November 27 th

10 MAUI Admissions Row Level Security: SQL Prospect Example ASSUMING EXISTING SECURITY LIMITS TO US_FRESHMEN NOW - count of prospects in the current daily snapshot select snapshot_type, snapshot_qualifier, count(*) from maui.prsp_prospect p where p.purge_session_cd = group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW select snapshot_type, snapshot_qualifier, count(*) from maui.prsp_prospect p where p.purge_session_cd = and p.admn_prospect_type_en = 'US_FRESHMAN' group by snapshot_type, snapshot_qualifier ; -- QUERY OF STUD_STUDENT ROUGH EXMPLE OF WHAT IS HAPPENING NOW -- IF THERE WERE MULTIPLE STUD, TEST, ETC table(s) the "sub select in clause" is repeated for every table. Thus the peformance win going forward! select snapshot_type, snapshot_qualifier, count(*) from maui.stud_student s where s.master_id in ( select p.master_id from maui.prsp_prospect p where p.purge_session_cd = and p.admn_prospect_type_en = 'US_FRESHMAN' ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - INNER JOIN to PROSPECT select s.snapshot_type, s.snapshot_qualifier, count(*) from maui.stud_student s join maui.prsp_prospect p on s.master_id = p.master_id where p.purge_session_cd = and p.admn_prospect_type_en = 'US_FRESHMAN' group by s.snapshot_type, s.snapshot_qualifier ;

11 MAUI Admissions Row Level Security: SQL Application Example ASSUMING EXISTING SECURITY LIMITS TO ADMITTED APPS IN CLAS NOW - count of applications in the current daily snapshot select snapshot_type, snapshot_qualifier, count(*) from maui.appl_application appl where appl.session_cd = group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW ( all apps ) select snapshot_type, snapshot_qualifier, count(*) from maui.appl_application appl where appl.session_cd = and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from maui.appl_program_of_study apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_acad_key = 'A' ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW ( all PRIMARY apps ) select appl.snapshot_type, appl.snapshot_qualifier, count(*) from maui.appl_application appl join maui.appl_program_of_study apos on apos.appl_application_id = appl.appl_application_id and apos.is_application_primary_pos = 'Y' where appl.session_cd = and appl.admn_decision_type_en = 'ADMITTED' and apos.program_college_acad_key = 'A' group by appl.snapshot_type, appl.snapshot_qualifier ; -- QUERY OF STUD_STUDENT ROUGH EXMPLE OF WHAT IS HAPPENING NOW -- IF THERE WERE MULTIPLE STUD, TEST, ETC table(s) the "sub select in clause" is repeated for every table. Thus the peformance win going forward! select snapshot_type, snapshot_qualifier, count(*) from maui.stud_student s where s.master_id in ( select appl.master_id from maui.appl_application appl where appl.session_cd = and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from maui.appl_program_of_study apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_acad_key = 'A' ) ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - INNER JOIN to PROSPECT select s.snapshot_type, s.snapshot_qualifier, count(*) from maui.stud_student s join maui.appl_application appl on s.master_id = appl.master_id join maui.appl_program_of_study apos on apos.appl_application_id = appl.appl_application_id and apos.is_application_primary_pos = 'Y' where appl.session_cd = and appl.admn_decision_type_en = 'ADMITTED' and apos.program_college_acad_key = 'A' group by s.snapshot_type, s.snapshot_qualifier ;

12 MAUI Snapshots Changes Eric Hudson ITS, Database Administrator

13 DW Maui Snapshots Snapshot is a picture of a point in time of MAUI Picture taken 6 days a week at midnight (end of the day) SNAPSHOT_TYPE + SNAPSHOT_QUALIFER = the identifier for the picture Type is the grouping Qualifier is the specifics Snapshot Type Snapshot Qualifier CATEGORY Snapshot Qualifier Example DAILY DATE DAY0 SESSION_CD DAY1 DAY9 SESSION_CD CENSUS SESSION_CD LASTDAY SESSION_CD EOM FISCAL FISCALYEAR YEAR 2015 CALENDARWK1 CALENDARWK52 YEAR 2014 WEEK1..WEEK19 SESSION_CD 20143

14 DW by the Numbers - May Oracle Database 6 TERABYTES of Storage 14 Subject Areas 27 Tables with over 100,000,000 rows 2,000 User Accounts 1,313,000 Avg. Connections Per Month 1997 Year first Module was Implemented 1,567 Subject Area Tables 3,637 Subject Area Indexes 4 Oracle DBA s

15 DW by the Numbers - Oct Oracle Database 1997 Year first Module was Implemented 18 TERABYTES of Storage 18 Subject Areas + 19 Legacy Subject Areas 50+ Tables with over 100,000,000 rows 1,872 User Accounts 1,313,000 Avg. Connections Per Month 7,709 Subject Area Tables 4,388 Subject Area Indexes 365 Subject Area Views 4 Oracle Database Administrators

16 Maui.Dw_Snapshot_Info SNAPSHOT_TYPE TOTAL SUMMER FALL WINTER SPRING SESSION CALENDARWK## CENSUS DAILY DAY DAY DAY DAY DAY DAY DAY DAY DAY DAY EARLYREG EOM FISCALYEAR LASTDAY ORIENTATION RESERVE WEEK## Total

17 Removing WEEK## Snapshots as CALENDARWK## is WEEK## Maui area of the Dw is 16.5 of the 18 Terabytes of storage Snapshots Retained Every Friday o o o CALENDARWK##_YEAR (ex CALENDARWK41_2016) o Week # in the calendar year WEEK##_SESSION (ex WEEK7_20163) o Week # in the session Exact same data 2 copies of the Friday Daily snapshots Difference is what is stored in fields snapshot_type and snapshot_qualifier Removing WEEK## snapshots and Keeping CALENDARWK## o Data can be retrieved in CALENDARWK## snapshots o Cuts Maui dw growth rate by ~1/3 o 5.5 terabytes are freed up

18 Impact What is the impact to me and how do I answer questions based on week in session?

19 Maui.Dw_Snapshot_Info additions SNAPSHOT_SESSION_WEEK number(2) o Week # within the session SNAPSHOT_CALENDAR_YEAR number(4) SNAPSHOT_CALENDAR_WEEK number(2) o Calendar year and week # of the snapshot Populated for every snapshot select * from maui.dw_snapshot_info order by snapshot_dt desc; o Weeks run from Saturday-Friday for # o Saturdays loads are not ran maui dw loaded 6 days a week o For example Jan 1 st is a Wednesday Sunday-Tuesday snapshots have calendar week 53 Wednesday-Friday snapshots have calendar week 1

20 Maui.Dw_Snapshot_Week_Vw View for crosswalk of CALENDARWK## to WEEK## o o o For use with queries and determining which snapshot to use SNAPSHOT_TYPE WEEK_TYPE SNAPSHOT_QUALIFIER WEEK_QUALIFIER select * from maui.dw_snapshot_week_vw order by SNAPSHOT_SESSION_WEEK, substr(snapshot_session_cd,-1), SNAPSHOT_DT;

21 Maui.Set_Snapshot (and _Compare) for WEEK## Maui.Set_Snapshot* functions modified to use corresponding Calendarwk## Maui.Set_Snapshot and Maui.Set_Snapshot_Compare o If WEEK## then set to CALENDARWK## o select maui.set_snapshot('week1', '20163') from dual; set to CALENDARWK returns CALENDARWK select snapshot_type, snapshot_qualifier, count(*) from maui.sess_session_info group by snapshot_type, snapshot_qualifier; --returns nothing because it is now CALENDARWK35 and not WEEK1 select snapshot_type, snapshot_qualifier, count(*) from maui.sess_session_info where snapshot_type = 'WEEK1' and snapshot_qualifier = '20163' group by snapshot_type, snapshot_qualifier; IMPACT - USE of WEEK## in sql must be removed or changed to use view maui.dw_snapshot_week_vw (or dw_snapshot_info)

22 Maui.Set_Snapshot_Multi not possible for WEEK## Maui.Set_Snapshot* functions modified to use corresponding Calendarwk## Maui.Set_Snapshot_Multi o WEEK1_20163 = CALENDARWK35_2016 o WEEK1_20123 = CALENDARWK34_2012 o WEEK1_20158 = CALENDARWK4_2016 o Snapshot_type is different so use of select set_snapshot_multi not possible for WEEK## IMPACT Comparison of WEEK## snapshots for year over year (or over sessions) must be done against ALL snapshots

23 WEEK## Comparison with Maui.Set_Snapshot_All select maui.set_snapshot_all from dual; WEEK## Comparison using Dw_Snapshot_Info --fall WEEK1 comparisons select sess.snapshot_type, sess.snapshot_qualifier, si.snapshot_session_week, si.snapshot_session_cd, count(*) from maui.sess_session_info sess inner join maui.dw_snapshot_info si on sess.snapshot_type = si.snapshot_type and sess.snapshot_qualifier = si.snapshot_qualifier where si.snapshot_type like 'CALENDARWK%' and si.snapshot_session_week = 1 and substr(si.snapshot_session_cd,-1) = '3' group by sess.snapshot_type, sess.snapshot_qualifier, si.snapshot_session_week, si.snapshot_session_cd; WEEK## Comparison using Dw_Snapshot_Week_Vw --fall WEEK1 comparisons select sess.snapshot_type, sess.snapshot_qualifier, v.snapshot_session_week, v.snapshot_session_cd, v.week_type, v.week_qualifier, count(*) from maui.sess_session_info sess inner join maui.dw_snapshot_week_vw v on sess.snapshot_type = v.snapshot_type and sess.snapshot_qualifier = v.snapshot_qualifier where v.week_type = 'WEEK1' and substr(v.snapshot_session_cd,-1) = '3' group by sess.snapshot_type, sess.snapshot_qualifier, v.snapshot_session_week, v.snapshot_session_cd, v.week_type, v.week_qualifier; In both examples you can replace red text to supply a list of sessions vs fall sessions and si.snapshot_session_cd in ('20163','20153','20158','20148') and v.snapshot_session_cd in ('20163','20153','20158','20148') and v.week_qualifier in ('20163','20153','20158','20148')

24 Timeline for Removing WEEK## Today Begin sql transition to use Calendarwk## Sunday November 27, 2016 Set_Snapshot and Set_Snapshot_Compare switched to Calendarwk## snapshots for Week## Sunday December 18, 2016 Set_Snapshot_Multi disabled for Week## Tuesday January 3, 2017 Turn off Week## snapshot creation Drop of Week## snapshots

25 CAMPUS DATA Ed Hill ITS-AIS Architecture Technical Lead Denny Crall ITS-AIS Developer Michael Alberhasky ITS-AIS Developer

26 Campus Data Team Attributes data.uiowa.edu

27 Campus Data Team Attributes catalog ˈkadlˌôɡ (also catalogue) noun a complete list of items, typically one in alphabetical or other systematic order, in particular: a descriptive list of works of art in an exhibition or collection giving detailed comments and explanations.

28 Campus Data Team Attributes

29 Campus Data Team Members

30 Campus Data Team Group

31 Campus Data Item List

32 Campus Data Item Filters

33 Campus Data Item Data

34 Campus Data Cover Image

35 Campus Data Item Security

36 Campus Data Security Options

37 Campus Data Access Approval

38 Campus Data Request Access

39 Campus Data Access Request (team view)

40 Campus Data Access Request (workflow)

41 IDUG December Session: Expert Challenge Solution Workshop Call for Questions you struggle to answer with DW Send by November 9 th to IDUG-FEEDBACK@IOWA.UIOWA.EDU Sample SQL, Approach to troubleshooting, data results and business rule clarifications Presented by Subject Experts

42 IDUG Next Session Wednesday, December 14th, 2016 UCC2520-D 8:30 10:00 For IDUG Executive Committee and Session Feedback or Ideas For IDUG Peer to Peer Discussion, Collaboration and Sharing IDUG Website Follow us on