INSTITUTIONAL DATA USERS GROUP October 12 th 2016 October Meeting
IDUG Agenda MAUI Admissions Row Level Security Adam Pyatt, Eric Hudson MAUI Snapshot Changes Eric Hudson Campus Data Portal Denny Crall, Michael Alberhasky, Ed Hill ITS-AIS ITS-AIS ITS-AIS
MAUI Admissions Row Level Security Adam Pyatt Eric Hudson ITS-AIS Enrollment Management Technical Lead ITS-AIS Database Administrator
MAUI Admissions Row Level Security Overview Quick Room Survey How many filter Application data using college or program How may filter Student data ( tests, GPA, etc ) using college or program Remove Row Level Security from both the Prospect and Application tables Impact on Student and Test tables Why Performance Increase Transparency
MAUI Admissions Row Level Security Prospect Row Level Security 11 prospect users impacted around campus Data Impacted Mail Item History Referral Codes STUD_* TST_* Current Security Program Prospect Type Prospect Status
MAUI Admissions Row Level Security Prospect Row Level Security User has Prospect Access for all US_FRESHMEN select count(*) from MAUI.PRSP_PROSPECT p where p.purge_session_cd = 20173 and p.admn_prospect_type_en = 'US_FRESHMAN
MAUI Admissions Row Level Security Application Row Level Security Data Impacted Applications Scores Institution Attended Current Security College Program Sub Program Admission Decision
MAUI Admissions Row Level Security Application Row Level Security User has Applicant Access to all Admitted apps in College A select count(*) from MAUI.APPL_APPLICATION appl where appl.session_cd = 20173 and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from MAUI.APPL_PROGRAM_OF_STUDY apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_key = 'A' )
MAUI Admissions Row Level Security Implementation Plan Communication with impacted users of the warehouse Examples will be sent with the communication Removal of Row Level Security on November 27 th
MAUI Admissions Row Level Security: SQL Prospect Example -------------------------------- ------- ASSUMING EXISTING SECURITY LIMITS TO US_FRESHMEN -------------------------------- -- NOW - count of prospects in the current daily snapshot select snapshot_type, snapshot_qualifier, count(*) from maui.prsp_prospect p where p.purge_session_cd = 20173 group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW select snapshot_type, snapshot_qualifier, count(*) from maui.prsp_prospect p where p.purge_session_cd = 20173 and p.admn_prospect_type_en = 'US_FRESHMAN' group by snapshot_type, snapshot_qualifier ; -- QUERY OF STUD_STUDENT ROUGH EXMPLE OF WHAT IS HAPPENING NOW -- IF THERE WERE MULTIPLE STUD, TEST, ETC table(s) the "sub select in clause" is repeated for every table. Thus the peformance win going forward! select snapshot_type, snapshot_qualifier, count(*) from maui.stud_student s where s.master_id in ( select p.master_id from maui.prsp_prospect p where p.purge_session_cd = 20173 and p.admn_prospect_type_en = 'US_FRESHMAN' ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - INNER JOIN to PROSPECT select s.snapshot_type, s.snapshot_qualifier, count(*) from maui.stud_student s join maui.prsp_prospect p on s.master_id = p.master_id where p.purge_session_cd = 20173 and p.admn_prospect_type_en = 'US_FRESHMAN' group by s.snapshot_type, s.snapshot_qualifier ;
MAUI Admissions Row Level Security: SQL Application Example -------------------------------- ------- ASSUMING EXISTING SECURITY LIMITS TO ADMITTED APPS IN CLAS -------------------------------- -- NOW - count of applications in the current daily snapshot select snapshot_type, snapshot_qualifier, count(*) from maui.appl_application appl where appl.session_cd = 20173 group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW ( all apps ) select snapshot_type, snapshot_qualifier, count(*) from maui.appl_application appl where appl.session_cd = 20173 and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from maui.appl_program_of_study apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_acad_key = 'A' ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - THIS IS WHAT YOU HAVE TO ADD NOW ( all PRIMARY apps ) select appl.snapshot_type, appl.snapshot_qualifier, count(*) from maui.appl_application appl join maui.appl_program_of_study apos on apos.appl_application_id = appl.appl_application_id and apos.is_application_primary_pos = 'Y' where appl.session_cd = 20173 and appl.admn_decision_type_en = 'ADMITTED' and apos.program_college_acad_key = 'A' group by appl.snapshot_type, appl.snapshot_qualifier ; -- QUERY OF STUD_STUDENT ROUGH EXMPLE OF WHAT IS HAPPENING NOW -- IF THERE WERE MULTIPLE STUD, TEST, ETC table(s) the "sub select in clause" is repeated for every table. Thus the peformance win going forward! select snapshot_type, snapshot_qualifier, count(*) from maui.stud_student s where s.master_id in ( select appl.master_id from maui.appl_application appl where appl.session_cd = 20173 and appl.admn_decision_type_en = 'ADMITTED' and exists ( select 1 from maui.appl_program_of_study apos where apos.appl_application_id = appl.appl_application_id and apos.program_college_acad_key = 'A' ) ) group by snapshot_type, snapshot_qualifier ; -- FUTURE - INNER JOIN to PROSPECT select s.snapshot_type, s.snapshot_qualifier, count(*) from maui.stud_student s join maui.appl_application appl on s.master_id = appl.master_id join maui.appl_program_of_study apos on apos.appl_application_id = appl.appl_application_id and apos.is_application_primary_pos = 'Y' where appl.session_cd = 20173 and appl.admn_decision_type_en = 'ADMITTED' and apos.program_college_acad_key = 'A' group by s.snapshot_type, s.snapshot_qualifier ;
MAUI Snapshots Changes Eric Hudson ITS, Database Administrator
DW Maui Snapshots Snapshot is a picture of a point in time of MAUI Picture taken 6 days a week at midnight (end of the day) SNAPSHOT_TYPE + SNAPSHOT_QUALIFER = the identifier for the picture Type is the grouping Qualifier is the specifics Snapshot Type Snapshot Qualifier CATEGORY Snapshot Qualifier Example DAILY DATE 20140908 DAY0 SESSION_CD 20143 DAY1 DAY9 SESSION_CD 20143 CENSUS SESSION_CD 20143 LASTDAY SESSION_CD 20141 EOM FISCAL 201502 FISCALYEAR YEAR 2015 CALENDARWK1 CALENDARWK52 YEAR 2014 WEEK1..WEEK19 SESSION_CD 20143
DW by the Numbers - May 2014 1 Oracle Database 6 TERABYTES of Storage 14 Subject Areas 27 Tables with over 100,000,000 rows 2,000 User Accounts 1,313,000 Avg. Connections Per Month 1997 Year first Module was Implemented 1,567 Subject Area Tables 3,637 Subject Area Indexes 4 Oracle DBA s
DW by the Numbers - Oct 2016 1 Oracle Database 1997 Year first Module was Implemented 18 TERABYTES of Storage 18 Subject Areas + 19 Legacy Subject Areas 50+ Tables with over 100,000,000 rows 1,872 User Accounts 1,313,000 Avg. Connections Per Month 7,709 Subject Area Tables 4,388 Subject Area Indexes 365 Subject Area Views 4 Oracle Database Administrators
Maui.Dw_Snapshot_Info SNAPSHOT_TYPE TOTAL SUMMER FALL WINTER SPRING SESSION CALENDARWK## 216 56 80 12 68 20123-20163 CENSUS 17 4 5 4 4 20123-20163 DAILY 13 0 13 0 0 20163-20163 DAY0 16 3 5 4 4 20123-20163 DAY1 13 4 5 0 4 20123-20163 DAY2 9 0 5 0 4 20123-20163 DAY3 9 0 5 0 4 20123-20163 DAY4 9 0 5 0 4 20123-20163 DAY5 13 0 5 4 4 20123-20163 DAY6 9 0 5 0 4 20123-20163 DAY7 9 0 5 0 4 20123-20163 DAY8 9 0 5 0 4 20123-20163 DAY9 9 0 5 0 4 20123-20163 EARLYREG 38 0 11 0 27 20148-20158 EOM 50 12 18 4 16 20123-20163 FISCALYEAR 4 3 1 0 0 20141-20163 LASTDAY 16 4 4 4 4 20123-20161 ORIENTATION 27 21 2 2 2 20148-20155 RESERVE 1 0 1 0 0 20123-20123 WEEK## 216 56 80 12 68 20123-20163 Total 703 163 265 46 229 20123-20163
Removing WEEK## Snapshots as CALENDARWK## is WEEK## Maui area of the Dw is 16.5 of the 18 Terabytes of storage Snapshots Retained Every Friday o o o CALENDARWK##_YEAR (ex CALENDARWK41_2016) o Week # in the calendar year WEEK##_SESSION (ex WEEK7_20163) o Week # in the session Exact same data 2 copies of the Friday Daily snapshots Difference is what is stored in fields snapshot_type and snapshot_qualifier Removing WEEK## snapshots and Keeping CALENDARWK## o Data can be retrieved in CALENDARWK## snapshots o Cuts Maui dw growth rate by ~1/3 o 5.5 terabytes are freed up
Impact What is the impact to me and how do I answer questions based on week in session?
Maui.Dw_Snapshot_Info additions SNAPSHOT_SESSION_WEEK number(2) o Week # within the session SNAPSHOT_CALENDAR_YEAR number(4) SNAPSHOT_CALENDAR_WEEK number(2) o Calendar year and week # of the snapshot Populated for every snapshot select * from maui.dw_snapshot_info order by snapshot_dt desc; o Weeks run from Saturday-Friday for # o Saturdays loads are not ran maui dw loaded 6 days a week o For example Jan 1 st is a Wednesday Sunday-Tuesday snapshots have calendar week 53 Wednesday-Friday snapshots have calendar week 1
Maui.Dw_Snapshot_Week_Vw View for crosswalk of CALENDARWK## to WEEK## o o o For use with queries and determining which snapshot to use SNAPSHOT_TYPE WEEK_TYPE SNAPSHOT_QUALIFIER WEEK_QUALIFIER select * from maui.dw_snapshot_week_vw order by SNAPSHOT_SESSION_WEEK, substr(snapshot_session_cd,-1), SNAPSHOT_DT;
Maui.Set_Snapshot (and _Compare) for WEEK## Maui.Set_Snapshot* functions modified to use corresponding Calendarwk## Maui.Set_Snapshot and Maui.Set_Snapshot_Compare o If WEEK## then set to CALENDARWK## o select maui.set_snapshot('week1', '20163') from dual; set to CALENDARWK35 2016 --returns CALENDARWK35 2016 359 select snapshot_type, snapshot_qualifier, count(*) from maui.sess_session_info group by snapshot_type, snapshot_qualifier; --returns nothing because it is now CALENDARWK35 and not WEEK1 select snapshot_type, snapshot_qualifier, count(*) from maui.sess_session_info where snapshot_type = 'WEEK1' and snapshot_qualifier = '20163' group by snapshot_type, snapshot_qualifier; IMPACT - USE of WEEK## in sql must be removed or changed to use view maui.dw_snapshot_week_vw (or dw_snapshot_info)
Maui.Set_Snapshot_Multi not possible for WEEK## Maui.Set_Snapshot* functions modified to use corresponding Calendarwk## Maui.Set_Snapshot_Multi o WEEK1_20163 = CALENDARWK35_2016 o WEEK1_20123 = CALENDARWK34_2012 o WEEK1_20158 = CALENDARWK4_2016 o Snapshot_type is different so use of select set_snapshot_multi not possible for WEEK## IMPACT Comparison of WEEK## snapshots for year over year (or over sessions) must be done against ALL snapshots
WEEK## Comparison with Maui.Set_Snapshot_All select maui.set_snapshot_all from dual; WEEK## Comparison using Dw_Snapshot_Info --fall WEEK1 comparisons select sess.snapshot_type, sess.snapshot_qualifier, si.snapshot_session_week, si.snapshot_session_cd, count(*) from maui.sess_session_info sess inner join maui.dw_snapshot_info si on sess.snapshot_type = si.snapshot_type and sess.snapshot_qualifier = si.snapshot_qualifier where si.snapshot_type like 'CALENDARWK%' and si.snapshot_session_week = 1 and substr(si.snapshot_session_cd,-1) = '3' group by sess.snapshot_type, sess.snapshot_qualifier, si.snapshot_session_week, si.snapshot_session_cd; WEEK## Comparison using Dw_Snapshot_Week_Vw --fall WEEK1 comparisons select sess.snapshot_type, sess.snapshot_qualifier, v.snapshot_session_week, v.snapshot_session_cd, v.week_type, v.week_qualifier, count(*) from maui.sess_session_info sess inner join maui.dw_snapshot_week_vw v on sess.snapshot_type = v.snapshot_type and sess.snapshot_qualifier = v.snapshot_qualifier where v.week_type = 'WEEK1' and substr(v.snapshot_session_cd,-1) = '3' group by sess.snapshot_type, sess.snapshot_qualifier, v.snapshot_session_week, v.snapshot_session_cd, v.week_type, v.week_qualifier; In both examples you can replace red text to supply a list of sessions vs fall sessions and si.snapshot_session_cd in ('20163','20153','20158','20148') and v.snapshot_session_cd in ('20163','20153','20158','20148') and v.week_qualifier in ('20163','20153','20158','20148')
Timeline for Removing WEEK## Today Begin sql transition to use Calendarwk## Sunday November 27, 2016 Set_Snapshot and Set_Snapshot_Compare switched to Calendarwk## snapshots for Week## Sunday December 18, 2016 Set_Snapshot_Multi disabled for Week## Tuesday January 3, 2017 Turn off Week## snapshot creation Drop of Week## snapshots
CAMPUS DATA Ed Hill ITS-AIS Architecture Technical Lead Denny Crall ITS-AIS Developer Michael Alberhasky ITS-AIS Developer
Campus Data Team Attributes data.uiowa.edu
Campus Data Team Attributes catalog ˈkadlˌôɡ (also catalogue) noun a complete list of items, typically one in alphabetical or other systematic order, in particular: a descriptive list of works of art in an exhibition or collection giving detailed comments and explanations.
Campus Data Team Attributes
Campus Data Team Members
Campus Data Team Group
Campus Data Item List
Campus Data Item Filters
Campus Data Item Data
Campus Data Cover Image
Campus Data Item Security
Campus Data Security Options
Campus Data Access Approval
Campus Data Request Access
Campus Data Access Request (team view)
Campus Data Access Request (workflow)
IDUG December Session: Expert Challenge Solution Workshop Call for Questions you struggle to answer with DW Send by November 9 th to IDUG-FEEDBACK@IOWA.UIOWA.EDU Sample SQL, Approach to troubleshooting, data results and business rule clarifications Presented by Subject Experts
IDUG Next Session Wednesday, December 14th, 2016 UCC2520-D 8:30 10:00 For IDUG Executive Committee and Session Feedback or Ideas IDUG-FEEDBACK@IOWA.UIOWA.EDU For IDUG Peer to Peer Discussion, Collaboration and Sharing IDUG-DISCUSSION@LIST.UIOWA.EDU IDUG Website http://itcommunities.uiowa.edu/content/institutional-data-users-group Follow us on Twitter @UIowaIDUG