APA INSTRUCTIONS

Transcription

1 APA INSTRUCTIONS For the Achievement Data Section of Instructional Program Review ONLY! Abstract Step-By-Step Instructions for Completing the Achievement Data (Section A) of the Annual Program Assessment (APA) Template for Instructional Program Review. Due Date: FEBRUARY 16, 2017 Paul Hert or

2 Introduction Program Review is a faculty-created process to improve student learning. It is a cyclical, comprehensive inquiry which relies on data and its analysis to improve programs, inform planning and drive budget allocation. Page 1 At MSJC we have a three-year program review cycle. The first year a comprehensive program review (CPR) is completed. The second and third years a follow-up annual program assessment (APA) is completed. Currently we are in the third year of our three-year cycle so the second and final ( ) APA will be completed. This particular template is not the full APA; it captures only the achievement data analysis. This includes analysis of data disaggregated by site and delivery modality. This template will be merged with other sub-templates that capture the CIP input and analysis of the learning outcome data analysis. Combined with a final summary, the compilation of all these templates will comprise the APA. This template (for section A of the APA) and the CIP input template (section C of the APA) are due 16 February Neither the data nor the template for the learning outcome analysis are ready; this subtemplate (section B of the APA) will be due later in the spring. If you are submitting RAPs, you only need to submit the sub-templates for APA sections A and C to provide full program review support of your RAPs. You will need to complete a program review for each degree. Therefore if you offer more than one degree, you will need to create more than one APA! Please remember these are step-by-step instructions so due to the level of detail they contain, they are lengthy! However, they should be a quick read, and, using the table of contents directly below, you can skip the portions you don t need! If you have any questions, please contact Paul Hert. Thank you for your participation!

3 Table of Contents Accessing the Template... 3 Entering Your Program (Degree) and Discipline... 5 Accessing the Data... 6 Page 2 A1. Program Achievement Data A1a: Program Fill Rate A1b: Program Retention Rate A1c: Program Success Rate A2. Demographic Success Rate: Ethnicity/Race A2a: Program Success Rate A2b: BAN Success Rate A2c: MVC Success Rate A2d: SJC Success Rate A2e: TEM Success Rate A2f: FTF v. Online/Hybrid Success Rate A3. Demographic Success Rate: Additional A3a: Program Success rate A3b: BAN Success Rate A3c: MVC Success Rate A3d: SJC Success Rate A3e: TEM Success Rate A3f: FTF v. Online/Hybrid Success Rate /31/2016, v1

4 To download a copy of the APA Section A template, APA Section A Achievement, go to the Program Review webpage: Page 3 Then click Templates, Instructions and Resources :

5 Click on the Word document to download it: Page 4 Click on the download (typically at the bottom left-hand corner of your screen) to open it:

6 First, please enter your program name in the header. Just double-click inside the header and type in your program degree name. For this example, we ll use a transfer degree for Geography (GEOG (AST OPT B)): Page 5 Next type in your discipline name. Here that would be Geography: NOTE: A separate program review must be completed for each (different) degree. Please do not write a single program review for multiple degrees!

7 Accessing the Achievement Data Here is a link to the gorgeous Tableau data presentation: %2f15%2fAuthenticate.aspx%3fSource%3d%252Fdepts%252Fresearch%252FReporting%252FShared%2 520Documents%252FReports%252FProgram%2520Review%252FProgram%2520review%252Exlsx%253F Web%253D1&Source=%2Fdepts%2Fresearch%2FReporting%2FShared%20Documents%2FReports%2FPr ogram%20review%2fprogram%20review%2exlsx%3fweb%3d1 Page 6 Click on the drop-down arrow and enter your MSJC username and password for credentials under Windows Authentication :

8 Here is the data: Page 7 To select your program, click on the Programs filter button

9 then hover over Programs to access the drop-down menu, click on Filter Page 8 and you will get a list of all programs:

10 Click on the current program s check box to deselect it and scroll down to select your program. For example, we ll choose AA.GEOG.OPTBAAT (the Associate of Arts Transfer degree in Geography, option B (Cal State)) and click OK : Page 9 Here are the Program Rates:

11 Template Section A1a: Program Fill Rate. Click anywhere outside the header to return control to the main document (APA template). To complete section A1, we need to compare the program fill rate to the institution's average fill rate and set standard. Here is what you'll see in the APA template: Page 10 A1a. Program Fill Rate From fall 2013 through spring 2016, how does the program fill rate compare to the Institutional Average Fill Rate and Institutional Set Standard? Click here to enter text. Let's focus on the fill rate data for our program (AA.GEOG.OPTBAAT, for our example):

12 Note the timeline for the data. The data begins with FA13 and ends SP16: Page 11 Note that from fall 2013 thorough spring 2016, the district average (yellow graph) has been steadily falling from about 92% in FA13...

13 ...to just below 84% in SP16: Page 12 The precise fill rate values can be found in the table below the graph: Let's report this trend in the template response (entering it where it says "Click here to enter text"):

14 Page 13 The institutional set standard fill rate (green line) is 85%. It is a district-set benchmark. Let's add this information to our response:

15 Now, over the same time frame, we'll compare the program (AA.GEOG.OPTBAAT) fill rate to the district average and set standard rates. First, let's compare the program rate to the district average rate. Since we don't need the set standard, remove it: click on the "filtered" icon... Page 14...hover over "Programs" and then click on "Filter"...

16 ...and a list of all our programs will appear. Scroll to the bottom to find the set standard: Page 15 Click on its checkbox to deselect the standard and then click "OK" to remove the set standard graph:

17 Since the only programs now selected are the program and institutional average fill rates, that's all you'll see (no more green lines!): Page 16 Note that like the district average rate (yellow graph), the program fill rate graph (orange graph) trends downward (from left to right) with time:

18 Only very recently is there an increase (upward trend) in program fill rate: Page 17 This means, like the district rate, the program rate is decreasing over time. This can also be seen from the table beneath the graphs, since the percentages (for both rates) are decreasing over time (i.e. reading each row of the table, left to right):

19 From the fill rate graphic, you can see the program fill rate (orange graph) is underneath the district average fill rate (yellow graph): Page 18 Moreover, this is true each and every semester. This means the program fill rate is less than the district average fill rate. However, the difference isn't much, usually 2-4%. And, while both rates are trending downward from FA13 through SP16, they are almost identical in SP16 (about 83-84%). Let's note this in the APA template:

20 Let's compare the program fill rate to set standard and we'll be done with this response. Let's "reselect" the institutional set standard and deselect the institutional average ("District Average" in (alphabetical) programs list; see previous instructions (pp ) for selecting and deselecting programs using the "filtered" icon). Then we'll see this: Page 19 Note that the institutional set standard is now the yellow graph (a straight line, since its value doesn't change with time; it is always 85%, each and every semester, as set by the district). Whenever the program fill rate (orange graph) is above the set standard, the program fill rate is larger than the set standard:

21 When the orange graph falls below the yellow line, the program fill rate is less than the institutional set standard: Page 20 Note that over the timeline (FA14 - SP16), the average rate values were almost the same: Let's enter the comparison between the set standard and program rates in the template: Now let's compare the individual site fill rates with the overall program fill rate.

22 First, let's get the overall average program fill rate from FA13 through SP16. From the table, it is 85.4%. Page 21 We'll use 85%. Here's the next template prompt: To answer this question for Online (ONLIN), we need to filter the data so only program courses offered online are utilized. Go to the data, click on the "Campus" drop down menu, hover over "Location" and click on "Filter..."

23 You'll see a list of the MSJC sites: Page 22 Click on the "Select All" checkbox to deselect everything and then click on the "ONLIN" checkbox:

24 Now click "OK": Page 23 The ONLIN average program fill rate is about 82%. Let's report this: Let's do the same thing for SJC. Go back to the data, deselect "ONLIN" and select "SJC" (under "Campus").

25 You should see this: Page 24 The SJC average program fill rate (from FA13 through SP16) is 94.9% (about 95%). If you do the same thing for Menifee (MVC), Banning (BAN) and Temecula (TEM), you'll get (approximate) rates of 87%, 63% and 91%, respectively. Let's enter these into the template:

26 Next we are asked to address the lowest of these site fill rates. Here is the prompt: Page 25 The lowest site fill rate was at Banning (63%, about 22% below the overall program average). To explain this, along with making suggestions that improve the rate, while maintaining access and equity, let's look at the Banning data one more time:

27 Notice that the typical number of sections offered is 2 or 3: Page 26 That's not very many students to begin with (90-120, if the class cap is somewhere between 30 and 40 students). If the economy picks up suddenly, or work is seasonal, and students leave school to find work, sections may suddenly not fill as they had the semester before. Or perhaps the offerings themselves are not an attractive mix for students (the actual courses offered, the days and times they are offered, the length of the courses (full-semester vs. short-term), etc.) It can be a tough balancing act. On the one hand, if you are trying to build a program schedule by offering both the discipline and elective courses, you'd want to offer enough of these sections semester by semester to build a student following (and live with lower fill rates in many of these courses as the program caught on at that particular site). On the other hand, to offer sections semester after semester that aren't very full is not an efficient use of resources (spending (excess) money for multiple instructors, electricity for multiple classrooms, etc.). We need an expert (i.e. Geography faculty member familiar with the courses and the history of program offerings at Banning) to interpret this data and make a reasonable and educated analysis!! Once that has been done, the response will be entered where it says: "Click here to enter text"!

28 Now that section A1a is complete, let's move on to section A1b (Retention Rate). Thankfully, you do the same analysis for retention as you did for fill. Here is the graphic: Page 27 You won't have all the tables you had for the fill rate, but they'll come back for the success rate (section A1c). So, do essentially what you did for fill rate (A1a) for retention (A1b) and success (A1c) and you'll be ready for Section A2! Here it is:

29 To complete the first prompt, we need to access the "Demographics Headcount" spreadsheet. Click on this tab to do so: Page 28 You should see this:

30 The three largest subpopulations are "White", "Hispanic" and "African American" since the bar graphs corresponding to these three subpopulations are the tallest three: Page 29 Let's report this: Next we determine the success rates for these three subpopulations. Click the "Demographics Success Rate" tab at the bottom of the data program...

31 ...and you will see this: Page 30 Here are the success rates for our three subpopulations:

32 Let's report these: Page 31 Next let's compare these to the program, district and set standard rates. Click on the "Program Rates" tab at the bottom of the data program; here's what you'll see: Here are the rates we need for our comparisons: Program Rate: 65%; District Rate: 68%; Set Standard: 66%. Here's the prompt: Using the Success Rate data on the Program Rates spreadsheet tab, how do the success rates for each of the three subpopulations above compare to a) the program success rate, b) the institutional average success rate and c) the institutional set standard success rate? Please list each subpopulation and its comparisons with these (three) rates. Click here to enter text.

33 Here's a response: Using the Success Rate data on the Program Rates spreadsheet tab, how do the success rates for each of the three subpopulations above compare to a) the program success rate, b) the institutional average success rate and c) the institutional set standard success rate? Please list each subpopulation and its comparisons with these (three) rates. The program average success rate is 65%, the district average success rate is 68% and set standard is 66%. The White rate (68%) is the same as the district, and 2-3 % above the program and set standard. The Hispanic rate (62%) is 4-6% below the three rates. The African American rate (52%) is 18% below the district, 13% below the program and 14% below the set standard. Page 32 To complete this section of the template, we would need to address the final prompt for the African American population: For any subpopulation success rates that are significantly above or below the program success rate, please offer an explanation for this difference. Click here to enter text. Let's continue, disaggregating by site. We'll start with Banning (BAN). Here is the prompt: A2b. BAN Sites (face-to-face only; no online, no hybrid) Using the Ethnic/Race Group Headcount data on the Demographics Headcount spreadsheet tab, what are the three largest student subpopulations? Please list each subpopulation and its size. Click here to enter text. Go back to the "Demographics Headcount" tab on the data program and select "BAN" as the only "Campus". Also, filter the "Delivery Mode" so "Face-to-Face" is the only selection.

34 Here's what you'll see: Page 33 Notice that the three largest subpopulations are now "White", "Hispanic" and "Multi-Race":

35 Here is the response: A2b. BAN Sites (face-to-face only; no online, no hybrid) Using the Ethnic/Race Group Headcount data on the Demographics Headcount spreadsheet tab, what are the three largest student subpopulations? Please list each subpopulation and its size. Page 34 Hispanic students are the largest subpopulation (123) followed by Whites (53) and Multi-Race (27). Note that we are doing here in section A2b for Banning exactly what we did for the program in the previous section, A2a. The remaining prompts in this section are the same as they were for A2a. Therefore, you can complete the prompts for BAN just as you did for the entire program in the previous template section. IMPORTANT: As you move between the spreadsheet tabs ("Program Rates", "Demographic Headcounts" and "Demographics Success Rates") in sections A2b, A2c, A2d and A2e, make sure the "Delivery Mode" filter has only "Face-to-Face" selected. Also, the site should not change when completing each of these template sections. Now complete sections A2c, A2d and A2e just as you did A2a and A2b. Let's compare the face-to-face rates to the online rates at each site (section A2f). Here is the prompt: A2f. Face-to-Face v. Online (all sites, hybrid included with online) For each site, compare the online success rate with the site face-to-face rate. If there is a significant difference (8% or more) for any of these comparisons, offer an explanation and suggestion for improvement. Click here to enter text. Let's go back to the data and get the face-to-face success rate for the program. Click on "Delivery Mode" and select only "Face to Face" (click on the "Delivery Mode" filtered icon, hover over the bottom "Delivery Mode" to open the next drop down menu and then select "Filter" at the bottom of that menu. Then select only "Face to Face"). You should see this:

36 Page 35 The program face-to-face success rate is 64.4% (we'll say 64%). Now go back to the "Delivery Mode" filter, deselect "Face to Face" and select both "Hybrid and Online":

37 Click "OK" and you'll see this: Page 36 The hybrid/online program success rate is higher: 65.3% (we'll say 65%). Here is our response: A2f. Face-to-Face v. Online (all sites, hybrid included with online) For each site, compare the online success rate with the site face-to-face rate. If there is a significant difference (8% or more) for any of these comparisons, offer an explanation and suggestion for improvement. For the program overall, the face-to-face success rate is 64%. The online/hybrid success is one point higher (65%). Now let's do the same thing for each site. Starting, with BAN, deselect all the other sites using the "Campus" filter:

38 Click "OK" and you'll see this: Page 37 There is no data. Note that "Hybrid" and "Online" are still the selected delivery modes ("Online" is invisible here but it is still checked). Therefore we can't make the comparison. Let's get the face-to-face rate at Banning and at least report that. Go back to the Delivery Mode filter and select only face-to-face to get that success rate: It is about 63%. Let's share this information:

39 A2f. Face-to-Face v. Online (all sites, hybrid included with online) For each site, compare the online success rate with the site face-to-face rate. If there is a significant difference (8% or more) for any of these comparisons, offer an explanation and suggestion for improvement. For the program overall, the face-to-face success rate is 64%. The online/hybrid success is one point higher (65%). BAN has no hybrid/online sections; the face-to-face success rate is 63%. Page 38 Now let's use the Campus filter to replace BAN with MVC. First click on the "Campus" filtered icon, hover over "Location" and then click on "Filter": Click on the BAN check box to deselect it and then check the MVC box:

40 Click "OK" and you'll see this: Page 39 The MVC face-to-face success rate is 69%. Now use the Delivery Mode filter to select only hybrid and online to get the MVC online success rate:

41 The rate here is about 63%: Here's the response: A2f. Face-to-Face v. Online (all sites, hybrid included with online) For each site, compare the online success rate with the site face-to-face rate. If there is a significant difference (8% or more) for any of these comparisons, offer an explanation and suggestion for improvement. Page 40 For the program overall, the face-to-face success rate is 64%. The online/hybrid success is one point higher (65%). BAN has no hybrid/online sections; the face-to-face success rate is 63%. The MVC face-to-face rate is 69%; online is 63%. Now do the same thing for SJC and TEM, and you're done with section A2f! On to our final section: A3. Demographics Success: Your Choice! Choose another demographic (age, gender, program enrolled, financial aid, enrollee type, first generation status, veterans, foster youth, athletes, DSPS) and analyze its data. If no data is available for a particular prompt, please say no data. If you choose "Age", please follow the instructions for ethnicity/race above (this template is not locked so you can copy/paste/delete as needed). Your choice: Click here to enter text. You choose a demographic and analyze it! Put your choice here: Click on the "Demographics Success Rate" tab at the bottom of the data program to see what interests you:

42 We've already done ethnicity above (section A2). Let's try Veterans Success Rate: Page 41 Note that even though the blue bar looks to be over twice as tall as the yellow one, the actual numbers aren't that different: the veteran population success rate for our program is about 67% while the nonveterans succeed at a rate about 3 points lower (64%). Here is the first prompt: Your choice: Veteran's Success Rate A3a. Program (all sites, all modalities (face-to-face and online and hybrid)) Using the data for your demographic from the Demographics Headcount spreadsheet tab, please list each subpopulation and its size. Click here to enter text. This should feel familiar; we already did this back in section A2. Click on the "Demographics Headcount" tab at the bottom of the data program to identify the subpopulation sizes:

43 There are 237 veterans and 3,627 non-veterans. That's our response for A3a: A3a. Program (all sites, all modalities (face-to-face and online and hybrid)) Using the data for your demographic from the Demographics Headcount spreadsheet tab, please list each subpopulation and its size. There are 237 veterans and 3,627 non-veterans. Page 42 Complete this section (A3a) just as you did A2a, and then do the same for the sites like you did to complete the entirety of section A2, and you are done! Please check your Assessment and Program Review Calendar from time to time; links for the remaining template and instructions will become live once the template and instructions are available! Once again, if you have any questions, please contact Paul Hert and he will assist you!