Collaborative Filtering Applied to Educational Data Mining

Transcription

1 Collaborative Filtering Applied to Educational Data Mining KDD Cup 200 July 25 th, 200

2 KDD Team Dataset Solution Overview Michael Jahrer, Andreas Töscher from commendo research

3 Dataset Team Dataset Solution Overview Algebra (9 million lines) Bridge to Algebra (20 million lines) Predict correct first attempt Similarities to collaborative filtering (CF) KDD Cup Student CF User 5 5 Step Item

4 Used Information Team Dataset Solution Overview Student and step Unit and section parsed from the provided hierarchy string Problem name Knowledge components (KC) Opportunity count Problem view

5 Solution Overview Team Dataset Solution Overview K Nearest Neighbor (KNN) Singular Value Decomposition (SVD) Factor Model (FM) Factor Model 2 (FM2) Factor Model 3 (FM3) Group Factor Model (GFM) Neural Network for blending 8-fold cross validation

6 KNN SVD FM

7 K Nearest Neighbor (KNN) KNN SVD FM Prediction is based on students with similar historic results Pearson correlation between students Use the K most similar students for prediction Problems Not all student correlations are defined with equal quality, so we use a correlation shrinkage Some steps are answered by no other students

8 KNN SVD FM ρ s s 2 = I s s 2 ρ s s 2 I s s 2 + α σ(x) = + e x ρ s s 2 = σ (δ ρ s s 2 + γ) c is = ĉ is = c is s S i (s;k) ρ s sc i s s S i (s;k) ρ s s s S i (s;k) ρ s s + µ s β s S i (s;k) ρ s s + β I s s 2... set of steps commonly answered by student s and student s 2 ĉ is... prediction for student s on step i ρ s s 2... Pearson correlation between student s and student s 2 S i (s; K)... to student s the K most similar students who answered step i α, β, γ, δ... meta parameters

9 KNN results KNN SVD FM Dataset RMSE Meta parameters K = 4, α = 2.9, β =.5, Algebra δ = 6.2, γ =.9 K = 4, α = 2.9, β =.5, B to A δ = 6.2, γ =.9 We have not used additional information = room for improvement Using step correlations does not work well

10 KNN SVD FM Singular Value Decomposition (SVD) Sparse student by step matrix C = [c is ], containing the correct first attempt values Represent a student s by a N dimensional feature vector b s Represent a step i by a N dimensional feature vector a i ĉ is = a T i b s

11 Results KNN SVD FM Parameters trained by stochastic gradient descent with L2 regularization λ Does not model biases and additional information Dataset RMSE Meta parameters Algebra N = 0, η = 0.002, λ = 0.02 B to A N = 0, η = 0.002, λ = 0.02 B to A N = 20, η = 0.002, λ = 0.0 B to A N = 20, η = 0.002, λ = 0.03

12 Factor Model (FM) KNN SVD FM ĉ is = µ + ˆµ i + µ s + µ p(i) + µ x(i) + µ u(i) + (ˇµ k + µ ks ) K(i, s) + a i + K(i, s) k K(i,s) k K(i,s) á k T b s µ... global bias ˆµ i... bias for step i µ s... bias for student s µ p(i)... bias for problem p(i) µ x(i)... bias for section x(i) µ u(i)... bias for unit u(i) K(i, s)... set of knowledge components á k... N dimensional KC feature vector

13 FM results KNN SVD FM Significant improvement over SVD Models: unit, section, knowledge component FM2, FM3 and GFM contain further improvements Dataset RMSE Meta parameters Algebra N = 50, η = , λ = 0.0 B to A N = 50, η = , λ = 0.0

14 Summary Thanks

15 Summary Thanks 2 layer neural network 80 sigmoid units on the first and second hidden layer 36 predictors for Algebra predictors for Bridge to Algebra Additional information

16 Feature Generation Summary Thanks

17 Results Summary Thanks Best individual results: Cross validation Leaderboard Algebra Bridge to Algebra Neural network blending: Cross validation Leaderboard Algebra Bridge to Algebra

18 Summary Summary Thanks Ideas from collaborative filtering are suitable for the KDD Cup 200 An ensemble of different models is a good approach for a competition For sure there is a way to build simpler models with similar or better performance

19 Summary Thanks Thanks for your attention commendo research & consulting GmbH Austria