AI in Education
Heffernan & Heffernan, ASSISTments
Last year one of the test questions was "What were the two lessons learned about ASSISTments?" Lesson 1: "Build a platform for learning sciences, not a curriculum" to get a community of teachers to use ASSISTments and provide a testbed for research Lesson 2: "Put the teacher in charge, not the computer.... Building a flexible system that allows teachers to use the tool in concert with the classroom routine." the system should be flexible enough for teachers to want to use it, and ASSISTments puts teacher in charge of their own content, students' data, etc. While they think that has contributed to widespread teacher adoption, they also acknowledge the tradeoff that learning to use the system takes more work because of this flexibility Future directions: making ASSISTments the Wikipedia of educational content
To Tutor or not to Tutor
Leena Razzaq This was a great simple study that showed quite large effects of who benefits from tutored problem solving (TPS) compared to solutions. -Found that low-proficiency students benefited more from problems with TPS feedback, while high-proficiency students benefited from problems with Solution feedback. -Also found that less-proficient students spent less than half as much time reading through solutions than high proficient students. This probably means that low-proficiency students were not engaged with these problems, while high-proficiency students were. -Attempted to answer the "assistance dilema" , how should tutoring systems balance giving and withholding information to optimize learning
The Lean Startup
Steve Blank Lean method: 1.) Business model canvas: set of hypotheses 2.) customer development: get out of the buidling to test minimal viable products and revise 3.) agile development: developing the product iteratively and incrementally "lean start-up," and it favors experimentation over elaborate planning, customer feedback over intuition, and iterative design over traditional "big design up front" development Traditionally- create a 5 year business plan Lean Startup- failing fast and continually learning
Open Science paper
Tim Van der Zee, Justin Reich Taylyn: I really like the idea of having open resources for each step of the research process (design, data, analyses, publishing). Other than making the process more transparent and open for close analysis by others, it forces researchers to truly think about their designs up front. I think I've even seen and have been a part of instances where there are loose research questions and hypotheses, but then the analyses change. Though it all still fits with the theory, if you change the analysis, then are you changing the hypothesis too? It's so difficult to publish null results, I think making the process more transparent will open some space for important null results. Or it may force more research to not be published because the results did not occur as hypothesized. I wonder how this will affect the publication process in other ways too. I've experienced submitting a paper to a publisher and one reviewer suggests a different analysis, which in turn slightly changes the interpretation of the results. Would this still happen with Open Education Science? Or would there be a part of the paper that describes changing the analyses? I think that so much of what we read is only published as idealized versions of what actually happened. Maybe following the Open Education Science framework, we can create opportunities to share the changes that are made from the lessons learned along the way. I do have a question about sharing data. Is there any concern of ownership of data in terms of uncovering interesting findings? Education researchers spend a lot of time and money collecting their data, which they then use to disseminate their results, show the strength of their research, and in turn use to get more grants. If we just share all of our data, does this mean some researchers can just use data from others to find new results, without spending the time and money to collect that data? John: Another flaw to publications is the file drawer problem. This is something which seems obvious, but has been overlooked. Essentially, the idea its very difficult to publish null effects and they look to predominantly publish research which contains positive outcomes. Therefore this leaves many of the null and inconclusive findings in the 'file drawer'. While this is seems minor, I have always wondered why studies that aren't positive aren't published. I have had the assumption that many times it has to do with the publishers assuming the study was poor or the analysis was improper. However, this takes away the chance that any information from the study could be influential to future studies. So while you cannot reject the null hypothesis, there are aspects that could be missing that another study could us to improve their study. Another interesting statistic from this study was that they discussed how a meta analysis was able to show that 33% of authors openly admitted to performing 'questionable' research practices such as dropping data based on their own instincts, not statistics. This clearly creates a bias in the results and creates an incomplete study.
Learning Styles
Hal Pashler Learning styles hypothesis individualizing instruction to the learner's style can allow people to achieve a better learning outcome Outcome of this review: There is no adequate evidence base to justify incorporating learning styles assessments into general educational practice. Best practices for answering these kinds of questions We urge investigators examining learning-styles concepts to embrace the factorial randomized research designs described in the earlier ''Inter- actions as the Key Test of the Learning-Styles Hypothesis'' section, because these alone have the potential to provide ac- tion-relevant conclusions. Making Education more evidence-based "If education is to be transformed into an evidence-based field, it is important not only to identify teaching techniques that have experimental support but also to identify widely held beliefs that affect the choices made by educational practitioners but that lack empirical support." Taylyn's opinion: The idea of learning styles is not bad. It comes from the motivation that students learn differently from each other as they are individuals. However, the harmful part of learning styles research is that it is not evidence-based and it is hugely costly for classrooms. It's not practical even if it were true.
Crowdsourcing Tutoring (M. Turk)
Jake Whitehill Goal: personalized learning at scale through crowdsourcing/learner sourcing How? Collect a large and diverse set of resources which can adapt to individual needs such as time Method of crowdsourcing to create and collect tutorial videos for students by using Mechanical Turk workers. Feasibility: yes with guidelines, 100 vidoes/wk @ $5 They first collected 399 videos that participants made regarding a logarithmic problem and found that 81% of those coded were mathematically correct videos. Effectiveness: Found that students benefited more from seeing one of those crowdsourced videos compared to a control video. Found that the best crowdsourced video led to comparable learning gains compared to a video from Khan Academy. Limitations Coding videos to see if they're mathematically correct
Online Mathematics Homework Increases Student Achievement: ASSISTments Efficacy Trial
Jeremy Rochelle, Mingyu Feng, Craig Mason Taylyn: Overall, the study showed how ASSISTments can be used as an easily adoptable homework intervention, which results suggest can help improve standardized tests scores for all students compared to a business-as-usual homework condition. Though the article had many of the pieces of an HLM paper, I think it was lacking in detail. There needed to be a descriptives table with all the variables and details on the ICC, as this statistic determines if HLM is an appropriate analysis for the data. The authors also could have done a better job at describing the process HLM as a whole and comparing the differences between each model they analyzed. The authors do a good job at pointing out limitations in the study sample, such as all students had access to laptops through the state of Maine, the population was primarily rural and homogeneous, they could not control for time spent on the homework, and that teachers were only assessed on impact the 2nd year of using the system and had time for trial and error in the first year. I think that the last limitation was actually a good study design choice because it gave teachers time to get used to using ASSISTments. However, it is limited in comparison to other studies that did not use the same method. They do a good job at emphasizing the practical implications that despite race, SES or IEP status, this intervention helped all students. In regards to theoretical implications, the authors are careful not to draw causal claims and address that they have other sources of data to be analyzed that get more at mechanisms that explain learning gains, such as teacher surveys and instructional logs. I would like to know more about the effect sizes used in this study. They explained the effect size in terms of standard deviations, an improvement index, and on a grade scale. I would like more information on the improvement index, what exactly it means, and where this is typically used.
Student Learning Benefits of a Mixed-reality Teacher Awareness Tool in AI-enhanced Classrooms
Ken Holstein, Bruce McLaren, Vincent Aleven Importance/Contribution: Experimental evidence on how real time analytic can improve student learning. Previously, research has focused on student performance and is typically not experimental. Pretest Intervention -glasses plus analytics -glasses -no analytics Posttest investigating the effects of this form of teacher/AI collaboration on the ways 1) teachers interact with students while using the software and 2) how student learning processes are affected Results: -The real-time analytics provided by Lumilo appear to have served as an equalizing force in the classroom: driving teachers' time towards students of lower prior ability and narrowing the gap in learning outcomes between students with higher and lower prior domain knowledge. -teacher's use of the glasses, with monitoring support (i.e., support for peeking at a student's screen remotely), but without advanced analytics, may reduce students' frequency of maladaptive learning behaviors (such as gaming/hint-abuse) without significantly influencing teachers' time allocation across students. These results sug- gest that the observed learning benefits of monitoring support may be due to a moti- vational effect, resulting from students' awareness that a teacher is monitoring their activities in the software (cf. [22, 42]), and/or due to a novelty effect
The Role of Student Choice Within Adaptive Tutoring
Korinn Ostrow Investigaing the effects of student choice of feedback (hints) within ASSISTments Study Design RCT 5th grade content in ASSISTments (N=82) at beginning of assignment, randomly assigned to choice (choose vid or text) or no choice (auto assigned vid or text) condition Results -Even if feedback isn't observed, students average significantly higher assignment scores after voicing a choice. p=.048 -trending: those who were given choice were more likely to master their assignment p=.097 Big limitation -only 14% of students overall requested hint feedback
Impact of Off-Task and Gaming Behaviors on Learning: Immediate or Aggregate?
Michale Cocea, Arnon Hershkovitz, Ryan Baker Goal: -investigate two hypotheses about the mechanisms that lead to reduced learning from gaming the system and off-task behavior -(a) less learning within individual steps (immediate harmful impact) and (b) overall learning loss due to fewer opportunities to practice (aggregate harmful impact). Study Design: 4 tutor lessons from a mathematics cognitive tutor pretest tutoring posttest already coded for gaming (harmful and nonharmful) and off-task behavior Measure immediate learning and aggregated learning Results: Gaming the system is associated with both immediate poorer learning (strongly) and aggregate poorer learning (more weakly). --by gaming, an opportunity to learn is wasted at the step level Off-task behavior, on the other hand, appears to only be associated with poorer learning at an aggregate level (strongly). --more cumulative for aggregate learning, fewer learning opportunities
Closing Global Achievement Gaps in MOOCS
Rene Kizilcec Study Design Intervention value affirmation social-belonging intervention control condition social identify threat for LDCs compared to MDCs Social belonging increased completion rates for LDCs but stayed the same for MDCs Value affirmation decreased completion for MDCs compared to LDCs Taylyn: This study showed that a simple intervention can have a powerful influence on narrowing the achievement gap based on sociodemographic differences. From this paper and other findings like this that I've heard of, it makes a lot of sense to take time to reflect on learning not only after learning but also to set intentions before learning. I liked that this paper discussed two studies and for the most part replicated their results from the first study. I think that makes this paper stronger. I thought it was interesting that they had two intervention conditions, value affirmation and social belonging. I was surprised that the value affirmation reliably decreased the persistence of learners from MDCs. I would have thought that reflecting on the value of the course would always increase persistence, or at least keep it the same. I wonder if these courses somehow seem less valuable for learners from MDCs. It might be nice to add a measure that asks learners to rate the value of the course on a scale from 1-5 for the affirmation condition to test this hypothesis. Similarly, it might be nice to ask learners in the social-belonging condition to actually rate how they feel on a belonging scale. I would like to know what the exact effect sizes were. The authors mention that there were "large" effect sizes, but I could not find the exact numbers to interpret myself. I also wish the figure of the results showed the actual gains for each condition by country development type, rather than just the outcome numbers. I know that they want us to compare to the control condition, but I think it's more accurate to show all the gains.
Scooter the Tutor
Ryan Baker System to reduce gaming continual reminder for students to focus and records for teachers which students were gaming invoking social norms by expressing negative emotion towards them giving gaming students another chance to actually learn the material they skipped. Other approaches to gaming have been preventative. This system acknowledges gaming when it happens and creates more opportunities to learn that would have been lost Study Design Pretest Initial instruction Intervention: 80 minutes across multiple periods Experimental condition: Scooter the tutor (N=51) Control condition: no tutor (N=51) Posttest Results Angry emotion expressed by scooter ≠ learning gains ≠ decrease in gaming Students using this system engage in less gaming Students who received a large number of supplementary exercises from Scooter had high learning gains, and caught up to the rest of the class. This result is quite different from the pattern observed in the control condition and past studies [4,5], where students who game harmfully start out with lower pre-test scores, and fall further behind the rest of the class by the post-test. **And Ryan practiced his keynote address for the LAK conference which posed 6-7 challenges for educational data mining/research that he hopes can at least start to be answered in the next decade i.e. How do we get ITSs to communicate with one another? Currently, if a student uses one ITS for a school year but then a different ITS the next year, that second ITS is starting from scratch to learn about the student even though there is existing data about that student's practice and performance - can someone figure out a model to get one ITS to learn from data that was collected in a different ITS?
Mind Wandering Detection and Intervention using Eye Trackers
Sidney D'Mello We developed the first educational technology capable of real- time mind wandering detection and dynamic intervention during computerized reading. Intervention asking questions on pages where mind wandering is detected and encouraging re- reading in response to incorrect responses will aid in re-directing attention to the text and correct knowledge deficits Like Scooter the Tutor, gives students who were behaving negatively another opportunity to learn where they would have missed out previously. Study Design No pretest (assumed students were not familiar with subject matter) Intervention MW Condition: Not told that they were using mind wandering detectors, just that sometimes they would be asked to answer a question and possibly re-read a page and answer another question "Yoked" Control Condition: Matched with an intervention condition and was asked an intervention questions on the same pages as matched pair, but despite their likelihood of mind wandering Posttest Results No difference in reading times Signs of a good detector Mind wandering likelihoods negatively correlated with performance on online and posttest questions showed promising effects for our intervention approach despite a very conservative experimental design, although the intervention itself did not lead to a boost in overall comprehension (because it is remedial), it equated comprehension scores when mind wandering was high (i.e., scores for the intervention group were comparable when the control group was low on mind wandering). cost of not intervening during mind wandering (i.e., scores for the intervention group were greater when the control group was high on mind wandering).
Instruction Based on Adaptive Learning Technologies
Vincent Aleven We define adaptivity as follows: A learning environment is adaptive to the degree that (a) its design is based on data about common learner challenges in the target subject matter, (b) its pedagogical. decision making changes based on psychological measures of individual learners, and (c) it interactively responds to learner actions (cf. Aleven et al., 2015; Aleven, Beal, & Graesser, 2013). Step-Loop --> changing instructional features within a task for the learner, like hints Task-Loop --> fchanging instructional tasks for the learner, like responding to student knowledge and Design-Loop --> instruction is adapted based on similarities among learners by course designers what should learning technologies adapt to? (evidence for all except learning styles) student knowledge student's paths affect & motivation self-regulation & metacogntion learning styles
Lipsey Effect Sizes
effect size is the magnitude of the difference between groups While a P value can inform the reader whether an effect exists, the P value will not reveal the size of the effect. Statistical significance is the probability that the observed difference between two groups is due to chance. With a sufficiently large sample, a statistical test will almost always demonstrate a significant difference, unless there is no effect whatsoever, that is, when the effect size is exactly zero; yet very small differences, even if significant, are often meaningless. Thus, reporting only the significant P value for an analysis is not adequate for readers to fully understand the results. I appreciated the section on accounting for basline differences. I often do this in my analyses by including the pretest as a covariate when predicting the posttest as an outcome. In some other courses I've had, people have advised me to use just posttest as the outcome or use learning gains (post-pre) as the outcome. I've also used a normalized learning gain (post-pre/100-pre) as an outcome measure.
