[PDF] Question answering on introductory Java programming concepts

View - Download Question answering on introductory Java programming concepts

Previous PDF

Next PDF

IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING,

SECOND CYCLE, 30 CREDITS

, STOCKHOLMSWEDEN2021

Question answering on

introductory Java programming concepts using the Transformer

LUKAS SZERSZEN

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

Question answering on

introductory Java programming concepts using the Transformer

LUKAS SZERSZEN

Date: March 14, 2021

Supervisor: Richard James Glassey

School of Electrical Engineering and Computer Science iii

Abstract

AI applications for education could help students learn in their introductory programming courses. Many applications for education try to simulate a hu- man tutoring session that engages the student in a dialogue. During the ses- sion, they can ask questions and have them answered while working through an exercise. Refining the question-answering capability of such applications mayprovetobeabaseforsupplementaryeducationtools. Thesecouldbeused by students in introductory programming courses to ask questions to review concepts in programming, facilitating the teaching done by professors. This thesis investigates question-answering on introductory Java program- ming using The Transformer model. The focus is on the extent to which the model can answer questions on Java concepts when trained on questions and answers from the online programming forum Stack Overflow. A total of five Transformer models with default parameters were trained on posts segmented with different granularities using byte-pair encoding. Each model was evalu- ated using perplexity as an automatic metric and a qualitative evaluation done by the author. The automatic metric evaluation scores a low perplexity indicating a high- quality model. However, the qualitative evaluation shows that the generated responses are short, generic, repetitive, and even contradicting, with the most common response being "You can do it like this:". That is, the model exhibits a fundamental inability to answers questions on Java programming concepts. iv

Sammanfattning

len. Fokus har lagts på i vilken grad modellen kan besvara frågor angående

Contents

1 Introduction

1

1.1 Problem Specification

3

1.2 Research Question

4

1.3 Limitations

4

2 Background

6

2.1 AI In Education

6

2.1.1 Intelligent Tutoring Systems

7

2.2 Machine Translation

9

2.2.1 Sequence to sequence model

9

2.2.2 Recurrent Encoder-Decoder

10

2.2.3 Transformer

11

2.2.4 Sub-word NMT

13

2.2.5 Word Embeddings

14

2.2.6 Transfer Learning

15

2.2.7 Evaluation

16

2.3 Related Work

18

2.3.1 Intelligent Tutoring Systems

18

2.3.2 Java question answering

20

2.3.3 Conversational Modeling & Dialogue Generation

21

3 Method

23

3.1 Dataset

23

3.1.1 Data collection

25

3.1.2 Dataset statistics

26

3.2 OpenNMT

27

3.3 Preprocessing

29

3.4 Model

32

3.5 Training

33
v vi CONTENTS

3.5.1 Training setup

33

3.5.2 Optimizer

34

3.5.3 Regularization

34

3.5.4 Pre-trained subword embeddings

34

3.6 Evaluation

35

3.6.1 Automatic evaluation - Perplexity

35

3.6.2 Qualitative evaluation

37

3.7 Method limitations

37

3.7.1 Hyperparameter optimization

37

3.7.2 Data preparation

38

4 Results

40

4.1 Metric Evaluation

40

4.2 Qualitative Analysis

44

4.2.1 Evaluation on test dataset

44

4.2.2 Out-of-corpus questions

45

5 Discussion

49

5.1 Generated Response Analysis

49

5.2 Segmentation

51

5.3 Pre-trained Embeddings

51

5.4 Metric & qualitative evaluation

52

5.5 Implications for AI in education

53

5.6 Limitations and future research

54

5.7 Sources of Error

55

5.8 Sustainability, ethics and societal aspects

55

6 Conclusion

57

Bibliography

58

A Appendix

65

A.1 Abbreviations

65

A.2 Code

66

Chapter 1

Introduction

Learning programming and Computer Science is a difficult task. Program- ming is not learned by memorization of facts or concepts from course text- books. It requires students to apply their knowledge to solve programming problems. This process develops the programming skills of the student, but moreimportantly,theirabilitytoproblem-solve. Astudent"sabilitytoproblem- solve is crucial to master programming as they continue in their education and are required to solve harder problems [ 1 Teachers responsible for introductory programming relate to the struggles oftheirstudentsinlearningthefundamentalsofprogramming[ 2 ]. Acommon factor is that students lack generic problem-solving skills, which are essential to learning programming [ 2 ]. As students progress through their introduc- tory course, they fail to connect skills and techniques learned from past pro- gramming assignments to new ones. The consequence becomes that students develop incorrect solutions in the long-term [ 2 3 Help and guidance from a teacher are essential to students learning. Intro- ductory programming courses are often comprised of a larger than usual body of students. Teachers and Teaching Assistants (TA) are limited in both quan- tity and the amount of time they can spend on helping students [ 4 ]. This is a problem throughout education. A large number of students make it difficult to provide the necessary individual help during the limited amount of dedicated time. Providing individual help, or individual one-on-one tutoring with a hu- man tutor has long been established as the most effective means of teaching and improving learning outcomes [ 5 6 ]. As it is not realistic to provide each student with an individual tutor, other means of facilitating learning have been sought after. For this reason, technology has had an increasingly integral role 1

2 CHAPTER 1. INTRODUCTION

in providing solutions that enhance learning. The hope is that technology can supply students and teachers with tools that can mimic, or supplement, indi- vidual tutoring, without the time and accessibility constraints of humans. A variety of computer-assisted learning tools have been developed and in- troduced into teaching. A common example is that of an Intelligent Tutor- ing System (ITS). ITS are applications for education that simulate one-on-one personal tutoring. These systems tutor students through practice problems on which they receive feedback. An integral feature is tracking the student"s de- velopment and knowledge. ITS have shown to be beneficial to learning when combined with traditional teaching methods, and when used on their own [ 7 The recent advancement of Artificial Intelligence (AI) has lead to an emer- genceofresearchlookingtoincorporateAIintoeducation. VariousAI-supported tutoringapproachesandtoolshavebeendevelopedtohelpstudentslearnbasic programming [ 1 ]. While different AI-supported ITS approaches to teaching programmingexist,onesuchapproachisdialogue-basedtutoring. Adialogue- based ITS approach centers around the student engaging with an AI-agent in a dialogue about specific programming concepts and problems [ 1 The dialogue-based approach to AI-facilitated tutoring centers on the stu- dent being able to ask questions to initiate a dialogue, and the system"s ability to respond. The responses provided by the agent progress the conversation through a tutoring session. Different configurations for dialogue-based ap- proaches to basic programming exist, and there is not one generic structure for the tutoring session. Instead, the level of dialogue is adapted to the task and the concept of programming which is being taught [ 1 The broader class of systems which answer questions are referred to as question answering (QA) systems. Such systems are related to the fields of neural machine translation, conversation modeling, and dialogue generation. The neural approach has become increasingly common due to the promising results shown by neural sequence-to-sequence models in predicting the next sentence given the previous sentence [ 4 8 ]. For example, they have shown to be able to find a solution to IT-help-desk technical problems over the course of a few dialogue turns in a conversation [ 8 Current neural sequence-to-sequence models have achieved new state-of- the-art results in question answering, as well as other areas of Natural Lan- guageProcessing(NLP).Ofparticularinterestisthemodel"TheTransformer", which has since its initial release been adapted and modified to further push the limits in various NLP tasks [ 9 10 11 12 ]. It is therefore of interest to investigate the potential use of "The Transformer" when applied to program- ming education. Adapting the Transformer for question answering according

CHAPTER 1. INTRODUCTION 3

quotesdbs_dbs8.pdfusesText_14

[PDF] java programming syllabus pdf

[PDF] java programs on arrays and strings

[PDF] java programs on strings and arrays

[PDF] java programs to practice

[PDF] java projects to practice

[PDF] java questions asked in interview

[PDF] java ring ppt presentation download

[PDF] java se 11 books

[PDF] java se 11 pdf download

[PDF] java send http get request example

[PDF] java series program examples

[PDF] java sort(comparator)

[PDF] java spring @componentscan example

[PDF] java spring boot componentscan

[PDF] java spring componentscan annotation