Loading...
Please wait, while we are loading the content...
Similar Documents
Automated Source Code Generation and Auto-Completion Using Deep Learning: Comparing and Discussing Current Language Model-Related Approaches
Content Provider | MDPI |
---|---|
Author | Cruz-Benito, Juan Vishwakarma, Sanjay Martin-Fernandez, Francisco Faro, Ismael |
Copyright Year | 2021 |
Description | In recent years, the use of deep learning in language models has gained much attention. Some research projects claim that they can generate text that can be interpreted as human writing, enabling new possibilities in many application areas. Among the different areas related to language processing, one of the most notable in applying this type of modeling is programming languages. For years, the machine learning community has been researching this software engineering area, pursuing goals like applying different approaches to auto-complete, generate, fix, or evaluate code programmed by humans. Considering the increasing popularity of the deep learning-enabled language models approach, we found a lack of empirical papers that compare different deep learning architectures to create and use language models based on programming code. This paper compares different neural network architectures like Average Stochastic Gradient Descent (ASGD) Weight-Dropped LSTMs (AWD-LSTMs), AWD-Quasi-Recurrent Neural Networks (QRNNs), and Transformer while using transfer learning and different forms of tokenization to see how they behave in building language models using a Python dataset for code generation and filling mask tasks. Considering the results, we discuss each approach’s different strengths and weaknesses and what gaps we found to evaluate the language models or to apply them in a real programming context. |
e-ISSN | 26732688 |
DOI | 10.3390/ai2010001 |
Journal | AI |
Issue Number | 1 |
Volume Number | 2 |
Language | English |
Publisher | MDPI |
Publisher Date | 2021-01-16 |
Access Restriction | Open |
Subject Keyword | AI Ai Software Engineering Deep Learning Language Model Source Code Natural Language Processing |
Content Type | Text |
Resource Type | Article |