Researchers from the ETRI Language Intelligence Research Laboratory have developed a small generative language model that performs well with less data. The model, which accelerates learning speed, can be improved into an application model depending on the purpose.
The team, led by Kwon Oh-uk, head of the Language Intelligence Research Laboratory at the Electronics and Telecommunications Research Institute (ETRI), announced on the 28th the development of ‘Eagle,’ a neural network-based small Korean generative language model (SLM) with 3 billion parameters.
Generative language models learn from sentence data to simulate human language abilities through artificial intelligence (AI) technology. They are designed to generate natural conversations based on user questions or instructions. While global IT companies lead large language models (LLMs) containing over 10 to 100 billion parameters in English, more recently, small open models with 1 to 4 billion parameters have been released.
The ETRI researchers increased the Korean language proportion to create a generation model specialized for Korea. In April, the team released a model with 1.3 billion parameters, which is half the size of those from global companies. Despite having only half the training data, the model showed a 15% improvement in performance in certain tasks.
The newly developed model was released as a basic model without fine-tuning. This foundational model can showcase excellent performance by shortening the training time by up to 20% and could be advanced into an application model to enhance new task performance.
The research team is also developing source technologies for predicting the quality of the basic model’s expression and for inference at the concept level. They are working on technologies to enable effective derivation of conceptual knowledge from the basic model and to solve complex problems, like math problems requiring multistep logical development. A model with 7 billion parameters, created with increased learning data, is also being developed and will be released sequentially by 2025.
Director Kwon stated, “Language models developed without abundant resources may not be superior to excellent overseas models in every aspect,” but expressed hope that “this relatively small Korean-native model will greatly aid research and development in various fields across academia and industry.”