ABSTRACTIVE-BASED AUTOMATIC TEXT SUMMARIZATION ON INDONESIAN NEWS USING GPT-2

Aini Nur Khasanah, Mardhiya Hayaty

Abstract


Automatic text summarization is challenging research in natural language processing, aims to obtain important information quickly and precisely. There are two main approach techniques for text summary: abstractive and extractive summary. Abstractive Summarization generates new and more natural words, but the difficulty level is higher and more challenging. In previous studies, RNN and its variants are among the most popular Seq2Seq models in text summarization. However, there are still weaknesses in saving memory; gradients are lost in long sentences so resulting in a decrease in lengthy text summaries. This research proposes a Transformer model with an Attention mechanism that can fetch important information, solve parallelization problems, and summarize long texts. The Transformer model we propose is GPT-2. GPT-2 uses decoders to predict the next word using the pre-trained model from w11wo/indo-gpt2-small, implemented on the Indosum Indonesian dataset. Evaluation assessment of the model performance using ROUGE evaluation. The study's results get an average result recall for R-1, R-2, and R-L were 0.61, 0.51, and 0.57, respectively. The summary results can paraphrase sentences, but some still use the original words from the text. Future work increase the amount of data from the dataset to improve the result of more new sentence paraphrases.

Full Text:

PDF

References


Q. Guo, J. Huang, N. Xiong, and P. Wang, “MS-pointer network: Abstractive text summary based on multi-head self-attention,” IEEE Access, vol. 7, pp. 138603–138613, 2019, doi: 10.1109/ACCESS.2019.2941964.

R. Adelia, S. Suyanto, and U. N. Wisesty, “Indonesian Abstractive Text Summarization Using Bidi-rectional Gated Recurrent Unit,” Procedia Comput Sci, vol. 157, pp. 581–588, 2019, doi: 10.1016/j.procs.2019.09.017.

N. Raphal, H. Duwarah, and P. Daniel, “Survey on Abstractive Text Summarization,” in 2018 In-ternational Conference on Com-munication and Signal Processing (ICCSP), IEEE, Apr. 2018, pp. 0513–0517. doi: 10.1109/ICCSP.2018.8524532.

R. Srivastava, P. Singh, K. P. S. Rana, and V. Kumar, “A topic modeled unsupervised approach to single document extractive text summarization,” Knowl Based Syst, vol. 246, p. 108636, Jun. 2022, doi: 10.1016/j.knosys.2022.108636.

D. Hu, “An Introductory Survey on Attention Mechanisms in NLP Problems,” Nov. 2018, [Online]. Available: http://arxiv.org/abs/1811.05544

A. Gupta, D. Chugh, and R. Ka-tarya, “Automated News Summa-rization Using Transformers.”

R. Nallapati, B. Zhou, C. N. dos santos, C. Gulcehre, and B. Xiang, “Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond,” Feb. 2016, [Online]. Available: http://arxiv.org/abs/1602.06023

A. Vaswani et al., “Attention is all you need,” Adv Neural Inf Process Syst, vol. 2017-Decem, no. Nips, pp. 5999–6009, 2017.

T. Klein and M. Nabi, “Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds,” Nov. 2019, [Online]. Available: http://arxiv.org/abs/1911.02365

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners.” [Online]. Available: https://github.com/codelucas/newspaper

N. Moritz, T. Hori, and J. Le, “Streaming Automatic Speech Recognition with the Transformer Model,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Pro-cessing (ICASSP), IEEE, May 2020, pp. 6074–6078. doi: 10.1109/ICASSP40776.2020.9054476.

G. Szucs and D. Huszti, “Seq2seq Deep Learning Method for Sum-mary Generation by LSTM with Two-way Encoder and Beam Search Decoder,” in 2019 IEEE 17th International Symposium on Intelligent Systems and Informatics (SISY), IEEE, Sep. 2019, pp. 221–226. doi: 10.1109/SISY47553.2019.9111502.

K. Ahmed, S. Keskar, and R. So-cher, “WEIGHTED TRANS-FORMER NETWORK FOR MA-CHINE TRANSLATION.”

A. Galassi, M. Lippi, and P. Tor-roni, “Attention in Natural Lan-guage Processing,” IEEE Trans Neural Netw Learn Syst, vol. 32, no. 10, pp. 4291–4308, Oct. 2021, doi: 10.1109/TNNLS.2020.3019893.

P. J. Liu et al., “Generating wik-ipedia by summarizing long se-quences,” 6th International Con-ference on Learning Representa-tions, ICLR 2018 - Conference Track Proceedings, pp. 1–18, 2018.

H. Batra et al., “CoVShorts: News Summarization Application Based on Deep NLP Transformers for SARS-CoV-2,” in 2021 9th Inter-national Conference on Reliability, Infocom Technologies and Op-timization (Trends and Future Di-rections), ICRITO 2021, Institute of Electrical and Electronics Engi-neers Inc., 2021. doi: 10.1109/ICRITO51393.2021.9596520.

G. Lample and A. Conneau, “Cross-lingual Language Model Pretraining,” Jan. 2019, doi: 10.48550/arXiv.1901.07291.

L. Perez, L. Ottens, and S. Viswa-nathan, “Automatic Code Genera-tion using Pre-Trained Language Models,” Feb. 2021, [Online]. Available: http://arxiv.org/abs/2102.10535

S. Shearing, A. Gertner, B. Well-ner, and L. Merkhofer, “Automat-ed Text Summarization: A Review and Recommendations,” 2020.

Q. Zhu, L. Li, L. Bai, and F. Hu, “Chinese text summarization based on fine-tuned GPT2,” in Third International Conference on Electronics and Communication; Network and Computer Technolo-gy (ECNCT 2021), M. K. Mohid-din, S. Chen, and S. F. EL-Zoghdy, Eds., SPIE, Mar. 2022, p. 109. doi: 10.1117/12.2629132.

“w11wo/indo-gpt2-small • Hug-ging Face.” https://huggingface.co/w11wo/indo-gpt2-small (accessed Feb. 24, 2023).

“kata-ai/indosum: A benchmark dataset for Indonesian text summarization.”, Accessed: Jun. 19, 2023. [Online]. Available: https://github.com/kata-ai/indosum

K. Kurniawan and S. Louvan, “In-doSum: A New Benchmark Da-taset for Indonesian Text Summa-rization,” in Proceedings of the 2018 International Conference on Asian Language Processing, IALP 2018, Institute of Electrical and Electronics Engineers Inc., Jan. 2019, pp. 215–220. doi: 10.1109/IALP.2018.8629109.

T. He et al., “ROUGE-C: A fully automated evaluation method for multi-document summarization,” in 2008 IEEE International Con-ference on Granular Computing, IEEE, Aug. 2008, pp. 269–274. doi: 10.1109/GRC.2008.4664680.

V. Kieuvongngam, B. Tan, and Y. Niu, “Automatic text summariza-tion of covid-19 medical research articles using bert and gpt-2,” arXiv preprint arXiv:2006.01997, 2020, [Online]. Available: https://arxiv.org/abs/2006.01997

J. D’Silva and U. Sharma, “Unsu-pervised Automatic Text Summa-rization of Konkani Texts using K-means with Elbow Method,” In-ternational Journal of Engineering Research and Technology, vol. 13, no. 9, pp. 2380–2384, 2020, doi: 10.37624/ijert/13.9.2020.2380-2384.

S. Esmaeilzadeh, G. X. Peh, and A. Xu, “Neural abstractive text summarization and fake news de-tection,” arXiv preprint arXiv:1904.00788, 2019, [Online]. Available: https://arxiv.org/abs/1904.00788

U. Sharma, “Development of a Konkani Language Dataset for Automatic Text Summarization and its Challenges.” [Online]. Available: http://www.irphouse.com




DOI: https://doi.org/10.33330/jurteksi.v10i1.2492

Article Metrics

Abstract view : 201 times
PDF - 144 times

Refbacks

  • There are currently no refbacks.


Lembaga Penelitian dan Pengabdian Kepada Masyarakat (LPPM) STMIK ROYAL 

Copyright © LPPM STMIK ROYAL

 

Lisensi Creative Commons
Ciptaan disebarluaskan di bawah Lisensi Creative Commons Atribusi-BerbagiSerupa 4.0 Internasional.