![\"\"](\"\/hic\/files\/2018\/10\/derry-wijaya-636x636.jpg\")
To create a dataset of colloquial Indonesian language, Wijaya and colleagues turned to social media. Shortening words like \u201cthanks\u201d to \u201cthx\u201d, typing \u201c2\u201d instead of \u201cto\u201d,\u00a0 and removing vowels like in the case of \u201clazy\u201d and \u201clzy\u201d are all examples of colloquial transformations that result in words that cannot be understood by natural language processing systems, despite how common they are on social media. The researchers collected Tweets and made a list of the most frequently used Indonesian words. They identified colloquial language in the list by finding words that are not part of the Indonesian dictionary.<\/p>\n
Indonesian linguists then determined how these colloquial words on Twitter are created from their formal counterparts in the Indonesian dictionary.\u00a0 The linguists\u2019 annotations detailing the different ways of creating colloquialisms were used to train the NLP models. By training a model using information on how to derive colloquial language from formal language, NLP systems might be able to do these language transformations themselves.<\/p>\n
Wijaya and colleagues tested their dataset using one of the most popular NLP models, known as the transformer model. Sure enough, the dataset enables the transformer model to learn and recognize some colloquial Indonesian word transformations, including the removal of vowels and shortening of words. But other transformations are harder for the model to recognize, like suffixes or similar additions to words. This might be because the context of a sentence can affect the meaning of the colloquial word. \u201cFor the model to do better, it needs to understand the context of the word within a sentence,\u201d said Wijaya.<\/p>\n
The dataset\u2019s success in training the transformer model is encouraging and other languages with lots of colloquialisms, like Korean and Arabic, could benefit from AI technology that can process and understand colloquial language. \u201cThe issue of chatbots or other AI technology understanding colloquial language is universal, and we will see more and more of it as social media becomes more prevalent all over the world,\u201d said Wijaya, \u201cThese resources can train AI models to process colloquial words that they aren\u2019t used to.\u201d<\/p>\n
\n