This seminar was hosted by Microsoft Research Africa, Nairobi together with the Microsoft AI for Good team in May 2024.<\/p>\n\n\n\n
User-generated content (UGC), e.g. social media posts written in “Internet language”, presents a lot of lexical variations and deviates from standard language. As a result, NLP models which were mostly trained on standard texts have been known to perform poorly on UGC, and sentence embedding models like LASER are no exception.<\/p>\n\n\n\n
In this talk, we focus on the robustness of LASER to UGC data. We evaluate this robustness by LASER\u2019s ability to represent non-standard sentences and their standard counterparts close to each other in the embedding space. Inspired by previous works extending LASER to other languages and modalities, we propose RoLASER, a robust English encoder trained using a teacher-student approach to reduce the distances between the representations of standard and UGC sentences. We also use data augmentation to generate synthetic UGC-like training data.<\/p>\n\n\n\n
We show that RoLASER significantly improves LASER\u2019s robustness to both natural and artificial UGC data by achieving up to 2\u00d7 and 11\u00d7 better alignment scores. We also perform a fine-grained analysis on artificial UGC data and find that our model greatly outperforms LASER on its most challenging UGC phenomena such as keyboard typos and social media abbreviations. Evaluation on downstream tasks shows that RoLASER performs comparably to or better than LASER on standard data, while consistently outperforming it on UGC data.<\/p>\n\n\n\n