ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
arXiv:2203.09127 · doi:10.1145/3534678.3539021
Abstract
Pre-trained models (PTMs) have become a fundamental backbone for downstream tasks in natural language processing and computer vision. Despite initial gains that were obtained by applying generic PTMs to geo-related tasks at Baidu Maps, a clear performance plateau over time was observed. One of the main reasons for this plateau is the lack of readily available geographic knowledge in generic PTMs. To address this problem, in this paper, we present ERNIE-GeoL, which is a geography-and-language pre-trained model designed and developed for improving the geo-related tasks at Baidu Maps. ERNIE-GeoL is elaborately designed to learn a universal representation of geography-language by pre-training on large-scale data generated from a heterogeneous graph that contains abundant geographic knowledge. Extensive quantitative and qualitative experiments conducted on large-scale real-world datasets demonstrate the superiority and effectiveness of ERNIE-GeoL. ERNIE-GeoL has already been deployed in production at Baidu Maps since April 2021, which significantly benefits the performance of various downstream tasks. This demonstrates that ERNIE-GeoL can serve as a fundamental backbone for a wide range of geo-related tasks.
arXiv commentAccepted by KDD 2022, camera-ready version
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- On the Opportunities and Risks of Foundation Models
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Publicly Available Clinical BERT Embeddings
- ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
- ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
- Understanding the Impact of the COVID-19 Pandemic on Transportation-related Behaviors with Human Mobility Data
- A Survey of Point-of-interest Recommendation in Location-based Social Networks
- Quantifying the Economic Impact of COVID-19 in Mainland China Using Human Mobility Data
- GEDIT: Geographic-Enhanced and Dependency-Guided Tagging for Joint POI and Accessibility Extraction at Baidu Maps