MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning
arxiv(2024)
摘要
The volume of unlabelled Earth observation (EO) data is huge, but many
important applications lack labelled training data. However, EO data offers the
unique opportunity to pair data from different modalities and sensors
automatically based on geographic location and time, at virtually no human
labor cost. We seize this opportunity to create a diverse multi-modal
pretraining dataset at global scale. Using this new corpus of 1.2 million
locations, we propose a Multi-Pretext Masked Autoencoder (MP-MAE) approach to
learn general-purpose representations for optical satellite images. Our
approach builds on the ConvNeXt V2 architecture, a fully convolutional masked
autoencoder (MAE). Drawing upon a suite of multi-modal pretext tasks, we
demonstrate that our MP-MAE approach outperforms both MAEs pretrained on
ImageNet and MAEs pretrained on domain-specific satellite images. This is shown
on several downstream tasks including image classification and semantic
segmentation. We find that multi-modal pretraining notably improves the linear
probing performance, e.g. 4pp on BigEarthNet and 16pp on So2Sat, compared to
pretraining on optical satellite images only. We show that this also leads to
better label and parameter efficiency which are crucial aspects in global scale
applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要