Foundation Models (FMs) serve as a general class for the development of artificial intelligence systems, offering broad potential for generalization across a spectrum of downstream tasks. Despite extensive research into self-supervised learning as the cornerstone of FMs, several outstanding issues persist in Graph Foundation Models that rely on graph self-supervised learning, namely: 1) Homogenization. The extent of generalization capability on downstream tasks remains unclear. 2) Scalability. It is unknown how effectively these models can scale to large datasets. 3) Efficiency. The training time and memory usage of these models require evaluation. 4) Training Stop Criteria. Determining the optimal stopping strategy for pre-training across multiple tasks to maximize performance on downstream tasks. To address these questions, we have constructed a rigorous benchmark that thoroughly analyzes and studies the generalization and scalability of self-supervised Graph Neural Network (GNN) models. Regarding generalization, we have implemented and compared the performance of various self-supervised GNN models, trained to generate node representations, across tasks such as node classification, link prediction, and node clustering. For scalability, we have compared the performance of various models after training using full-batch and mini-batch strategies. Additionally, we have assessed the training efficiency of these models by conducting experiments to test their GPU memory usage and throughput. Through these experiments, we aim to provide insights to motivate future research.
We perform a comprehensive benchmark of state-of-the-art self-supervised GNN models through four key aspects: dataset scale, training strategies, Graph Self-Supervised Learning (GSSL) methods for Graph FMs, and adaptability to different downstream tasks.
After training GSSL to obtain node representations, GraphFM evaluates three downstream tasks simultaneously: node classification, link prediction, and node clustering.
Node Classification results on Cora, Citeseer, Pubmed based on full batch training.
Link Prediction results on Cora, Citeseer, Pubmed based on full batch training.
Node Clustering results on Cora, Citeseer, Pubmed based on full batch training.
To investigate the scalability of the Graph Foundation Model, GraphFM conduct experiments on both small and large-scale datasets using a sampling method.
The result of GraphFM in Pubmed dataset with node sampling training strategy.
The result of GraphFM in Pubmed dataset with subgraph sampling training strategy.
The result of GraphFM in Flickr dataset.
To understand the training speed and memory usage of the GSSL methods using different sampling strategies, we report throughput and actual memory usage during training.
Cora
Citeseer
Pubmed
Flickr
Arxiv
GraphFM explore the viability of saving pre-trained models based on their results across different downstream tasks, such as link prediction.
The result of GraphFM in Flickr dataset by saving valid model with the best performance in link prediction.
@article{xu2024graphfm,
title={GraphFM: A Comprehensive Benchmark for Graph Foundation Model},
author={Xu, Yuhao and Liu, Xinqi and Duan, Keyu and Fang, Yi and Chuang, Yu-Neng and Zha, Daochen and Tan, Qiaoyu},
journal={arXiv preprint arXiv:2406.08310},
year={2024}
}