Context
This paper was part of the Machine Learning Reproducibility Challenge (MLRC) 2024 for which it was published in Transactions on Machine Learning Research (TMLR) (the original publication can be found here).
The MLRC was then also incorporated into NeurIPS 2024 as a workshop track, where this paper was presented at the conference as a poster in Vancouver, Canada. The official NeurIPS link can be found here.
This paper started as a project for the course 'Fairness, Accountability, Confidentiality and Transparency in AI' of the Master AI at the University of Amsterdam.
Context
This paper was part of the Machine Learning Reproducibility Challenge (MLRC) 2024 for which it was published in Transactions on Machine Learning Research (TMLR) (the original publication can be found here).
The MLRC was then also incorporated into NeurIPS 2024 as a workshop track, where this paper was presented at the conference as a poster in Vancouver, Canada. The official NeurIPS link can be found here.
This paper started as a project for the course 'Fairness, Accountability, Confidentiality and Transparency in AI' of the Master AI at the University of Amsterdam.
Summary
In this paper we reproduced the results of an earlier paper that introduced a technique called GNNInterpreter to validate their results. GNNInterpreter is a method that aims to explain the features learned by a Graph Neural Network (GNN) classifier by generating graphs that are the prime examples of each class according to the features that the GNN has learned. For example, in one dataset NO2 bonds in molecules caused a higher degree of mutagenicity, as such we would expect a GNN to pick up on that feature for the mutagenic class. GNNInterpreter would then be able to show us if the GNN had actually learned this feature by generating molecules that the GNN has a very high of classifying as mutagenic. In theory, if the generated molecules include these features than the GNN has learned correctly and will generalize well and if not it might have picked up on some erroneous other features that do not actually correlate to mutagenicity.
Summary
In this paper we reproduced the results of an earlier paper that introduced a technique called GNNInterpreter to validate their results. GNNInterpreter is a method that aims to explain the features learned by a Graph Neural Network (GNN) classifier by generating graphs that are the prime examples of each class according to the features that the GNN has learned. For example, in one dataset NO2 bonds in molecules caused a higher degree of mutagenicity, as such we would expect a GNN to pick up on that feature for the mutagenic class. GNNInterpreter would then be able to show us if the GNN had actually learned this feature by generating molecules that the GNN has a very high of classifying as mutagenic. In theory, if the generated molecules include these features than the GNN has learned correctly and will generalize well and if not it might have picked up on some erroneous other features that do not actually correlate to mutagenicity.
In our work we investigated the main claims made by the original authors, namely that GNNInterpreter (i) generates faithful and realistic explanations without requiring domain-specific knowledge, (ii) has the ability to work with various node and edge features, (iii) produces explanations that are representative for the target class and (iv) has a much lower training time compared to XGNN, the current state-of-the-art model-level GNN explanation technique.
To investigate the claims we reproduced the original code, performed additional quantitative and qualitative evaluation and performed experiments on a new dataset.
In the end, based on our experiments we rejected claims (i) and (iii) while confirming claims (ii) and (iv). Since the rejected claims were crucial to the conclusions of the paper, our reproduction called into question the validity of original paper.