Supplementary Material: Neural Predictor for Neural Architecture Search

Figure 1 includes ablation study of different architectures for the Neural Predictor on NASBench-101. We compared Graph Convolutional Networks (GCN) and Multi-layer Perceptrons (MLP) in the figure. To generate inputs for a MLP, we simply concatenate the one-hot codes of node operations with the upper triangle of the adjacency matrix. From the figure, we can see such a simple MLP can outperform state-of-the-art Regularized Evolution; more importantly, the GCN that we selected achieves the best. We also tried Convolutional Neural Networks (CNN) but completely failed with a performance near to random search. During our development, we also proposed a data augmentation to improve the performance of MLP and CNN. In this augmentation, we randomly permute the order of nodes to generate new inputs online. However, we needed to perform the permutation during validation; otherwise, the validation data distribution is different from training data distribution. More importantly, GCN encodes the inductive bias that the prediction should be permutation invariant. Therefore, GCN is our final decision.