Path Forward Beyond Simulators: Fast and Accurate GPU Execution Time Prediction for DNNWorkloads

Today, DNNs' high computational complexity and sub-optimal device utilization present a major roadblock to democratizing DNNs. To reduce the execution time and improve device utilization, researchers have been proposing new system design solutions, which require performance models (especially GPU models) to help them with pre-product concept validation. Currently, researchers have been utilizing simulators to predict execution time, which provides high flexibility and acceptable accuracy, but at the cost of a long simulation time. Simulators are becoming increasingly impractical to model today's large-scale systems and DNNs, urging us to find alternative lightweight solutions. To solve this problem, we propose using a data-driven method for modeling DNNs system performance. We first build a dataset that includes the execution time of numerous networks/layers/kernels. After identifying the relationships of directly known information (e.g., network structure, hardware theoretical computing capabilities), we discuss how to build a simple, yet accurate, performance model for DNNs execution time. Our observations on the dataset demonstrate prevalent linear relationships between the GPU kernel execution times, operation counts, and input/output parameters of DNNs layers. Guided by our observations, we develop a fast, linearregression-based DNNs execution time predictor. Our evaluation using various image classification models suggests our method can predict new DNNs performance with a 7% error and new GPU performance with a 15.2% error. Our case studies also demonstrate how the performance model can facilitate future DNNs system research.