Interactive Hyperparameter Optimization with Paintable Timelines

We propose a method to integrate more interactivity into automatic hyperparameter optimization systems to leverage the user's prior knowledge on parameter distribution. In our method, the user continuously observes automatic optimization's progress and dynamically specifies where to search in the parameter space. We present a prototype implementation of an interactive dashboard for an optimizer to show our method's feasibility. The interactive dashboard's main feature is "paintable timeline" where the user can not only observe the past parameter values tested as in standard timeline but also specify the range of future parameters to be tested with simple painting operations. We show three examples where user intervention might improve the performance of automatic optimizations. We run a user study with experts and the results show that, with prior knowledge about parameter distribution of the target problem, interactive optimization can reach better results compared to fully automatic optimization.