Curiosity in data science -Curious Hyper-parameter Optimization of Machine Learning Algorithms (CHyPO)

Mor Krispil, M.Sc. student  Advisor: Prof. Goren Gordon

Via Zoom click here

Abstract:
Building an effective machine-learning model is a complex and time-consuming process that involves determining the appropriate algorithm and obtaining an optimal model architecture by tuning its hyper-parameters (HPs). Selecting the best hyper-parameter configuration for machine learning (ML) models directly impacts the model's performance. A traditional way to tune hyper-parameters is through manual testing, which requires a deep knowledge of ML algorithms and their hyper-parameter value settings. The manual testing method is problematic for many reasons, including high number of hyper-parameters, model evaluation's complexity and time constraints, as well as non-linear hyper-parameter interactions. Due to these factors, research for automatic optimization techniques of hyper-parameters has prolifirated in recent years. In this work, we address this challenge by implementing concepts from the field of intrinsically motivated computational learning, also known as artificial curiosity (AC). In AC, an autonomous agent acts to optimize its learning about itself and its environment by receiving internal rewards based on prediction errors. We present a novel method of intrinsically motivated learning and curiosity loops to optimize the hyper-parameters of common machine learning models in large and varied data-sets. An autonomous agent learns to obtain an optimal model architecture over time by tuning its hyper-parameters without requiring external supervision. We first present a detailed analysis of the suggested method, called the Curious Hyper-Parameter Optimization (CHyPO) algorithm, on four data sets taken from the AutoML challenge. We then compare CHyPO to seven hyper-parameter optimization techniques on four OpenML benchmark (classification and regression) data-sets. We show that our method (CHyPO) is in the top three performing algorithms across all models and datasets examined, whereas other algorithms have much higher performance variance. This shows that CHyPO is a more robust and adaptive alternative to state-of-the-art algorithms. Furthermore, the importance of hyper-parameters in machine learning models is crucial to optimize for improved performance, and techniques such as fANOVA are used to evaluate the impact of hyperparameters on the performance of an algorithm. We show that CHyPO also provides additional insights into hyper-parameter importance.

Bio:

Mor Krispil is an M.Sc. student at the Department of Industrial Engineering at Tel Aviv University. Mor holds a B.Sc. degree in Industrial Engineering and Management from Ben Gurion University. She works as a data scientist at Meta and has experience in Business & Product analytics in several global online companies. Her research, supervised by Prof. Goren Gordon, focuses on Hyper-parameter Optimization of Machine Learning Algorithms by implementing concepts from the field of intrinsically motivated computational learning, also known as artificial curiosity.

Contact:

• E-Mail: krispilmor@gmail.com

• LinkedIn: https://www.linkedin.com/in/mor-krispil-631706144/