Tutorial: Large Scale Global Optimization Part 1: Introduction and Decomposition Methods

"Many real-world optimization problems involve a large number of decision variables. The trend in engineering optimization shows that the number of decision variables involved in a typical optimization problem has grown exponentially over the last 50 years, and this trend continues with an ever-increasing rate. The proliferation of big-data analytic applications has also resulted in the emergence of large-scale optimization problems at the heart of many machine learning problems. The recent advances in the area of machine learning has also witnessed very large scale optimization problems encountered in training deep neural network architectures (so-called deep learning), some of which have over a billion decision variables. It is this ``curse-of-dimensionality�� that has made large-scale optimization an exceedingly difficult task. Current optimization methods are often ill-equipped in dealing with such problems. It is this research gap in both theory and practice that has attracted much research interest, making large-scale optimization an active field in recent years. We are currently witnessing a wide range of mathematical, metaheuristics and learning-based optimization algorithms being developed to overcome this scalability issue. This Tutorial is dedicated to exploring the recent advances in this field, covering topics in large-scale black-box continuous optimization and large-scale combinatorial optimization. In particular we focus on:

The methods that learn and exploit problem structures to tackle large-scale black-box optimization problems such as decomposition methods based on variable interaction analysis.
The opportunities and challenges of applying Evolutionary Algorithms (EAs) to deep learning are also discussed.
The problem reduction and decomposition techniques to solve large-scale combinatorial optimization problems, with a special focus on the emerging topic of leveraging machine learning and data mining for combinatorial optimization"