Welcome to OPT 2011, the 4th International Workshop on Optimization for Machine Learning, held as a part of the NIPS workshops series. This workshop builds on precedent established the previous OPT workshops:

  1. (@NIPS*08):
  2. (@NIPS*09):
  3. (@NIPS*10):

Both these workshops had packed (often overpacked) attendance almost throughout the day. This enthusiastic reception reflects the strong interest, relevance, and importance enjoyed by optimization in the greater ML community.

One could ask why does optimization attract such continued interest? The answer is simple but telling: optimization lies at the heart of almost every ML algorithm. For some algorithms textbook methods suffice, but the majority requires tailoring algorithmic tools from optimization; moreover, this tailoring depends on a deeper understanding of the ML requirements. In fact, ML applications and researchers are driving some of the most cutting-edge developments in optimization today. The intimate relation of optimization with ML is the key motivation for our workshop, which aims to foster discussion, discovery, and dissemination of the state-of-the-art in optimization, especially in the context of ML.

Background & Objectives

Optimization is indispensable to many machine learning algorithms. What can we say beyond this obvious realization?

Previous talks at the OPT workshops have covered frameworks for convex programs (D. Bertsekas), the intersection of ML and optimization, especially in the area of SVM training (S. Wright), large-scale learning via stochastic gradient methods and its tradeoffs (L. Bottou, N. Srebro), exploitation of structured sparsity in optimization (Vandenberghe), and randomized methods for extremely large-scale convex optimization (A. Nemirovski), among others.

The ML community's interest in optimization continues to grow. Invited tutorials on optimization will be presented this year at ICML (N. Srebro) and NIPS (S. Wright). The traditional "point of contact" between ML and optimization - SVM - continues to be a driver of research on a number of fronts. Much interest has focused recently on stochastic gradient methods, which can be used in an online setting and in settings where data sets are extremely large and high accuracy is not required. Regularized logistic regression is another area that has produced a recent flurry of activity at the intersection of the two communities. Many aspects of stochastic gradient remain to be explored, for example, different algorithmic variants, customizing to the data set structure, convergence analysis, sampling techniques, software, choice of regularization and tradeoff parameters, parallelism. There also needs to be a better understanding of the limitations of these methods, and what can be done to accelerate them or to detect when to switch to alternative strategies. In the logistic regression setting, use of approximate second-order information has been shown to improve convergence, but many algorithmic issues remain. Detection of combined effects predictors (which lead to a huge increase in the number of variables), use of group regularizers, and dealing with the need to handle very large data sets in real time all present challenges.

We also do NOT ignore the not particularly large scale setting, where one has time to wield substantial computational resources. In this setting, high-accuracy solutions and deep understanding of the lessons contained in the data are needed. Examples valuable to MLers may be exploration of genetic and environmental data to identify risk factors for disease; or problems dealing with setups where the amount of observed data is not huge, but the mathematical model is complex.

Operational Details
  • one day long with morning and afternoon sessions;
  • three invited talks optimization and ML experts
  • discussion: this year we plan to bolster discussion by having an open problems session;
  • contributed talks;
  • an interactive poster session;

The planned submission/reviewing dates are planned as follows.

  • Deadline for submission of papers: 24th October 2011
  • Notification of acceptance: 12th November 2011
  • Final version of submission: 20th November 2011
  • Workshop date: 16th December 2011
  • Stochastic, Parallel and Online Optimization:
  • Large-scale learning, massive data sets
  • Distributed algorithms
  • Optimization on massively parallel architectures
  • Optimization using GPUs, Streaming algorithms
  • Decomposition for large-scale, message-passing and online learning
  • Stochastic approximation
  • Randomized algorithms
  • Non-Convex Optimization:
  • Nonsmooth, nonconvex optimization
  • Non-convex quadratic programming, including binary QPs
  • Convex Concave Decompositions, D.C. Programming, EM
  • Training of deep architectures and large hidden variable models
  • Approximation Algorithms
  • Algorithms and Techniques (application oriented):
  • Global and Lipschitz optimization
  • Algorithms for non-smooth optimization
  • Linear and higher-order relaxations
  • Polyhedral combinatorics applications to ML problems
  • Combinatorial Optimization:
  • Optimization in Graphical Models
  • Structure learning
  • MAP estimation in continuous and discrete random fields
  • Clustering and graph-partitioning
  • Semi-supervised and multiple-instance learning
  • Practical techniques
  • Optimization software and toolboxes
  • GPU, Multicore, Distributed implementations
  • Applications close to machine learning
  • Sparse learning, compressed sensing, signal processing
  • Computational Statistics
  • Large scale scientific computing

Submission Instructions

  • The submissions should be ideally 4 pages long. Hard-limit: 6 pages.
  • Open Problems may be of any length within the hard-limits
  • The review process will be double-blind
  • Please use the NIPS 2010 format for your submissions
  • Submission Website