[PDF] 1 Convex Optimization with Sparsity-Inducing Norms

View - Download 1 Convex Optimization with Sparsity-Inducing Norms

Previous PDF

Next PDF

1 Convex Optimization with

Sparsity-Inducing Norms

Francis Bachfrancis.bach@inria.fr

INRIA - Willow Project-Team

23, avenue d"Italie, 75013 PARIS

Rodolphe Jenattonrodolphe.jenatton@inria.fr

INRIA - Willow Project-Team

23, avenue d"Italie, 75013 PARIS

Julien Mairaljulien.mairal@inria.fr

INRIA - Willow Project-Team

23, avenue d"Italie, 75013 PARIS

Guillaume Obozinskiguillaume.obozinski@inria.fr

INRIA - Willow Project-Team

23, avenue d"Italie, 75013 PARIS

1.1 Introduction

The principle of parsimony is central to many areas of science: the simplest explanation to a given phenomenon should be preferred over more compli- cated ones. In the context of machine learning, it takes the form of variable or feature selection, and it is commonly used in two situations. First, to make the model or the prediction more interpretable or computationally cheaper to use, i.e., even if the underlying problem is not sparse, one looks for the best sparse approximation. Second, sparsity can also be used given prior knowledge that the model should be sparse.

2 Convex Optimization with Sparsity-Inducing Norms

For variable selection in linear models, parsimony may be directly achieved by penalization of the empirical risk or the log-likelihoodby the cardinality of the support of the weight vector. However, this leads to hardcombinatorial problems (see, e.g., Tropp, 2004). A traditional convex approximation of the problem is to replace the cardinality of the support by the1-norm. Estimators may then be obtained as solutions of convex programs. Casting sparse estimation as convex optimization problemshas two main benefits: First, it leads to efficient estimation algorithms-and this chapter focuses primarily on these. Second, it allows a fruitful theoretical analysis answering fundamental questions related to estimation consistency, predic- tion efficiency (Bickel et al., 2009; Negahban et al., 2009) ormodel con- sistency (Zhao and Yu, 2006; Wainwright, 2009). In particular, when the sparse model is assumed to be well-specified, regularization by the1-norm is adapted to high-dimensional problems, where the number of variables to learn from may be exponential in the number of observations. Reducing parsimony to finding the model of lowest cardinality turns out to be limiting, andstructured parsimonyhas emerged as a natural extension, with applications to computer vision (Jenattonet al., 2010b), text processing (Jenatton et al., 2010a) or bioinformatics (Kimand Xing, 2010; Jacob et al., 2009). Structured sparsity may be achieved by regularizing by other norms than the1-norm. In this chapter, we focus primarily on norms which can be written as linear combinations of norms on subsets of variables (Section 1.1.1). One main objective of this chapter is to present methods which are adapted to most sparsity-inducing norms with lossfunctions potentially beyond least-squares. Finally, similar tools are used in other communities such assignal process- ing. While the objectives and the problem set-up are different, the resulting convex optimization problems are often very similar, and most of the tech- niques reviewed in this chapter also apply to sparse estimation problems in signal processing. This chapter is organized as follows: in Section 1.1.1, we present the optimization problems related to sparse methods, while in Section 1.1.2, we review various optimization tools that will be needed throughout the chapter. We then quickly present in Section 1.2 generic techniques that are not best suited to sparse methods. In subsequent sections, we present methods which are well adapted to regularized problems, namely proximal methods in Section 1.3, block coordinate descent in Section1.4, reweighted

2-methods in Section 1.5, and working set methods in Section 1.6. We

provide quantitative evaluations of all of these methods inSection 1.7.

1.1 Introduction3

1.1.1 Loss functions and Sparsity-Inducing Norms

We consider in this chapter convex optimization problems ofthe form min wRp() +Ω()(1.1) where:RRis a convex differentiable function and Ω:RRis a sparsity-inducing-typically nonsmooth and non-Euclidean-norm. In supervised learning, we predict outputsinYfrom observationsinX; these observations are usually represented by-dimensional vectors, so that X=R. In this supervised setting,generally corresponds to the empirical risk of a loss function:YRR+. More precisely, givenpairs of data points(()())RY;= 1, we have for linear models ():=1 =1(()()). Typical examples of loss functions are the square loss for least squares regression, i.e.,(ˆ) =1

2(ˆ)2withinR,

and the logistic loss(ˆ) = log(1 +ˆ) for logistic regression, within

11. We refer the readers to Shawe-Taylor and Cristianini (2004) for a

more complete description of loss functions. When one knowsa priorithat the solutionsof problem (1.1) only have a few non-zero coefficients, Ω is often chosen to be the1-norm, i.e., Ω() =?=1. This leads for instance to the Lasso (Tibshirani, 1996) with the square loss and to the1-regularized logistic regression (see, for instance, Shevade and Keerthi, 2003; Koh et al., 2007) with the logistic loss. Regularizing by the1-norm is known to induce sparsity in the sense that, a number of coefficients of, depending on the strength of the regularization, will beexactlyequal to zero. In some situations, for example when encoding categorical variables by binary dummy variables, the coefficients ofare naturally partitioned in subsets, orgroups, of variables. It is then natural to select or remove simultaneouslyall the variables forming a group. A regularization norm exploiting explicitly this group structure can be shown to improve the prediction performance and/or interpretability of the learned models (Yuan and Lin, 2006; Roth and Fischer, 2008; Huang and Zhang, 2009;Obozinski et al., 2009; Lounici et al., 2009). Such a norm might for instance take the form G

2(1.2)

whereGis a partition of1, ()Gare some positive weights, and denotes the vector inRrecording the coefficients ofindexed byinG. Without loss of generality we may assume all weights ()Gto be equal to

4 Convex Optimization with Sparsity-Inducing Norms

one. As defined in Eq. (1.2), Ω is known as a mixed12-norm. It behaves like an1-norm on the vector (2)GinRG, and therefore, Ω induces group sparsity. In other words, each2, and equivalently each, is encouraged to be set to zero. On the other hand, within the groupsinG, the2-norm does not promote sparsity. Combined with the square loss, it leads to the group Lasso formulation (Yuan and Lin, 2006). Note that whenG is the set of singletons, we retrieve the1-norm. More general mixed1- norms for 1 are also used in the literature (Zhao et al., 2009): G:=? G? 1 In practice though, the12- and1-settings remain the most popular ones. In an attempt to better encode structural links between variables at play (e.g., spatial or hierarchical links related to the physicsof the problem at hand), recent research has explored the setting whereGcan contain groups of variables thatoverlap(Zhao et al., 2009; Bach, 2008a; Jenatton et al., 2009; Jacob et al., 2009; Kim and Xing, 2010; Schmidt and Murphy, 2010). In this case, Ω is still a norm, and it yields sparsity in the form of specific patterns of variables. More precisely, the solutionsof problem (1.1) can be shown to have a set of zero coefficients, or simplyzero pattern, that corresponds to a union of some groupsinG(Jenatton et al., 2009). This property makes it possible to control the sparsity patterns ofby appropriately defining the groups inG. This form ofstructured sparsityhas notably proven to be useful in the context of hierarchical variable selection (Zhao et al.,

2009; Bach, 2008a; Schmidt and Murphy, 2010), multi-task regression of

gene expressions (Kim and Xing, 2010) and also for the designof localized features in face recognition (Jenatton et al., 2010b).

1.1.2 Optimization Tools

The tools used in this book chapter are relatively basic and should be acces- sible to a broad audience. Most of them can be found in classical books on convex optimization (Boyd and Vandenberghe, 2004; Bertsekas, 1999; Bor- wein and Lewis, 2006; Nocedal and Wright, 2006), but for self-containedness, we present here a few of them related to non-smooth unconstrained opti- mization.

Subgradients

Given a convex function:RRand a vectorinR, let us define the

1.1 Introduction5

subdifferentialofatas ():=R()+()() for all vectorsR The elements of() are called thesubgradientsofat. This definition admits a clear geometric interpretation: any subgradientin() defines an affine function() +() which is tangent to the graph of the function. Moreover, there is a bijection (one-to-one correspondence) between such "tangent affine functions" and the subgradients. Let us now illustrate how subdifferential can be useful for studying nonsmooth opti- mization problems with the following proposition:

Proposition 1.1(Subgradients at Optimality).

For any convex function:RR, a pointinRis a global minimum ofif and only if the condition0()holds. Note that the concept of subdifferential is mainly useful fornonsmooth functions. Ifis differentiable at, the set() is indeed the singleton (), and the condition 0() reduces to the classical first-order optimality condition() = 0. As a simple example, let us consider the following optimization problem min R1 2()2+ Applying the previous proposition and noting that the subdifferential is+1for 0,1for 0 and [11] for= 0, it is easy to show that the unique solution admits a closed form called thesoft-thresholding operator, following a terminology introduced by Donoho andJohnstone (1995); it can be written =?0 if (1 )otherwise(1.3) This operator is a core component of many optimization techniques for sparse methods, as we shall see later.

Dual Norm and Optimality Conditions

The next concept we introduce is the dual norm, which is important to study sparsity-inducing regularizations (Jenatton et al., 2009; Bach, 2008a; Negahban et al., 2009). It notably arises in the analysis of estimation bounds (Negahban et al., 2009), and in the design of working-set strategies as will be shown in Section 1.6. The dual norm Ω of the norm Ω is defined for any

6 Convex Optimization with Sparsity-Inducing Norms

vectorinRby ():= maxwRpsuch that Ω()1

Moreover, the dual norm of Ω

is Ω itself, and as a consequence, the formula above holds also if the roles of Ω and Ω are exchanged. It is easy to show that in the case of an-norm,[1;+], the dual norm is the-norm, within [1;+] such that1 +1= 1. In particular, the1- and-norms are dual to each other, and the2-norm is self-dual (dual to itself). The dual norm plays a direct role in computing optimality conditions of sparse regularized problems. By applying Proposition 1.1 to Eq. (1.1), a little calculation shows that a vectorinRis optimal for Eq. (1.1) if and only if1 ()Ω() with

R; Ω()1if= 0

R; Ω()1 and= Ω()otherwise(1.4)

As a consequence, the vector 0 is solution if and only if Ω ?(0)?. These general optimality conditions can be specified to the Lasso prob- lem (Tibshirani, 1996), also known as basis pursuit (Chen etal., 1999): min wRp1

222+1(1.5)

whereis inR, andis a design matrix inR. From Equation (1.4) and since the-norm is the dual of the1-norm we obtain that necessary and sufficient optimality conditions are = 1? () if= 0 () =sgn() if= 0(1.6) wheredenotes the-th column of, andthe-th entry of. As we will see in Section 1.6.1, it is possible to derive from theseconditions inter- esting properties of the Lasso, as well as efficient algorithms for solving it. We have presented a useful duality tool for norms. More generally, there exists a related concept for convex functions, which we now introduce.

Fenchel Conjugate and Duality Gaps

Let us denote bythe Fenchel conjugate of(Rockafellar, 1997), defined by ():= sup wRp[()] The Fenchel conjugate is related to the dual norm. Let us define the indicator functionΩsuch thatΩ() is equal to 0 if Ω()1 and +otherwise.

1.1 Introduction7

Then,Ωis a convex function and its conjugate is exactly the dual norm Ω. For many objective functions, the Fenchel conjugate admitsclosed forms,quotesdbs_dbs4.pdfusesText_8

[PDF] bragard chef jacket dubai

[PDF] bragard chef jacket singapore

[PDF] bragard chef jacket size chart

[PDF] bragard chef jacket sizes

[PDF] bragard chef jackets canada

[PDF] bragard chef jackets uk

[PDF] bragard chef jackets usa

[PDF] bragard outlet

[PDF] branches of sociology and their definition pdf

[PDF] branches of sociology in nursing

[PDF] branches of sociology in pakistan

[PDF] branches of sociology of education

[PDF] branches of sociology wikipedia

[PDF] brassage interchromosomique drosophile

[PDF] brassage interchromosomique en anglais