Portfolio optimization is the process of selecting an optimal portfolio (asset distribution), out of a set of considered portfolios, according to some objective. The objective typically maximizes factors such as expected return, and minimizes costs like financial risk, resulting in a multi-objective optimization problem. Factors being considered may range from tangible (such as assets, liabilities, earnings or other fundamentals) to intangible (such as selective divestment).

Modern portfolio theory was introduced in a 1952 doctoral thesis by Harry Markowitz, where the Markowitz model was first defined. The model assumes that an investor aims to maximize a portfolio's expected return contingent on a prescribed amount of risk. Portfolios that meet this criterion, i.e., maximize the expected return given a prescribed amount of risk, are known as efficient portfolios. By definition, any other portfolio yielding a higher amount of expected return must also have excessive risk. This results in a trade-off between the desired expected return and allowable risk. This risk-expected return relationship of efficient portfolios is graphically represented by a curve known as the efficient frontier. All efficient portfolios, each represented by a point on the efficient frontier, are well-diversified. While ignoring higher moments of the return can lead to significant over-investment in risky securities, especially when volatility is high, the optimization of portfolios when return distributions are non-Gaussian is mathematically challenging. Hierarchical Risk Parity is a sophisticated approach to portfolio optimization introduced in 2016 as an alternative to the traditional mean-variance optimization model developed by Harry Markowitz.

The portfolio optimization problem is specified as a constrained utility-maximization problem. Common formulations of portfolio utility functions define it as the expected portfolio return (net of transaction and financing costs) minus a cost of risk. The latter component, the cost of risk, is defined as the portfolio risk multiplied by a risk aversion parameter (or unit price of risk). For return distributions that are Gaussian, this is equivalent to maximizing a certain quantile of the return, where the corresponding probability is dictated by the risk aversion parameter. Practitioners often add additional constraints to improve diversification and further limit risk. Examples of such constraints are asset, sector, and region portfolio weight limits.

Portfolio optimization often takes place in two stages: optimizing weights of asset classes to hold, and optimizing weights of assets within the same asset class. An example of the former would be choosing the proportions placed in equities versus bonds, while an example of the latter would be choosing the proportions of the stock sub-portfolio placed in stocks X, Y, and Z. Equities and bonds have fundamentally different financial characteristics and have different systematic risk and hence can be viewed as separate asset classes; holding some of the portfolio in each class provides some diversification, and holding various specific assets within each class affords further diversification. By using such a two-step procedure one eliminates non-systematic risks both on the individual asset and the asset class level. For the specific formulas for efficient portfolios, see Portfolio separation in mean-variance analysis.

One approach to portfolio optimization is to specify a von Neumann–Morgenstern utility function defined over final portfolio wealth; the expected value of utility is to be maximized. To reflect a preference for higher rather than lower returns, this objective function is increasing in wealth, and to reflect risk aversion it is concave. For realistic utility functions in the presence of many assets that can be held, this approach, while theoretically the most defensible, can be computationally intensive.

Harry Markowitz developed the "critical line method", a general procedure for quadratic programming that can handle additional linear constraints and upper and lower bounds on holdings. Moreover, in this context, the approach provides a method for determining the entire set of efficient portfolios. Its application here was later explicated by William Sharpe.

The complexity and scale of optimizing portfolios over many assets means that the work is generally done by computer. Central to this optimization is the construction of the covariance matrix for the rates of return on the assets in the portfolio.

Techniques include: