L2, R41: Backtesting and Simulation

What are the two types of simulation analysis?

1) historical simulation - constructs results by selecting returns at random from many different historical periods (windows) without regard to time-ordering - A simulation method that uses past return data and a random number generator that picks observations from the historical series to simulate an asset future returns 2) monte carlo simulation - each key variable is assigned a statistical distribution and observations are drawn at random from the assigned distribution

Describe Step 1 of the backtesting process: Strategy design

1) identify the investment goals - for example to achieve excess returns over the relevant benchmark 2) identify the investment hypothesis - a method aimed at achieving the goal 3) translate the hypothesis into rules and processes - specify key parameters so that the hypothesis an be backtested: The key parameters include: - investment universe: refers to all the securities in which we can potentially invest - return definition: choosing paramerters such as currency we calculate our returns the benchmark related to our investment universe, etc - rebalancing frequency and transaction costs - start and end date / backtesting period

Regarding rolling-window backtesting, which one of the following statements is inaccurate? A The data are divided into just two samples. B Out-of-sample data become part of the next period's in-sample data. C Repeated in-sample training and out-of-sample testing allow managers to adjust security positions on the basis of the arrival over time of new information.

A is correct, because the statement is inaccurate. B and C are incorrect, because they accurately describe the rolling-window backtesting technique.

Which of the following situations is least likely to involve scenario analysis? A Simulating the performance and risk of investment strategies by first using stocks in the Nikkei 225 Index and then using stocks in the TOPIX 1000 Index. B Simulating the performance and risk of investment strategies in both "trade agreement" and "no-trade-agreement" environments. C Simulating the performance and risk of investment strategies in both high-volatility and low-volatility environments.

A is correct, because there is no structural break or different structural regime. B and C are incorrect because they involve structural breaks/different structural regimes and thus represent different scenarios.

Which of the following situations concerning simulation of a multifactor asset allocation strategy is most likely to involve sensitivity analysis? A Changing the specified multivariate distribution assumption from a normal to a skewed t-distribution to better account for skewness and fat tails B Splitting the rolling window between periods of recession and non-recession C Splitting the rolling window between periods of high volatility and low volatility

A is correct, because this choice represents sensitivity analysis. B and C are incorrect because these choices represent scenario analysis.

Point-in-time data are useful for avoiding the following problems that may affect backtesting except: A data snooping. B survivorship bias. C look-ahead bias.

A is correct. An analyst can still use a point-in-time dataset to make an inference based on statistical results rather than testing a prior inference. B and C are incorrect, because point-in-time data are useful for avoiding look-ahead bias and survivorship bias (a special case of look-ahead bias). Point-in-time data explicitly corrects for what is not known at a given point in time.

Risk parity

A portfolio allocation scheme that weights stocks or factors based on an equal risk contribution. The key takeaway is that asset classes with lower risk will have higher weights when the risk parity approach is applied.

What is a structural break?

A structural break otherwise known as a regime change are depression, recessions, geopolitical events, major shifts in monetary and fiscal policies, technological changes etc.

Which of the following is incorrect? Monte Carlo simulation is popular because: A. it is very flexible B. it is not complex C. it does not rely on past data in forecasting future data

B is correct - Monte carlo simulation is complex and computationally intensive

Which one of the following is not a metric or visual used in assessing backtesting of a factor-based investment strategy? A Distribution plots of factor returns B A word cloud of text describing the characteristics of the factor C Maximum drawdown

B is correct, because a word cloud is not a visual used in assessing backtesting of a factor-based investment strategy. A and C are incorrect, because they are visuals and metrics, respectively, used to assess backtests of factor-based strategies.

Which of the following situations is most likely to involve data snooping? A A researcher performs rolling-window backtesting of a new momentum strategy using 20 years of point-in-time (PIT) data from the United States. She cross-validates results by similarly analyzing PIT data from the following markets: mainland China, Asia ex-Japan, Europe, the United Kingdom, and Canada. B A researcher tries many different modeling techniques, backtesting each of them, and then picking the best-performing model without accounting for model selection bias. C A researcher sets a relatively high hurdle, a t-statistic greater than 3.0, for assessing whether a newly discovered factor is statistically significant.

B is correct, because this situation most likely involves data snooping. A and C are incorrect because these are approaches to avoiding data snooping.

Which one of the following statements concerning historical simulation and Monte Carlo simulation is false? A Historical simulation randomly samples (with replacement) from the past record of asset returns, where each set of past monthly returns is equally likely to be selected. B Neither historical simulation nor Monte Carlo simulation makes use of a random number generator. C Monte Carlo simulation randomly samples from an assumed multivariate joint probability distribution in which the past record of asset returns is used to calibrate the parameters of the multivariate distribution.

B is correct, because this statement is false. A and C are incorrect because they are true statements about historical and Monte Carlo simulation, respectively.

Which of the following is an example of cross-validation? A Maximum drawdown B Backtesting with out-of-sample data C Incorporating point-in-time data

B is correct. Cross-validation is a technique that involves testing a hypothesis on a different set of data than that which was used to form the inference or initially test the hypothesis. Choice B is the definition of cross-validation.

Which of the following backtesting problems can not be mitigated using point-in-time data? A. Survivorship Bias B. Look-Ahead Bias C. Data Snooping

C - data snooping is correct

Which of the following is a drawback of the long-short hedged portfolio approach for implementing factor-based portfolios? A The hedged portfolio is formed by going long the top quantile (with the best factor scores) and shorting the bottom quantile (with the worst factor scores). B Securities must be ranked by the factor being scrutinized and then grouped into quantiles based on their factor scores. C Not every manager can short stocks.

C is correct, because it best describes a drawback of the long-short hedged portfolio approach. A and B are incorrect because they describe the approach itself.

The following are caveats regarding the use of rolling-window backtesting in assessing investment strategies except: A this technique implicitly assumes that the same pattern of past performance is likely to repeat itself over time. B this technique may not fully account for the dynamic nature of financial markets and potentially extreme downside risks. C this technique is intuitive, because it mimics how investing is done in reality—that is, forming ideas, testing strategies, and implementing periodically.

C is correct, because it is not a caveat in using rolling-window backtesting. A and B are incorrect because they are caveats in the use of this technique.

Regarding the use of rolling-window backtesting in assessing factor allocation to a risk parity-based strategy, which statement is correct? A The procedure is used once for estimating factor returns over the rolling window. B The procedure is used once for dividing the data into just two samples. C The procedure is used twice—once for estimating factor returns over the rolling window, and a second time for estimating the covariance matrix of factor returns (for deriving risk parity weights) over the rolling window.

C is correct, because the procedure must be used a second time for estimating the covariance matrix of factor returns (for deriving risk parity weights) over the rolling window. A is incorrect because the procedure must be done twice: once for estimating factor returns over the rolling window and a second time for estimating the covariance matrix of factor returns (for deriving risk parity weights). B is incorrect because the rolling-window procedure divides the sample into many samples.

Which one of the following statements concerning Monte Carlo simulation is false? A When simulating multiple assets (factors) whose returns are correlated, it is crucial to specify a multivariate distribution rather than modeling each asset on a standalone basis. B Regression and distribution-fitting techniques are used to estimate the parameters underlying the statistical distributions of the key decision variables. C The Monte Carlo simulation process is deterministic and non-random in nature.

C is correct, because this statement is false. A and B are incorrect because they are true statements about Monte Carlo simulation.

Which of the following is not a potential concern of using a short time period for a backtest? A The backtest will cover a limited number of business cycle, inflation, and interest rate regimes. B The backtest may not be useful because the findings may apply only under the conditions present in the time frame. C The backtest is likely to cover multiple business cycle, inflation, and interest rate regimes.

C is correct. Covering multiple macroeconomic regimes is not a concern associated with using a short time period for a backtest, because macroeconomic regimes tend to be multi-year in length. A and B are incorrect because they are concerns associated with using a short time period: The backtest may capture only a limited experience, and thus the findings may be relevant for only that experience.

An analyst develops an investment strategy by picking the strategy with the highest t-statistic and lowest p-value after backtesting dozens of different strategies. This approach is an example of which common problem in backtesting? A Reporting lag B Survivorship bias C Data snooping

C is correct. Data snooping refers to making an inference—such as formulating an investment strategy—after looking at statistical results rather than testing a prior inference. A is incorrect because reporting lag refers to the fact that data describing a period is often available only after the period ends and is often subject to revision. B is incorrect because survivorship bias is a form of look-ahead bias in which results are based on a limited, biased sample of subjects (e.g., only surviving companies).

The fact that GDP figures for a quarter are not released by government statistical agencies until approximately 30 days after the quarter ends and often undergo several revisions thereafter creates a problem known as: A data snooping. B survivorship bias. C reporting lag.

C is correct. Reporting lag refers to the fact that data describing a period is often available only after the period ends and is often subject to revision, which certainly is true of GDP data.

An analyst performed a backtest on an investment strategy in June 2019, selecting the constituents of the Russell 3000 Index as the investment universe, and December 1985 and May 2019 as the start and end dates, respectively. While discussing the results with some colleagues, the analyst was shown lists of the Russell 3000 Index constituents as of December 2005 and December 1995. She noticed that the lists included only 2,250 and 1,500 companies, respectively, of the Russell 3000 companies at May 2019. The analyst must correct her backtest for which problem? A Data snooping B Reporting lag C Look-ahead bias

C is correct. The dataset the analyst uses assumes that the Russell 3000 Index constituents as of May 2019 are the same companies that constituted the index throughout the entire backtesting period. The backtest suffers from look-ahead bias, so conclusions drawn from it will be erroneous because it includes companies that did not exist (or were not index members) over the period starting in December 1985. To correct this problem, the analyst should use a dataset of point-in-time constituents of the Russell 3000 Index.

Historical Scenario Analysis / Historical Stress Testing

Examines the efficacy of a strategy in discrete historical environments such as during recessions or periods of high inflation

How does simulation differ from scenario analysis?

Explores how a strategy would perform in a hypothetical environment specified by the user rather than a historical setting

T/F: Unlike maximum drawdown, VaR is not a downside risk measure

False - they are both downside risk measure

T/F: like backtesting in historical simulation historical period returns are selected with time ordering

False - they are selected WITHOUT time ordering

T/F: Sensitivity analysis determines the efficacy of a strategy in discrete historical environments such as during recessions or periods of high inflation

False - this is true about historical scenario analysis

T/F: survivorship bias is not a common problem in backtesting?

False! Three common problems in backtesting include survivorship bias, lookahead bias and data snooping

Describe step 2 of the backtesting process: historical investment simulation

In this step we construct the portfolio to be tested and ensure that it is rebalanced based on the predetermined frequency

Sensitivity Analysis

Often combine with simulation to uncover the impact of changing key assumptions

If performing a monte carlo analysis that found that the retrurn distributions of value and momentum factors exhibit non-normal based distributions with negative skewness, excess kurtosis and tail dependence, how could you mitigate the situation?

Performing sensitivity analysis because the user could test different probability distributions that relax the assumptions of the normal distribution

What are the three steps of backtesting?

Strategy design Historical investment simulation analysis of backtesting output

Investment universe

The set of assets that may be considered for investment.

Describe step 3 of the backtesting process: analysis of backtesting output

This final step in backtesting is used to generate results for presentation and interpretation. Analysts often use metrics such as the Sharpe ratio, Sortino ratio, volatility and maximum drawdown.

T/F: Asset (and factor) returns are often negatively skewed and exhibit excess kurtosis (fat tails) and tail dependence compared with a normal distribution. As a result, standard rolling-window backtesting may be unable to fully account for the randomness in asset returns, particularly on downside risk.

True

T/F: Data snooping bias is overanalyzing the data in an attempt to find the desired results

True

T/F: For an investment strategy, backtesting is typically employed as acceptance or rejection criteria with the implied assumption that the future will resemble history to some extent

True

T/F: Historical simulation uses past return data but randomly changes the sequencing of historical periods

True

T/F: In Monte Carlo simulation, the most important decision is the choice of functional form of the statistical distribution of decision variables/return drivers. Multivariate normal distribution is often used in investment research, owing to its simplicity. However, a multivariate normal distribution cannot account for negative skewness and fat tails observed in factor and asset returns.

True

T/F: In rolling window structure, the strategy uses data from a historical in-sample period. Then the data is tested on out-of-sample period. The process is repeated as time moves forward.

True

T/F: Look ahead bias is the most common bias made by practitioners

True

T/F: One limitation of the monte carlo simulation is that it assumes a multivariate normal distribution as a starting point

True

T/F: Point in time data takes into account the 'low volatility anomaly' which states that low volatility stocks tend to outperform high volatility stocks

True

T/F: Sensitivity analysis using a multivariate skewed t-distribution considers skewness and kurtosis but requires estimation of more parameters and thus is more likely to suffer from larger estimation errors?

True

T/F: The main objective of backtesting is to understand the risk-return trade-off of an investment strategy by approximating the real-life investment process?

True

T/F: The more frequent the rebalancing the greater the transaction costs

True

T/F: There is no guarantee that strategies that perform well in a backtest will produce excess returns in the future

True

T/F: backtesting describes going back in time to see how well a model would have done in the past, while simulation means going forward in time to see how well a model might do in the future

True

T/F: data snooping is also known as p hacking?

True

T/F: data snooping occurs when intentionally or unintentionally data is manipulated in a way to produce statistically significant results, and often the ultimate results of data snooping are false positives

True

T/F: historical simulation is also considered 'non-deterministic rolling-window backtest'

True

T/F: most quantitative selectin models employ a multifactor structure

True

T/F: random sampling with replacement, also known as bootstrapping is often used in historical simulations because the number of simulations needed is often larger than the size of the historical data set

True

T/F: rolling windows approach to backtesting replicates 'live investing process' as managers adjust/revise their positions for new information.

True

T/F: simulation analysis explores how a strategy would perform in a hypothetical environment specified by the user rather than a historical setting.

True

T/F: the rolling method is implemented twice when backesting a multifactor strategy

True - once at the factor level and once at the factor portfolio level

T/F: sensitivity analysis explains the limitations of monte carlo simulation

True - sensitivity analysis is often combined with simulation such as the monte carlo method to uncover the impact of changing key assumptions

T/F: historical scenario analysis is a type of backtesting that helps to assess the effects of regime changes and structural breaks on the performance and risk of the investment strategy?

True - two common types of regime changes are expansions and recessions and high-and-low volatility regimes

T/F: monte carlo simulation requires a specification of the functional form for each key decision variable

True - usefulness of monte carlo simulation depends on how well the functional forms reflect the true distribution of the underlying data

T/F: sensitivity analysis explores the impact of revising key assumptions

True it will evaluate whether these assumptions have a small, big, or no impact on the model - examples of assumptions include price of oil increasing, fed tigtening etc.

T/F: Investment managers prefer to use long back testing periods

True so long as they are RELEVANT periods

T/F: Monte Carlo and Historical simulation are non-deterministic

True they are random in nature

T/F: Financial data often face structural breaks - scenario analysis can help investors understand the performance of an investment strategy in different structural regimes?

True!

T/F: reporting lags, revisions and index additions are all components of look ahead bias

True!

T/F: Data snooping can be mitigated using cross-validation?

True! Cross-validation is when data is partitioned into training data and validation/testing data. THe model built from training data is tested for validation

T/F: one downside of historical simulation is that it assumes that the past only happened in one way and that data is stationary

True- stationary data is NOT true for financial variables

Cross-validation

Verifying the results obtained from a validation study by administering a test on a different sample Can help mitigate data snooping

Return definition

When designing a backtesting strategy this refers to specifying the benchmark and determining which currency the returns will be computed

What are the two common multifactor equity portfolio strategies?

a) benchmark factor portfolio - each stock or factor is given equal risk b) risk parity factor portfolio - constructed by weighing factors by their risk contribution, each stock or factor is weighted based on risk / volatility, and requires a complete variance-covariance matrix at each rebalancing date

Monte carlo simulation

each key variable is assigned an assumed statistical distribution and observations are drawn from these assigned distributions

rolling windows approach

is a backtesting method that uses rolling window structure to rebalance portfolio after each period and then tracks the performance overtime

Look-ahead bias

occurs when a study tests a relationship using sample data that was not available on the test date. this form of bias is created by using informatino that was unknown or unavailable during the historical periods over which the backtest is conducted Can be overcome by using point in time data!! ex - reporting lags, revisions and index additions

Data snooping

otherwise known as data dredging and p-hacking, is making an inference after looking at statistical results rather than testing a prior inference

Survivorship Bias

refers to deriving conclusions from data that reflects only those entities that have survived to that date . To avoid this problem use point-in-time data that allows analysts to use the complete data for any given prior time period

bootstrap sampling

take small samples from the larger sample and take the average of these small samples

T/F: because the monte carlo simulation uses assumed statistical distributions, it allows analysts to incorporate non-normality, fat-tails and tail dependence

true

Backtesting

use of historical data to test a strategy that was developed subsequent to the observation of the data

point-in-time data

when analysts use all-inclusive data for any given prior time period to backtest the most realistic investment strategies

stale data

when the value of an item is dependent on other data and that item is not updated when the other data is changed - for example a reporting lag can cause stale data