Statistically sound backtesting of trading strategies

1. April 201419. May 2021 Christoph

ParagraphFormats 

In this series of posts I will look at some aspects of backtesting trading strategies. The posts are based on David Aronson’s book Evidence-Based Technical Analysis.

In this series of posts I will look at some aspects of backtesting trading strategies. The posts are based on David Aronson’s book Evidence-Based Technical Analysis.

Naive backtesting

A trading strategy in the sense used here refers to an algorithm that uses market data in some way to generate a trading recommendation that can either be long (1), short (-1) or neutral (0). An example for a simple trading strategy is

• If price is above 200-day moving average, trade LONG (1)
• If price is below 200-day moving average, trade SHORT (-1)

In R, this translates to the following piece of code:

if (!require(quantmod)){
install.packages(quantmod); require(quantmod)
} else{require(quantmod)}

if (!require(PerformanceAnalytics)){
install.packages(PerformanceAnalytics); require(PerformanceAnalytics)
} else{require(PerformanceAnalytics)}

if (!require(TTR)){
install.packages(TTR); require(TTR)
} else{require(TTR)}

#Load data
dax.d<-get.hist.quote(“^GDAXI”,start=”1990-01-01″, quote=”Close”,retclass=”zoo”,compression=”d”)
current.dax<-getQuote(“^GDAXI”)$Last
#Add current data
if(dax.d[nrow(dax.d),]==getQuote(“^GDAXI”)$Last){
paste(“Values are up to date”)
} else{
dax.d<-rbind(dax.d,zoo(print(current.dax,style=”vertical”)))
}

#Set up data frame
ROC(dax.d,na.pad=T)-&gt;dax.d.roc # This is the percentage change per day
df<-data.frame(
dax.d, #Acutal market data
SMA(dax.d,200), #Moving average for 200 days
dax.d.roc #Percentage Change per day
)
names(df)<-c(“Close”,”SMA.200″,”ROC”) #Simplify names

df$ind<-ifelse(df$Close&gt;df$SMA.200,1,-1) #Is current price above Moving average?
df$ind.1<-c(NA,df$ind[-length(df$ind)]) #The actual indicator is for the next day

df$perf<-df$ROC*df$ind.1 #This is the performance using the indicator

mean(df$perf,na.rm=T)*100 #Daily average return market
mean(df$ROC,na.rm=T)*100 #Daily average return strategy

Concerns about naive backtesting

The strategy seems to work, since it outperforms the market. However, a number of concerns about this kind to naïve backtesting can be raised:

1. If the market data has a certain bias such as 70% of days has a positive return, the market data should be centered on 0. Otherwise, a random but positively biased strategy could still yield a positive return.
2. A mere outperformance of the market data is not enough to indicate that a strategy actually offers value. The question that needs to be answered is: How likely it is that a random strategy has created this kind of return? The Monte-Carlo Permutation test offers a good way to answer this question.
3. The Monte-Carlo Permutation tests needs to be adapted, if the number of strategies tests is increased (Data-Mining Bias). Imagine that we are just looking at random strategies. The more strategies we look at, the higher the probability that we find a strategy that apparently offers a high expected return.
4. Another bias that is difficult to take into account is the Data-Snooping bias. By having looked at previous studies, it is hard for me not to incorporate these ideas into my trading strategies. These studies usually do not state the number of strategies they have looked at to get to the final result.
5. By choosing the strategy with the highest average return we have a good chance of choosing the best strategy with true predictive power. However, since we choose the strategy with the highest average performance, it is likely that a high percentage of the strategies’ return will be due to random factors. This means that the average return of the strategy systematically overestimates future returns.

Here is a code for the Monte-Carlo Permutation test with returns that are centered on zero:

With 5000 repetitions the calculation takes a while.

require(tseries)
 require(scales)
 require(quantmod)
 require(TTR)
 require(ggplot2)
 require(PerformanceAnalytics)
 t1<-365*5
 t2<-0
 SysD<-Sys.Date()
require(tseries)
require(scales)
require(quantmod)
require(TTR)
require(ggplot2)
require(PerformanceAnalytics)
t1<-365*5
t2<-0
SysD<-Sys.Date()

s1<-get.hist.quote(“^GDAXI”,start=SysD-t1,end=SysD, quote=”Close”,retclass=”zoo”,compression=”d”)
dax<-Delt(s1)

#Center return on 0
s0<-Delt(s1)[-1]-mean(Delt(s1)[-1])


#Monte Carlo Permutation Test
length(s0)
r<-c()
R<-c()

for(j in 1:5000){
for(i in 1:length(s0)){
r[i]<-ifelse(rnorm(1)>0,1,-1)*sample(s0,1)
}
R[j]<-mean(r)
}

The last command gives a value of 0.0576155 which means that this value should be exceeded if we want our trading strategy to be meaningful at 10%.

So this gives a framework that can be used to evaluate trading strategies. The data-mining bias and data snooping bias are not yet covered, but this test already gives a good indication on whether a trading strategy has performed well due to chance or whether it offers some predictive value.