Not all Backtester Engine are the same

Not All Backtester Engine are the same.

 

How a backtester “organise” it’s historical data will dramatically impact the reliability and validity of the software.

 

In every scientific experiment, we need to evaluate the validity of the test before we assess the reliability of the results.

 

It is invalid to use a weighing machine to measure your height and hence the “results” are irrelevant, invalid for comparison.

 

In addition, if each time you step on the weighing machine (within the span of 2 min interval) and assuming every other variable remain the same, but you keep getting different results, then this weighing machine is a valid instrument but a highly unreliable one.

 

Likewise, if a forex software doesn’t use the historical data “correctly” to simulate as close as possible how a real market behaves (as in your live account), then no matter how fanciful features a forex software has, it is INVALID.

 

We put so much faith in all those statistically testing, yet often we don’t examine or evaluate in the 1st place how does the software backtester engine work.  We don’t evaluate whether the use of “previous bar values” in multi time frame strategy is being employed correctly.

 

I understand the SQ is in the process of adding Multi time frame features to SQ4 (and I’m waiting to trial out to see whether it is any better than other software) and therefore I want to write in to raise the awareness of the “common problem” I faced with other forex software that employ faulty methodology in their multi time frame logic, so as to hope that SQ would avoid these pit fall.

 

Common use of “Previous Bar Value (PBV) of Higher time frame” in multi time frame feature that is fundamentally wrong in my opinion.

 

Here is an example.

 

I use a 1 min time chart and add in M30, H1, H4 indicators/rules to my strategy.

 

When we use the close bar of the PBV of the higher time frame as part of my EA’s calculation, true/false and whether to execute the trade or not, we faced this problem…

 

at 11:57 am the EA will have to gather various bar closing values from different/multi time frame to determine whether my EA rules are true or false.

 

For 1min rules, it will take 1min’s bar close price value at 11:57

 

For M30 rules, it will take M30’s previous bar close at 11:30 (which has a time gap/difference of 27min from my current bar close)

 

For H1 rules, it will take H1’s previous bar close at 11:00 (which has 57min time gap from my current bar close)

 

For H4 rules, it will take H4’s previous bar close at 8:00 (because H4’s interval is at 00:00, 04:00, 08:00 and 12:00 etc) that is a 3hrs and 57 time gap/difference from my current bar close).

 

 

Let’s look at another example:  Imagine your EA at different interval will use 4 different price values (instead of using only 1min’s current bar close price level) to calculate it’s opening/closing rules to determine whether to execute a trade or not.

 

At 3:46 pm

 

1min rules will take/use bar close price value at 3:46 current bar close price value in it’s calculation

 

M30 rules will take/use bar close price value at 3:30 (time gap =16min)  (previous example, has 27min time gap)

 

H1 rules will take/use bar close price value at 3:00 (time gap = 46min) (previous example, has 57min time gap)

 

H4 rules will take/use bar close price value at 00:00 (time gap = 3hrs 46min) (previous example, has 3hrs and 57 time gap)

 

 

So what’s the implication of such “wrongful” use of Higher time frame’s Previous Bar and what difficulties does it pose to the users?

 

 

1.  For any EA aim to achieve consistent results, it need to have a consistent formula.  

 

Is your EA’s formula consistent? (ie. what method does it use to calculate market prices/historical data in order to determine whether to execute a trade or not, whether the rules are true/false).

 

If we use 1 price level and apply it across all the various/multi time frames, eg. Use only the time chart’s bar close/open/current price value and apply it across all other multi time frame, we are “produce” a consistent formula.

 

However, if we use multi previous bar values taken from multi time frame’s previous bar value, we created a “variable” formula.  A formula that is changing all the time because you are utilizing different price values taken at different time gap/intervals.

 

It is common sense, a Variable/changing formula will produce Variable/changing results (inconsistent results) and a fix formula will produce fixed results, consistent to it’s application.  It stays true to the user’s intention and execution.

 

Hence, I don’t expect any consistent results from EA that were generated from software that uses different “previous bar value” when I use multi time frame rules in my EA.  In fact, I won’t waste my time/money with such software.

 

2. Imagine how are you going to lay down your trading rules, if you are using H4 (or other higher time frames) previous bar values.  

 

Imagine you want your EA to enter at current bar close when H4’s MA crossover.  How are you going to put this rule down if the software is using H4’s previous bar value?  

 

You have to think “backward”, ie. How does the condition look like before H4 crossover, in another words, what do you think your H4 indicators have to be 4 hours before a breakout happened (because the software uses H4’s previous bar, which was 4 hours ago).  

 

Sometimes, the H4’s indicator remain same for many bars before it really crossover, how are you going to put such “previous bar” rules down.  Won’t it be easier to put down the rules as it is, at current bar close?

 

if software uses the 1min’s current bar close to calculate H4’s indicator, it will then I can simply put my rules as: execute a trade when H4’s MA 100 cross MA 200, whenever at 1min’s bar close.  So much easier then to input H4’s previous bar close condition that is 4 hours ago.

 

 

The problem lies in the software developer’s methodology in using historical data.  Hence not all backtester are (designed) the same.  

 

The use of other multi time frames’ previous bar values as a point of decision/execute produce unrealistic and inaccurate (EA) performance.

 

Realistic backtestingEven though no approximation can be 100% perfect, we have done everything to accurately recreate past market conditions and order execution for strategy trading. Typical backtesting engines have a lot of assumptions and shortcuts, which result in unrealistic testing and unreliable results. MultiCharts is an institutional-level trading platform that minimizes assumptions and considers many factors.

 

 

I’m not here to promote MC, but to highlight and add the emphasis on the importance of using historically data correctly to recreate past market conditions so as to produce reliable/valid backtesting results for you to select which EA to discard or keep.

 

 

 

The use of Historical data has 2 different/primary functions

 

1) To provide price tracking/movement over a certain period of time (how far back is your historical data) 

 

     Hence, these data are package in OHLC time bars.  So that you can see on the MT4 chart, the price movements over months or years.

 

2) For the software to utliize the price values to calculate whether your EA rules are true/false.  For Execution or open/close a trade in a simulated backtesting environment.

 

 Although I’m using H4’s indicator, I don’t have to depend on H4’s OHLC price values to make a decision whether at current price level/bar close, is there a crossover in H4’s MA?  That’s the common mistake I think a lot of software fall into.  Just because I’m using H4 indicators as part of my EA’s formula doesn’t mean that H4’s OHLC bar values can only be utilized.  

 

The use of historical data at this point is for execution, not for price tracking.  since H4 is made up of multiple 1min’s bar values, it is therefore more accurate to use current bar value or 1 min’s bar close and apply this value across all the indicators for calculation and thus determine whether to execute the trade in a timely fashion instead of using “out data” H4’s previous bar value.  Given such volatile market, what happened 4 hours ago, the market/trend/price difference would have changed dramatically.

 

 

Accuracy is the Key

    

In fact, if multi time frames are available in SQ, ideally, we should be using only 1min time chart to input all our trading rules.  Which time chart you use determine how fast/often you want your EA to scan the market for trading opportunities.  It has nothing to do with what strategies you are using.  You can input H4, H1 or any other time indicators in 1min’s time chart.  The issue is a matter of speed.  How fast do you want your EA to react to market changes.  A laggy EA would not likely to have accurate entry/exit executions.  

 

In this current volatile market conditions, it is eminent that our EA are fast and accurate.  If it is so laggy with such time gap/difference, it is very difficult to achieve highly accurate and profitable EA.  Most of my EA can make between $3000 – $5000 per week (given it’s current volatile market).  I won’t be able to achieve such results if I use any time frame higher than 1min.

 

How fast do you want your EA to scan the market and keep looking for the right opportunities?  Even minute or every hour or every 4 hours?  How fast do you want your EA to respond to market changes, every minute or every hourly.  

 

Using 1min’s time chart EA enables me to develop EA that has high entry/exit rules.  With the use of multi time frames and the avoidance of “wrongful” use of multiple previous bar values (instead of 1 current bar value), we have a better hope of making good profit from every market movements (of cos the rest depend on how sound is your trading plan/strategies).

 

 

 

Why do you think Multi Chart has this function below?

 

Tick-by-tick simulation

Bar Magnifier is essential for increasing precision during backtesting. MultiCharts can construct larger bars out of smaller components ‘” second and minute bars out of ticks, hour and day bars out of minutes. You can recreate exact price movements within each bar by using the Bar Magnifier. For example, Bar Magnifier can invisibly load minutes that make up the hour, and strategy will be backtested on a minute-by-minute basis. 

 

Hence by using 1min’s time chart for your EA, you are similarly backtesting your EA on a minute by minute basis (provided the developer is using 1min’s bar close as point of calculation for all other multi time frames input/rules).

 

Ps: I’m not here to promote MC, cos I think MC and SQ are 2 very separate software.  SQ are more for non programmers.  But we can learn from other people’s expertise.

 

 

Conclusion

Since SQ4 is incorporating Mult time frame features, kindly don’t use the higher time frame’s previous bar as your backtesting methodology.  I would definitely never buy a software with such backtesting methods.  Because whatever EA, I developed using the Previous bar value features, these EA (with variable formula/time gap) will never produce any consistent and reliable results for me to put my EA in live account. 

I’m impressed with SQ development team and hence I do want SQ to succeed in finding ways to provide effective and powerful tools to their dedicated users, who are all so supportive and appreciative of SQ team members.  SQ success will results to users’ success (in their search for profitable EA). 

thank you for the long article, I understand what you mean. Generally you are saying that it is not a good idea to use previous bar value in multi-TF strategies.

 

I don’t see it this strict, it depends, and sometimes it has advantages to use previous bar values. But new SQ4 won’t limit you in your choice, it has possibility to use current bar value (for example H4 bar that is currently in the development), you are not limited to the previous bar value.

 

You mentioned MultiCharts, it is no doubt a good software, but SQ has comparable or even better backtesting engine, using real tick precision in a clean and understandable manner.

I always found Tradestation/MultiCharts choice of use of inside bar simulation confusing.

Great!!!

 

Mark,

 

It is assuring to know that SQ4 doesn’t have such restriction but offer much more flexibility than I can imagine.

 

Now it’s worth the waiting cos I was wondering how is SQ4 backtest engine organised and whether it would repeat the some kind of pit fall.  

 

Very happy to hear that SQ4 is much more “advance” than I want it to be.  Good job.  

 

Good things are worth the waiting….I’m waiting with great anticipation.