Backtesting

Backtesting
Author: Roy Warren
Date: 06-07-05 20:10

Hi all
I have just finished backtesting my system, made a change to my trailing stop and then retested my system. The periods I have chosen are as follows;

Bull Period - Sept, Oct, Nov 2004
Sideways Period - Dec 99, Jan, Feb 2000
Bear Period - May, June, July 2002

I seem to have an anomally that needs an explanation if anyone can help?

For my Bull period I have a net profit of $3937, for my sideways period I have a net profit of $17553 and my bear period a net profit of -$3129.
The same result was achieved in both systems with one producing more profits than the other.

Firstly, I was wondering if the periods I have chosen are in fact a Bull, a Sideways and a bear period? And secondly, shouldn't I have achieved greater profits in my Bull period?

Any thoughts or comments greatly appreciated.
Roy Warren

Author: Roy Warren
Date: 07-07-05 20:33

To Vaughan and Adrian, thanks for the reply. I will try and clarify a few things.

I am trading a long system for a medium term. I am trading FPO's and using the 200Mav with a 50 day look back period as a trend gate.

In my first system I used a trailing stop of 3ATR until the (highest price since entry - 3ATR) equalled my entry price (i.e I broke even) and then switched to a trailing stop of (highest price since entry - 2ATR) to lock in more profit.
This had the following results
Bull Period = $1884
Sideways Period = $16872
Bear Period = -$2910

In my second system I used a trailing stop of 3ATR.
This had the following results
Bull Period = $3937
Sideways Period = $17553
Bear Period = -$3129

So from these results, an overall profit from my first system was $15845 and the overall profit from my second system was $18361, a difference of $2515 in favour of the second system.

By saying, in my previous post, that both systems produced the same result, I was talking about the greater profits in my sideways period in both systems.

Hope this makes sense and am looking forward to all and any comments

Ta
Roy

Author: Chris Potulski
Date: 08-07-05 14:46

That is absolutely correct.

For the result to be statistically valid the testing period has to be long enough (10 years seems to be just fine).

The problem is that we do not have 10 years of data with a couple of thousand HT (and other) members trading in similar fashion. (Trading has became one of the fast growing activity in recent times).

And for a relatively small market (ASX) a few thousand people (even trading a modest amount each) have a capacity to affect the outcome.

This by itself does not make testing invalid. Not at all.
But we should realise that testing being very (if not extremaly) usefull is not without possibly quite serious limitations.

It is said that what has happened on the market in the past will happen again in the future. Based on human nature it is probably right. However never before trading on the market was so easily accessible to so many. And this is a bit of unknown factor how this may affect (especially smaller) markets.

The obective of this posting is not to discourage people from doing what after all is the right thing to do but to make sure that possible limitations of testing are considered.

To be forewarned is to be forearmed.

Author: Jeff Bryant
Date: 08-07-05 15:36

If this is a concern there's a couple of things that could be considered:

1. Trade higher liquidity instruments (eg Top 100)

2. Also test over a shorter timeframe, say 5 years.
This will still give you a look at your rules over a period covering Tech crash, Sept 11 & the bear market to the Iraq war as well as a the recent bull run and several sideways markets.

The main reason we test over the times we do is to make sure we see how our rules have worked over a sufficient variety of market conditions.

Ultimately it's a personal thing, but it works for me.

Jeff

Author: Chris Potulski
Date: 11-07-05 08:25

Jeff,

LIQUIDITY.

Would $1,000,000 daily T/O be enough to consider stock liquid?
Based on HT teachning (for use of modest float) - rather yes.
(I use $600,000 T/O as minimum for positions up to $40,000)

Let's see then. HT has ~6,000 members and say only 20% or 1,200 are actively trading.

Say their positions are modest $5,000 each and they have maximum 4 positions opened at any given time.

So their potential buying/selling power is 1,200 x $5,000 x 4 =
$24,000,000.

Using common gates (one or two) usually yields a few stocks in a good day. Say 6 would be a maximum to be expected.

If this is the caset the above trading group has a capacity to "cover" a full daily T/O of the selected shares.

I would say it is a significant market force. Force that is in action for only say last 3 years (if that) and not always in its full strength )it grew up gradually) And if HT is doing their job well (as they undoubtely do), this group is ever growing.

TESTING PERIOD

The back test to be statistically valid has to be based on statistically valid sample. At least of a decent size (hence 10 years commonly advocated by HT) but also of decent quality ie relevant.

Testing period of 5 years may not be long enough, also dynamic changes (described above) make the data less reliable.

All depend on sensitivity of the system of course, but described above force can have noticeable influence on market.

As you said above ..." over sufficient variety of market conditions"... It is this "sufficient variety" that is in question.

CONCLUSION

1. There is no doubt whatsoever about validity of back testing process. It is a very useful tool.

The system has to work based on past data or it is no good.
If it works there is a increaased probability that it just might work in the future.

2. Fact that back testing has got serious limitations should not stop us from using it as a tool (Democracy is a lousy system but we do not know a better one - W Churchill).

3. When we use a tool, any tool, for decision making we need to be aware of this tool's limitations.

4. It does not appear to me that HT brings these points up clearly enough in education provided (unless I am a slow one).

I have deliberately used term "tool". Because what I also want to say is that the tool can not make the master craftsmen.

Regards
Chris


PS

During American Civil War both armies experienced a great deal of casualties. More then expected, more then ever before. One reason for it was that the tactics used was mostly an old one (often "the last men standing" type) but weapons used were greatly improved in their effectivenes and efficiency such as for example the rate of fire.

Technological advances present opportunities, but also challanges and need for change in current thinking - or risk an increased probability to become a casualty.

Author: Jeff Bryant
Date: 11-07-05 12:56

Chris,

Having 8 different triggers on each side of the market will dilute the impact.. Further dilution might arise when different people have their funds at different levels of utilisation & may not all be able to buy. Then there will times when, per above, 6 stocks trigger & fund limit is 4 – (natural?) selection needs to take place, diluting yet again.

Where that might end up in relation to overall average turnover, who ultimately knows, BUT – the final impact might be a positive anyway.

If it continues to bother you, perhaps you should head off into the currency market – it will be a little while before we as a group have the same impact in there J

The issues/limitations relating to backtesting are covered in the SD4 module. The advantages of having access to or being able to use trading simulators are enormous (particularly compared to the labour-intensive alternative).

In the end whichever way someone goes, the purpose is to give people the appropriate level of confidence to go to the market with their own trading strategy. “Sufficiency” of testing is still a personal thing. My only concern would be where people shortcut this phase of the process trying to get to market more quickly. It’s common, & inevitably very bad.

Jeff

Author: Chris Potulski
Date: 11-07-05 16:09

Jeff,

My intellectual faculties must became sluggish lately.

Where exactly is a hole in my logic?

As Jason pointed out several times (and I happen to agree with that) triggers are of limited value. Gates are important. Gates trunkate the number of trades way down from the number indicated as possibly showing some promise by triggers alone. And commonly used gates are fewer than triggers.

In any case I assumed 20% of HT members are active (to illustrate the point how little it takes) what if this number is higher, significantly higher? What if the floats are bigger? etc.

Where did I say back testing is not worth doing? However, I firmly believe that I don't have to worship back testing or consider it without a limitations or faults to see its value.

It is not back testing that bothers me, it is the high pedestal it is put on that, in my opinion, is not quite justified.

FX - I am doing it so the advice received is wasted on me as being too late.

In education process it is believed that not only students learn from teachers but also teachers can learn from students....well some of them.

I am taking a break for a few weeks. No trading, no laptop or mobile phone. So the up side is that I won't bother people with my non-conforming opinions - at least for a while.

Chris

Author: Jeff Bryant
Date: 11-07-05 17:32

Chris,

“Where did I say back testing is not worth doing?”
I didn’t suggest you said that, rather I was addressing your concerns regarding its limitations.
My response to your post was aimed at those (including yourself) who might feel that our weight of numbers could be an issue that skews the results to an extent that invalidates the test itself.

As to your ‘pedestal’ concept… I may have to admit guilt in this regard, however I make no apologies for trying (& not yet, to my knowledge, succeeding) to over-state the importance of backtesting.

It forms the basis for our expectations, & it is against these that we judge success &/or failure.
Nowhere is this more important than when we are forming our expectations of drawdown.

Jeff

PS: Out of interest – the positions I hold currently are:
Long – CTX, ALL, TCL
Short – WYL, GUD
Anyone else got the same?

Author: Daniel Watson
Date: 11-07-05 18:09

I think it also may be worthwhile pointing out that a lot of HT members (myself included) Only trade CFD's and therefore have no impact on the market (unless cfd provider is direct market access) I think HT members may have a small effect on the market but i dont think it is anything that should cause concern. although i find it interesting when you purchase tradesim it gives you an example system using a MACD as a trigger, Then it gives you a system which uses MACD as a trigger but completely opposite to the way it is meant to be used, Guess which one is the most profitable? The reversed system! When i first got tradesim i tested loads of indicators and found that the ones you read most about in books work the least, maybe too many people use them so they have become obscelete.Ok i think im getting a bit off track now.

Author: Adrian Miller
Date: 12-07-05 08:27

Long

ALL
MCG
WOR
MCC
CBA
GCL
EXL
MBL
PDN
BNB

Short
FKP
PBB
GUD

Adrian

Author: Rick Cortese
Date: 19-07-05 00:32

Hi Roy. I find it quite interesting that of all the posts offered in reply to your issue, only 1 came from a (current) HT trainer to comment, and that person had later mentioned that some of his trades were NOT system trades.

As a HT trainer myself, the best information I can possibly offer to you at this time, given the limited information you have provided, is for you to book a 1 on 1 coaching session with the trainer in the HT office nearest to you. Every HT office has at least 1 assigned trainer who performs coaching for exactly these type of situations. I'm surprised that this has not been mentioned already, considering I am the 28th post on this subject and am currently enjoying sunny Brasil with some of my ION profits. (Maybe I didn't read all the posts in sufficient detail due to my interest being prioritised in a different direction at the moment)

Your issue is deeper than any post on this forum can possibly offer as a deserved solution or help. Your issue requires attention to the detail of your trading plan as well as the specific results obtained. And it is the start of developing your eventual trading system, not the end of it.

So please take up HT's membership offer and book a coaching session with your office trainer to progress your hard work. Hope this helps.

Rick Cortese