Category Archives: Algorithmic Trading

Predicting Stock Prices with Machine Learning – Part 1- Introduction

2 Replies

This post is the first in a series that will explore the forecasting of stock prices using machine learning (ML) methods (for a quick intro to ML see my previous post). If you’ve spent any reasonable amount of time with me then you’ll know that I tend not to talk too kindly of papers that attempt to forecast equity prices using machine learning. However, here we are simply using equity price time-series as a great example of a non-stationary process.

Price prediction or, more generally, models for generating alpha are not the best use of ML in the quantitative trading process. Far from it. At some point I’ll dedicate a post solely to describing the architecture of a quantitative trading system but for now let’s just say that portfolio optimisation, the mixture of prediction models and algorithmic execution are tasks better suited to ML.

All the same, it is possible to forecast long term price movement with ML. This series of posts will be based on work for a paper that I published earlier this year called Automated trading with performance weighted random forests and seasonality where I demonstrated the power of the online generation of ML models and suggested a novel and highly successful way to combine the predictions of multiple models.

Before we get to the nitty-gritty of combining model outputs, we first need to cover some housekeeping essentials . Initially, we’ll look at the input data and how we turn this into useful features for our model. Without data, we’re nothing so this step is arguably our most important. Next we’ll go on to look out how to measure the performance of our prediction systems including a number of important and all-too-often forgotten metrics for understanding the long term success (or not) of our model. Finally we’ll get to the fun stuff and begin to train some ML models. We’ll start simple and add layers of complexity with associated justifications along the way.

I leave it at that for now. Below is a list of the posts to come and I’ll hyperlink the items as I get them written:

  • Part 2 – Data and features
  • Part 3 – Performance Metrics
  • Part 4 – Standard Methods
  • Part 5 – Ensemble Methods
  • Part 6 – Incorporating “online” performance weighting
  • Part 7 – Summary

What happens inside a quantitative hedge fund?

1 Reply

quantitative hedge fund

Those of us working in academia have a tendency to try and do everything ourselves, often reinventing the wheel in the process. That said, due to the inherent secrecy of the quantitative hedge fund industry, academic researchers in automated trading often do need to do everything themselves. That is: modelling, algorithm design, optimisation, programming, backtesting and simulation.

But how does it work in a quantitative hedge fund? Well, from my (somewhat limited) experience with such funds I’ve found the following general structure to be fairly common.

At the beginning of the investment process you have the quant strategists/researchers. These guys and girls tend to have a PhDs in maths or physics and have a very strong understanding of  probability theory and statistics. Their job is to generate models and ideas that systematically capture investment opportunities and to develop algorithms based on these ideas. These algorithms are usually backtested by the quants and/or a research team.

Once an algorithm is deemed viable, it is passed on to the quant. developers. These developers are highly skilled (often PhD) programmers that focus on optimising the code for speed and robustness before passing it on to the programmers. The programmers’ job lies in interfacing the code with the trading platform and connectivity with the exchanges.

Next in line are the traders who work on unleashing the algorithms into the market at specific times  dependant on prevailing market conditions and instruction from the portfolio manger/s. In quantitative funds this can be quite a low touch job with traders simply launching and babysitting algorithms. However, there are often hundreds of algorithms running at once with traders in charge of allocating capital between them (according to some framework laid out by the portfolio managers).

Naturally, this tends to be more of a circular process. Given that the traders are in direct contact with the markets, they will often will come up with trading ideas and will talk with the quants who will explore, refine and generate algorithms from the ideas as the process starts again.

So you want to predict the stock market…

2 Replies

At least once a week, a Google Scholar Alert pops into my inbox featuring another bored academic that though they’d try using their “expertise” to “predict the stock market”. Said paper tends to follow this structure:

  • Starts with a paragraph about how humans are obsolete and that the market is being run by computers.
  • Next comes the ‘novel’ machine learning technique, which is really just a well known algo. (SVM, neural net., logistic regression) with an adaptive learning rate.
  • Some (usually 5) simple technical analysis indicators are used on unprocessed daily price data to create some features for the new super-algo.
  • Said algorithm is applied to predict whether price(t+1) > price(t) or vice versa.
  • Results report prediction accuracy of ~65% (And we know they tried 100 different stocks before settling on the 3 that they reported results for).
  • There are usually no out of sample results at all!

Some will stop here, reporting that they’ve cracked it and that their technique beats all others. If we’re lucky, however, they’ll go on to assume that they can trade at the close price of each day, buying when they predict upward movement and selling before a fall with a constant size strategy, reporting annual returns of 40%.

Sigh…

Putting aside the lack of out of sample results and non-existant estimation of transaction costs, they’re really missing the point. Attempting to predict the price return over the next 24 hours is not the best way to go about using machine learning to create investment opportunities.

Predicting the direction of price movement is certainly an essential part of automated trading models. But it’s not the only part. A simple automated trading model is better based around event detection: bull, bear or neutral markets, insider trading or institutional trading. Detecting such schemes can easily be approached with either binary classification of hypothesis testing.

In binary classification, a feature matrix, X of size (N*K), is used to predict a binary vector, Y of size N. In this case we would interpret each row as representing one day with K features that we hope relate in some way to the occurrence of the market event that we are trying to predict, Y. Once in this format, binary classification is a simple matter of applying one of the many machine learning algorithms to map each row, Xi to Yi. If our out of sample predictions are consistent, we have a means of quickly identifying regimes and adjusting our trading strategy accordingly. However, given that this kind of binary classification is a standard machine learning problem, it suffers all the pitfalls of overfitting.

For hypothesis testing, we make an assumption about the distribution of an observable variable before applying a test for low probability events. As an example, let’s say we’re into high-frequency volatility trading and we’re looking for unusual activity in a stock. We can describe the number of orders arriving per second with a Poisson distribution, X, and the individual order volume * price with a Gamma distribution, Y. Now we can fit X and Y through maximum likelihood estimation and calculate the probability, p, of the market acting “as normal” each second. Taking p<0.05 as a rare event, if we see more than 5 such events in a 100 second period we might be tempted to make a move. It’s important to bare in mind, however, that hypothesis testing assumes structure in data and thus requires a stationary distribution for decent results.

These kinds of techniques produce far more stable predictions than forecasting daily returns and provide us with information upon which it is much easier to trade.

The difference between automated, algorithmic and high-frequency trading

4 Replies

On telling people that I work on automated and algorithmic trading systems a common response tends to be:

“Oh, that high-frequency trading stuff?”

And I guess that’s because of the current (mostly negative) hype surrounding high-frequency trading. A truthful answer to their question is “sometimes but not always”. In order to explain, let me quickly  review the process of trading (for the sake of brevity I will stick to the buy-side’s point of view).

The diagram above shows the processing of an order from investment decision down to the execution of a trade. In this example, the process begins with an analyst’s idea that leads to a decision to trade. The idea is assessed and approved by the portfolio manager who generates an instruction to buy or sell a certain quantity of asset ‘abc’. The order is then passed on to a trader whose job it is to decide the best approach/broker to use.

Direct-Market access (DMA) represents the client’s ability to access brokers order routing infrastructure, allowing them to issue their orders almost directly to the exchanges. Sponsored access is a step up from DMA, these tend to be ultra low latency, direct connections to the market. In this situation, the client uses their own infrastructure but with the broker’s trading identifier.  Sponsored access is (predominantly) used by clients exploiting high-frequency trading strategies.

Algorithmic trading refers to trade execution strategies that are typically used by fund managers to buy or sell large amounts of assets. They aim to minimise the cost of these transactions under certain risk and timing constraints. Such systems follow preset rules in determining how to execute each order. People often think of these systems as buying low and selling high – making investment decisions – but this is not the case.  Algorithmic trading systems are offered by many brokers and simply execute the orders that they are given. Their job is to get a good price (as compared to various benchmarks) and minimise the impact of trading. This is done by slicing orders and dynamically reacting to market events.

Of course there are algorithms that deal with investment decision making and this is where automated trading comes in.

Automated trading, often confused with algorithmic trading, is the complete automation of the quantitative trading process. Thus, automated trading must encapsulate: quantitative modelling and indicator tracking to determine trade initiation and closeout; monitoring of portfolio risk; as well as algorithmic trading. This type of trading tends to be done by quantitative hedge funds that use propriety execution algorithms and trade via DMA or sponsored access.

High-frequency trading (HFT) is a subset of automated trading. Here, opportunities are sought and taken advantage of on very small timescales from milliseconds up to hours. Some high-frequency strategies adopt a market maker type role, attempting to keep a relatively neutral position and proving liquidity (most of the time) while taking advantage of any price discrepancies. Other strategies invoke methods from time series analysis, machine learning and artificial intelligence to predict movements and isolate trends among the masses of data. Specifics of the strategy aside, for HFT, monitoring the overall inventory risk and incorporating this information into pricing/trading decisions is always vital.

See, it’s simple!!

Just remember, algorithmic trading should have been called “algorithmic execution”; automated trading does what it says on the tin; and HFT is a specific type of ultra-fast automated trading.