As quantitative finance has matured and the importance of computation has exploded, it’s time to use machine learning to harvest the new low hanging fruit. Traditionally, quants might work alongside engineers and computer scientists — the quants provide the statistical expertise, and the computer scientists and engineers provide the computational expertise. Machine learning folks combine statistics and computation in one brain to build models that leverage new levels of scale and richness to generalize better to unseen data and tackle new problems.
Quants
Finance and statistics have overlapped for over 100 years, at least since Louis Bachelier published The Theory of Speculation in 1900. Modern quantitative finance folks, called quants, have strong statistical backgrounds and come from a broad set of fields like finance, economics, physics, statistics, actuarial science, and so on. Pure quants are stronger in statistics and less so in computation. As the importance of computation has exploded, a traditional pure quant skill has become outmoded. This is actively changing over the past decade, so the non-computational characterization of quants is by no means universal. Many statisticians argue that computational statistics should be part of the primary education of a statistician.
Scale
Machine learning folks have both strong statistical and computational backgrounds. That said, there are those focused more on machine learning theory, especially in academia, who are incredibly deep at the statistics end of the spectrum. For those focused more on applying machine learning in industry, there is no need to be as statistically deep as a stats PhD. Rather, there is a need to combine statistics and optimization with distributed systems to tackle large scale problems that present complicated engineering challenges. Leveraging state of the art systems engineering and big data allow applied machine learning folks to tackle data problems on a new scale (size of data) and complexity (richness of data — for example working with text or images). Much of this industry work was pioneered at Google during the early 2000’s — for example the now widely used MapReduce model for distributed computing was first developed to meet the needs of training a machine learning model on web scale data sets.
Richness
A good way to understand the difference between the quant and machine learning toolbox is to consider how each would throw a linear model at a problem. A quant will tend to use a data sample, an out of the box model from a software package like R, and fit it in a standard way. A machine learning person might train it on a larger dataset over a larger parameter space with a loss function and optimization algorithm tailored to the specific problem. This combination of model, features, large data, loss function, and optimization algorithm allow machine learning folks to use a richer toolbox to build models that generalize better to unseen data.
Newness
Many unexploited opportunities have been evident to quants for decades, but although the solutions may be clear statistically at some level, the limiting factor has been computation. A simple example is incorporating unstructured data like online content, or semi-structured data like company reports and transaction data, into predictive models. Feature engineering is the process machine learning folks use to generate inputs to statistical models from raw input data. There are approaches for automated feature learning with techniques like deep learning — recently, this has allowed us to unlock the potential of understanding and labeling images. Then there are approaches that require collaboration with subject matter experts. Effective feature engineering requires an understanding of how to feed the right kind of information into an optimization algorithm that allows the model to learn what it needs to in order to perform well at its task. The approaches that machine learning folks take to feature engineering are computationally sophisticated in a way that harnesses much more information from the raw data than traditional quants. There are also techniques like unsupervised learning and distant supervision that enable building models that perform well on tasks for which there is either little or no training data to learn from. Lastly, fields like natural language processing and deep learning allow us to capture information contained in raw text and images, and use this information to solve totally new problems.
Data Science
While ‘quant’ and ‘machine learning’ are clearly defined terms at this point, ‘data science’ is a place to be careful, as the definition is still in flux. This title could be used by someone with a background in applied machine learning whose deliverable is working production models that accomplish a specific task. It could also be used by someone who plays an analyst role leveraging environments like matlab, R or python whose deliverables are visualizations, studies, and presentations. Both are valuable, but it is important to work backward from your desired deliverable to define your staffing needs. If you are searching for people on linkedin to build out a team that’s meant to deliver working models, I’d search for ‘machine learning’ rather than ‘data science.’
Quantitative finance has matured to the point that it yields terrific results for many problems — delivering models that eclipse the performance of humans for many tasks. Machine learning is not magic, depending on the problem and available data, a traditional quant approach might be state of the art. In these cases, quants are doing the same thing that a machine learning person would do. Machine learning shines when the scale of the available data and richness of the toolbox enables models that generalize better to unseen data, or allows us to tackle new problems. Due to the maturity of quantitative finance and the economic benefits of better performance on important finance problems, the market for applying these traditional approaches is pretty efficient. This means that much of the new low hanging fruit lies in applying machine learning approaches pioneered in the tech world at larger scale with new data, richer models, and to new problems. If your financial services organization is looking at running a machine learning initiative, try to find a problem where machine learning provides a real edge above and beyond traditional quant approaches you are already employing. Then work backward from a clear deliverable to define the team composition most likely to yield success.
Follow Bradford on Twitter to stay up to date on machine learning in finance.