ENSPIRING.ai: ENSPIRING.org: Jim Simons Trading Secrets 1.1 MARKOV Process

In this video, we delve into the trading secrets of Jim Simons, particularly focusing on the Markov process, which plays a significant role in his remarkable trading strategy. We explore concepts from a book related to his methods, use coding to simulate strategies, and discuss how the Markov model can be applied to trading. This detailed explanation offers insights into Simons' quantitative strategy, which has outperformed markets even during recessions.

This video is worth watching for anyone interested in quantitative trading and the secrets behind the success of one of the greatest traders, Jim Simons. It provides practical coding demonstrations and historical performance analyses of mean-reverting strategies, offering a unique look into how a sophisticated model like Markov chains can be leveraged in trading. Additionally, it highlights various strategies taught in Quant courses, bolstering both beginner and advanced traders' understanding of market dynamics.

Main takeaways of this video:

Please remember to turn on the CC button to view the subtitles.

Study Notes on Key Vocabulary and Common Phrases:

1. Quant [kwɒnt] - (n.) - Short for quantitative analyst, a Quant is a financial analyst who specializes in mathematical and statistical methods to understand and predict market trends.

Jim Simons’s strategy has been purely a Quant based strategy.

2. Secretive [ˈsiːkrətɪv] - (adj.) - Describes something that is kept hidden or mysterious, often suggesting secrecy for a particular purpose or benefit.

What he does in his fund is extremely Secretive, but there are certain ideas...

3. Stochastic [stəˈkæstɪk] - (adj.) - Pertaining to a process involving a randomly determined sequence of observations or variables.

They go on to create a Stochastic equation based on this Markov chain.

4. Revert [rɪˈvɜːrt] - (v.) - To return to a previous state, condition, or practice.

The strategies were often based on the idea that prices tend to Revert after an initial move higher or lower...

5. Volatility [ˌvɒləˈtɪləti] - (n.) - Refers to the degree of variation of a trading price series over time, usually measured by the standard deviation of returns.

You've got to understand that during recessionary periods, the Volatility is extremely high.

6. Transition matrix [trænˈzɪʃən ˈmeɪtrɪks] - (n.) - A mathematical representation of the probabilities of different states transitioning into one another in a Markov process.

This is called Transition matrix.

7. Diagonal [daɪˈæɡənəl] - (adj./n.) - In this context, referring to a straight line connecting opposite corners of a matrix, often used in data visualization to indicate a perfect correlation or alignment.

You can see it's literally a Diagonal, literally across.

8. Hypothetical [ˌhaɪpəˈθetɪkl] - (adj.) - Based on or involving a hypothesis or something supposed but not necessarily real or true.

So these numbers are Hypothetical numbers.

9. Machine learning [məˈʃiːn ˈlɜːrnɪŋ] - (n.) - A subset of artificial intelligence involving algorithms that allow computers to learn from and make decisions based on data.

They've been using Machine learning models.

10. Algorithm [ˈælɡərɪðəm] - (n.) - A process or set of rules to be followed in problem-solving operations, especially by a computer.

If you don't know anything about Python, then I would suggest you to go to our video in our channel, algorithmic trading in Python.

Jim Simons Trading Secrets 1.1 MARKOV Process

What do you think of quants? Jim Simons Medallion Fund has done 39% net of fees for three decades, which proves that it worked. They were very, very smart. Yes. They got very rich. Very, very smart. Very smart and very rich and very high grade, by the way. Yeah. Jim Simons. Jim Simons is considered to be one of the greatest traders of all time, who has beaten the likes of Warren Buffett or Charlie Munger. And his strategy has been purely a Quant-based strategy.

What he does in his fund is extremely Secretive, but there are certain ideas, and there are certain concepts that we could get from what he does through this book that I've been reading. And most of my strategies that I've come across, which I do in my personal life, have also been inspired by this book. So, what we're going to do today is we're going to take some of the information that we can find in this book and start coding and try to see the results and try to figure out what Jim Simons has been doing in this fund.

So, one of the pages in the book here, it's about Axe. Axe used to work for Jim Simons. He was part of the fund, and he's also some kind of a mathematical genius. I think he's got, like, amazing papers written by him. And if you can see in this paragraph, he focuses on a thing called Markov Chain. So, in a Markov Chain, each step along the way is impossible to predict with certainty. But future steps can be predicted with some degree of accuracy if one relies on a capable model. And they go on to create a Stochastic equation based on this Markov Chain.

Another important thing, which is just a few pages prior to this, was this one loafer. Again, another mathematical genius working for Simons. And they did more of a mean reverting strategy. So, here, the strategies were often based on the idea that prices tend to Revert after an initial move higher or lower, and they would buy futures contracts if they opened at unusually low prices. So, that is a typical example of mean reverting strategies.

So, at the end of the book, one of the things that I noticed was his trading results. And if you can see, in 2008, 2007, which was basically the recessionary timeframe, he went on to make a 152% return, 136% return. That's substantially higher than any of those years. And you've got to understand that during recessionary periods, the Volatility is extremely high, and mean reverting strategies perform extremely well.

The past two years have been really good for mean reverting base strategy. So this is the trading result of that mean reverting strategy on the SPY. And if I can look into that buy and hold equity line, this period you see here, that was the 2008 recession. You can literally see the blue line, which is the S&P 500. Buy and hold has crashed almost 50%. But the mean reverting strategy performed extremely well.

Now, if you can go back to the 2001-2002 period, it's literally an X. The blue line went down considerably while our strategy performed extremely well. The same thing can be seen the past two years because 2021-2022 and even now, the market still hasn't recovered. You can see from the peak it's been going down and still in a consolidation, not recovering to the highs. But the strategy has performed extremely well. And the reason why is because of the recessionary environment. Recessionary Volatility-based, high Volatility-based environment gives great results for mean reverting strategies.

So what we're going to do in this video is, we're going to discuss what the Markov Process is. Because Markov Process is what we saw from the book, and how we can create trading strategies from the Markov Process. So to start off with, a Markov Process is basically a random sequence of events where the probabilities of the future are based on the current state. It's not based on the past. So tomorrow's probabilities depend upon today, not yesterday.

So, for instance, if I have to predict the weather, the weather prediction for tomorrow is based on today and not yesterday. So, Markov Process is used in many different fields, not just in the quantitative trading field. It's also used in weather forecasting and many other fields. So I hope you guys understood the definition of Markov Process. Now I'm going through the example of it so you guys get an idea in simple terms.

So let's take two scenarios. One is a Markov guy, and the other is a human being. So let's take the case of a human being. Let's take myself as a condition. So I wake up in the morning, I wake up at home, and then I go to the shop to buy some stuff. And I buy the stuff, and then I go to work. So, when I reach the shop, I know I came from home, so there's no reason for me to go back home, so I can go straight to work.

So, Markov, on the other hand, he goes from home and goes straight to the shop. And now he's in the current state, right? So, in the current state, he can go either home or to work because he doesn't know what happened the previous instance, as compared to a human being. So the whole Markov probabilities are based on this current state and future state because the shop is where the current state is. And once Markov is in that current state of the shop, he can go either to home or to work.

But when Markov is at work, he has nowhere else to go, so he goes straight to the shop. Same thing goes when he's at home. He has nowhere else to go, and then he goes straight to the shop. So, if you're calculating the probabilities of the Markov, that's when things get slightly not complicated, but the numbers start to come into play.

So, when he is at home, there's only one place for him to go, and that is to the shop. So, there's a 100% probability that he will go to the shop. So then we write one. Now, in the other scenario, when he's at the shop, as I said before, he doesn't know where he came back from, so he can go either home or to work. So now there's a 50% chance for him to go either home or work. Now, once Markov reaches work again, he has nowhere else to go. So he's got a 100% probability that he will go to the shop.

So this is how simple the Markov probability is. Now, if you are putting this into trading perspective, let's take a trading example into consideration. So forget the thing that's going on here. Let's just focus on this one here. So these numbers are Hypothetical numbers. So I'm going to explain to you what it's all about.

So you see this positive percentage and negative percentage. So whenever you see the news, you always see the market went up 5% and the market went down 2% and things like that. So that's a percentage move for the specific day. So in this positive percentage move, the 0.7 depicts the probability of the next day being a positive percentage. So if today is a positive percentage close, if today is an up day, then the next day's probability is 0.7.

Now, the 0.7 is just a Hypothetical number, so don't go deep into it as of now. So this positive percentage for the next day to be a positive percentage is 0.7. So what will be a negative percentage? It's pretty simple. One minus 0.7 that is 0.3. So you can see the arrow here, that's minus percentage.

So similarly, when it's today’s negative percentage, what is the probability that the next day will be negative? Well, here I put in 0.2. So what's the probability that it will be a positive? It's one minus 0.2 which is 0.8. Now, how did I come across all these numbers? Well, you can calculate in many ways. You can calculate just based on historical data.

We can calculate the number of up days and the number of down days and divide it by the up days and the total number of down days. And we'll get the probability of the up days and down days. And then there are Machine learning models as well. So if you can go through that book, one of the things that they have done is even before many years, you're talking about 30 years or so, they've been using Machine learning models. But now these days you can use Machine learning models with just a few lines of code.

So I hope you guys understood the whole idea of this thing, of this Markov probabilities. These numbers are just Hypothetical, but now we can put this into a matrix, right? This is called a Transition matrix. So you've got the positive percentage, you've got the negative percentage, you've got the positive percentage, you've got the negative percentage here as well in the columns.

So, a positive percentage and the next day is a positive percentage is 0.7, as you saw here. Similarly, a positive percentage and the next day is a negative percentage is 0.3. Again, a negative percentage day and the next day is a positive percentage day is 0.8. As you can see here, and a negative percentage followed by the next day, a negative percentage is 0.2.

So if you can observe something 0.7 plus 0.3 is one and 0.8 plus 0.2 is again one. So this is basically a Transition matrix. So here we have just taken two days in a row, so we can actually have more rows and more columns where you can have plus, plus, minus, minus here, or plus, plus, plus, minus, minus, minus. You can have many kinds of permutations and combinations in this, but this is basically a Markov trading, and this is how we calculate the trading probability.

So now what we're going to do is, we're going to go to a real-world example on SPY, and we're going to calculate the probabilities of the SPY getting a positive percentage on the next day following the previous day being a positive percentage, and similarly negative and negative and negative and positive.

This is the Anaconda notebook where we'll be calculating the Markov process and transition probabilities. And if you don't know anything about Python, then I would suggest you to go to our video in our channel, algorithmic trading in Python. So you get the basics of how to do Python. So it'll be really beneficial for you in your Quant trading journey. You can also do the TradingView Pinescript if you fancy, as well.

So now going to the Anaconda notebook. So first thing we do is basically we download the y finance library and the Pandas and the numpy, which is necessary for us to calculate many things. Then we download the data. So we download the data for spy from 2010 to 2022. You can download more data or you can keep the data smaller. So you can assess different time periods. So for example, if you want to just assess a recessionary time period, you can just do the 2008 or the 2001 dot bubble crash. So you get the recessionary environment data as well. So it's up to you, really.

So I've just randomly chosen 2010 to 2022 and then we've actually downloaded the data. And basically you can see the open, high, low, close, the adjusted close, and the volume. So we need to get the daily return. So we're going to take the adjusted close and dot percentage change function, and that will give us the percentage difference between yesterday and today. And also the state. So basically state is where the daily return is greater or equal to zero. We have got up. So we got the numpy, pandas, numpy as NP. So that's why we use NP here. So daily return is greater than equal to zero. Then it's an update, else it's a down day. And then we have stored it in data of state.

So then here is the data frame of the data and you can see the daily return here and whether it's an up percentage close or a down percentage close. So you can see whenever there's a positive one, it's up, and whenever there's a negative one, it's down negative here again it's down negative. Here it's again down as well. So basically we're using just pure math to find out the probabilities as compared to using Machine learning models. However, in the book, they have talked about Machine learning models and that was years ago. So now you can do Machine learning models just with a few lines of code. So if you guys are stoked about doing this, making this more efficient, then go ahead with the Machine learning model as well. But as of now, we're just going to make it simple so you guys can understand the process.

So we've got the up counts and the down counts. Upcounts is basically you take the length of the data of the state where it's up. So how many days has there been up? And then similarly down counts give you the length, how many days it's been down. So if you can get that information, then we can calculate the properties. We're not going to use these two lines of codes anywhere, but it's just created to give you an understanding of how to calculate the probabilities. So up to up is like two consecutive positive percentage close. Down to up is a negative day followed by a positive day, and up to down similarly and down to down, two consecutive down days. So we calculate the length of the how many times the days has been like consecutive updates, and then we divide it by the update and that will give us the probability of a two consecutive updates similarly, down to up to down and down to down.

Then we'll do a Transition matrix where we've got like a pandas data frame and we're going to put all these results into like a matrix kind of a fancy kind of a way. And then we'll print the Transition matrix and we've got the information. So you can see up to up is 54%, up to down is 45%, down to up is 57 and down to down is 42. So the best performing is an update after a down day. So that is 57%. So if I'm going to take any bet in all these four conditions, my bet will be to go for an update after a down day because there's a 57% chance for that to work out. Now, down to down is significantly lower. So it's just 42%. So there is nothing significant for us to make a trade in. So in all these numbers, these numbers are not that significant. It's on the 50% and 40% area. So I want something more effective.

Let's do another one. Let's do what's the probability of up day if there are five consecutive down days? So down day, down day, down day, down day, down day. And then we divide it by the length that's here again, length of the five or six down days. And what is the probability of that? So that probability is 66%. Now that is a pretty good probability. 66% is something that I can work with. So what I'm going to do is I'm going to take this information that I've got, like five to six days of down days, and the probability of the next day being an up day is pretty good. So I'm going to take this information. I'm going to back test it. So nine times out of ten, I would back test in Amibroker. But now, just to make things simple, I'm just going to go into Pine editor and I'm going to do a condition where close is less than one, close of one is less than close of two. So basically, yesterday's close is lower than the prior day. So we've got that condition for five to six days, and that is our entry if that condition is met. And then we are going to close our position if the next day close is higher than today's close.

So it's pretty simple. Now, you can create a Markov model for the exit condition as well. So what is the probability of us having a greater return if the close is tomorrow or two days later? So then we can calculate a better probability and create even better strategy. When you run this strategy, you would see the result as 46% with a drawdown of just 5%. So this is not at all significant when you look at simple terms. But when you look in the overall perspective, it's pretty good because you can see there's only 20 trades placed. That's from 1994, and it's only based on one condition. So imagine if you create multiple conditions or Markov models. Here we did six days of consecutive close below. So what about five days? What about, what about a combination like up, down, up followed by an update or down, down, up, followed by an update. So then you can add up more and more conditions. If the probability is suiting you, like 60% up, and then your net profit starts climbing up and up and up again. And because you've got more conditions, the drawdowns start to be better as well. Because if you're basing it on just one strategy, then the drawdowns won't be that significant. So you need to have more strategy.

So the whole idea of this video is for you guys to understand what a Markov process is, to code it in python and then apply it in your trading goal. So, for example, the Q five strategy that we did in our course, if you can look at it, because it's a mean reverting strategy, and also the fact that it's kind of inspired from the Markov model, you can see how amazingly it performs. So this is the 2001 dot bubble. The market just crashed considerably. Well, and you can see it's literally a Diagonal, literally across. Similarly, here again, 2008 crash, where it went down 54%. And look at the results of that strategy again, recently past two years, the market went down and this strategy has outperformed the market. And the drawdown is quite minimal as well. So this is the advantage of Markov model.

So in this strategy, I took the Markov model and I effectively did. I don't want to give out much information about it, but the people who know the course know how I combine two different things. I've chosen the days similar to what we saw here. The close is less than one kind of thing, but then I use the exit condition to be slightly tweaked. So these are some of the things that you can efficiently do in creating good strategies, great strategies based on good probabilities. So in this case, we got a 66% probability. And that is why it kind of worked. So even if I go to the ETF of QQQ again, you will see a 21% return and Microsoft and one. And the list just goes on. So you will basically see just based on one condition, right? So imagine having multiple conditions and applying this to multiple stocks.

And this is pretty much what Jim Simmons is doing, and we don't know exactly what he's doing, but all we can do is to get information from the book, information from any kind of interviews he does and combine it together and kind of improve. But regardless, when you are in a Quant trading journey, you're trying to get the probabilities in your favor. So any tool including the Markovian process and calculate the transition properties is highly efficient.

So in this case, we actually did, you know, just based on historical data probabilities. Now you can actually tweak that to use a Machine learning model to calculate the probabilities of these. You can go a step further, create a for loop and change these down datas and, you know, down and up to different combinations and calculate more properties, create a massive matrix, not just up, up, down, like, you know, like ten rows here, ten columns there, and then you can tweak the data points. You know, instead of this, you can actually do the recessionary environment. So you can create a recessionary based environment strategy or trend following strategy.

So sky's the limit when you have data and when you have the tools and the skills to process it. So I hope you guys liked this video. If you have any queries, any doubts or clarifications, feel free to leave a comment and I'll be more than happy to help you guys out. Have a great, great day.

Quantitative Trading, Markov Process, Economics, Finance, Innovation, Advanced Mathematics