Implementing Kalman Filter for Pair Trading Strategy Enhancement #7

chraberturas · 2024-02-12T09:44:33Z

Overview

This pull request introduces the implementation of the Kalman Filter to dynamically adjust hedge ratios between trading pairs, aiming to enhance the accuracy and profitability of our pair trading strategy. The Kalman Filter, a recursive algorithm, is used to estimate the state of a linear dynamic system from a series of noisy measurements. In the context of pair trading, it allows for more adaptive and responsive hedge ratio adjustments based on the evolving price movements of the stock pairs.

Solves: /issues/3

Changes Made

Kalman Filter Implementation: Added a new script kalman_filter.q which encapsulates the Kalman Filter logic applied to pair trading.
Strategy Integration: Integrated the Kalman Filter with the existing pair trading strategy, allowing dynamic hedge ratio adjustments.

Kokechacho · 2024-02-13T14:57:34Z

I've uploaded a new temporary file named kf.q, which contains the initial implementation of the Kalman Filter function.
One issue I've encountered is related to appending to the covariance matrix, which is currently causing errors:

// Append new updates estimates,: mt; covariances,: (,/)Ct;

Additionally, I'm still figuring out the best approach for returning the values from the function. One idea I'm considering is to return an array containing all the matrices concatenated together.

// Return (estimates;covariances)}

I'll continue to work on refining the Kalman Filter function and addressing these issues. Once I have made further progress and resolved the existing challenges, I'll provide another update.

chraberturas · 2024-02-13T19:38:23Z

Why are you flattening your matrix (,/) Ct? We should preserve the full matrix in its original format, so something like covariances, :Ct should work.

Yes, we need to consider how to approach this. I've thought that perhaps we could use set with global variables. However, I believe that returning everything is the best method here. For example, returning a dictionary instead of a list with multiple items would be more efficient. In this case, we need the opinion of our dear great expert @neutropolis.

Kokechacho · 2024-02-14T08:24:20Z

Okey I will return everything as a dictionary, but then, It does not makes sense to give to the function previous estimations and covariances because it ill never use them inside.
Input: (X,Y, Delta)
Output: (EstAlphaBeta, EstCO)

The function works fine I think so far, I need to test if it gives same results as in python.

Fixed covariances Results

chraberturas

Review the formulas; this is quite important

chraberturas · 2024-03-06T09:05:38Z

kalman_filter.q

+    et: y - ft; // e_t = y_t - f_t (EQ 3)
+    Qt: ((Ft mmu Rt) mmu transpose[Ft]) + V; // Q_t = T(F_t) * R_t * F_t + V_t (EQ 6)
+    At: ((Rt mmu transpose[Ft]) mmu 1%Qt); // A_t = R_t * F_t * inv(Q_t) (EQ 7)
+    mt: (At *\: et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)


Why are you using each right\:, i think you don't need it

Suggested change

mt: (At *\: et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)

mt: (At * et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)

I do not remeber right know why I did it exactly. Maybe it is useless i will try to do it without *: and see if it is still the same result

Done, as you said it works the same

chraberturas · 2024-03-06T09:07:25Z

kalman_filter.q

+    Qt: ((Ft mmu Rt) mmu transpose[Ft]) + V; // Q_t = T(F_t) * R_t * F_t + V_t (EQ 6)
+    At: ((Rt mmu transpose[Ft]) mmu 1%Qt); // A_t = R_t * F_t * inv(Q_t) (EQ 7)
+    mt: (At *\: et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)
+    Ct: (eye[2] - (At mmu Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)


Could you review this? I think correct formula is this

Suggested change

Ct: (eye[2] - (At mmu Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)

Ct: (eye[2] - (flip[enlist At] mmu enlist Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)

Okey I will try it as well

chraberturas · 2024-03-06T09:07:52Z

kalman_filter.q

+    At: ((Rt mmu transpose[Ft]) mmu 1%Qt); // A_t = R_t * F_t * inv(Q_t) (EQ 7)
+    mt: (At *\: et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)
+    Ct: (eye[2] - (At mmu Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)
+    (mt;-1*Ct)}  // Return new updates


Why are you negating C? I think that doesnt make sense

The result is correct if I negate C. I think it has to do with the order of the operation. from above.

Ct: ( (At mmu Ft) . eye[2]) mmu Rt; (mt;Ct)} // Return new updates

This should do the trick. I will change it

I've been thinking about this and I think that the best option is to replicate the formula. If my math is correct, what you have without negating it is correct, but hard to understand. So this is quite easier: Ct:Rt-flip[enlist At] mmu enlist first[Qt] * At

Okey this is correct, the spreads works fine now

chraberturas · 2024-03-06T09:10:00Z

kalman_filter.q

+
+// Auxiliary functions
+// Creates a diagonal matrix
+eye:{(2#x)#1f,x#0f}


Consider this shorter version :)

Suggested change

eye:{(2#x)#1f,x#0f}

eye:{{x=/:x}til x}

I can try that but I dont know if it is going to cause some problems as we need floats and not booleans or integers

It shouldn't cause problems, try e.g.: 4f*eye 4

This implementation causes problems, I think it has to do with types

chraberturas · 2024-03-06T09:12:21Z

kalman_filter.q

+    At: ((Rt mmu transpose[Ft]) mmu 1%Qt); // A_t = R_t * F_t * inv(Q_t) (EQ 7)
+    mt: (At *\: et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)
+    Ct: (eye[2] - (At mmu Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)
+    (mt;-1*Ct)}  // Return new updates


Consider returning e_t as well, in order to have your spreads calculated so you save one calculation.

Ok I will try that

But returning e_t doesnt save me from any calculations no? It will need to calculate e_t again on the next iteration

Kokechacho · 2024-03-07T08:47:38Z

I found this repository, very similar to ours https://github.com/nicholasferguson/kdb.q.kalman.filter.beta.ETFs/blob/master/beta.kalman.q

Updated some mstakes on EQ 8 and EQ 4

Add files via upload

d3b7cba

chraberturas added the enhancement New feature or request label Feb 12, 2024

chraberturas requested a review from neutropolis February 12, 2024 09:44

chraberturas assigned Kokechacho and chraberturas Feb 12, 2024

chraberturas linked an issue Feb 12, 2024 that may be closed by this pull request

Study Using Kalman Filter #3

Open

3 tasks

chraberturas added this to the Enhancements to Pair Trading Strategy milestone Feb 12, 2024

Kokechacho added 4 commits February 12, 2024 10:51

Add files via upload

7d46624

Delete KalmanInQ.q

bc05865

Add files via upload

97bbbd6

Add files via upload

404ec75

Kokechacho added 2 commits February 14, 2024 09:18

Add files via upload

d2bfdf8

Delete kf.q

dc874a4

Kokechacho added 2 commits February 14, 2024 14:01

Add files via upload

1347796

Update kalman_filter.q

34833fe

Fixed covariances Results

chraberturas commented Mar 6, 2024

View reviewed changes

Update kalman_filter.q

444d54c

Updated some mstakes on EQ 8 and EQ 4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing Kalman Filter for Pair Trading Strategy Enhancement #7

Implementing Kalman Filter for Pair Trading Strategy Enhancement #7

chraberturas commented Feb 12, 2024

Kokechacho commented Feb 13, 2024 •

edited

Loading

chraberturas commented Feb 13, 2024 •

edited

Loading

Kokechacho commented Feb 14, 2024

chraberturas left a comment

chraberturas Mar 6, 2024

Kokechacho Mar 6, 2024

Kokechacho Mar 7, 2024

chraberturas Mar 6, 2024

Kokechacho Mar 6, 2024

chraberturas Mar 6, 2024

Kokechacho Mar 6, 2024 •

edited

Loading

chraberturas Mar 6, 2024

Kokechacho Mar 7, 2024

chraberturas Mar 6, 2024

Kokechacho Mar 6, 2024

chraberturas Mar 6, 2024

Kokechacho Mar 7, 2024

chraberturas Mar 6, 2024

Kokechacho Mar 6, 2024

Kokechacho Mar 7, 2024

Kokechacho commented Mar 7, 2024

	mt: (At \: et) + alphat; // m_t = a_t + A_t e_t (EQ 4)
	mt: (At * et) + alphat; // m_t = a_t + A_t * e_t (EQ 4)

	Ct: (eye[2] - (At mmu Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)
	Ct: (eye[2] - (flip[enlist At] mmu enlist Ft)) mmu Rt; // C_t = R_t - A_t * Q_t * T(A_t) (EQ 8)

Implementing Kalman Filter for Pair Trading Strategy Enhancement #7

Are you sure you want to change the base?

Implementing Kalman Filter for Pair Trading Strategy Enhancement #7

Conversation

chraberturas commented Feb 12, 2024

Overview

Changes Made

Kokechacho commented Feb 13, 2024 • edited Loading

chraberturas commented Feb 13, 2024 • edited Loading

Kokechacho commented Feb 14, 2024

chraberturas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kokechacho Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kokechacho commented Mar 7, 2024

Kokechacho commented Feb 13, 2024 •

edited

Loading

chraberturas commented Feb 13, 2024 •

edited

Loading

Kokechacho Mar 6, 2024 •

edited

Loading