Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reverse lookup prototype #599

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

sinisaos
Copy link
Member

@sinisaos sinisaos commented Aug 25, 2022

Related to #378. Prototype for reverse foreign key lookup. Any suggestions and help are welcome. If I missed the point or it's not a good api design feel free to close PR.
Usage of reverse FK lookup:

from piccolo.table import Table
from piccolo.columns import Varchar, ForeignKey
from piccolo.columns.reference import LazyTableReference
from piccolo.columns.reverse_lookup import ReverseLookup

class Manager(Table):
    name = Varchar()
    artists = ReverseLookup(
        LazyTableReference("Artist", module_path=__name__),
    )

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

rows = Manager.select(
        Manager.name, Manager.artists(Artist.id, Artist.name, table=Manager) # or as list Manager.artists(Artist.name, as_list=True, table=Manager)
    ).run_sync()

@codecov-commenter
Copy link

codecov-commenter commented Aug 26, 2022

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 88.79310% with 13 lines in your changes missing coverage. Please review.

Project coverage is 91.33%. Comparing base (ff26a88) to head (a7405b7).
Report is 157 commits behind head on master.

Files Patch % Lines
piccolo/columns/reverse_lookup.py 88.73% 8 Missing ⚠️
piccolo/query/methods/select.py 87.80% 5 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.02%     
==========================================
  Files         108      109       +1     
  Lines        7517     7629     +112     
==========================================
+ Hits         6867     6968     +101     
- Misses        650      661      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dantownsend
Copy link
Member

Thanks for this! It looks very promising.

@sinisaos
Copy link
Member Author

sinisaos commented Aug 26, 2022

Thanks. I used a lot of code (with some modifications) from M2M fields. I tried to use the best names of variables and methods. The main obstacle is that the user would have to enter reverse_lookup_name. I will try to make that optional (if table has only one FK default index value for foreign_key_columns will be 0), so that the user has to specify reverse_lookup_name only if the table has multiple FK so ReverseLookup knows which column to use. If you have any ideas, any help is welcome.
UPDATE:
I removed the reverse_lookup_name and added a table parameter to the ReverseLookupSelect class. With that, Piccolo can identify which table to use for reverse lookup if the table has multiple FKs. I think that is a better option.

@powellnorma
Copy link

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

Also I like how in peewee one can just set the backref parameter on the ForeignKeyField, see here
Could we have a similar API in piccolo?

i._meta.name
for i in reverse_lookup_table._meta.foreign_key_columns
]
return fk_columns.index(self.table._meta.tablename)
Copy link

@powellnorma powellnorma Dec 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we compare column names with table names here. They don't have to match. I think instead of

Manager.artists(table=Manager)

it would be better to be able to specify:

Manager.artists(column="manager")

Since the column that is used for the reverse lookup doesn't change(?), I think it would be best if we could also specify it in the ReverseLookup constructor.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@powellnorma Thanks for your interest and review. It's been a while since I wrote this code, but if I remeber it has to be a table because we refer to the foreign key of parent table later in the code. We need this function to find the foreign key column index if there are multiple foreign keys per single table.

@dantownsend
Copy link
Member

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

I haven't had chance to review it yet - but I agree that it's a feature we need.

reverse_lookup_fk_table_name = (
reverse_lookup_table._meta.foreign_key_columns[
self.foreign_key_columns_index
]._meta.name

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think instead of _meta.name it should be _meta.table._meta.tablename. Otherwise reverse_lookup_fk_table_name is a column name

SELECT
"{reverse_lookup_table_name}"."{column_name}"
FROM {reverse_select}
) AS "{reverse_lookup_table_name}s"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use AS "{original_column_name}s". The table must give us that name. See m2m.py:

) AS "{m2m_relationship_name}"

@sinisaos
Copy link
Member Author

@dantownsend It would be great if you could find the time to review this. @powellnorma This code works and passes all tests. If you think that things should be changed, it would be best after Daniel's review. Thank you both.

Comment on lines 86 to 93
reverse_select = f"""
"{reverse_lookup_table_name}"
JOIN "{reverse_lookup_fk_table_name}" "inner_{reverse_lookup_fk_table_name}" ON (
"{reverse_lookup_table_name}"."{reverse_lookup_fk}"
= "inner_{reverse_lookup_fk_table_name}"."{reverse_lookup_pk}"
) WHERE "{reverse_lookup_table_name}"."{reverse_lookup_fk}"
= "{reverse_lookup_fk_table_name}"."{reverse_lookup_pk}"
""" # noqa: E501

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this JOIN here? Isn't the following enough?

        reverse_select = f"""
            "{reverse_lookup_table_name}"
            WHERE "{reverse_lookup_table_name}"."{reverse_lookup_fk}"
                = "{reverse_lookup_fk_table_name}"."{reverse_lookup_pk}"
        """

@powellnorma
Copy link

powellnorma commented Dec 16, 2022

This code works and passes all tests

I haven't looked, but I guess the tests always use a ForeignKey named the same as the referenced Table. As e.g. here:

class Manager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

the sections I pointed lead to errors / wrong queries

@powellnorma
Copy link

When I query an M2M field of the referenced table, I get:

AttributeError: type object 'M2MSelect' has no attribute 'value_type'

Make it work when the ForeignKey is named differently than the
referenced Table.
@powellnorma
Copy link

powellnorma commented Dec 16, 2022

I have fixed the parts that I reviewed on my branch reverse_lookup, see: https://github.com/powellnorma/piccolo/commits/reverse_lookup

However querying an M2M field (of the referenced table) seems to be rather slow at the moment. When I set self.serialisation_safe = False in the case of the above mentioned AttributeError, it seems that all m2m joining keys are being used as arguments to a second query. This is overall (in my case) slower than using a for loop in python (which is located in a transaction) instead of the ReverseLookup.
@dantownsend what could we do to make this (querying m2m of reverse-referenced table) faster?

@sinisaos
Copy link
Member Author

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

the sections I pointed lead to errors / wrong queries

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist. Shouldn't it be like this

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

I want to try your branch. In my reverse lookup attempt I use LazyTableReference like this

class Manager(Table):
    name = Varchar()
    bands = ReverseLookup(
        LazyTableReference("Band", module_path=__name__),
    )


class Band(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

# query for all managers bands
query = await Manager.select(
        Manager.name, Manager.bands(Band.name, as_list=True, table=Manager)
    )

Can you show me how to write a table schema and an example query because right now I get TypeError: __init__() missing 1 required positional argument: 'reverse_fk'. Thanks in advance

@powellnorma
Copy link

powellnorma commented Dec 16, 2022

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist.

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

Can you show me how to write a table schema and an example query because right now I get TypeError: init() missing 1 required positional argument: 'reverse_fk'.

Like this:

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

@sinisaos
Copy link
Member Author

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

You are right, it doesn't work if the column is different from the table name. I tried to use foreign_key_columns_index so that if the table has multiple fk columns we know the index of those columns, but it obviously doesn't work well.

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

@powellnorma
Copy link

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

Hm, for me that example works. I (already) made changes so that _name should (indeed) be set from the Table Metaclass here.
Have you made sure that you checked out my branch correctly? Maybe only parts of my changes got applied?

@sinisaos
Copy link
Member Author

Sorry, that was my mistake. I tried everything from scratch because I probably messed something up and you are right, the tests from my PR pass with your code.
The only thing I've noticed is that if the column name is plural the tests fail, but singular name pass. I tried to strip column name if it ends with s and it works in Sqlite but not in Postgres.
I have to go now and will be gone for 2 days, but you should continue with this because your version is much better and easier to understand and it works. I'll be happy to delete my PR so you can make a new one. I'll try to help as much as I can.

@powellnorma
Copy link

@sinisaos I created a PR on your fork/branch. If you accept it, my changes should be included in this PR.

The only thing I've noticed is that if the column name is plural the tests fail

Do you have an example that demonstrates this problem? How can I reproduce this? Thank you

@powellnorma powellnorma mentioned this pull request Dec 18, 2022
@powellnorma
Copy link

The only thing I've noticed is that if the column name is plural the tests fail

Does it still occur after this commit? sinisaos@d6c6453

@sinisaos
Copy link
Member Author

@powellnorma I have added your changes. Thanks again for your help.

@lqmanh
Copy link
Contributor

lqmanh commented May 2, 2023

Hi, just wonder what's the status quo of this PR, as it's been around for a long time. Are there any blockers left? I'd really love to see this feature getting merged.

Thanks you guys in advance.

@dantownsend
Copy link
Member

@lqmanh Will try and get this merged in soon. It just needs reviewing in detail to make sure we're happy with the API 👍

@powellnorma
Copy link

One thing this is currently missing is the ability to use .order_by on the ReverseLookupSelect. Similarly with .where etc. @dantownsend Do you think this should be supported? And if so, do you see a way in which we could reuse the existing implementations for these methods?

@erhuabushuo
Copy link

Wow, I'm waiting for this PR.

@py-radicz
Copy link

it would be cool to have reverse lookup in the api, cant wait for it

@dantownsend
Copy link
Member

dantownsend commented Oct 26, 2023

OK, seems like a lot of people want this. Getting Piccolo v1 out has taken most of my attention. Will take another look at this once I've released Python 3.12 support. Bear with me, I know this PR has been around for a while 🙏

@kayprogrammer
Copy link

You guys are doing well.
Still waiting for this PR, including composite unique constraints and check constraints.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants