Initial Pass at Lambda Filters - Seeking Feedback / Help

[bcameron1231]

This is the initial pass at building Lambda Expression for OData Filters.

Overall feelings - Not loving it, it feels a bit kludgy. It tried to implement the operations where there was a clear separation of AND/OR conditionals but that proved difficult and will need some thoughts on. Additionally, constructing without lambda using the FilterBuilder object, is also a bit kludgy and probably needs to be re-worked (as well as re-named).

I have another branch where I'm working on improving this, but haven't invested a ton yet to show. Would love to see if this effort can be improved.

1. Allows for the ability pass in Standard Odata filter (for backwards compatibility).
2. Allows for agility to use Lambda expressions with Intellisense
3. Supports passing in date objects that can be automatically handled by the library
4. Supports creating chained and/or conditionals
5. Supports for re-using the underlying filter builder objects.

Sample usage

  //standard odata
  const r = await sp.web.lists.getByTitle("SPFx List").items.filter("Title eq 'Beau'")();    
  
  //lambda
  const r2 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=> i.field("Title").equals("Beau").or(i=>i.field("SPFxCostCenter").equals("Information")))();
  const r3 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=> i.field("Title").equals("Test").and(i=>i.field("SPFxCostCenter").equals("Information")))();
  const r4 = await sp.web.lists.getByTitle("SPFx List").items.filter(i=>(i.field("Title").equals("Beau")).or(i=>i.field("Title").equals("Cameron")))();
  
  //filter builder - very kludgy, needs to be re-worked
 let builder = new FilterBuilder();
 builder.field("Title").equals("Test");
 const r5 = await sp.web.lists.getByTitle("SPFx List").items.filter(builder.build())();

[Tanddant]

Hi @bcameron1231

I've been working on a similar concept as well - I went for less of a Lambda, and more of a CAMLjs style

OData.Where<ITask>()
    .NumberField("taskProductId").EqualTo(ProjectId)
    .And()
    .DateField("Created").LessThanOrEqualTo(new Date())
    .ToString();

Where The fields passed into DateField or NumberField (and so on) are required to be keys of the ITask interface

Output

taskProductId eq 58 and Created le '2023-11-20T14:18:45.455Z'

I've also implemented support for nested queries

I.e

OData.Where().Some([
    OData.Where<IProject>().NumberField("EditorId").EqualTo(60),
    OData.Where<IProject>().NumberField("EditorId").EqualTo(6),
    OData.Where<IProject>().NumberField("EditorId").EqualTo(85),
]).ToString()

( EditorId eq 60 or EditorId eq 6 or EditorId eq 85 )

These can also be nested (my main gripe with lambda expressions, is how unreadable they get at scale)

OData.Where().Some([
    OData.Where().All([
        OData.Where<any>().TextField("ProjectStatus").NotEqualTo("Done"),
        OData.Where<any>().DateField("Deadline").LessThan(new Date()),
    ]),
    OData.Where().All([
        OData.Where<any>().TextField("ProjectStatus").EqualTo("Critical"),
    ]),
]).ToString()

"( ( ProjectStatus neq 'Done' and Deadline lt '2023-11-20T14:29:02.608Z' ) or ( ProjectStatus eq 'Critical' ) )

[bcameron1231]

Thanks @Tanddant. Some great ideas here. Do you think you could create a branch with some of these changes implemented, or do you need some help with that?

We think there there is definitely a lot to like to about your implementation, but there may be a middle ground between the approach that I have and what you've implemented that could take this to the next level. I would love to see the inner workings of it.

Let me know how you'd like to best collaborate on this.

[Tanddant]

Hi @bcameron1231

Would love to help out - bit busy this week (need to finalize a bunch of stuff ahead of ESPC) - I spent 30 minutes trying to get the repo to work locally, but failed 😅

I've added my code at the end of the spqueryable.ts in my own fork, let me know if you have any questions - I'm at that phase of the solution where I'm like "I like this, it could work, and scale, but there's probably a reason no one has done this before if it's this easy, so I'm missing something obvious"

Also the naming could use a rework or two

Tanddant@4c43d58#diff-bdc963fcde94659fffe052eb963737c9440c80ed8435a039055a7892e3d8a7bb

[Tanddant]

There are also some filters I'm unaware of how works (never used) day, month, year, hour, minute, second, that maybe we should also think of a way to incorporate

docs

[Tanddant]

@bcameron1231 - I've moved my code here for now, as I keep iterating on it a bit, maybe we can have a chat post ESPC about what makes sense, I like the ideas you're working with using lambda expressions, but can certainly see the benefits of having type validation, maybe there's a golden middle ground

[bcameron1231]

@bcameron1231 - I've moved my code here for now, as I keep iterating on it a bit, maybe we can have a chat post ESPC about what makes sense, I like the ideas you're working with using lambda expressions, but can certainly see the benefits of having type validation, maybe there's a golden middle ground

That sounds good to me. I have some ideas we can start playing with. Let's connect in December.

[Tanddant]

Hey @bcameron1231, My calendar (and post-conference flu) is starting to clear up a bit!

Before I left i rewrote it to be based upon passing around a single class rather then just a string array (Rewrite branch), other then that I haven't had much time to look at it - I'm a bit stuck on getting PnPjs to run on my machine - so I can't really try it out in the full context, but let me know if there's anything you need me to do 😊

[bcameron1231]

@Tanddant Sent you a message on discord. We can try and get it resolved :)

[patrick-rodgers]

@Tanddant - what's the latest status here? Do we want to merge this into v4 so other can contribute? Are you still actively developing? No pressure on timeline, just not sure where we are with this work. Thanks!

[Tanddant]

@patrick-rodgers we have a well functioning solution, but for some odd reason I can't build it without commenting out some of it, and then "uncommenting" it while the code is running, @bcameron1231 wanted to take a look, but to my knowledge has been super busy 😊

[Tanddant]

@bcameron1231 Have you had a chance to look at this? 😊

[bcameron1231]

@Tanddant Not yet. :( I've been extremely busy recently.

[Tanddant]

@bcameron1231 All good, no worries - take care of yourself and relax when there's a bit of down time 😊

[Tanddant]

@bcameron1231 Are you down to see if we can get this working? 😊

[Tanddant]

No pressure, but I think I have this working like below, but would like either some "challenging queries" to test with, or someone else to review what I've got

Feel free to check my branch https://github.com/Tanddant/pnpjs/tree/v4-ODataFilterQuery

My Model (and lists) look like this

export interface IEmployee {
    Firstname: string;
    Lastname: string;
    Age: number;
    Department: IDepartment;
    Employed: boolean;
    Manager: IEmployee;
    SecondaryDepartment: IDepartment[];
}

export interface IDepartment {
    Title: string;
    DepartmentNumber: number;
    Alias: string;
    Manager: IEmployee;
    HasSharedMailbox: boolean;
}

Basic usage, there are three different ways to use these functions, all demonstrated here, my personal preference is number 1, but all three work, so I'm not planning on actively blocking it, all three have full on intellisense including validation that you can't use a numberfield to query a choice field and so on, so it'll only suggest the relevant fields for you, even knowing which fields are queryable on lookups.

// Get all active employees who either:
// 1. Work in Consulting, are managed by Dan, and are over 20
// 2. Work in Marketing and are over 30

// Method 1
const r = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.All(
    f.BooleanField("Employed").IsTrue(),
    f.Some(
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f.LookupField("Manager").TextField("Firstname").EqualTo("Dan"),
            f.NumberField("Age").GreaterThan(20)
        ),
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            f.NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Dan' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

// Method 2
const r1 = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.All(
    f => f.BooleanField("Employed").IsTrue(),
    f => f.Some(
        f => f.All(
            f => f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f => f.LookupField("Manager").TextField("Firstname").EqualTo("Dan"),
            f => f.NumberField("Age").GreaterThan(20)
        ),
        f => f.All(
            f => f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            f => f.NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Dan' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

// Method 3
const r2 = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(SPOData.Where<IEmployee>().All(
    SPOData.Where<IEmployee>().BooleanField("Employed").IsTrue(),
    SPOData.Where<IEmployee>().Some(
        SPOData.Where<IEmployee>().All(
            SPOData.Where<IEmployee>().LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            SPOData.Where<IEmployee>().LookupField("Manager").TextField("Firstname").EqualTo("Beau"),
            SPOData.Where<IEmployee>().NumberField("Age").GreaterThan(20)
        ),
        SPOData.Where<IEmployee>().All(
            SPOData.Where<IEmployee>().LookupField("Department").TextField("Alias").EqualTo("Marketing"),
            SPOData.Where<IEmployee>().NumberField("Age").GreaterThan(30)
        )
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Manager/Firstname eq 'Beau' and Age gt 20) or (Department/Alias eq 'Marketing' and Age gt 30)))

Some other random examples I could come up with

let list: IItems = sp.web.lists.getByTitle("Employees").items;

// Get all employees who work in Consulting and were hired before 2020
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.DateField("HireDate").LessThan(new Date(2020, 1, 1))
))();
// (Department/Alias eq 'Consulting' and HireDate lt '2020-01-31T23:00:00.000Z')

// Get all employees who work in Marketing, were hired today, and have an 'a' in their first name
await list.filter<IEmployee>(f => f.All(
    f.DateField("HireDate").IsToday(),
    f.LookupField("Department").TextField("Alias").EqualTo("Marketing"),
    f.TextField("Firstname").Contains("a")
))();
// ((HireDate gt '2024-10-18T22:00:00.000Z' and HireDate lt '2024-10-19T21:59:59.999Z') and Department/Alias eq 'Marketing' and substringof('a', Firstname))

// Get all employees who work in Consulting, are not employed, and have a birthday in the next 30 days
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.BooleanField("Employed").IsFalse(),
    f.DateField("Birthday").IsBetween(new Date(), new Date(new Date().getTime() + 30 * 24 * 60 * 60 * 1000))
))();
// (Department/Alias eq 'Consulting' and Employed eq 0 and (Birthday gt '2024-10-19T15:05:15.410Z' and Birthday lt '2024-11-18T15:05:15.410Z'))

// Get all employees named Dan, Jeppe, or Beau
await list.filter<IEmployee>(f => f.TextField("Firstname").In(["Dan", "Jeppe", "Beau"]))();
// (Firstname eq 'Dan' or Firstname eq 'Jeppe' or Firstname eq 'Beau')

// Get all employees who work in Consulting and are managed by Beau
await list.filter<IEmployee>(f => f.All(
    f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
    f.LookupField("Manager").TextField("Firstname").EqualTo("Beau")
))();
// (Department/Alias eq 'Consulting' and Manager/Firstname eq 'Beau')

// Get all employees who's fist name is Dan or last name is Toft
await list.filter<IEmployee>(f => f.TextField("Firstname").EqualTo("Dan").Or().TextField("Lastname").EqualTo("Toft"))();
// Firstname eq 'Dan' or Lastname eq 'Toft'

// Get alle active employees in consulting under the age of 30 who's name has a 'd' in it
// Or any employees who still works with us but are over the age of 60
await list.filter<IEmployee>(f => f.All(
    f.BooleanField("Employed").IsTrue(),
    f.Some(
        f.All(
            f.LookupField("Department").TextField("Alias").EqualTo("Consulting"),
            f.NumberField("Age").LessThan(30),
            f.TextField("Firstname").Contains("d")
        ),
        f.NumberField("Age").GreaterThan(60)
    )
))();
// (Employed eq 1 and ((Department/Alias eq 'Consulting' and Age lt 30 and substringof('d', Firstname)) or Age gt 60))


// Get everyone who is 30 or above and works in a department with an alias containing 'd' or 't'
await list.filter<IEmployee>(f => f.NumberField("Age").GreaterThanOrEqualTo(30).And().Some(f.LookupField("Department").TextField("Alias").Contains("d"), f.LookupField("Department").TextField("Alias").Contains("t")))();
// Age ge 30 and (substringof('d', Department/Alias) or substringof('t', Department/Alias))


// Get all employees who are employed and over 20
await list.filter<IEmployee>(f => f.All(f.BooleanField("Employed").IsTrue(), f.NumberField("Age").GreaterThan(20)))();
// (Employed eq 1 and Age gt 20)

[Tanddant]

I also need some opinions on what folks like more for the "grouping" operators

Pass in an array

await list.filter<IEmployee>(f => f.All([
    f => f.BooleanField("Employed").IsTrue(),
    f => f.NumberField("Age").GreaterThan(20)
]))();

Or just pass in multiple arguments?

await list.filter<IEmployee>(f => f.All(
    f => f.BooleanField("Employed").IsTrue(),
    f => f.NumberField("Age").GreaterThan(20)
))();

(The only difference is the [...])

[Tanddant]

Would also love some opinions on where it makes sense to put this change, it's technically speaking mergeable right now, but I feel like it belongs in a separate file, but I'm not familiar enough with the project architecture to know where that would be.

It also currently is applied to every spqueryable object, but it really only makes sense on list items, unless I come up with some more generic naming for the filters, and we provide default model classes for everything else, that could also work, but not sure how much value that provides compared to the list items filter

let lists = sp.web.lists.filter(L => L.Boolean("Hidden").IsFalse())();

[patrick-rodgers]

Hey @Tanddant!

Nice work on this, really love the progress. Some feedback from the team:

• We like "method 1", seems simpler
• Drop "Field" from all the names so Lookup() vs LookupField().
• switch to camel case so Lookup becomes lookup()
• could simplify the names, equal instead of EqualsTo and notEqual vs NotEqualTo
• Swap and/or for All/Some, just seems clearer to someone showing up with no knowledge of the filter methods
• pass in multiple objects you can use ...[] to gather them into an array within the method.
• this should be able to support all the filter cases, another reason to drop the "Field" name from the builder functions. Like for a site if you give it ISiteInfo you could filter on Text("Title").EqualsTo("blah");

Awesome stuff - when you have a chance if you can make those few updates we should look to get it documented and out for people to try. We can document it is Beta so folks are aware it might break.

[Tanddant]

Thanks @patrick-rodgers - Really appreciate the feedback! 🙌

I'll get right on those (will have something ready later this week) 👌

A few quick follow ups, just so I'm headed the right direction!

We like "method 1", seems simpler

I can't technically prevent people from using the other methods (maybe 3 by setting some variables as private) - but we'll just only demonstrate 1 in the docs.

could simplify the names, equal instead of EqualsTo and notEqual vs NotEqualTo

I did consider that, how do we feel about having multiple synonyms? i.e .equal() and .eq() doing the same thing, and just calling along internally - would that be too confusing for very few characters saved?

Swap and/or for All/Some, just seems clearer to someone showing up with no knowledge of the filter methods

I'm not sure I fully understand what you mean, but to make sure I'm on the right track, you want .some(...) to be .or(...)? I'm worried it might confuse some folks that you can both do f.text("field").equals("blah").or().number("field1).equals(2) but also do ftext("field").equals("blah").or(f.number("field1).equals(2),f.number("field1).equals(3))

[Tanddant]

Doing a bit of playing around, I'm not sure I fully like the .some and .all becoming .and( / .or(

Here's a before/after

const r = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.all(
    f.boolean("Employed").isTrue(),
    f.some(
        f.all(
            f.lookup("Department").text("Alias").equal("Consulting"),
            f.lookup("Manager").text("Firstname").equal("Dan"),
            f.number("Age").greaterThan(20)
        ),
        f.all(
            f.lookup("Department").text("Alias").equal("Marketing"),
            f.number("Age").greaterThan(30)
        )
    )
))();

const r = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f => f.and(
    f.boolean("Employed").isTrue(),
    f.or(
        f.and(
            f.lookup("Department").text("Alias").equal("Consulting"),
            f.lookup("Manager").text("Firstname").equal("Dan"),
            f.number("Age").greaterThan(20)
        ),
        f.and(
            f.lookup("Department").text("Alias").equal("Marketing"),
            f.number("Age").greaterThan(30)
        )
    )
))();

It very quickly becoming confusing when this is also a valid way to do .or()

list.filter<IEmployee>(f => f.text("Firstname").equal("Dan").or().text("Lastname").equal("Toft"));

Ultimately I'll do which ever you guys as the maintainers prefer, just want to flag some potential confusion 😊

[juliemturner]

I guess we're all confused by our own thing but and/or seem imminently clearer than some/all which is why we suggested it but if @patrick-rodgers and @bcameron1231 think otherwise I can get used to it.

[Tanddant]

I've gone with and/or in the other PR - I think I'm just biased coming from a C# background 😂

[Tanddant]

@juliemturner I changed it, but as I'm writing the docs, I'm realizing why I initially didn't go with and / or

The following looks a bit off to me

sp.web.lists.getByTitle("Employees").items.filter<IListItem>(f => f.boolean("Employed").isTrue().and().and(
    f.text("Department").in("Consulting", "Marketing", "Sales"),
    f.lookup("Manager").text("FirstName").equal("John"),
    f.number("Age").greaterThan(30)
))();

// compared to 
sp.web.lists.getByTitle("Employees").items.filter<IListItem>(f => f.boolean("Employed").isTrue().and().all(....

Let me know your thoughts 😊

[juliemturner]

I can't really comment as I don't understand without looking further why there are 2 'ands'... @bcameron1231 maybe you have some feedback.

[Tanddant]

@juliemturner - Basically because every "result" needs either a .and() / .or() to lead you on to the next "field type", I've been doing some playing around, and I can get it to where this works

sp.web.lists.getByTitle("Employees").items.filter<IListItem>(f => f.boolean("Employed").isTrue().and(...).or().text("Title").equals("blah))

But that then means you can do .and().or().and() without any paramaters, and break the query, I'll give it some more work and see what I can come up with 👍

[Tanddant]

The more I play around with this, the more I confuse myself, Now I've got it working, but I feel like it makes reading the queries harder

f => f.number("Age").lessThan(30).or(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

It is not clear if the or refers to the two departments, or to the age

This is (to me) more clear

f => f.number("Age").lessThan(30).and().or(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

but still somewhat confusing, I think that might be where the initial some/all same from, to name a "group of queries" (it's all a little vague, started working on it late last year, and then kinda forgot about it)

What about something like f => f.number("Age").lessThan(30).and().orJoin(....)

[Tanddant]

I can't really comment as I don't understand without looking further why there are 2 'ands'... @bcameron1231 maybe you have some feedback.

@juliemturner To answer this, (now that I fully understand it myself, after fully confusing myself for a little there)

If the query is like this

const TheYoungKidsOnTheBlock = await sp.web.lists.getByTitle("Employees").items.filter<IEmployee>(f =>
    f.number("Age").lessThan(30).and().and(
        f.lookup("Department").text("Title").equal("Cowboy"),
        f.lookup("Department").text("Title").equal("Cowgirl")
    )
)();

That translates to this:

Age lt 30 and (Department/Title eq 'Cowboy' and Department/Title eq 'Cowgirl')

Where the first and goes between 30 and ( and the second one goes between 'Cowboy' and Department

So if we instead changed the query to

f.number("Age").lessThan(30).and().or(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

It would come out to

Age lt 30 and (Department/Title eq 'Cowboy' or Department/Title eq 'Cowgirl')

Keeping the first and is needed to determine what goes before the parentheses

[juliemturner]

Ok, so the second example... .and().or(...) makes sense to me... the .and().and(...) you would never do because it should be needed... to me it should be f.and(...) assuming the entire chain is an "inclusive" statement...

[Tanddant]

But in that case wouldn't just doing

f => f.and(
    f.number("Age").lessThan(30),
    f.lookup("Department").text("Title").equal("Cowboy"),
    ....
)

be neater anyways?

The problem only arises when using the .and() after another statment

The same could be said for .or().or()

I'll see if I can't find a way to make it so that if you drop filters into the first one they'll be joined with that operand, so that this also works

f.number("Age").lessThan(30).and(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

[juliemturner]

Yes I'm saying this is correct

f => f.and(
    f.number("Age").lessThan(30),
    f.lookup("Department").text("Title").equal("Cowboy"),
    ....
)

this may work but is weird becuase you'd just do the above...

f.number("Age").lessThan(30).and(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

and this is how you'd do an and/or

f.number("Age").lessThan(30).and().or(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
)

I'm coming into this late, so if you want to just table it we'll talk about it when we have our next maintainers meeting.

[Tanddant]

All is good, I'll find a way to get it to work! - I agree with your ideas, seems like the best approach!

Might need a review on the code once it's functional, but I'll get it there 😂

[juliemturner]

Could maybe do and/or like this... again not sure

f.and(number("Age").lessThan(30).or(
    f.lookup("Department").text("Title").equal("Cowboy"),
    f.lookup("Department").text("Title").equal("Cowgirl")
))

[Tanddant]

So if I run my current code against that, this is what I get, I don't know where the and would go

(Age lt 30 or (Department/Title eq 'Cowboy' or Department/Title eq 'Cowgirl'))

Maybe if another query was added

f.and(f.number("Age").lessThan(30).or(
        f.lookup("Department").text("Title").equal("Cowboy"),
        f.lookup("Department").text("Title").equal("Cowgirl")
    ),
    f.boolean("Employed").isFalse())
)

(Age lt 30 or (Department/Title eq 'Cowboy' or Department/Title eq 'Cowgirl') and Employed eq 0)

Or am I missing something here 😊