Trustfall Language vs others?

[leeola]

Not an issue but i'd love to see documentation outlining what the design goals and benefits of the Trustfall Language has over, say, SQL, EdgeQL or etc.

Not that it needs justification of course, but i have a bit of friction when deciding if i really want to embrace the Trustfall query language. Given that the language will be the primary UX for interaction it feels very important. Yet i'm left a bit unsold on if it's the "right" choice. Not that i'm the biggest fan of SQL, for example, but i'd love to see what Trustfall QL (what do we call it?) does better than existing solutions - or where existing solutions fail in this context.

edit: As an aside, Github Discussions might be a useful feature to enable for this repo

[obi1kenobi]

Good question and good suggestion — Discussions are now enabled. Thanks! I'll move this there.

[obi1kenobi]

I plan to write more on the topic in the future, but I'll write a bit here in the hopes of making your decision easier.

TL;DR: The Trustfall language is production-tested (~7yrs and counting) across a variety of domains and at huge scale. While it's a small language, it is future-proofed with "escape hatches" to allow expressing concepts its syntax can't capture today. But ultimately, I think you should adopt Trustfall not for the query language, but for what's underneath.

Trustfall aims to be the "LLVM of data." LLVM is a compiler framework that revolutionized how we write compilers: the first 1min of this video is an excellent summary.

Query languages and data systems alike are stagnating today, because it's too hard to build them and make them compatible with each other.

LLVM was built to address the equivalent problem: programming languages and chip architectures were stagnating because it was too much work to port every language to every piece of hardware. LLVM gave us Rust, Clang, Swift, etc. on one end, and x86 / ARM / microcontroller support for nearly all those languages on the other.

SQL is the defacto standard for querying, but it's an atrocious standard: it's a monster language to even parse let alone run, and has awful edge cases in the semantics. Here's an amazing writeup of how SQL is failing us right now: https://www.scattered-thoughts.net/writing/against-sql/

It's an explicit goal to make SQL / GraphQL / EdgeQL / Cypher possible to be parsed and executed as Trustfall queries. It's also an explicit goal to make it possible to compile Trustfall queries to each of those languages (and also languages not invented yet!), for execution on a data source that supports them natively.

Why a custom language, then?

To highlight the advantages of this unified approach, I couldn't afford to just make a worse SQL / GraphQL / <insert lang here>. It would have gotten in the way of building some of the most awesome features.

A few examples:

• Users can see each result as soon as it is available instead of waiting until the last result in the answer is fetched. This is because the interpreter is lazily evaluated — which you can't do in GraphQL.
• Queries are composable and can also ergonomically query APIs, raw files, ML models, etc. — some SQL DBs have foreign data wrappers, but performance can be poor and relational schemas aren't a good fit for APIs or ML. In the SQL world, non-SQL data sources plugged into it are always going to be a second-class capability.
• Modern schema-aware editing UX — GraphQL is incredible at this, whereas SQL is stuck in the stone ages, forcing you out of your editor and into dedicated querying apps if you want schema-aware tools in your workflow.
• Queries must be very expressive and optimizable without rewriting the query.
  • GraphQL isn't expressive enough (no aggregations, recursion, left joins, ...) and also has pretty terrible performance (e.g. N+1 query problem) so large complex queries are a no-no.
  • SQL claims to be able to optimize queries without changing them, but ultimately fails because the language is too large and there are too many equivalent ways of writing the same query for the optimizer to catch. Over the course of years, I've seen countless SQL queries get sped up by 100-1000x by doing non-obvious SQL query changes.

Worst of all, had I picked any existing language, all the other query language communities would have instantly dismissed Trustfall as "just another tool in that other language's community" and not applicable to them.

Rather than being all-encompassing from day 1, the Trustfall query language is compact in scope but porous to give us viable "escape hatches" for concepts it can't express today. Its syntax and semantics are derived from a prior project I worked on, GraphQL compiler, which has been deployed in production for nearly 7 years now and has filled every need asked of it.

If I haven't convinced you yet, try this: can any commonly-used query language automatically paginate your database queries without you having to do SKIP / LIMIT? SQL — no, GraphQL — no, any others? Can they automatically parallelize your query execution over a cluster of database machines to make it run faster? I've made sure Trustfall (both the language and the overall system) will be able to do both of these things. The key is in that LLVM-like design.

[Kinrany]

Thank you!

[obi1kenobi]

@leeola and @Kinrany, I'd love to hear what you both think!

Did my answers make you more or less likely to use Trustfall?

Are there any other things you'd like to see Trustfall do? Any things you want it to approach differently?

[leeola]

Sounds good! Right now i'm just grappling with the API, no issues with the Language for this discussion. It still, at first blush, feels a bit.. interesting, but the value proposition is enough to have me move beyond that.

Appreciate the depth of replies!

[obi1kenobi]

Of course, happy to help! Yes, the API does need a bit of getting used to ... and many more examples and docs. Thanks for powering through it!

[Kinrany]

I second the need for docs. Examples are very good, but an explanation of the semantics of the query language would be very helpful.