Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use ISO 646 alternative operators for bit ops + proper error messages for spaces around the |filter operator #18

Merged
merged 4 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions book/src/template_syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -537,27 +537,44 @@ block to use includes more dynamically.
Rinja supports string literals (`"foo"`) and integer literals (`1`).
It supports almost all binary operators that Rust supports,
including arithmetic, comparison and logic operators.
The parser applies the same precedence order as the Rust compiler.
The parser applies the same [operator precedence] as the Rust compiler.
Expressions can be grouped using parentheses.
The HTML special characters `&`, `<` and `>` will be replaced with their
character entities unless the `escape` mode is disabled for a template.
Methods can be called on variables that are in scope, including `self`.

```
{{ 3 * 4 / 2 }}
{{ 26 / 2 % 7 }}
{{ 3 % 2 * 6 }}
{{ 1 * 2 + 4 }}
{{ 11 - 15 / 3 }}
{{ 4 + 5 % 3 }}
{{ 4 | 2 + 5 & 2 }}
{{ (4 + 5) % 3 }}
```

The HTML special characters `&`, `<` and `>` will be replaced with their
character entities unless the `escape` mode is disabled for a template,
or the filter `|safe` is used.

Methods can be called on variables that are in scope, including `self`.

**Warning**: if the result of an expression (a `{{ }}` block) is
equivalent to `self`, this can result in a stack overflow from infinite
recursion. This is because the `Display` implementation for that expression
will in turn evaluate the expression and yield `self` again.

[operator precedence]: <https://doc.rust-lang.org/reference/expressions.html#expression-precedence>

### Expressions containing bit-operators

In Rinja, the binary AND, OR, and XOR operators (called `&`, `|`, `^` in Rust, resp.),
are renamed to `bitand`, `bitor`, `xor` to avoid confusion with filter expressions.
They still have the same operator precedende as in Rust.
E.g. to test if the least significant bit is set in an integer field:

```jinja
{% if my_bitset bitand 1 != 0 %}
It is set!
{% endif %}
```


## Templates in templates

Expand Down
82 changes: 50 additions & 32 deletions rinja_parser/src/expr.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ use std::str;
use nom::branch::alt;
use nom::bytes::complete::{tag, take_till};
use nom::character::complete::char;
use nom::combinator::{cut, map, not, opt, peek, recognize};
use nom::combinator::{cut, map, not, opt, peek, recognize, value};
use nom::error::ErrorKind;
use nom::error_position;
use nom::multi::{fold_many0, many0, separated_list0};
use nom::sequence::{pair, preceded, terminated, tuple};

use super::{
char_lit, filter, identifier, not_ws, num_lit, path_or_identifier, str_lit, ws, Level,
char_lit, filter, identifier, keyword, not_ws, num_lit, path_or_identifier, str_lit, ws, Level,
PathOrIdentifier,
};
use crate::{ErrorContext, ParseResult, WithSpan};
Expand All @@ -22,10 +22,7 @@ macro_rules! expr_prec_layer {
let (_, level) = level.nest(i)?;
let start = i;
let (i, left) = Self::$inner(i, level)?;
let (i, right) = many0(pair(
ws(tag($op)),
|i| Self::$inner(i, level),
))(i)?;
let (i, right) = many0(pair(ws($op), |i| Self::$inner(i, level)))(i)?;
Ok((
i,
right.into_iter().fold(left, |left, (op, right)| {
Expand All @@ -34,23 +31,6 @@ macro_rules! expr_prec_layer {
))
}
};
( $name:ident, $inner:ident, $( $op:expr ),+ ) => {
fn $name(i: &'a str, level: Level) -> ParseResult<'a, WithSpan<'a, Self>> {
let (_, level) = level.nest(i)?;
let start = i;
let (i, left) = Self::$inner(i, level)?;
let (i, right) = many0(pair(
ws(alt(($( tag($op) ),+,))),
|i| Self::$inner(i, level),
))(i)?;
Ok((
i,
right.into_iter().fold(left, |left, (op, right)| {
WithSpan::new(Self::BinOp(op, Box::new(left), Box::new(right)), start)
}),
))
}
}
}

#[derive(Clone, Debug, PartialEq)]
Expand Down Expand Up @@ -189,15 +169,26 @@ impl<'a> Expr<'a> {
))(i)
}

expr_prec_layer!(or, and, "||");
expr_prec_layer!(and, compare, "&&");
expr_prec_layer!(compare, bor, "==", "!=", ">=", ">", "<=", "<");
expr_prec_layer!(bor, bxor, "|");
expr_prec_layer!(bxor, band, "^");
expr_prec_layer!(band, shifts, "&");
expr_prec_layer!(shifts, addsub, ">>", "<<");
expr_prec_layer!(addsub, muldivmod, "+", "-");
expr_prec_layer!(muldivmod, filtered, "*", "/", "%");
expr_prec_layer!(or, and, tag("||"));
expr_prec_layer!(and, compare, tag("&&"));
expr_prec_layer!(
compare,
bor,
alt((
tag("=="),
tag("!="),
tag(">="),
tag(">"),
tag("<="),
tag("<"),
))
);
expr_prec_layer!(bor, bxor, value("|", tag("bitor")));
expr_prec_layer!(bxor, band, token_xor);
expr_prec_layer!(band, shifts, token_bitand);
expr_prec_layer!(shifts, addsub, alt((tag(">>"), tag("<<"))));
expr_prec_layer!(addsub, muldivmod, alt((tag("+"), tag("-"))));
expr_prec_layer!(muldivmod, filtered, alt((tag("*"), tag("/"), tag("%"))));

fn filtered(i: &'a str, mut level: Level) -> ParseResult<'a, WithSpan<'a, Self>> {
let start = i;
Expand Down Expand Up @@ -315,6 +306,33 @@ impl<'a> Expr<'a> {
}
}

fn token_xor(i: &str) -> ParseResult<'_> {
let (i, good) = alt((value(true, keyword("xor")), value(false, char('^'))))(i)?;
if good {
Ok((i, "^"))
} else {
Err(nom::Err::Failure(ErrorContext::new(
"the binary XOR operator is called `xor` in rinja",
i,
)))
}
}

fn token_bitand(i: &str) -> ParseResult<'_> {
let (i, good) = alt((
value(true, keyword("bitand")),
value(false, pair(char('&'), not(char('&')))),
))(i)?;
if good {
Ok((i, "&"))
} else {
Err(nom::Err::Failure(ErrorContext::new(
"the binary AND operator is called `bitand` in rinja",
i,
)))
}
}

#[derive(Clone, Debug, PartialEq)]
pub struct Filter<'a> {
pub name: &'a str,
Expand Down
23 changes: 18 additions & 5 deletions rinja_parser/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ use std::{fmt, str};
use nom::branch::alt;
use nom::bytes::complete::{escaped, is_not, tag, take_till, take_while_m_n};
use nom::character::complete::{anychar, char, one_of, satisfy};
use nom::combinator::{cut, eof, map, opt, recognize};
use nom::combinator::{cut, eof, map, not, opt, recognize};
use nom::error::{Error, ErrorKind, FromExternalError};
use nom::multi::{many0_count, many1};
use nom::sequence::{delimited, pair, preceded, terminated, tuple};
Expand Down Expand Up @@ -727,14 +727,27 @@ impl Level {
const MAX_DEPTH: u8 = 128;
}

#[allow(clippy::type_complexity)]
fn filter<'a>(
i: &'a str,
level: &mut Level,
) -> ParseResult<'a, (&'a str, Option<Vec<WithSpan<'a, Expr<'a>>>>)> {
let (i, _) = char('|')(i)?;
*level = level.nest(i)?.1;
pair(ws(identifier), opt(|i| Expr::arguments(i, *level, false)))(i)
let (j, _) = take_till(not_ws)(i)?;
let had_spaces = i.len() != j.len();
let (j, _) = pair(char('|'), not(char('|')))(j)?;

if !had_spaces {
*level = level.nest(i)?.1;
cut(pair(
ws(identifier),
opt(|i| Expr::arguments(i, *level, false)),
))(j)
} else {
Err(nom::Err::Failure(ErrorContext::new(
"the filter operator `|` must not be preceeded by any whitespace characters\n\
the binary OR operator is called `bitor` in rinja",
i,
)))
}
}

/// Returns the common parts of two paths.
Expand Down
84 changes: 0 additions & 84 deletions rinja_parser/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -684,20 +684,6 @@ fn test_odd_calls() {
})),
)]
);
assert_eq!(
Ast::from_str("{{ a(b) |c }}", None, &syntax).unwrap().nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Call(
Box::new(WithSpan::no_span(Expr::Var("a"))),
vec![WithSpan::no_span(Expr::Var("b"))]
))),
Box::new(WithSpan::no_span(Expr::Var("c")))
),)
)]
);
}

#[test]
Expand Down Expand Up @@ -847,19 +833,6 @@ fn test_parse_tuple() {
})),
)],
);
assert_eq!(
Ast::from_str("{{ () | abs }}", None, &syntax)
.unwrap()
.nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Tuple(vec![]))),
Box::new(WithSpan::no_span(Expr::Var("abs")))
)),
)],
);
assert_eq!(
Ast::from_str("{{ (1)|abs }}", None, &syntax).unwrap().nodes,
vec![Node::Expr(
Expand All @@ -872,21 +845,6 @@ fn test_parse_tuple() {
})),
)],
);
assert_eq!(
Ast::from_str("{{ (1) | abs }}", None, &syntax)
.unwrap()
.nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Group(Box::new(WithSpan::no_span(
Expr::NumLit("1")
))))),
Box::new(WithSpan::no_span(Expr::Var("abs")))
)),
)],
);
assert_eq!(
Ast::from_str("{{ (1,)|abs }}", None, &syntax)
.unwrap()
Expand All @@ -901,21 +859,6 @@ fn test_parse_tuple() {
})),
)],
);
assert_eq!(
Ast::from_str("{{ (1,) | abs }}", None, &syntax)
.unwrap()
.nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Tuple(vec![WithSpan::no_span(
Expr::NumLit("1")
)]))),
Box::new(WithSpan::no_span(Expr::Var("abs")))
)),
)],
);
assert_eq!(
Ast::from_str("{{ (1, 2)|abs }}", None, &syntax)
.unwrap()
Expand All @@ -931,22 +874,6 @@ fn test_parse_tuple() {
})),
)],
);
assert_eq!(
Ast::from_str("{{ (1, 2) | abs }}", None, &syntax)
.unwrap()
.nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Tuple(vec![
WithSpan::no_span(Expr::NumLit("1")),
WithSpan::no_span(Expr::NumLit("2"))
]))),
Box::new(WithSpan::no_span(Expr::Var("abs")))
)),
)],
);
}

#[test]
Expand Down Expand Up @@ -1050,17 +977,6 @@ fn test_parse_array() {
}))
)],
);
assert_eq!(
Ast::from_str("{{ [] |foo }}", None, &syntax).unwrap().nodes,
vec![Node::Expr(
Ws(None, None),
WithSpan::no_span(Expr::BinOp(
"|",
Box::new(WithSpan::no_span(Expr::Array(vec![]))),
Box::new(WithSpan::no_span(Expr::Var("foo")))
)),
)],
);
}

#[test]
Expand Down
4 changes: 2 additions & 2 deletions testing/templates/allow-whitespaces.html
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@
{{-1}}{{ -1 }}{{ - 1 }}
{{1+2}}{{ 1+2 }}{{ 1 +2 }}{{ 1+ 2 }} {{ 1 + 2 }}
{{1*2}}{{ 1*2 }}{{ 1 *2 }}{{ 1* 2 }} {{ 1 * 2 }}
{{1&2}}{{ 1&2 }}{{ 1 &2 }}{{ 1& 2 }} {{ 1 & 2 }}
{{1|2}}{{ 1|2 }}{{ 1 |2 }}{{ 1| 2 }} {{ 1 | 2 }}
{{1 bitand 2}}{{ 1 bitand 2 }}{{ 1 bitand 2 }}{{ 1 bitand 2 }} {{ 1 bitand 2 }}
{{1 bitor 2}}{{ 1 bitor 2 }}{{ 1 bitor 2}}{{1 bitor 2 }} {{1 bitor 2}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this weird tab change? Wouldn't adding whitespace characters before 1 and after 2 provide the expected result?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had no idea what to do in the last test. :) If I remove it, testing/tests/whitespace.rs:42 changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh well, let's keep it as is for now, I'll try to take a look later on.


{{true}}{{false}}
{{!true}}{{ !true }}{{ ! true }}
Expand Down
6 changes: 3 additions & 3 deletions testing/templates/operators.html
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@
{% if c >> b == a -%}
lsh
{%- endif -%}
{% if a & b == b -%}
{% if a bitand b == b -%}
band
{%- endif -%}
{% if b ^ c == a + c -%}
{% if b xor c == a + c -%}
bxor
{%- endif -%}
{% if (b | c) == a + c -%}
{% if b bitor c == a + c -%}
bor
{%- endif -%}
{% if a == b && a + b == c -%}
Expand Down
2 changes: 1 addition & 1 deletion testing/templates/precedence.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
{{ 1 * 2 + 4 -}}
{{ 11 - 15 / 3 -}}
{{ 4 + 5 % 3 -}}
{{ 4 | 2 + 5 & 2 -}}
{{ 4 bitor 2 + 5 bitand 2 -}}
Loading