-
Notifications
You must be signed in to change notification settings - Fork 2
Conditional tokens example
SwiLex provides a mechanism for conditionally activating tokens. This tokens are only available for specific Mode
.
In the following example we want to make the .quotedString
token available only when the .quote
mode is active.
Start by defining the possible Mode
s.
enum Modes {
case normal
case quote
}
Then define a SwiLaxable enum to list the possible tokens with their corresponding regular expressions (like for a simple Lexer):
enum QuotedStringTextNumberTokens: String, SwiLexable {
static var separators: Set<Character> = [" "]
case text = "[A-Za-z]*"
case number = "[0-9]*"
case doubleQuote = "\""
case quotedString = #"[^"]*"#
}
Next, define the tokens' availability for the possible modes by implementing the activeForModes
property:
extension QuotedStringTextNumberTokens {
var activeForModes: Set<Modes> {
switch self {
case .doubleQuote:
return [.quote, .normal]
case .quotedString:
return [.quote]
default:
return [.normal]
}
}
}
.doubleQuote
is available for both.normal
and.quote
modes allowing to switch between them (by defining the end and the begining of a quote).
.quotedString
is only available for the.quote
mode.
- All the other tokens are only availbale for the
.normal
mode.
The last step is to tell when to change the curent mode.
For that implement the function changeMode(current mode: Modes?) -> Modes?
:
extension QuotedStringTextNumberTokens {
func changeMode(current mode: Modes?) -> Modes? {
switch self {
case .doubleQuote:
return mode == .normal ? .quote : .normal
default:
return mode
}
}
}
Finally create an instance of SwiLex and call the lex function with the initial mode:
let input = #"4lb3rt Einstein said "If you can't explain it to a 6 year old, you don't understand it yourself.""#
var lexer = SwiLex<QuotedStringTextNumberTokens>()
let tokenList = try lexer.lex(input: input, initialMode: .normal)
// [number[4], text[lb], number[3], text[rt], text[Einstein], text[said], doubleQuote["],
// quotedString[If you can't explain it to a 6 year old, you don't understand it yourself.],
// doubleQuote["]]
This will return a list of tokens with the type of the token and its value (the matched string).
A mistake? A bug? Please report it by opening an issue or via twitter.