Conditional tokens example

SwiLex provides a mechanism for conditionally activating tokens. This tokens are only available for specific Mode.

In the following example we want to make the .quotedString token available only when the .quote mode is active.

Start by defining the possible Modes.

enum Modes {
    case normal
    case quote
}

Then define a SwiLaxable enum to list the possible tokens with their corresponding regular expressions (like for a simple Lexer):

enum QuotedStringTextNumberTokens: String, SwiLexable {
    static var separators: Set<Character> = [" "]
    
    case text = "[A-Za-z]*"
    case number = "[0-9]*"
    case doubleQuote = "\""
    case quotedString = #"[^"]*"#
}

Next, define the tokens' availability for the possible modes by implementing the activeForModes property:

extension QuotedStringTextNumberTokens {
var activeForModes: Set<Modes> {
    switch self {
        case .doubleQuote:
            return [.quote, .normal]
        case .quotedString:
            return [.quote]
        default:
            return [.normal]
        }
    }
}

• .doubleQuote is available for both .normal and .quote modes allowing to switch between them (by defining the end and the begining of a quote).
  
• .quotedString is only available for the .quote mode.
  
• All the other tokens are only availbale for the .normal mode.

The last step is to tell when to change the curent mode.

For that implement the function changeMode(current mode: Modes?) -> Modes?:

extension QuotedStringTextNumberTokens {
    func changeMode(current mode: Modes?) -> Modes? {
        switch self {
        case .doubleQuote:
            return mode == .normal ? .quote : .normal
        default:
            return mode
        }
    }
}

Finally create an instance of SwiLex and call the lex function with the initial mode:

let input = #"4lb3rt Einstein said "If you can't explain it to a 6 year old, you don't understand it yourself.""#

var lexer = SwiLex<QuotedStringTextNumberTokens>()
let tokenList = try lexer.lex(input: input, initialMode: .normal)

// [number[4], text[lb], number[3], text[rt], text[Einstein], text[said], doubleQuote["], 
//  quotedString[If you can't explain it to a 6 year old, you don't understand it yourself.], 
//  doubleQuote["]]

This will return a list of tokens with the type of the token and its value (the matched string).