Class Tokenizer

Converts raw source code into a stream of tokens. Line-based, BASIC-style tokenizer.

Inheritance

object

Tokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: VM.Parser

Assembly: VM.dll

Syntax

public class Tokenizer

Constructors

Tokenizer(string)

Converts raw source code into a stream of tokens. Line-based, BASIC-style tokenizer.

Declaration

public Tokenizer(string sourceCode)

Parameters

Type	Name	Description
string	sourceCode

Methods

Tokenize()

Tokenizes the source code line by line, returning a stream of tokens.

Declaration

public IEnumerable<Token> Tokenize()

Returns

Type	Description
IEnumerable<Token>