| Module Info
| Add a review of String-Tokenizer
Nice module. There is a way to skip whitespace, but not token separators. Automatically skipping the latter would be a nice feature.
I'm not sure what this module offers that the older String::Tokeniser doesn't, other than Americanised spelling in its name.
When I was working on a perl port of a java query parser (see CQL::Parser) I was looking for a perl equivalent to java.util.StringTokenizer for the lexing part of my parser. I dug around and tried some other modules on CPAN, but discovered Stevan's elegant String::Tokenizer, and was sold. It fit like a glove.