SUMMARY:  INNER | FIELD | CONSTR | METHOD | FRIEND DETAIL:  FIELD | CONSTR | METHOD | FRIEND


Platform
Class String::Tokenizer

NonCopyableType
   |
   +-Tokenizer

Implemented in:
Platform.String.h
Enclosing class:
String

class Tokenizer:
private NonCopyableType

A string tokenizer allows to break a string into tokens based on a set of delimeters (the characters that separate tokens), or a delimiter function object of type String::Delimiter.

For example the following code will produce a sequence of tokens "this", "is", "a", "test":
    String text = appText("this is a test");
    for(String::Tokenizer tokenizer(text, appText(" ")); tokenizer.hasNext();)
        Runtime::StdOut::println(tokenizer.next());
The same example with delimiter function object:
    String text = appText("this is a test");
    for(String::Tokenizer tokenizer(text); tokenizer.hasNext();)
        Runtime::StdOut::println(tokenizer.next(String::WhiteSpaceDelimiter()));
See Also:
String::Delimiter

Constructor Summarycollapse
Tokenizer(const String& inString)
          Constructs a string tokenizer for the specified string with the default delimiter characters: " \t\n\r\f".
Tokenizer(const String& inString, const String& inDelimiters)
          Constructs a string tokenizer for the specified string with the given delimiter characters.
Tokenizer(const String& inString, const String& inDelimiters, Int inPosition)
          Constructs a string tokenizer for the specified string with the given delimiter characters and initial position.

Method Summarycollapse
Bool hasNext() const
          Tests whether there are tokens available from this tokenizer's string (i.e.
Bool hasNext(const Delimiter& delimiter) const
          Tests whether there are tokens available from this tokenizer's string (i.e.
String next() const
          Retrieves the next token from this string tokenizer.
String next(const Delimiter& delimiter) const
          Retrieves the next token from this string tokenizer.
static String nextToken(const String& text, Int& fromIndex, const Delimiter& delimiter)
          Retrieves the next token from the specified string starting at the given position.
static Bool skipToken(const String& text, Int& fromIndex, const Delimiter& delimiter)
          Skips the next token from the specified string starting at the given position.
Containers::Array<String> toArray() const
          Retrieves an array of all the tokens from this string tokenizer.
Containers::Array<String> toArray(const Delimiter& delimiter) const
          Retrieves an array of all the tokens from this string tokenizer.
Void toFirst() const
          Rewinds the internal position of this tokenizer to the first token.

 

Constructor Detail

Tokenizer[inline]

Tokenizer(const String& inString)

Constructs a string tokenizer for the specified string with the default delimiter characters: " \t\n\r\f".

Parameters:
inString - a string to be tokenized

Tokenizer[inline]

Tokenizer(const String& inString, const String& inDelimiters)

Constructs a string tokenizer for the specified string with the given delimiter characters.

Parameters:
inString - a string to be tokenized
inDelimiters - the delimeters

Tokenizer[inline]

Tokenizer(const String& inString, const String& inDelimiters, Int inPosition)

Constructs a string tokenizer for the specified string with the given delimiter characters and initial position.

Parameters:
inString - a string to be tokenized
inDelimiters - the delimeters
inPosition - initial position within the string

 

Method Detail

hasNext[nothrow, inline]

Bool hasNext() const

Tests whether there are tokens available from this tokenizer's string (i.e. if it is possible to subsequently call next()). If this method returns True, then a subsequent call to next() will successfully return a token.

Returns:
True if there is at least one token in the string after the current position; False otherwise

hasNext[nothrow, inline]

Bool hasNext(const Delimiter& delimiter) const

Tests whether there are tokens available from this tokenizer's string (i.e. if it is possible to subsequently call next()). If this method returns True, then a subsequent call to next() will successfully return a token.

Parameters:
delimiter - delimiter function object
Returns:
True if there is at least one token in the string after the current position; False otherwise

next[nothrow, inline]

String next() const

Retrieves the next token from this string tokenizer. Use the hasNext() method to ensure that the next token is actually available from this tokenizer's string.

Returns:
the next token from this string tokenizer

next[nothrow, inline]

String next(const Delimiter& delimiter) const

Retrieves the next token from this string tokenizer. Use the hasNext() method to ensure that the next token is actually available from this tokenizer's string.

Parameters:
delimiter - delimiter function object
Returns:
the next token from this string tokenizer

nextToken[nothrow, inline]

static String nextToken(const String& text, Int& fromIndex, const Delimiter& delimiter)

Retrieves the next token from the specified string starting at the given position. The position is updated to point to the character after the retrieved token if the operation was successful.

Parameters:
text - a string to be tokenized
fromIndex - initial position within the string
delimiter - delimiter function object
Returns:
the next token from the specified string if available, empty string otherwise

skipToken[nothrow, inline]

static Bool skipToken(const String& text, Int& fromIndex, const Delimiter& delimiter)

Skips the next token from the specified string starting at the given position. The position is updated to point to the character after the skipped token if the operation was successful.

Parameters:
text - a string to be tokenized
fromIndex - initial position within the string
delimiter - delimiter function object
Returns:
True if the next token from the specified string was skipped, False otherwise

toArray[nothrow]

Containers::Array<String> toArray() const

Retrieves an array of all the tokens from this string tokenizer.

Returns:
an array of all the tokens from this string tokenizer

toArray[nothrow]

Containers::Array<String> toArray(const Delimiter& delimiter) const

Retrieves an array of all the tokens from this string tokenizer.

Parameters:
delimiter - delimiter function object
Returns:
an array of all the tokens from this string tokenizer

toFirst[nothrow, inline]

Void toFirst() const

Rewinds the internal position of this tokenizer to the first token.



SUMMARY:  INNER | FIELD | CONSTR | METHOD | FRIEND DETAIL:  FIELD | CONSTR | METHOD | FRIEND