scone.util.tokenstream
Interface Tokenizer

All Known Implementing Classes:

HtmlTokenizer

public interface Tokenizer

A Tokenizer tokenizes (parses) a Reader.

Version:

1.0

Author:

Volkert Buchmann, IBM Almaden Research Center, (c) Copyright IBM Corp 2000

Method Summary

void	assign(java.io.Reader reader) assigns a Reader to the Tokenizer.
Token	nextToken() extracts the next Token from the Reader.

Method Detail

nextToken

Token nextToken()

extracts the next Token from the Reader.

Returns:

the next Token, or null if there are no more Tokens

assign

void assign(java.io.Reader reader)

assigns a Reader to the Tokenizer. Only one Reader can be assigned to the Tokenizer.

Parameters:

reader - the Reader