[OpenBIOS] bug in toke?

[Mark Wenning]

Hi -
We're looking at using toke, detok in our development process.

Our current setup to tokenize device drivers uses a "script" similar to 
this:
-----   start kng_sr_script -----
    \  define these words for the tokenizer
    tokenizer[
        0           constant ?SYSTEM-ROM
        0           constant COMMENT-OUT

        0           constant ?DEBUG-TX
        0           constant ?DEBUG-RX

        h# 0110 constant ibm-Code-Revision-Level

        h# 17D5     constant ibm-VendorId
        h# 5831     constant ibm-king-DeviceId
        h# 020000   constant ClsCode
    ]tokenizer

    fload kng_main.of
----- end kng_sr_script -----

Then in kng_main.of, we do:
----- start kng_main.of   -----
    ....
   tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
 
   tokenizer[ hex ibm-VendorId ibm-king-DeviceId  ClsCode decimal 
]tokenizer PCI-HEADER
  
   FCODE-VERSION2

    ...
----- start of code...-----

The version of toke I downloaded doesn't like this for 2 reasons:
1.  Constant declarations in TOKENIZER[    ...   ]TOKENIZER blocks  emit 
bytes/tokens to the output,
so we get extra stuff before the PCI header.   I don't think it should 
do this; code inside tokenizer blocks
should not be in the output FCODE.  Is this correct?
2.  Previous constant declarations aren't being looked up/interpreted 
when in Tokenizer mode.  For
example,
    tokenizer[ hex ibm-Code-Revision-Level decimal ]tokenizer  SET-REV-LEVEL
causes a "empty stack: error.  I have to change it to
    tokenizer[ 0110 ]tokenizer  SET-REV-LEVEL
to make it work.  Is this a bug? 

I'm a little new to this; basically what I'm asking is if this behaviour 
is what was intended, or should we attempt to fix it and submit a fix to 
you.