How does the Windows Command Interpreter (CMD.EXE) parse scripts?

I made some/many experiments, and this seems to be the main results.

To better understand how batch works, and why sometimes escaping works and other times it seems to fail. I work this out by many experiments, and I build tests so I can identify the order of the discrete phases.

There exists multiple areas to examine. I got the

• BatchLineParser - The parser inside of batch files, for lines or blocks
• CmdLineParser - Like the BatchLineParser, but directly at the command prompt, works different
• LabelParser - How call/goto and labels work
• CommandBlockCaching - How parenthesis and caching works
• Tokenizer - How single tokens(groups of characters) build and in which phases

The BatchLineParser:

A line of code in a batch file has multiple phases (on the command line the expansion is different!).

The process starts with phase 1

Phase/order
1) Phase(Percent):

• A double %% is replaced by a single %
• Expansion of argument variables (%1, %2, etc.)
• Expansion of %var%, if var does not exists replace it with nothing
• For a complete explanation read this from dbenham Same thread: percent expansion

1.5) Remove all <CR> (CarriageReturn 0x0d) from the line

2) Phase(Special chars, " <LF> ^ & | < > ( ): Look at each character

• If it is a quote (") toggle the quote flag, if the quote flag is active, the following special characters are no longer special: ^ & | < > ( ).
• If it is a caret (^) the next character has no special meaning, the caret itself is removed, if the caret is the last character of the line, the next line is appended, the first charater of the next line is always handled as escaped charater.
  • <LF> stops the parsing immediatly, but not with a caret in front
• If it is one of the special characters & | < > split the line at this point, in case of the pipe (|) both parts gets a phase restart (a bit more complex ...) For more info on how pipes are parsed and processed, look at this question and answers: Why does delayed expansion fail when inside a piped block of code?
• In this phase the primary token list is build, token delimiters are <space> <tab> , ; = and <0xFF> (also known as non-breaking space)
• Process parenthesis (provides for compound statements across multiple lines):
  • If the parser is not looking for a command token, then ( is not special.
  • If the parser is looking for a command token and finds (, then start a new compound statement and increment the parenthesis counter
  • If the parenthesis counter is > 0 then ) terminates the compound statement and decrements the parenthesis counter.
  • If the line end is reached and the parenthesis counter is > 0 then the next line will be appended to the compound statement (starts again with phase 1)
  • If the parenthesis counter is = 0, and the parser is looking for a commmand, then ) and all remaining characters on line are ignored
• In this phase REM, IF and FOR are detected, for the special handling of them.
• If the first token is "rem", only two tokens are processed, important for the multiline caret

3) Phase(echo): If "echo is on" print the result of phase 1 and 2

• For-loop-blocks are echoed multiple times, first time in the context of the for-loop, with unexpanded for-loop-vars
• For each iteration, the block is echoed with expanded for-loop-vars

---- These two phases are not really follows directly, but it makes no difference
4) Phase(for-loop-vars expansion): Expansion of %%a and so on

5) Phase(Exclamation mark): Only if delayed expansion is on, look at each character

• If it is a caret (^) the next character has no special meaning, the caret itself is removed
• If it is an exclamation mark, search for the next exclamation mark (carets are not observed anymore), expand to the content of the variable
  • Consecutive opening ! are collapsed into a single !
  • Any remaining ! that cannot be paired is removed
• If no exclamation mark is found in this phase, the result is discarded, the result of phase 4 is used instead (important for the carets)
• Important: At this phase quotes and other specical characters are ignored
• Expanding vars at this stage is "safe", because special characters are not detected anymore (even <CR> or <LF>)

6) Phase(call/caret doubling): Only if the cmd token is CALL

• If the first token is "call", start with phase 1 again, but stops after phase 2, delayed expansion are not processed a second time here
• Remove the first CALL, so multiple CALL's can be stacked
• Double all carets (the normal carets seems to be stay unchanged, because in phase 2 they are reduced to one, but in quotes they are effectivly doubled)

7) Phase(Execute): The command is executed

• Different tokens are used here, depends on the internal command executed
• In case of a set "name=content", the complete content of the first equal sign to the last quote of the line is used as content-token, if there is no quote after the equal sign, the rest of the line is used.

CmdLineParser:

Works like the BatchLine-Parser, but:

• Goto/call a label isn't allowed

Phase1(Percent):

• %var% will be replaced by the content of var, if the var isn't defined, the expresssion will be unchanged
• No special handling of %%, the second percent could be the beginning of a var, set var=content, %%var%% expands to %Content%

Phase5(exclamation mark): only if "DelayedExpansion" is enabled

• !var! will be replaced by the content of var, if the var isn't defined, the expresssion will be unchanged

for-loop-command-block

e.g. for /F "usebackq" %%a IN (command block) DO echo %%a

The command block will be parsed two times, at first the BatchLineParser(the loop is inside a batch) or the CmdLineParser(loop on the cmd-line) is active, at the second run always the CmdLineParser is active. In the second run, DelayedExpansion is active only if it is enabled with the registry key

The second run is like calling the line with cmd /c

Setting of variables are therefore not persistent.

Hope it helps Jan Erik

Here is an expanded explanation of Batch processing Phase 1 in jeb's answer

1)(Percent) Starting from left, scan each character for %. If found then

• 1.1 (escape %)
  If followed by another % then
  Replace %% with single % and continue scan
• 1.2 (expand argument)
  • Else if followed by * and command extensions are enabled then
    Replace %* with the text of all command line arguments
  • Else if followed by <digit> then
    Replace %<digit> with argument value (replace with nothing if undefined) and continue scan
  • Else if followed by ~ and command extensions are enabled then
    
    • If followed by optional valid list of argument modifiers followed by required <digit> then
      Replace %~[modifiers]<digit> with modified argument value (replace with nothing if not defined or if specified $PATH: modifier is not defined) and continue scan.
      Note: modifiers are case insensitive and can appear multiple times in any order, except $PATH: modifier can only appear once and must be the last modifier before the <digit>
    • Else invalid modified argument syntax raises fatal error: batch processing aborts!
• 1.3 (expand variable)
  
  • Else if command extensions are disabled then
    Look at next string of characters, breaking before % or <LF>, and call them VAR (may be an empty list)
    
    • If next character is % then
      Replace %VAR% with value of VAR (replace with nothing if VAR not defined) and continue scan
    • Else goto 1.4
  • Else if command extensions are enabled then
    Look at next string of characters, breaking before % : or <LF>, and call them VAR (may be an empty list). If VAR breaks before : and the subsequent character is % then include : as the last character in VAR and break before %.
    • If next character is % then
      Replace %VAR% with value of VAR (replace with nothing if VAR not defined) and continue scan
    • Else if next character is : then
      • If VAR is undefined then
        Remove %VAR: and continue scan.
      • Else if next character is ~ then
        
        • If next string of characters matches pattern of [integer][,[integer]]% then
          Replace %VAR:~[integer][,[integer]]% with substring of value of VAR (possibly resulting in empty string) and continue scan.
        • Else goto 1.4
      • Else if followed by = or *= then
        Invalid variable search and replace syntax raises fatal error: batch processing aborts!
      • Else if next string of characters matches pattern of [*]search=[replace]% then
        Replace %VAR:[*]search=[replace]% with value of VAR after performing search and replace (possibly resulting in empty string) and continue scan
      • Else goto 1.4
• 1.4 (strip %)
  Else remove % and continue with scan

The above helps explain why this batch

@echo off
setlocal enableDelayedExpansion
set "1var=varA"
set "~f1var=varB"
call :test "arg1"
exit /b  
::
:test "arg1"
echo %%1var%% = %1var%
echo ^^^!1var^^^! = !1var!
echo --------
echo %%~f1var%% = %~f1var%
echo ^^^!~f1var^^^! = !~f1var!
exit /b

Gives these results:

%1var% = "arg1"var
!1var! = varA
--------
%~f1var% = P:\arg1var
!~f1var! = varB

Note 1 - Phase 1 occurs prior to the recognition of REM statements. This is very important because it means even a remark can generate a fatal error if it has invalid argument expansion syntax or invalid variable search and replace syntax!

@echo off
rem %~x This generates a fatal argument expansion error
echo this line is never reached

Note 2 - Another interesting consequence of the % parsing rules: Variables containing : in the name can be defined, but they cannot be expanded unless command extensions are disabled. There is one exception - a variable name containing a single colon at the end can be expanded while command extensions are enabled. However, you cannot perform substring or search and replace operations on variable names ending with a colon. The batch file below (courtesy of jeb) demonstrates this behavior

@echo off
setlocal
set var=content
set var:=Special
set var::=double colon
set var:~0,2=tricky
set var::~0,2=unfortunate
echo %var%
echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%
echo Now with DisableExtensions
setlocal DisableExtensions
echo %var%
echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%

Note 3 - An interesting outcome of the order of the parsing rules that jeb lays out in his post: When performing search and replace with normal expansion, special characters should NOT be escaped (though they may be quoted). But when performing search and replace with delayed expansion, special characters MUST be escaped (unless they are quoted).

@echo off
setlocal enableDelayedExpansion
set "var=this & that"
echo %var:&=and%
echo "%var:&=and%"
echo !var:^&=and!
echo "!var:&=and!"

As pointed out, commands are passed the entire argument string in μSoft land, and it is up to them to parse this into separate arguments for their own use. There is no consistencty in this between different programs, and therefore there is no one set of rules to describe this process. You really need to check each corner case for whatever C library your program uses.

As far as the system .bat files go, here is that test:

c> type args.cmd
@echo off
echo cmdcmdline:[%cmdcmdline%]
echo 0:[%0]
echo *:[%*]
set allargs=%*
if not defined allargs goto :eof
setlocal
@rem Wot about a nice for loop?
@rem Then we are in the land of delayedexpansion, !n!, call, etc.
@rem Plays havoc with args like %t%, a"b etc. ugh!
set n=1
:loop
    echo %n%:[%1]
    set /a n+=1
    shift
    set param=%1
    if defined param goto :loop
endlocal

Now we can run some tests. See if you can figure out just what μSoft are trying to do:

C>args a b c
cmdcmdline:[cmd.exe ]
0:[args]
*:[a b c]
1:[a]
2:[b]
3:[c]

Fine so far. (I'll leave out the uninteresting %cmdcmdline% and %0 from now on.)

C>args *.*
*:[*.*]
1:[*.*]

No filename expansion.

C>args "a b" c
*:["a b" c]
1:["a b"]
2:[c]

No quote stripping, though quotes do prevent argument splitting.

c>args ""a b" c
*:[""a b" c]
1:[""a]
2:[b" c]

Consecutive double quotes causes them to lose any special parsing abilities they may have had. @Beniot's example:

C>args "a """ b "" c"""
*:["a """ b "" c"""]
1:["a """]
2:[b]
3:[""]
4:[c"""]

Quiz: How do you pass the value of any environment var as a single argument (i.e., as %1) to a bat file?

c>set t=a "b c
c>set t
t=a "b c
c>args %t%
1:[a]
2:["b c]
c>args "%t%"
1:["a "b]
2:[c"]
c>Aaaaaargh!

Sane parsing seems forever broken.

For your entertainment, try adding miscellaneous ^, \, ', & (&c.) characters to these examples.

Rob van der Woude has an awesome Batch scripting and Windows Command prompt reference on his site.

This link is a good starting reference for string manipulation in cmd.exe scripts.

edit: see accepted answer, what follows is wrong and explains only how to pass a command line to TinyPerl.

Regarding quotes, I have the feeling that the behaviour is the following:

• when a " is found, string globbing begins
• when string globbing occurs:
  • every character that is not a " is globbed
  • when a " is found:
    • if it is followed by "" (thus a triple ") then a double quote is added to the string
    • if it is followed by " (thus a double ") then a double quote is added to the string and string globbing ends
    • if the next character is not ", string globbing ends
  • when line ends, string globbing ends.

In short:

"a """ b "" c""" consists of two strings: a " b " and c"

"a"", "a""" and"a"""" are all the same string if at the end of a line