Scanner generation using JLex (6 Points)

Due date: Feb 6/8, 2019

Where to submit

You must submit your assignment at the web submission site. You are enrounraged to submit many times until you obtain the full marks. We will use your last submission for your final marks for this assignment. For each submission you will get your marks instantly on the web and by email.

Purpose

Understand the lexical definition of a language. Generate a scanner using JLex. 

Assignment Specification

The task is to write a JLex specification for Tiny language.  Please note that in this assignment  we don't need to use all the grammar definitions there.  Only the lexical part is needed.

You will write a JLex specification named "A2.lex". We will run the following commands to generate the scanner, compile it, and run it. 

  > java JLex.Main A2.lex
  > javac A2.lex.java
  > java A2

You should take extra care on the file names. Make sure all the three commands can run without problem, and the A2.output file is generated. If any of the 3 commands fails, you will receive very low marks, even 0, no matter how good the other part of your program is.

The A2.class program will read a text file named "A2.input", and produce a file named "A2.output" which contains following five lines:

     identifiers: NumberOfIdentifiers
     keywords: NumberOfKeyowrds
     numbers: NumberOfIntergersOrRealNumbers
     comments: NumberOfComments
     quotedString: NumberOfQuotedStrings

Here are the sample A2.input and the corresponding output file A2.output. Note that this time you only need to count the occurrences of the identifiers, keywords, etc. You do not need to remove the duplciates as in last assignment.

Note that you don't need to write any Java programs. The scanner is generated from your lex specification. 

Marking scheme

This assignment will give you bonus up to 1 point depending on the length of your program. If your program counts everything correct, you shall receive 5. On top of that, there is a bonus calculated using your program length.

	yourMark=0;
	if ( A2.lex file is not sent properly or file name is incorrect) return; 
	if (java program named A2.lex.java is generated from your lex file) 
	   yourMark+=0.5;
	if ( generated program is compiled correctly && A2.class is generated) {
            yourMark+=0.5;
            if (your java program reads A2.input && generates result file A2.output){
                     for (each of the 5 counts in A2.output) {
                           if (count is correct) {  
                                 yourMark+=0.8;
                            }
                      }
             }   
	}
	yourMark += (lengthOfYourProgram>140)?0:96/lengthOfYourProgram;  // length is counted by PHP in terms of words; Note that different programs count the words in different ways. 
	for (each day of your late submission)  yourMark=yourMark*0.7;

What to submit

You only need to submit A2.lex     

How to get started

Before starting this assignment, you must run our examples on the slides, especially the simple.lex example and the one immideatly after this example.

Common Errors

• main method not found: Probably your code does not have a main method, or you forget to change class name from MyLexer to A2 using

   %class A2 

• Infinite loop: when you encounter an infinite loop, you need to check the terminating condition in your program. make sure that your code should look like as below

   while (yy.yylex()>=0){};
     ...
  %integer 
  	       

  Note that while (true){} in simple.lex will cause an infinite loop.
• "Unmatched" error: Some input symbols can not be matched by any of your regular expressions. To ensure that all the symbols are matched by at least one of your regular expressions, you need to add the following, meaning that matching with any symbol except for new line (that is the meaning of dot) and new line (that is \n|\r). `

   .|\n|\r {} 

• ID count incorrect: probably your regular expression for ID is wrong. In some examples we have "[a-zA-Z_][a-zA-Z0-9_]*" as the RE for ID, which is incorrect according to our language definition: we do not allow underscore, hence you need to change the RE accordingly. There are some other subtle errors, e.g., in [a-za-Z] the second 'a' should be in upper case. Another possbile cause is that you put ID in front of KEYWORD like below:

  {ID} {idCount++;}
  {KEYWORD} {keywordCount++}
  

  In this case, keywords like INT will be recognized as an ID instead of a keyword. To solve the problem, put line for KEYWORD in front of the line for ID.
• Number count incorrect: double check your RE. Most common error is [0-9]+(.[0-9]+), where the dot should be replaced by '\.'. A dot means mathching with any character.\. means to match with a dot. Another common error is to write the RE as

   
  [0-9]+(\.[0-9]+)*
  

  In this case 2.3.4.5 would be recognized as one integer, which is wrong. To solve this problem, change * to ?. '?' means repeating zero or once.
• Comment count incorrect: you need to use state like below. The details are in the slide.

  %state comment;
  %%
  <YYINITIAL>"/**" {...}
  <comment>i"**/" {... }