Friday, 10 October 2014

Compiler 2 Part 2: Language Specification

Part 1 gave an overview of the topics that will be discussed in this blog series.

With Calc 1, I bored you with some broad definitions of a compiler and basic overviews of how everything works. Since we've already done that, we can dispense with the niceties and delve right in!

I’ll now discuss the additions needed for Calc 2 in a bit finer detail to provide us the framework to implement our new language features. Later, in Part 4, I will explain some of the decisions behind the way I implemented these features.

Functions


Depending on the language, functions can have a few different features. They might:

Have a name. Though, you could also have anonymous functions.
They may take parameters. Some languages may enforce rules about the parameters, like whether they’re optional, require a type and whether they’re variable.
A function almost always have a body but you could have an empty implementation for a stub.
They could have a return type, too.
Could be a type inof themselves, for use in function pointers, closures, etc.

Calc 2 will use the decl keyword to declare a function. Let’s look at what form it will take:

func-decl ::= “(“ “decl” identifier {arg-list} type expr “)”
arg-list ::= “(“ (ident whitespace)+ type {“,” (ident whitespace)+ type}* “)”
expr ::= expr | expr-list
expr-list ::= “(“ expr+ “)”
identifier ::= [a-zA-Z][a-zA-Z0-9]*
type ::= “int”

Example:

(decl add (a b int) int (+ a b))

Declare a function called add which takes two integer arguments, a and b, whose return type is an integer and returns the result of adding a to b.

A function declaration is special in that it may only occur at the top-level scope. This means that it can’t be nested within other expressions.

Function Call


A function is called like a binary expression is. It takes the form:

call-expr ::= “(“ identifier expr* “)”

Example:

(add 2 3)
=> 5

Call a function named ‘add’ with the arguments ‘2’ and ‘3’. It outputs the expected result of ‘5’.

Note that a function name may only contain alphanumeric characters and must start with a letter.

Assignment


It’s typical to have the ability to assign a value to a variable which has already been declared. It will look like this:

assign-expr ::= “(“ “=” ident value “)”

Example:

(= a 42)

Assign the variable ‘a’ the value ‘42’.


Variable Declaration


Since functions tend to take arguments it stands to reason to have also have variable declarations. A nice, simple keyword will suffice.


var-expr ::= “(“ “var” (ident type) | (assign-expr {type})“)”
ident ::= [a-zA-Z][a-zA-Z0-9]*
type ::= “int”

Example 1:

(var a int)

Declare a variable called ‘a’ of the type int.

Example 2:

(var (= b 5))

Declare a variable called ‘b’ and assign it the value ‘5’. Since the optional type, when using an assign-expr, has been omitted the type is inferred.

Variables will have a default zero value.

Looping


To keep things simple, looping constructs like for and while will be held in reserve for Calc 3. To implement our required looping needs we’ll use recursion instead.

Branching

Everybody loves a good if statement!

if-expr ::= “(“ “if” binary-expr {type} expr {expr} “)”
operator ::= “>” | “>=” | “<” | “<=” | “=” | “!=” | “and” | “or”

If bool resolves to 1 then the first expr is executed, else, the other is executed.

Example:

(if (< a 2) int 2 a)

If ‘a’ is less than ‘2’ evaluates to true then return ‘2’ else return the value of ‘a’.

Of course, there are other constructs like elif chains and select-case. Neither will be implemented for Calc 2.

Putting it All Together

Taking the specification from our previous compiler, we can insert our new definitions.

package ::= file+
file ::= func-decl+
func-decl ::= “(“ “decl” identifier {arg-list}] type expr “)”
expr ::= assign-expr | binary-expr | call-expr | expr-list | if-expr | basic-lit | var-expr
expr-list ::= “(“ expr+ “)”
arg-list ::= “(“ (ident whitespace)+ type {“,” (ident whitespace)+ type}* “)”
assign-expr ::= “(“ “=” ident value “)”
binary-expr ::= “(“ (operator whitespace expr (whitespace expr)+ ”)”
call-expr ::= “(“ identifier expr* “)”
if-expr ::= “(“ “if” bool-expr {type} expr-list {expr-list} “)”
var-expr ::= “(“ “var” (ident type) | (assign-expr {type})“)”
basic-lit ::= integer
identifier ::= [a-zA-Z][a-zA-Z0-9]*
operator ::= [+-*/%><=] | “>=” | “<=” | “!=” | “and” | “or”
whitespace ::= [ \t\n\r]+
integer ::= digit+
digit ::= [0-9]
type ::= “int”

Calc 2


We can now write the two functions fibonacci and factorial like this:

(decl fibonacci (n int) int (
    (if (<= 0) int 0)
    (if (== n 1) int
        1
        (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))

(decl factorial (n int) int (
    (if (<= n 0) int 0)
    (if (== n 1) int
        1
        (* n (factorial (- n 1))))))

Amusingly, aside from multiplication vs. addition, the two functions are exactly the same.

I think you can agree that Calc will be a bit more useful in it’s next incarnation even though it still has a long way to go to be a proper, general purpose language.

That about sums up the new language additions in Calc 2. I think that we’ll discuss some of the architectural changes that will need to come to support the user-facing features.

Wednesday, 8 October 2014

Compiler 2 Part 1: Introduction to Apprentice Compiler Design

Table of Contents
  1. Introduction to Apprentice Compiler Design
  2. Language Specification
  3. Infrastructure
  4. Language Design Decisions
  5. Tokens
  6. Scanner
  7. AST
  8. Parser
  9. Assembly
  10. C Runtime
  11. Using the C Runtime (Assembly Part 2)
  12. Compiler, Code Generation
  13. Front End

Introduction


Welcome! It's been a long time coming!

This series is the second I've written on compilers. The first one can be found here and if you've not already done so, I highly recommend starting there first because this series builds upon the code and ideas started there.

Calc 2 is an evolution of the language. Calc 1 achieved my goal of being a basic calculator and a teaching tool but it leaves much to be desired for being a much more useful, general purpose tool.

While Calc 1 demonstrated how to take raw source code and compile it into a working binary, it did little to demonstrate how languages are actually transformed into machine language. Short of doing some basic arithmetic, it does little else.

I also did not answer many questions I posed in the first series. Questions about different types of parsers, abstract syntax trees and language design decisions. I hope to address more of these questions while still keeping the discussion on the task at hand.

In this second series, I have set a new goal for Calc. We will give it the ability to calculate a Fibonacci number or a factorial.

So, hold on tight because here we go!

Forewarning


You need to understand the material covered in the first series before tackling anything covered here. This series takes a considerable jump in difficulty and complexity.

I’ll do my best to explain what I can but I will, like in the last series, be making some assumptions about your level of programming skill. A moderate level of competency in Go is a must.

I should also caution you that I am not a professional programmer nor am i an expert on compilers. I am merely trying to pass on some of the knowledge I’ve gained in the hopes of helping you learn how to write a compiler yourself.

Also, when we reach the topic of the C runtime and assembly you will find that having experience with pointers and understanding how memory addressing works will be an asset.

Consider yourself warned.

More Credit


Calc 2 deviates considerably further away from Go but the fact remains that there are parallels between this work and Go and it’s standard library package by the same name. As before, any and all credit must go to the Go developers for any code that resembles any of their collective works.

You can find the source code for Go at golang.org.

Runtime


To help make some of our tasks easier, I wrote a C runtime for Calc 2. This runtime is statically linked into the resulting binary. An end user need never know it even exists.

This runtime is very small and implements a tiny subset of pseudo-assembly instructions to make code generation easier. The library resides within Calc’s own directory so once Calc is purged from your system so is the runtime. It is never installed into the main OS’s system directories to lay forgotten, taking up space. Lurking...waiting...hungering…

You need never look at it if you don’t want to but you probably should.

Assembly


You got it! I’ll be talking a bit about the big, scary assembly language in this series!

Not to mislead you, because we won’t be actually using assembly, but in order to understand how the C runtime works you will need to know the strengths and advantages of assembly as it pertains to language implementation.

Language Features


With the goal of calculating a fibonacci number in mind, there are several features which will need to be added to Calc.

Calc 1 already does the arithmetic we require to make the needed calculations but it lacks the ability to actually facilitate our needs in a meaningful way.

In order to add a meaningful level of abstraction, Calc 2 will need to have the following implemented:
  • Functions
  • Variables
  • Looping
  • Branching
These four language features alone allow us to implement the design target for Calc 2.

Compiler Features


The compiler section of Calc 1 didn't actually do a whole lot. By optimizing away the complexity, Calc 1 merely prints the result of the arithmetic operations. Calc 2 will need to actually generate unoptimized code.

I will also be including the ability to compile a directory containing multiple source files.

Bugs


The potential for undiscovered bugs is always a spectre looming over every project’s shoulder. Compilers, especially, have the potential for odd and unpredictable bugs.

I have attempted to be as thorough as possible but I can’t make a 100% bug-free claim. If you discover a bug, please report it in the issue tracker.

Moving Forward


That concludes the overview of the series. In Part 2 we’ll dive right in to the language specification.

There is a lot to digest in this series. It is a considerable amount longer and a lot of new information will be introduced. I hope you bear with me and take the time to make it all the way through.

I wish you the best of luck!

Tuesday, 19 August 2014

A Further Update

Well, here it is the middle of August and still no second series for learning how to write a compiler.

All I can say is, it's coming!

To reduce it's length a bit, I am going to prune some of the code I have implemented and will add it back in the next series.

I've been stuck at 95% too long!

For those who want to skip ahead/preview, you can view the code at: https://github.com/rthornton128/calc.

Thursday, 5 June 2014

Calc 2 is Coming Along Nicely

If you haven't been following github, then you might be wondering what's happening with the next series of articles. Then again, maybe you aren't. Who am I to know?

Calc 2 is ripening nicely! The compiler is essentially finished and I'm now into re-factoring and bug fixing. I've actually had a lot of fun writing it and I am looking forward to showing it off.

I expect to fork Calc 2 off the master branch within the next week or two, signaling that the compiler is reasonably stable. That's the dream, anyway.

The articles themselves are coming along nicely. I'm a bit concerned about the length and am trying to figure out where I can trim some fat. This second series is already three articles longer than the first one and a couple of the posts are quite long. Hopefully, I can hold everyone's attention that long!

Here I thought that adding only four new features to the language would be short!

Eleven of the thirteen articles are fully drafted and the majority of them have gone through several editing passes, already. I don't expect the last two articles to take too long to finish so, outside of some personal life obstructions, I might even have everything complete by months end!

Don't hold me to that though. My spare time is more limited than ever so any number of things could push that date out.

Saturday, 3 May 2014

Compiler Part 10: Compiling to C

Part 1: Introduction
Part 2: Compilers, Transpilers and Interpreters
Part 3: Overview of Compiling
Part 4: Overview of Language Design
Part 5: Calc 1 Language Specification
Part 6: Tokens
Part 7: Scanning
Part 8: Abstract Syntax Tree
Part 9: Parsing

The final stage at last!

With our language spec being so simple I could have easily skipped C and gone straight to Assembly. I have two reasons for not doing so. First, is portability. I don’t have to write any architecture specific code in C for this tutorial. C has been ported to lots of different systems so I’ll let your C compiler do the work for me.

Second, assembly is going to be a lot more foreign to many programmers than C. Even if you've never written anything in C before it will be easier to understand than Assembly.

Compiling

This should be starting to look familiar. We’re going to be walking the AST and either generating C code or doing a calculation.

The output generated by our compiler will go into a file which shares the same name as the source file with a “.c” extension appended to it. CompileFile creates the output file, parses the source code and starts the output process.

As with the parser, the first element of the AST will be the File object. While the File object itself doesn't contain anything useful outside our root expression, it does provide us with the opportunity to produce some output.

This part actually isn't that exciting. It’s just a bunch of boilerplate. We provide an entry point for the program, an exit status, and a printf statement which requires we also include the C standard input/output header.

Optimization


I said at the beginning of this series that I wouldn’t be talking much about optimization, and I won’t. That said, our language is ideally suited to one type of optimization: pre-calculation.

We could go through the task of emitting code for each element of our tree but to what end? It is surprisingly complex and provides no benefit. Instead, why not do all the calculations at this stage and make the final output simpler and faster?

If the root expression is a number then we pass that number to printf. If it’s a binary expression, we can have some fun.

First stop is a generic function called compNode where we determine what kind of object we have: a BasicLit or a BinaryExpr.

If it is a basic literal, which we know is an integer, we simply convert it to an actual integer and return the result.

The binary expression is just as easy. The first element of the list of expressions is always the start point. Order of the operands is important for some operations like division and subtraction. The first element of the expression list is the starting point and is stored in a temporary variable (as an aside, this would be like using the eax or rax register in assembly).

Then, based on the operator, we compute the result and store it back into the same variable for each additional operand. Once complete, the result is returned. This is done recursively until everything is resolved.

Final Stage


Once the compiler is complete there’s still more work to be done. First off, we need to create a command to read in the Calc 1 source code and call the compiler library. Next, a C compiler (likely gcc or clang) should be run on the output from the compiler command. Finally, depending on how the C compiler was run, you might need to run a linker on the object code output from the C compiler.

This is all handled by a program called calcc, which stands for Calc Compiler. Another example of my brilliant naming skills.

I don't think there's anything spectacular going on in this program. It merely opens the input file, verifying that it has the .calc extension, and calls comp.CompileFile on it. It then uses exec.Command from the Go standard library to execute the C compiler and linker.

It has several flags which can be used to control the C compiler a bit.

Last Words


I sincerely hope I've helped someone learn a little something about compiling. It is a massive subject and it can be pretty overwhelming.

For some, this series is probably a little too simple and for that I’m sorry. I hope you can bare with me when I take things to the next level in Calc 2.

There is a tremendous amount I've skipped over and I hope to continue in the next series to start addressing these shortcomings. Topics I hope to include in Calc 2 are:

  • symbol tables 
  • scope 
  • function declarations 
  • variable declarations 
  • comparisons and branching
  • variable assignment
  • typing 
  • stack memory

I will be basing Calc 2 directly on the code from Calc 1 so anything you've learned here will, I think, be directly applicable in the next series.

If a Calc 3 comes to fruition, and I hope it does, I certainly plan to tackle assembly. If I do, then I might include a few articles solely on assembly itself, possibly as a separate series prior to Calc 3. Other possible ideas are: objects, methods, multiple-assignment, multiple files and libraries.

Thanks for reading!

I bid thee, adieu!

Compiler Part 9: Parsing

Part 1: Introduction
Part 2: Compilers, Transpilers and Interpreters
Part 3: Overview of Compiling
Part 4: Overview of Language Design
Part 5: Calc 1 Language Specification
Part 6: Tokens
Part 7: Scanning
Part 8: Abstract Syntax Tree

We've come a long way so far. We've covered the basic principles of scanning and the abstract data tree. Now, we can finally move on to parsing.

If you’re struggling to keep up at the point then I’ll warn you that it just gets harder from here on out. Parsing is probably the hardest part to wrap your head around conceptually. We will be taking the lexemes our scanner detects, give them meaning and storing the resulting objects into the AST.

Make sure you understand the previous material before moving on.

The Parser Object


The parser for our little language is pretty simple. Like our scanner, we have a pointer to the information on the file we’re scanning. We have a way to track errors and a scanner object.

The last three fields directly reflect the same three values returned by the scanner: the position, the token and the literal string.

Entry Point


ParseFile is where the magic begins. We initialize the parser and begin scanning. If one or more errors are encountered we stop and print out the errors. Otherwise, we return the file object, which is the entry point into the program.

Just like with the scanner, init primes the pump to get things started. Any calls to any other parser functions will advance the parser and scanner and we can’t go back. Fuel is in the line so lets start the engine!

parseFile is a companion to it’s exported cousin. This function creates the first object for our AST. Going back to Part 5 yet again (see how important grammar rules are?) we can see that a file object is the root of everything. That’s exactly what we've done. The very first object is the File object.

Note that we don’t really check anything before proceeding to find the first expression. That’s because our grammar rules tell us this has to be true. Anything else is wrong. We expect to find an expression of some kind and we’ll be mighty upset if we don’t get one so we’ll just plow on ahead!

Last, we ensure that the very last thing in our file is the End of File (EOF) token. Our grammar rules show that after we find our root expression there should be nothing else. Report an error if there if something else is found and parade it around for everyone to look at and ridicule!

Expressions


We've discussed what an expression is a few times. Now, our job is to find one with parseGenExpr but we don’t know what kind to expect at first. Well, the first token we find is going to tell us all we need to know. If we find an opening bracket then we have a binary expression. If we find an integer, that’s what we have. Otherwise, we generate an error and move on.

The integer is the easiest element to parse. There’s no point into going into detail on it. The code is self-explanatory.

However, the binary expression is a bit trickier. Calc 1 only has binary expressions but later on we’re going to have more types of expressions. Just like with the Expression object we should have a generic way of handling expressions in general before dealing with the specifics. parseExpr handles this.

First off, we expect to find a left paren. It’s bad if we don’t. Next off, we need to figure out what kind of expression we have. We know that the only expression we have is a binary expression so we’ll test to find out what operator is the next token. If we get something else, we need to report an error.

The BinaryExpr exemplifies the recursive nature of our parser. We've already found the opening paren and operator so now we need to find our operands. This is done via a loop which recursively calls parseGenExpr. We keep building up our tree deeper and deeper until we reach the end.

After we find all the operands we expect a right paren to close the expression. Finally, we return the resulting BinaryExpr object and it gets inserted into our tree.

Expect, Next and AddError


Expect is a great little utility function. We tell it what token we expect to find. If it’s there, great. If not, we report an error. Regardless, we return the position of that element.

Next doesn't really need explanation. It just takes exactly what the scanner finds and stores it in the parser object.

AddError appends a new error message to our ErrorList. If more than ten errors have been found there’s really no point in continuing. It prints the errors and forces the parser to exit with an error code.

Syntax Analysis


If it isn't clear, our parsing stage does syntax analysis as it works. It ensures that the source code we’re parsing adheres to our grammar rules.

Anything less is unacceptable. No, seriously.

Finished


That’s it for this stage. If you got through this then the final bits should be easy. The parser will get more and more complicated as our grammar gets more in-depth. Calc 2 ought to serve as a good example of that.

Compiler Part 8 - Abstract Syntax Tree

Part 1: Introduction
Part 2: Compilers, Transpilers and Interpreters
Part 3: Overview of Compiling
Part 4: Overview of Language Design
Part 5: Calc 1 Language Specification
Part 6: Tokens
Part 7: Scanning

Before we can build the parser we need to take a detour and talk about how we’re going to handle the data we’re going to process.

We need to use some kind of abstract data type to hold all the data we’re parsing. As it turns out, a tree data structure is perfect for our needs. This tree describes the syntax of our programming language and is appropriately called the Abstract Syntax Tree (AST).

AST


Tree data structures always start with a root and ours is no different. Typically, in a full-fledged compiler, you’d probably have an object which represents a package or program. In our case, we only have one file so we’ll have an object called File.

The rest of the objects we need to create can be found in our grammar blueprint. I refer you, again, back to Part 5 where we created our grammar rules. This handy layout tells us exactly what we need. Aren’t you glad we created it? It’s also going to come in very handy when we start parsing and syntax analysis.

The other objects we’ll need to create: an expression and an integer.

Go Interfaces


Before we create our objects we need to consider what they’re going to look like. What is it we know?

We know that our scanner is going to provide us with three things:

  • The literal string of the element we've scanned. A number might be “42”.
  • The token representing the element. An opening parenthesis would be token.LPAREN.
  • The position at which the element starts. A number between the file’s base and base + size.

One thing that will be common to all objects in our tree is their position. Where do they start and where do they end? Error reporting, for example, won’t care what kind of object it has. It only cares what it’s position is. So, we create a Node interface which has two methods: Pos, for the start position, and End, for the end position.

Next, there is a recurring element in our grammar rules: expressions. An expression can either be an integer or a sequence of elements encapsulated in parentheses. The interface will be called an Expr. Any Expr objects must implement the method exprNode as well as fulfill the Node interface.

We can now pass our objects around between different parts of our compiler based on their interface and use type assertions to discover their exact type, if required. When we get to parsing, this ought become more clear.

Objects


There are three we have to implement as well as one extra helper object:

  1. Integer
  2. Expression
  3. File

We’ll start with the Integer. We have to be a bit forward thinking with our objects and the integer type is no different. We could spend a lot of time creating an object for each type in our language. We could, however, just create a single object to hold any basic type.

BasicLit is just such an object. Whether we have an integer, float, string, or character it doesn't matter. This object satisfies our needs for all of them. Of course, we only have integers at the moment but eventually we want to add more. This object holds the token telling us what type of object it is, the position where the literal starts and the string representing the literal.

Skipping over the Expression for a moment, we’ll move on to File. This object is even simpler. This is where it all starts. It’s our starting point and has one field, Root. Looking back at our grammar rules we can see that a file holds an expression and that expression is the root of everything else.

Our expressions are of a specific type right now. They’re binary expressions. Mathematics. We have several elements we have to keep track of: An operator, opening and closing parentheses and two or more operands which are, in of themselves, expressions.

A binary expression won’t always be our only type of expression. Eventually, we’ll want more. One thing that will be common to every expression in our language is an opening and closing paren. That’s where the generic Expression object comes in handy. It’s only job is to handle the parens. This object will be nested into all our sub-expressions.

The beauty of Go is that any objects with an embedded  type sort of “inherit” the nested object’s methods. It’s not true inheritance but rather the compiler being clever. Regardless, the result is the same for our purposes.

As already stated, the BinaryExpr must have an operator and a list of operands. We have fields that hold the operand and where it is located is the sources. Our operands are held in the field List which is a slice of objects which fulfill the Expr interface.

AST Utilities


If we need to inspect our AST, a method of walking the tree would be handy. The simple Walk function does just that.

Printing the AST is done via the Print method which merely Walks the tree and prints out some information about each node in order.

Moving On


We’ll finally get to parsing soon but I want to be sure the code for the AST is clear. There’s that nasty recursive definition of an Expression and likely some methods that don’t provide any context to their use. Make sure you review this article and any previous posts before moving on.

Thankfully, parsing ought to make things more clear how all the pieces fit together.