Edit – I have been looking at other alternatives to writing my own parser/language, although I did learn a lot when trying to write my own. Nemerle seems like the most likely candidate to me, given how extensible and C#-like it is. Among mainstream languages, I am also beginning to favor F#. Thanks to everyone in the comments section who pointed out various options to me.

I guess it’s finally time to let people know what I’ve been working on (and thinking about) for the past six months or so.

The driving force behind this new project is my increasing frustration with existing languages. While most of them have been crafted masterfully, there’s always something in each that isn’t available and prevents me from writing the most elegant or readable code for that compiler. With C#, my list of gripes is long and would eat up a whole post. Let’s just say that while I love C#, there are many features I wish it had. And most of the time, it’s not something that can’t be added, it’s just not in the compiler because

• The feature probably wasn’t considered or was cut because it wasn’t an important use-case
• The development team couldn’t add it in due to time constraints (that’s fair)
• Or they believe it’s not something that the language needs (these are far more frustrating, because you know they’re never going to be in the language)

Again, I state these without mentioning what it is I want, which is indeed the point of this post.

Without further ado, I present a proposed implementation for my (.NET) compiler which I have dubbed Quenya for the time-being. I intended this compiler to be syntax agnostic (or at the very least, extremely extensible) and therefore the bulk of the idea revolves around a way for users to re-define source code and for the compiler to be able to parse it.

Before we begin, I would like to point you to this awesome post by Luke Hoban, who is one of my favorite blog authors ever. I have used the source code for his monadic parser to build Quenya’s parsing logic.

To allow Quenya to parse something, it is not enough to provide it with source code. Quenya needs a grammar/parser (or a reference to one) and the source to parse. The parser is then expected to return a set of known AST nodes when called upon to parse the source code (linked to that grammar/parser). The rest of the compiler phases then proceed as usual and the compiler spits out IL. The figure below shows an overview of the parsing process in Quenya.

Quenya parsing overview

But it’s probably more cumbersome to write an entire parser each time you want to add or modify a single feature of the base grammar you’re using right? That’s where the monadic parser written in C# comes in. To illustrate, let’s take a simple example. We’ll write a simple parser that can parse words (alphabets) given a string and then extend it to have more features without rewriting the entire parser.

// A simpleton parser, one that doesn't understand the . (period) and that will fail if it encounters one.
public abstract class SimpleSentenceParser: CharParsers
{
    // Defines the set of all whitespace characters
	private Parser _whitespaceChars;
    public virtual Parser WhitespaceChars
    {
        get
        {
            return _whitespaceChars ??
                   (_whitespaceChars = Char(' ').OR(Char('\t').OR(Char('\n')).OR(Char('\r'))));
        }
    }
 
    // A whitespace is 0 or more repetitions of a valid Whitespace character
	private Parser _whitespace;
    public virtual Parser Whitespace
    {
        get
        {
            return _whitespace ??
                   (_whitespace = Rep(WhitespaceChars));
        }
    }
 
    // A word is multiple (alphabetic) characters that are not whitespaces
	private Parser _word;
    public virtual Parser Word
    {
        get
        {
            return _word ??
                   (_word = from w in Whitespace
                            from c in Char(char.IsLetter)
                            from word in Rep(Char(char.IsLetter))
                            select new WordTerm(word.Aggregate(new string(new[] {c}), (acc, ch) => acc + ch)));
        }
    }
 
    // A sentence is a set of words - simplistic, but will work for single sentences
	private Parser _sentence;
    public virtual Parser Sentence
    {
        get
        {
            return _sentence ??
                   (_sentence = from words in Rep(Word)
                                select new SentenceTerm(words));
        }
    }
 
    // Set of all sentences in the input (start symbol for this grammar)
	private Parser _all;
    public virtual Parser All
    {
        get
        {
            return _all ??
                   (_all = from sentence in Sentence
                           select (Term) sentence);
        }
    }
}
 
// A concrete implementation of the above parser that can accept string input
public class SimpleSentenceParser: SimpleSentenceParser
{
    public override Parser AnyChar // Pick one character at a time
    {
        get
        {
            return input => input.Length > 0 ? 
				new Result(input[0], input.Substring(1)) : 
				null;
        }
    }
}

Note that the code above is quite readable (as a grammar) due to the use of C# query expressions. Now let’s look at a parser that “extends” the capabilities of this parser.

// A slightly more intelligent parser, it can identify sentences properly
// Note that this parser does NOT redefine what a word means. It only redefines a sentence.
public abstract class BetterSentenceParser: SimpleSentenceParser
{
    private Parser _whitespaceChars;
    public override Parser WhitespaceChars // Adds to existing whitespace characters
    {
        get
        {
            return _whitespaceChars ??
                (_whitespaceChars = base.WhitespaceChars.OR(Char(',')).OR(Char(';')));
        }
    }
 
    // New, defines sentence terminators to allow this parser to understand them
	private Parser _sentenceTerminators;
    public virtual Parser SentenceTerminators
    {
        get
        {
            return _sentenceTerminators ??
                   (_sentenceTerminators = Char('.'));
        }
    }
 
    // Redefines what a sentence means and takes sentence terminators
	// into account
	private Parser _sentence;
    public override Parser Sentence
    {
        get
        {
            return _sentence ??
                   (_sentence = from words in Rep(Word)
                                from terminator in SentenceTerminators
                                select new SentenceTerm(words));
        }
    }
 
    // This now returns multiple sentences
	private Parser _all;
    public override Parser All // Actually selects multiple sentences
    {
        get
        {
            return _all ??
                   (_all = from sentences in Rep(Sentence)
                           select (Term)new ParagraphTerm(sentences));
        }
    }
}
 
// A concrete implementation of a parser that can accept string input
public class BetterSentenceParser : BetterSentenceParser
{
    public override Parser AnyChar
    {
        get
        {
            return input => input.Length > 0 ? new Result(input[0], input.Substring(1)) : null;
        }
    }
}

Ah, that looks fairly simple! The observant among you no doubt noticed that this parser extends (Sentence) and modifies existing behavior (WhitespaceChars) without having to do all the work by itself. Using the concrete version of these two parsers, we can now parse stuff!

internal class Program
{
    private const string SimpleText = @"Lorem ipsum dolor sit amet";
 
    private const string ComplexText =
        @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus enim lorem," +
        @" tincidunt ac sollicitudin fringilla, vulputate a dolor. Sed eu viverra ligula.";
 
    private static void Main(string[] args)
    {
        // Will parse SimpleText into multiple words and one sentence
		{
			var parser = new SimpleSentenceParser();
			var result = parser.All (SimpleText);
		}
 
        // Will parse ComplexText partially, fail when it encounters the '.'
		{
			var parser = new SimpleSentenceParser();
			var result = parser.All (ComplexText);
		}
 
        // Will parse ComplexText into multiple words and multiple sentences
		{
			var parser = new ComplexSentenceParser();
			var result = parser.All (ComplexText);
		}
    }
}

Let’s now extend this concept given a fully functional C# parser definition. The user will now be able to “modify” the syntax of the language by defining his own parser that changes certain productions and/or terminals in the grammar. Add in a mechanism to link a given source file to a given grammar/parser (attribute on the parser = file extension of source) and you have Quenya, a language that encourages people to define and modify it.

Granted, this is probably tougher than just learning one language, but considering that the code you write using your DSL or modified base language will be much, much more suited to your purposes (if you use it wisely) and that most of your code will be written in the straight-up base grammar for your language makes this worth it (to me, at least – I’d be happy to hear your thought/comments). Also, people who don’t care about language extensibility can just use a vanilla grammar for the language of their choice and continue doing what they’ve been doing.

The next post on Quenya will deal with the implications of a language like this and how it can be used (or abused) in a real context.

Download the source code here: QuenyaParsing

I just published my version of .NET bindings for OpenCL over on Codeplex last night. Why another, you ask? There’s already so many out there … Cloo, OpenCL.NET from hoopoe and another OpenCL.Net over at Sourceforge (and more?). Well, …

• Every API out there has an object-oriented version of the API that’s easily usable from .NET. Sure, it’s easy … but looks very different from the raw OpenCL API if I wanted to use it, right? There’s reason #1 right there. I wanted a set of .NET bindings that look, feel and act like the C-style API so I (read: the users) could port existing samples or read books/tutorials and understand what’s going on.
• Which brings us to our second question: Don’t all of the mentioned bindings also have the C-style API exposed? Sure, they do. But some of them mark the exposed API as unsafe, use of which would force users to mark their own projects as unsafe too. Secondly, every set of bindings I saw had raw translations of the API transliterating <anything>* to IntPtr (and others along those lines) meaning we have to use their OO abstractions to get anything done easily. I wanted something more .NET friendly at the API level, however.

So I guess the reason I wrote my own bindings is: I wanted to strike the middle ground between OO + .NET friendly and raw API translation. You can find this over at Codeplex. The API isn’t complete yet, but some things I’m particularly happy about are:

1. No unsafe code (yet).
2. Everything is .NET friendly AND you can use it as you would the raw API (overloads).
3. As little explicit marshaling as possible, in fact all the exposed methods that invoke the extern’d methods in OpenCL.dll are only one call/line.

Initially, I started with these bindings for my project Brahma, but decided it was a large enough effort to put up separately. I hope people find it useful alongside the other tools for OpenCL for .NET that are already available. I hope to have the API completed and released soon. Stay tuned.

Whenever I sit down to write an application (mostly a console application), I find myself wishing I had some code I could just drop in to parse command line arguments with. A friend of mine has authored the excellent ConsoleFX library, and it’s really good if you want to do heavy processing, validations and complex command line argument structures with help. Most of the time, however, I don’t.

So I spent the better part of my evening writing something that I hope I will be able to carry around as a code snippet small enough to use anytime, anywhere in an elegant fashion.

#region CommandLine



static class CommandLine


{


	public class Switch // Class that encapsulates switch data. Not meant for direct use.


	{


		public Switch(string name, Action<IEnumerable<string>> handler, string shortForm = null)


		{


			Name = name;


			Handler = handler;


			ShortForm = shortForm;


		}



		public string Name


		{


			get;


			private set;


		}


		public string ShortForm


		{


			get; private set;


		}


		public Action<IEnumerable<string>> Handler


		{


			get; private set;


		}



		public int InvokeHandler(string[] values)


		{


			Handler(values);


			return 1;


		}


	}



	// The regex that extracts names and comma-separated values for switches 


	// in the form (<switch>[:value,value,value])+


	private static readonly Regex _regex =


		new Regex(@"s*(?<switch>(?<name>w+)(:s*(?<value>w+(s*,s*w+)*))?)",


			RegexOptions.Compiled | RegexOptions.CultureInvariant |


			RegexOptions.ExplicitCapture | RegexOptions.IgnoreCase);



	private const string NameGroup = "name"; // Names of capture groups


	private const string ValueGroup = "value";



	public static void Process(this string[] args, Action printUsage, params Switch[] switches)


	{


		// Run through all matches in the concatenated argument list and if any


		// of the switches match, get the values and invoke the handler we were


		// given. We do a Sum() here for 2 reasons; a) To actually run the handlers


		// and b) see if any were invoked at all (each returns 1 if invoked).


		// If none were invoked, we simply invoke the printUsage handler.


		if ((from Match match in _regex.Matches(string.Join(" ", args))


			 from arg in switches


			 where match.Success &&


				   ((string.Compare(match.Groups[NameGroup].Value, arg.Name, true) == 0) ||


					(string.Compare(match.Groups[NameGroup].Value, arg.ShortForm, true) == 0))


			 select arg.InvokeHandler(match.Groups[ValueGroup].Value.Split(','))).Sum() == 0)


			printUsage(); // We didn't find any switches


	}


}



#endregion

All of the magic happens inside the Process method, where the query tries to match the arguments against the ones we want to check for. The .Sum() aggregate method does two things.

1. It enumerates the elements of the IQueryable, actually calling the handlers in the process. Nothing happens till then!
2. Each of the InvokeHandler calls returns 1, so the result of the .Sum() is 0 if there were no matches (which is when we call the supplied printUsage handler).

Neat, huh? Using it is even simpler!

            args.Process(


                () => Console.WriteLine("Usage is switch0:value switch:value switch2"),


                new CommandLine.Switch("switch0",


                    val => Console.WriteLine("switch 0 with value {0}",


                        string.Join(" ", val))),



                new CommandLine.Switch("switch1",


                    val => Console.WriteLine("switch 1 with value {0}",


                        string.Join(" ", val)), "s1"),



                new CommandLine.Switch("switch2",


                    val => Console.WriteLine("switch 2 with value {0}",


                        string.Join(" ", val))));


        }

Any handlers you provide will be called for each switch or it’s shortform(if found, of course). Now get started writing those fantastic console applications.

Obligatory disclaimer and License: The code above is for you to do whatever you want to. Even if it blows you up (like Dr. Manhattan blows Rorschach up), don’t come back from the dead to haunt me. I would, however appreciate it if you kept the comment attributing that code to me.

What does Windows 7 have to do with music? Nothing, really. But I’ve noticed that no one has been enterprising enough to put the Window 7 taskbar features to REALLY good use and make an mp3 player that we can use while we work (WMP team, are you listening?). I mean, who doesn’t listen to music while working? Everyone who raised their hands can leave the room now. Just kidding.

Onto the good stuff. Lyre sits in your taskbar, updates the application icon with the album art (if present), shows play progress on the taskbar button and provides all the basic play controls as thumb-buttons on the taskbar. There are no visualizations yet, but they should be easy enough to do if there’s enough demand for it. Here are some screenshots:

Here’s the player with controls, progress and tag-reading.

You can control the player by using it’s jump list.

True, this project only took me a week to complete (to this level), but there were a lot of unknowns and problems with the managed Windows7 integration wrapper. Well, I guess I should write a blog post on those later

Download it here, give it a whirl and let me know what you think.

Lyre-setup.zip

One of the features I spent a lot of time thinking about, and implementing are: “deferred” ProvideValue calls on MarkupExtensions. In a XAML markup extension, ProvideValue is called when the tree is being loaded, and sometimes, it may be impossible to provide a value at that time. Also, since XAML makes use of internal classes like ResourceReferenceExpression (along with some other quirks like requiring derivation from FrameworkElement. More on this here), it becomes quite a messy task resolving these references and values later.

In Xoml, however, every markup extension can tell XomlLoader to defer its ProvideValue call, by setting defer to true.

public abstract object ProvideValue(object context, PropertyInfo property, out bool defer);

Of course, at this point the markup extension is also expected to provide a default value, one that can be assigned to the context without errors. It is important to remember that the “context” can also be other markup extensions, in the case of a nested sub-expression.

After the entire object tree has been loaded, DeferredProvideValue

public virtual object DeferredProvideValue(object root, object context, PropertyInfo property)

is called once for each ProvideValue call that was deferred. Since the tree has been loaded now, the partially-initialized(?) root object is also passed in. XomlLoader then calls the OnAfterLoad method on each markup extension that was found.

protected internal virtual void OnAfterLoad(AfterLoadEventArgs e)

This calling order is undefined, and extensions that have not been used in the currently loaded XOML are not called upon for OnAfterLoad processing. OnAfterLoad is called exactly once for each type of extension, its calling order is undefined. Like DeferredProvideValue, the root of the tree is provided to this method. The difference here, however is that OnAfterLoad can return a different object back to XomlLoader (through AfterLoadEventArgs), thus changing the whole tree! I’m not sure if this is an irresponsible thing to do, I thought it might help do radical things to the object tree and make it more extensible.

I would love to hear your thoughts on the features I’ve currently implemented, and future directions for development.

For a long time, I’ve been writing about how messed up Xaml is because of WPF-interdependency, and how Microsoft should have decoupled the Xaml bits from WPF, etc (Some of these rants were on my old blog). Well, I finally decided to do something about it, and I’m writing a XAML-like library that loads, and writes markup using its own XomlLoader and XomlWriter. At this point I’m calling the markup eXtensible Object Markup Language (I know that the Workflow Foundations’ designer files have that extension…) because that’s what its meant to be. It is also aimed to run on the .NET Framework, Mono and the .NET CF.

Using Xoml, you will be able to data-bind, forward reference, resource reference, write markup-extensions and more. I will be putting this project up for download shortly, look out for more news on it.

PS: If there’s enough interest, it will have its own mini-site here!