LINQ Explained - Index

For your convenience, I am adding the following index of my on going series on Language Integrated Query or LINQ.

LINQ Explained - 1: Introduction to LINQ
LINQ Explained - 2: Features used by LINQ
LINQ Explained - 3: Some more features used by LINQ
LINQ Explained - 4: LINQ Syntax
LINQ Explained - 5: LINQ to SQL
LINQ Explained - 6: LINQ to Objects
LINQ Explained - 7: LINQ to XML

LINQ Explained – Part 4

This is the fourth part of my on-going series on Language Integrated Query or LINQ. I have been away from this series for a while. However my following posts (including this one) are aimed at completing this series. In the second and third posts, we had an overview of the features which are important to understand to have a full grasp of LINQ. In this post, we will have a detailed look at LINQ syntax and explore its different features. So let us get started.


Introduction

LINQ is a set of language extensions added to C# and VB.NET which makes queries a first-class concept to these languages. It provides a unified programming model to different data domains for data management. As I mentioned in my first post, using LINQ we can query and operate on different data domains including relational databases, XML, custom entities, DataSets or any third party data source. Above all, the concept of queries is now applicable to in-memory data as opposed to using queries with a persistent medium only. From a developer’s point of view, the interface to each data domain remains the same. But the LINQ engine is responsible for converting the queries to target the domain being referenced.

Since LINQ is a first-class concept in C# and VB.NET, these languages come loaded with support for LINQ. LINQ queries take advantage of features including IntelliSence and compile-time syntax checking. LINQ queries rely on standard query operators (discussed shortly) which are a set of functions used to fetch, parse, sort and filter the data.

Query Expression and Method-Based Queries

A LINQ query is a reminiscent of standard SQL-Query syntax. The purpose of LINQ is to add data querying capabilities to the .NET Framework such that any data-domain can be processed with the same ease. LINQ relies on concepts including Extension Methods, Anonymous Types, Anonymous Methods and Lambda Expressions discussed in earlier posts.

A LINQ Query is also known as Query Expression or Query Syntax. A query expression is a declarative syntax for writing queries which allows us to filter, group and order data. According to MSDN, “a query expression operates on one or more information sources by applying one or more query operators from either the standard query operators or domain-specific operators”. This means that LINQ can operate on different data-domains using operators specific to LINQ or developed by a third party. To me this is a polymorphic behavior of LINQ. The result of a query expression is an in-memory sequence of elements (or objects). Any object which implements the IEnumerable <T> interface is a sequence. The resultant sequence can be iterated through by built-in language iterators.

Let us see a simple example in listing 1 which shows a LINQ Query in action:

Listing 1

int [] prime = new int [8] {1, 3, 5, 7, 11, 13, 17, 19}; // Line 1

var primeNumbers =  // Line 2
        from p in prime      // Line 3
        where p > 0         // Line 4
        select p;            // Line 5

foreach (int p in primeNumbers) // Line 6
{
    // use p
}

In the above listing, a list of prime numbers is queried and the result is assigned to a variable primeNumbers. The LINQ Query (line 2–5) resembles a SQL statement with the standard from, where and select clauses. But it is being applied to an in-memory collection of numbers rather than a persistent medium (such as database or XML). Although simple but I am sure you can visualize the strength of LINQ upfront from the above example. The rest of the example is what we talked about (var, foreach) in the previous post.

A query expression can also be represented by a Method-based Query syntax. A method-based query utilizes extension methods and lambda expression. It is a rather concise way of writing query expressions. There is no performance difference between the two. A query expression is more readable while a method-based query is concise to write. It really comes down to your choice of syntax. But do keep in mind that all query expressions are translated into method-based queries. Using method-based query, listing 1 can be written as following:

Listing 2

IEnumerable <int> primeNumbers =
                                    prime           
                                    .Where (p => p > 0)
                                    .Select  (p => p);

Avid readers must have figured out the reason for applying the foreach loop on the variable primeNumbers in listing 1. This is because the type primeNumbers is converted to IEnumerable <T> which represents a collection of elements. This collection be iterated using foreach loop.

I am sure by now you can spot many of the features explained in part 2 and part 3 of this series. The above queries are just making use of concepts including var keyword, extension methods, lambda expressions and enumerators.


LINQ Syntax

LINQ is a reminiscent of standard SQL Language and thus has a sharp resemblance to it. Like its counterpart, query expression consists of clauses. There are three main clauses in a LINQ expression including from, select and the where clause. The general syntax of a LINQ query is as following:

var [query] = from …
where …
select …

The first clause in a LINQ Query is the from clause. You may be wondering why a LINQ query begins with the from clause unlike a standard SQL query which is preceded by the select clause. The reason for this precedence is to support Intellisence when working with Visual Studio IDE. Since the from clause specifies the data source ahead of the query, the compiler becomes data-source aware and hence supports Intellisence. The from clause is then followed by the where and select clauses. You can find the full syntax of a LINQ Query here .

Let us see listing 3 to analyze a LINQ query piece by piece. We start by defining a simple class followed by object initialization:

Listing 3

public class Car
{
        public string Type { get; set; }
        public string Color { get; set; }
}

Car[] cars = 
{
        new Car { Type = "SUV",   Color = "Green" }, 
        new Car { Type = "SUV",   Color = "Black" }, 
        new Car { Type = "4x4",   Color = "Red" }, 
        new Car { Type = "Truck", Color = "Orange" }, 
        new Car { Type = "Jeep",  Color = "Black"}
};      

Now that we have an array of cars with their properties initialized, we use a LINQ Query to find all the cars with a specific make:

IEnumerable<Car> search =
            from myCar in cars
            where myCar.Type == "SUV"
            select myCar;

The query begins with the from clause. A from clause only operates on sequences implementing the IEnumerable interface. This clause is actually made up of two parts; from and in. The in part points to the source-sequence which must be of type IEnumerable while the from part is a variable used for iterating through the source-sequence.

Next is the where clause used for filtering. Behind the scene, this clause is converted to Where Query Operator which is a member of the Enumerable class. This method accepts a lambda expression as parameter to apply the filter.

Next in the sequence is the select clause. This clause defines an expression which is assigned to a variable. The expression can be of any type including an instance of a class, string, number, boolean etc. Indeed this clause lets a type be created on the fly and assigned to a variable.

Finally, we can iterate through the variable ‘search’ since it is of type IEnumerable using the following code:

foreach (Car c in search)
{
// use c.Type, c.Color
}

Standard Query Operators

So far we have hardly scratched the surface of LINQ syntax and have seen some very simple LINQ queries, but in reality; the discussion of query expressions is incomplete without Standard Query Operators. The standard query operators represent an API defined in the Enumerable and Queryable classes under the System.Linq namespace. These operators are extension methods which accept lambda expressions as argument. These operators operate on sequences where any object which implements the IEnumerable<T> interface qualifies for a sequence. These operators are used to traverse, filter, sort, order and perform various functions on the given data. In other words they provide many of the features of a standard SQL Query including Distinct, Group, Set, Order By, Select etc.

I have stated above that a query expression is converted to a method-based query. In a method-based query, a Clause is converted to its respective Standard Query Operator (an extension method) . For example, the where clause is converted to a Where operator while the select clause is converted to a Select Operator. To keep it simple, just remember that the same clause in a query expression is represented by an operator when converted to a method-based query.

According to LINQ’s official documentation, Standard Query Operators can be categorized into the following:

• Restriction operators
• Projection operators
• Partitioning operators
• Join operators
• Concatenation operator
• Ordering operators
• Grouping operators
• Set operators
• Conversion operators
• Equality operator
• Element operators
• Generation operators
• Aggregate operators

A detailed explanation of each of the above is beyond the scope of this post. However, in the following sections, we will look at some of these operators and their use.

Select / SelectMany – Projection Operators

A Select operator performs a projection over a sequence and returns an object of type IEnumerable<T>. When this object is enumerated, it enumerates through the source sequence and produces one output element for each item in the sequence. The signature of the Select operator is as following:

public static IEnumerable<S> Select<T, S> (
this IEnumerable<T> source,
Func<T, S> selector);
public static IEnumerable<S> Select<T, S> (
this IEnumerable<T> source,
Func<T, int, S> selector);

The first argument of the selector predicate is the source sequence while the selector argument is a zero-based index of elements within the source sequence. I will skip an example for this operator as all the above examples make use of this operator :)

The SelectMany operator is used with nested sequences or in other words sequence of sequences. It merges all the sub-sequences into one single enumerable sequence. The SelectMany operator first enumerates the source sequence and converts its respective sub-sequence into an enumerable object. It then enumerates each element in the enumerable object to form a flat sequence. The operator has the following signature:

public static IEnumerable<S> SelectMany<T, S>(
this IEnumerable<T> source,
Func<T, IEnumerable<S> > selector);
public static IEnumerable<S> SelectMany<T, S>(
this IEnumerable<T> source,
Func<T, int, IEnumerable<S>> selector);

The source is the sequence to be enumerated. The selector predicate represents the function that that is applied to each element in the sequence.

Listing 4 shows the use of the SelectMany operator:

Listing 4

public class Region
{
     public int RegionID;
     public string RegionName;
     public List<Product> Products;
}

public class Product
{
    public string ProductCode;
    public string ProductName;
}

Now we will initialize a list of type Region with a child object of type Product:

List<Region> products = new List<Region>
        {
            new Region { RegionID = 1, RegionName = "North",
                            Products = new List<Product> { 
                                    new Product { ProductCode = "EG", ProductName = "Eggs" },
                                    new Product { ProductCode = "OJ", ProductName = "Orange Juice" },
                                    new Product { ProductCode = "BR", ProductName = "Bread" }
                            }
            },

            new Region { RegionID = 2, RegionName = "South",
                            Products = new List<Product> { 
                                    new Product { ProductCode = "CR", ProductName = "Cereal" },
                                    new Product { ProductCode = "HO", ProductName = "Honey" },
                                    new Product { ProductCode = "MI", ProductName = "Milk" },
                            }
            },

            new Region { RegionID = 3, RegionName = "East",
                            Products = new List<Product> { 
                                    new Product { ProductCode = "SO", ProductName = "Soap" },
                                    new Product { ProductCode = "BS", ProductName = "Biscuits" },
                            }
            }
        };

Now we apply the SelectMany operator to select products from the North and East region:

var ProductsByRegion =
            products
            .Where (p => p.RegionName == "North" || p.RegionName == "East")
            .SelectMany (p => p.Products); // using SelectMany operator

foreach (var product in ProductsByRegion)
{
    string code, name;

    code = product.ProductCode;
    name = product.ProductName;
    // use code & name
}

The above listing begins by defining two classes, Region and Product. The region class consists of a child collection property Products. Next a list of regions is initialized such that each Region in turn has multiple products. The LINQ Query is applied using the SelectMany operator. This will flat out the sub-lists (products in this case) for the selected regions (North and East) and create a single sequence to be iterated. If you run the above code, you get the following result:

EG: Eggs, OJ: Orange Juice, BR: Bread, SO: Soap, BS: Biscuits


Where– Restriction Operator

The Where operator, also known as restriction operator, filters a sequence based on a condition. The condition is provided as a predicate. The Where operator enumerates the source sequence and yield those elements for which the predicate returns true. We can also use ‘where’ keyword in place of the Where operator. The signature for the Where operator is as follows:

public static IEnumerable<TSource> Where<TSource>(
this IEnumerable<TSource> source,
Func<TSource, bool> predicate);
public static IEnumerable<TSource> Where<TSource>(
this IEnumerable<TSource> source,
Func<TSource, int, bool> predicate);

The source represents the sequence to be enumerated while the predicate defines the condition or filter to be applied on the given sequence.


Join / GroupJoin - Join Operators

The Join operator is a counter part of Inner Join used in SQL Server and serves the same purpose. It returns a sequence of elements from two different sequences with matching keys. This operator has the following signature:

public static IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult> (
this IEnumerable<TOuter> outer,
IEnumerable<TInner> inner,
Func<TOuter, TKey> outerKeySelector,
Func<TInner, TKey> innerKeySelector,
Func<TOuter, TInner, TResult> resultSelector);
public static IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult> (
this IEnumerable<TOuter> outer,
IEnumerable<TInner> inner,
Func<TOuter, TKey> outerKeySelector,
Func<TInner, TKey> innerKeySelector,
Func<TOuter, TInner, TResult> resultSelector,
IEqualityComparer<TKey> comparer);

In the above overloads, outer and inner represent the two source sequences. The predicates outerKeySelector and innerKeySelector represent the keys on which the join will be performed. They should be of the same type. The resultSelector predicate represents the projected result for the final output. In the second overload, we can also use a custom comparer to perform the join between the two sequences based on custom logic.

Listing 5 shows the use of Join operator joins two different lists based on a condition:

Listing 5

public class Developer
{
    public string name {get; set; }
    public string projecttitle { get; set; }
}

public class Project
{
    public string title { get; set; }
    public int manDays {get; set; }
    public string company { get; set; }
}

List<Developer> developers =
                new List<Developer>
                {
                    new Developer { name = "Steven", projecttitle = "ImageProcessing" },
                    new Developer { name = "Markus", projecttitle = "ImageProcessing" },
                    new Developer { name = "Matt",   projecttitle = "ImageProcessing" },
                    new Developer { name = "Shaza",  projecttitle = "GraphicsLibrary" },
                    new Developer { name = "Neil",   projecttitle = "WebArt" },
                };

        List<Project> projects =
                new List<Project>
                {
                    new Project { title = "ImageProcessing",    company = "Future Vision",  manDays = 120 },
                    new Project { title = "DatabaseFusion",     company = "Open Space",     manDays = 30 },
                    new Project { title = "GraphicsLibrary",    company = "Kid Zone",       manDays = 88  },
                    new Project { title = "WebArt",             company = "Web Ideas",      manDays = 57 },
                    new Project { title = "GamingZone",         company = "Play with Us",   manDays = 50},
                };

        var ProjectDetails =
            from dev in developers
            join pro in projects
            on dev.projecttitle
            equals pro.title
            select new
            {
                Programmer = dev.name,
                ProjectName = pro.title,
                Company = pro.company,
                ManHours = pro.manDays
            };

        foreach (var detail in ProjectDetails)
        {
            // use detail.Programmer, detail.Company, detail.ProjectName, detail.ManHours
        }

In the above code, the one thing to notice is the use of ‘equals’ rather then the ‘=’ sign. This is different from what we use in regular sql join statement. The example returns a sequence with matching ‘titiles’ from both sequences.

The above example works well for a 1:1 mapping between keys. However, if we wanted information on all ‘projects’ irrespective of any matching ‘developer’ then the above query doesn’t work. In a sql environment, a left join would do the trick since it will return all ‘projects’ and matching ‘developer(s)’. However it will return a ‘null’ for all ‘developers’ which do not have an associated ‘project’. In case of LINQ, the same purpose is surved by the GroupJoin operator.

The GroupJoin operator does not return a single sequence of elements returns a hierarchical sequence of elements. This sequence consists of one element each from the outer sequence. For each element in return, matching elements from the inner sequnce are grouped and attached with it. So it represents a hierarchical grouping of all elements from the outer sequence with each having a child-sequence (grouped together) of matching values from the inner sequence. This operator has the following signature:

public static IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey,
TResult> (
this IEnumerable<TOuter> outer,
IEnumerable<TInner> inner,
Func<TOuter, TKey> outerKeySelector,
Func<TInner, TKey> innerKeySelector,
Func<TOuter, IEnumerable<TInner>, TResult> resultSelector);
public static IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey,
TResult> (
this IEnumerable<TOuter> outer,
IEnumerable<TInner> inner,
Func<TOuter, TKey> outerKeySelector,
Func<TInner, TKey> innerKeySelector,
Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,
IEqualityComparer<TKey> comparer);

In the above overload, the arguments are similar to the one for the Join operator but how it works is interested. When the sequence returned by GroupJoin is iterated, it first enumerates through the inner sequence based on the innerKeySelector and groups them together. Grouping elements are stored in a hash table against they key. Next elements from the outer sequence are enumerated based on the outerKeySelector. For each element, matching elements from the hash table are searched. If found, the sequence from the hash table is associated with the element in the outer sequence. This way we get a parent-child grouping of elements. Listing 6 shows how to use GroupJoin operator (the data sample is from listing 5):

Listing 6

var query = projects.GroupJoin( // outer sequence
                    developers,           // inner sequence 
                    p => p.title,          // outer key selector 
                    d => d.projecttitle, // inner key selector 
                    (p, g) => new
                    {     // result projector
                        ProjectTitle = p.title,
                        Programmers = g
                    });

foreach (var detail in query)
{
        // use detail.ProjectTitle
        foreach (var programmer in detail.Programmers)
        {
                // use programmer.name, programmer.projecttitle;
        }
}

You must have noticed that we are using nested loops to access the elements. This is further proof that the elements are arranged in a parent-child hierarchy such that for each element in the outer sequence, we have matching elements (grouped together) from the inner sequence. The above example will produce the following resultset where all the ‘projects’ are displayed irrespective of a developer(s) assigned to them:

ImageProcessing
Steven, Markus, Matt
DatabaseFusion
GraphicsLibrary
Shaza
WebArt
Neil
GamingZone


OrderBy..ThenBy / OrderByDescending..ThenByDescending - Ordering Operators

The OrderBy operator is used for ordering the elements in a sequence by one or more keys. It also determines the direction of the order i.e. in an ascending order. The signature of this operator is as following:

public static IOrderedSequence<T> OrderBy<T, K>(
this IEnumerable<T> source,
Func<T, K> keySelector);
public static IOrderedSequence<T> OrderBy<T, K>(
this IEnumerable<T> source,
Func<T, K> keySelector,
IComparer<K> comparer);

In both the overloads, The source is the source sequence on which the operator will operator. The keySelector represents a function that extracts a key of type K from each element of type T from the source sequence. In the second overload, comparer is a custom comparer where we can write custom code to perform the comparison.

You must have noticed that the return type of this operator is IOrderedSequence and not IEnumerable. Before I explain this, let me mention that the OrderBy operator is supported by the ThenBy operator. In a regular sql command, we can order the resultset by any number of fields in addition to the primary field. The same concept is supported by the ThenBy operator. The primary ordering is done by the OrderBy operator followed by the ThenBy operator. The ThenBy operator defines the seconary ordering and can be used n-number of times in a LINQ Query. Both operators together work as following:

source-sequence.OrderBy ().ThenBy ().ThenBy ()…

The above shows that the output of OrderBy is input to the ThenBy operator. Going back to the return type of IOrderedSequence, the ThenBy operator can only be applied to IOrderedSequence and not IEnumerable<T>. For this reason, the return type of OrderBy operator is IOrderedSequence. The example in listing 7 will sort the projects by manHours (OrderBy) and then by title (ThenBy):

Listing 7

List<Project> projects =
                new List<Project>
                {
                    new Project { title = "ImageProcessing",    company = "Future Vision",  manDays = 120 },
                    new Project { title = "DatabaseFusion",     company = "Open Space",     manDays = 30 },
                    new Project { title = "GraphicsLibrary",    company = "Kid Zone",       manDays = 88  },
                    new Project { title = "WebArt",             company = "Web Ideas",      manDays = 57 },
                    new Project { title = "GamingZone",         company = "Play with Us",   manDays = 50},
                };

IEnumerable<Project> details =
    projects.OrderBy (p => p.manDays).ThenBy (p => p.title);

foreach (var project in details)
{
            // use project.manDays , project.title , project.company
}

The concept of ordering can also be achieved by using the OrderByDescending and ThenByDescending operators. In this case, as the name implies, the direction of ordering is descending. The signature of OrderByDescending operator is as following with identical arguments to OrderBy operator:

public static OrderedSequence<TSource> OrderByDescending<TSource, TKey>(
this IEnumerable<TSource> source,
Func<TSource, TKey> keySelector);
public static OrderedSequence<TSource> OrderByDescending<TSource, TKey>(
this IEnumerable<TSource> source,
Func<TSource, TKey> keySelector,
IComparer<TKey> comparer);


Distinct / Union - Set Operators

The Distinct operator removes duplicate items from a given sequece. It is just the counterpart of the Distinct keyword used in regular SQL statements. It has the following signature:

public static IEnumerable<TSource> Distinct<TSource>(
this IEnumerable<TSource> source);
public static IEnumerable<TSource> Distinct<TSource>(
this IEnumerable<TSource> source,
IEqualityComparer<TSource> comparer);

In both the overloads, the source is the sequence on which the operator will operator. Using the second overload, we can specify a custom comparer to compare an element. Listing 8 shows the use of the Distinct operator:

Listing 8

public class Fruit
    {
        public string name { get; set; }
    }

List<Fruit> fruits =
                new List<Fruit>
                {
                    new Fruit { name = "Orange"},
                    new Fruit { name = "Orange"},
                    new Fruit { name = "Orange"},
                    new Fruit { name = "Apple"},
                    new Fruit { name = "Apple"},
                    new Fruit { name = "Pappaya" }
                };

        var query =
            (from fruit in fruits
             select new { fruit.name}
            ).Distinct();

        foreach (var fruit in query)
        {
            // use fruit.name
        }

The output of the above query will be following:

Orange
Apple
Papaya

Another of the Set Operators includes the Union operator. A Union operator returns unique elements from two sequences while ignoring the duplicates. The Union operator has the following signature:

public static IEnumerable<TSource> Union<TSource>(
this IEnumerable<TSource> first,
IEnumerable<TSource> second);
public static IEnumerable<TSource> Union<TSource>(
this IEnumerable<TSource> first,
IEnumerable<TSource> second,
IEqualityComparer<TSource> comparer);

In both the overloads, the first and second represents the two sequences on which the Union operator is applied. The third parameter in the second overload is a custom comparer for comparison. Listing 9 shows the use of Union operator:

Listing 9

int[] A = { 1, 2, 3, 4, 5 };
int[] B = { 4, 5, 6, 7, 8};

var union = A.Union (B);

foreach (var n in union)
{
        // use n
}

The result of the above query will be 1, 2, 3, 4, 5, 6, 7, 8. It will ignore the duplicates 4 and 5 and yield one element each.



Summary

In this post, we had a look at the basic LINQ Syntax. A LINQ query is also known as a Query Expression. A query expression is a declarative way of writing query where we can perform different operations such as filtering, sorting, grouping etc. The yield of a query expression is a sequence.

Another way of writing a query expression is a Method-based Query which is just a concise way of writing LINQ Query. It makes use of Lambda Expression and Extension Methods. In the background, every query expression is converted to a method-based query. However there is no performance difference between the two and it comes down to the preference of usage.

A query expression makes use of Standard Query Operators. These operators represent an API defined in the System.Linq namespace. They operate on a sequence to perform different functions such as sorting, filtering, projection, grouping and much more.

With this we come to an end of this post. In the next post, you will see the use of LINQ in real world applications. We will begin with LINQ-to-SQL (a provider of LINQ) which is used to query relational databases. So stay tuned for more…

LINQ Explained – Part 3

This is the third part of my on-going series on LINQ. In the second part, we looked at some of the underlying C# features which are important to understand to work with LINQ. In this part, we will conclude with the rest of the features. So let us dive in straight.


Yield Statement

The Yield statement was introduced with C# 2.0. In my previous post, we talked about Enumerators. Enumerators help us iterate through collections and custom classes. Collections and custom classes must implement the IEnumerable interface. This interface has one method, GetEnumerator, which returns an instance of IEnumerator interface. The IEnemrator interface performs the actual iteration.

All this is pretty straight forward but does require some coding on behalf of the developers. Implementing IEnumerator interface for complex classes can be time consuming. Wouldn’t it be nice if the compiler could handle the enumeration process for us? Thanks to the yield statement, this is still possible.

Let me explain this concept by the help of the following simple example:

Listing 1

protected void Button1_Click (object sender, EventArgs e)
{
    foreach (string fruit in BuyFruits ()) // ref 1
    {
        ListBox1.Items.Add (fruit);
    }
}

 
private IEnumerable BuyFruits ()
{  
    string [] fruits = new string [] {"apple", "orange", "coconut", "papaya", 
                                      "mango"};

    for (int i = 0; i <= fruits.Length - 1; i++)
    {
        yield return fruits [i]; // ref 2
    }
}

If you have noticed, BuyFruits method has a return type of IEnumerable (reference type) but it returns a string (value type) using the yield return statement. Although BuyFruits is a method, it is acting as a class which performs iteration. This may look strange but under the hood, two things are happening.

First, the yield statement, a compiler directive, instructs the compiler to generate an inner (nested) class which implements the IEnumerator interface. It is this inner class which handles iteration for us. Fig 1 shows this class which has been generated using the ILDASM tool. The nested class is named as d__0 and implements the generic and non-generic versions of IEnumerable interface. It also implements the member functions including MoveNext, Reset method and Current property.

Fig 1



Second, the yield statement returns a single element at a time but maintains state between calls. This means that each subsequent call to yield return statement will return the next element in the collection. This is possible because the compiler maintains a state engine which resumes execution from the previously returned value.

The first time the yield statement is executed in the loop, a new object of the inner (mentioned above) class is created. This instance is used across the loop until it iterates through and reaches the end of the entire collection. Each yield return is delegated to the MoveNext method of the inner class. After the loop terminates, the instance of the inner class is also disposed. A new instance is created for each new call. This makes it type-safe across calls.

The code in Listing 1 works under the above principles. First, a class implementing the IEnumerator interface is generated (Fig 1). Next in the Button1_Click event, the foreach loop calls the BuyFruits method. The first call will create an instance of the generated class. Each yield return call is then delegated to the MoveNext method. This way it iterates through the entire collection.

We can also implement enumeration for a custom collection using the yield statement. Following is the code for the custom collection:

Listing 2

public class FruitCollection : IEnumerable
{ 
    private string[] fruits;

    public FruitCollection ()
    {
        fruits = new string[] {"banana", "apple", "mango", "apricot", 
                                          "kiwi"};
    }
    
    public IEnumerator GetEnumerator ()
    {
        foreach (string fruit in fruits)
        {
            yield return fruit;
        }
    }
}
protected void Button1_Click(object sender, EventArgs e)
{
    FruitCollection basket = new FruitCollection ();

    foreach (string fruit in basket)
    { 
        ListBox1.Items.Add (fruit);
    }
}

The FruitCollection class implements the IEnumerable interface so GetEnumerator method must be implemented. Within this method, a list of strings is iterated using the yield statement (Note: We can use any looping technique). As mentioned above, a private nested class is generated. This class performs enumeration on behalf of FruitCollection for each yield return statement. The rest of the process is the same as mentioned above. We can then iterate throught the collection as shown in the Button1_Click event.

The yield break statement is another dialect of the yield statement. This statement can stop iteration at any point in the loop. The following snippet will only return the first string in the array.

Listing 3

private IEnumerable BuyFruits()
{
    string[] fruits = new string[] {"apple", "orange", "coconut", "papaya", 
                                      "mango"};

    for (int i = 0; i <= fruits.Length - 1; i++)
    {            
        if (i > 1)
            yield break;

        yield return fruits[i];
    }
}

One last point to mention is that yield statement is equally applicable to generic and non-generic types. For generic types, the IEnumerable interface is used.


Extension Methods

Extension methods are one of the new features added to C# 3.0. According to MSDN “Extension method enable you to ‘add’ methods to existing types without creating a new derived type, recompiling or otherwise modifying the original type”. As the definition implies, we can add new functionality to existing types, primitive or custom.

Extension methods are a special breed of static methods which are invoked like regular instance methods. Extension methods can be added to existing primitive types, classes, structures and interfaces. These methods are static and are defined in a separate static class. Importantly, the first parameter in an extension method defines the type the method will operate on. This parameter must be preceded by this keyword. For example an extension method with first parameter as this string input is available for string data types and input represents the string which invoked the extension method.

Let us see a simple example of an extension method. The following method adds a greeting message to a string type:

Listing 4

public static class GreetingsClass
{ 
    public static string AddGreetings (this string name) //note the input parameter
    {
        return String.Concat ("We welcome you ", name);
    }
}

We can now invoke the AddGreetings method with the following code:

protected void Button1_Click(object sender, EventArgs e)
{
    string emp = "Employee";

    txtMessage.Text = emp.AddGreetings ();//invoked as a regular method
}

The extension method AddGreetings is defined in a separate static class. The first parameter of the method preceded by this keyword makes it an extension method. Using Intellisence, if we look at the list of available methods for the string ‘emp’, we can locate AddGreetings method along other methods.

Extension methods can also be added to existing .NET classes. In listing 5, an extension method is added to the built-in Stack class. The extension method finds all items in an integer stack which have a value greater than the input parameter.

Listing 5

public static class StackExtender
{
    public static int ItemsGreaterThanInput (this Stack stack, int maxValue)
    {
        int count = 0;

        foreach (object o in stack)
        {
            if (Convert.ToInt32(o) > maxValue)
                count++;
        }

        return count;
    }
}

protected void Button1_Click(object sender, EventArgs e)
{
    Stack intStack = new Stack ();

    intStack.Push(5);
    intStack.Push(4);
    intStack.Push(1);
    intStack.Push(8);

    TextBox1.Text = "Total integers greater than (2) = " + intStack.ItemsGreaterThanInput(2).ToString();
}

The above code is pretty straight forward but it has two parameters. The first parameter is the type (Stack) on which it operates. The second is a required parameter which should be provided when this method is invoked.


Var keyword

C# 3.0 introduced another useful feature, the var keyword which allows us to declare implicitly typed variables. An explicit type declaration of these variables is not required which is handled by the compiler. The following line declares a variable of type string which is inferred by the compiler:

var greet = “Hello World”; // no use of string type

It is important to note that these variable types are strongly typed. A proof of this is that the type of these variables cannot be changed later on. For example, the above statement will result in an error “Cannot implicitly convert type 'int' to 'string' if later used as:

greet = 10;

The var keyword can only be used with local variables. An attempt to use it with class level variables will result in an error ”The contextual keyword 'var' may only appear within a local variable declaration”. Since the compiler infers the type of these variables so they ‘must be’ initialized when declared.

The var keyword also rids us of extra type declaration. For example, the following code:

ArrayList alst = new ArrayList ();

can also be written as:

var alst = new ArrayList ();

Similarly, the value returned from a method can also be assigned to a ‘var’ variable. For example, the following method returns an ArrayList:

private ArrayList GetMyList()
{
    ArrayList alst = new ArrayList ();
    // use alst;
    return alst;
}

This method can be directly assigned to the ‘var’ variable:

var myList = GetList ();

If we change the implementation of the above method to return an instance of IList, the compiler is still smart enough to infer the type.


Anonymous Types

Anonymous types are a new feature added to C# 3.0. The idea is to create a type on the fly without actually declaring it. The compiler infers the type at runtime. This gives us a lot of flexibility in defining new types without actually declaring them.

How a type is created ‘just like that’ is based on the concept of object initialization. Object initialization was also introduced with C# 3.0. The idea is to initialize properties when creating objects without invoking the constructor. Let us consider the class in listing 6 (for those who are new, we can now use ‘automatic properties’ where a variable per property is not required):

Listing 6

public class Fruit
{
    // automatic properties
    public string FruitName { get; set; } 
    public int FruitPrice { get; set; }
}

The above class can be instantiated as following:

Fruit fruit = new Fruit ();
fruit.FruitName = “apple”;
fruit.FruitPrice = 10;

The above snippet is straightforward but can be time consuming for large classes. Using object initialization, we can reduce the amount of work required to initialize a class with the following syntax:

Fruit fruit = new Fruit {FruitName = “apple”, FruitPrice = 10 };

The concept can be further extended to generic types as following:

List<Fruit> basket = new List<Fruit>{
 new Fruit {FruitName = “apple”, FruitPrice = 10 },
 new Fruit {FruitName = “mango”, FruitPrice = 15}
};

Coming back to anonymous types, these are created at runtime. Creating an anonymous type is facilitated by ‘object initialization’ and ‘var’ keyword. The following snippet creates an anonymous type - Employee:

var Employee = new { Name = “Emp1”, Age = 35, Salary = 3000 };

We can even create a nested anonymous type using the same syntax. For example, the above code can be extended to add a nested type Address as:

var Employee = new { Name = “Emp1”, Age = 35, Salary = 3000,
Address = new { HouseNo = “H1”, Block = 3, City = “MyCity” }
};

Anonymous methods are extensively used in LINQ. The following post will describe this concept further.


Lambda Expressions and Anonymous Methods

Lambda Expressions are a new feature added to C# 3.0. They provide a handy way to write concise code. But before we jump on to this topic, it is important to understand anonymous methods.

C# 2.0 introduced the concept of ‘anonymous methods’. As the name implies, these are name-less methods which can be declared inline. The method is declared and implemented at the same place. The idea is to avoid defining a separate method which is not reused.

An anonymous method can be used implicitly at a place where a delegate is expected. As we know, when a delegate is called, the referenced method is invoked. In case of an anonymous method, the delegate is replaced by piece of code - inline. The anonymous method is defined using the delegate keyword followed by an optional list of parameters (within parenthesis) and the body of the method. If an anonymous method doesn’t have any parameters, we can omit the parameter parenthesis. An expression defining an anonymous method is known as anonymous-method-expression and has following syntax:

delegate (optional_signature) { // method body - code }

It is important to mention that the signature of the anonymous method must match with the delegate signature (this is how delegates basically work) being replaced. The return type and input parameter list must be the same.

Let us see a simple example of using anonymous methods. Delegates play a de-facto role in events-based programming. Consider the following code for an event:

protected void Page_Load (object sender, EventArgs e)
{
    Button1.Click += new EventHandler (Button1_Click);
}

protected void Button1_Click (object sender, EventArgs e)
{
    // implementation
}

Using anonymous method we can re-write the above as following:

Listing 7

protected void Page_Load (object sender, EventArgs e)
{
    Button1.Click += delegate (object sender1, EventArgs s) { /* implementation */ };
}

In the code above, the EventHandler delegate has been replaced by an anonymous method. If we use ILSADM tool to view the assembly, you will find a new private method with matching signature as illustrated in figure 2. Under the hood, it is this method (b__0) which is invoked by the compiler:


Fig 2



An anonymous method can be declared within a static or an instance method. This also determines the type of the anonymous method, though private. Let us see another example where we actually declare a delegate. This anonymous method will perform some (complex :-) computation:

Listing 8

public delegate int MathDelegate (int n1, int n2, int n3); // declare delegate

protected void Button1_Click (object sender, EventArgs e)
{
    int result;

    MathDelegate mathdel = delegate (int num1, int num2, int num3) /* anonymous type */
                                        {
                                            return (((num1 * num2) / num3) + 50); // any calculation
                                        };

    result              = (int) mathdel (10, 10, 2); // invoke delegate and cast reture-value
    TextBox1.Text = result.ToString ();
}

Usually for the above code to work, the delegate needs to reference a method with matching signature. However, in the above code, an anonymous method is defined inline to perform the calculation. Under the hood, again a private method is declared which is invoked by the compiler.

The above is a trivial example; however anonymous methods can be useful in many scenarios such as collections and custom classes. Let us see an example of using an anonymous method within a collection. We first define a custom (my favorite :-) Fruit class:

public class Fruit
{
    public Fruit (string n, string d)
    {
        FruitName        = n;
        FruitDescription = d;
    }

    // automatic-property
    public string FruitName
    {   get; set;   }
    
    // automatic-property
    public string FruitDescription
    {   get; set;   }
}

Next we define a generic collection of type Fruit. This collection will let us find a particular fruit using an anonymous method:

Listing 9

protected void Button1_Click (object sender, EventArgs e)
{
    string fruitName    = "kiwi";

    List<Fruit> fruits  = new List<Fruit> ();

    Fruit f1 = new Fruit ("mango", "A tropical fruit"); 
    Fruit f2 = new Fruit ("apple", "Have an apple a day"); 
    Fruit f3 = new Fruit ("kiwi", "Good for health");

    fruits.Add (f1);
    fruits.Add (f3);
    fruits.Add (f2);

    // using anonymous method
    Fruit searchFruit = fruits.Find (delegate (Fruit f)
                                        {
                                            return f.FruitName == fruitName; // out of scope
                                        });

    if (searchFruit != null)
    {
        txtname.Text        = searchFruit.FruitName;
        txtdescription.Text = searchFruit.FruitDescription;
    }
    else
    {
        Label1.Text = "Fruit not found...";
    }
}

The above example has a few points to note. First, the Find method expects a parameter of type Predicate. This parameter represents a delegate (a boolean expression) used by generic lists to filter and search an element. In the above code, the delegate gap is filled by an anonymous method.

Second, if you watch closely, the anonymous method accesses a local variable (fruitName) which is in the scope of the outer method. How this works is pretty interesting. Earlier in Figure 2 we saw that the compiler generated a method for an anonymous method. But for a variable outside the scope of the anonymous method, the compiler generates a class. The generated method and variable are members of this class. When the anonymous method is invoked, the compiler creates and invokes an instance of this class. All the local variables maintain their state across calls made by the same instance. Figure 3 illustrates this concept:


Fig 3



It is also important to note that an anonymous method cannot access ref or out variables of the outer method.

Returning back to Lambda Expressions, they offer a convenient way to write anonymous methods. Using lambda expressions, we can omit much of the syntactical requirement of an anonymous method. For example, using lambda expression, listing 7 can be re-written as:

protected void Page_Load (object sender, EventArgs e)
{
    Button1.Click += (object s, EventArgs ea) => { // implementation }
}

To understand how the above works, you should know that a lambda expression offers the following liberty to developers:

1. The delegate keyword is not required.
2. For a single statement, braces can be avoided. We use the lambda operator => (pronounced as goes to) in place of braces. The left side of this operator represents the input parameters while the right side is the expression block.
3. The return keyword is not required.
4. Since C# 3.0 supports type inference, it is perfectly alright to drop the type definition for variables and let the compiler infer it (a very strong feature of lambda expressions).

Using the above features, let us rewrite listing 8 as following:

public delegate int MathDelegate (int n1, int n2, int n3); // declare delegate

protected void Button1_Click(object sender, EventArgs e)
{
    int result;

    MathDelegate mathdel  = (num1, num2, num3) =>
                            (((num1 * num2) / num3) + 50); // lambda-expression

    result = (int) mathdel (10, 10, 2); // invoke delegate and cast reture-value
    TextBox1.Text = result.ToString ();

}

Clearly, the above is a neat and concise expression written using lambda expression. The delegate and return keyword are not omitted. Similarly braces have been avoided and the compiler infers the data type of the input parameters. The same applies to the code in listing 9.

To use a lambda expression, we need a delegate. .NET 3.5 facilitates us with two built-in generic delegate types, Func and Action, so that we don’t have to define our own delegates. The former returns a value while the later does not. In a Func delegate, the last parameter represents the return type. It has the following overloads:

public delegate TResult Func<TResult> ()
public delegate TResult Func<T, TResult> (T t)
public delegate TResult Func<T1, T2, TResult> (T1 t1, T2 t2)
public delegate TResult Func<T1, T2, T3, TResult> (T1 t1, T2 t2, T3 t3)
public delegate TResult Func<T1, T2, T3, T4, TResult> (T1 t1,T2 t2, T3 t3, T4 t4)

Similarly, an Action delegate accepts input parameter(s) but has a return type of void. It has the following overloads:

public delegate void Action();
public delegate void Action<T> (T t1);
public delegate void Action<T1, T2> (T1 t1, T2 t2);
public delegate void Action<T1, T2, T3> (T1 t1, T2 t2, T3 t3);
public delegate void Action<T1, T2, T3, T4> (T1 t1, T2 t2, T3 t3, T4 t4);

To help you understand the above, let me give you a simple example of using the Func delegate with three parameters. The Action delegate is no different (except with no return type).

Listing 10

Func<string, string, string /*return type */> GreetPerson = (message, person) => 
  message + " " + person;

protected void Button1_Click (object sender, EventArgs e)
{
 TextBox1.Text = GreetPerson (“Hello”, “Scott”);
}

I am sure you can evaluate the above code with ease. Func defines a delegate which accepts two input parameter (message, person) of type string and the last parameter, also a string, as the return type. Clearly, it has simplified the process of defining a delegate. Otherwise we had to define a delegate with matching signature.


Summary

In this post, we looked at different C# language features which make up LINQ. The yield statement lets us implement enumerators without implementing any enumerator interface. Also, the yield maintains state between calls. This is possible since the compiler maintains a state engine.

Extension methods enable us to add functionality to existing types. The types can be primitive or custom. Extension methods are static methods but are invoked like instance methods. These methods are defined in a separate static class. The first input parameter is preceded by ‘this’ keyword which makes it an extension method. This parameter also defines the type on which the extension method will operate.

The var keyword is used to declare implicitly typed local variables. These are strongly typed variables and their types cannot be changed later on. The compiler is responsible for determining the type at runtime. We can either explicitly assign a value or return a value from a method to a ‘var’ variable. These variables must be instantiated with a value so that the compiler can infer the type at runtime.

Anonymous Types facilitate us to create types without actually declaring it. The type is inferred by the compiler at runtime. Anonymous types in turn use a feature known as ‘object initialization’ to work. Object initialization lets us initialize properties without invoking the constructor, when creating objects.

Anonymous methods allow us to use a piece of code inline without defining a separate method. The code is declared and used at the same place. Anonymous methods can be used where a delegate is anticipated. An anonymous method is defined using the delegate keyword followed by an optional list of parameters (within parenthesis) and the body of the method. If an anonymous method doesn’t have any parameters, parenthesis can be omitted. The signature of anonymous method must match with the signature of the delegate being replaced.

Lambda expressions offer a concise way to write anonymous methods. Using lambda expression, we can avoid the extra syntactical requirement of an anonymous method. We can omit the delegate and return keyword. Also, the compiler can infer the data type of the input parameters. To use a lambda expression, a delegate is required. .NET framework has two built in function, ‘Action’ and ‘Func’ respectively, which help us use a lambda expression without actually defining it.

With this we come to the end of this post. To concentrate more on LINQ, I had to sum up all the above concepts in two posts otherwise each feature is worth a separate post. In the next post we will start looking at LINQ Syntax and how it can be leveraged into our code. So stay tuned for more…

LINQ Explained– Part 2

This is the second part of my on going series on LINQ. In the first installment, we had an overview of LINQ. In this post, we will look at some of the underlying concepts which are important to understand to work with LINQ. Though you are not required to master them but an understanding of these concepts gives you the extra level of confidence to work with LINQ. (Note: I will use C# as language of preference in this series).

LINQ and C# Language

LINQ works with C# 3.0 and Visual Basic 9.0. It relies heavily on the features provided by these languages. Although these features may be used separately, they are fundamental to the working of LINQ. Some of these features were delivered with C# 1.x and 2.0 – the predecessor to C# 3.0. The following sections describe these features in more detail.

Generics

Let us look at a simple example of comparing two numbers. The following method accepts two integers as argument, compares them and returns the result:

private int Compare (int x, int y)
{
 return x < y ? x : y;
}

This code works fine for comparing two integers. But suppose we wanted to compare two floating values or even two strings. For this purpose, either we change the method signature to accept the respective types or we end up writing entirely new methods for each type. But as developers, we would be inclined towards using a generalized method to perform the same function irrespective of the argument types.

One solution to the above problem is to accept object as arguments. In C#, every type is driven from the base type object so it can be cast to and back from the object type. We can rewrite the above method as following:

private object Compare (object x, object y)
{
           // comparison logic goes here
}

The above method now accepts object as argument. This makes the method more generalized but with some shortcomings. First of all, C# is a type-safe language, that is, objects have associated type and only operations defined by the associated types can be performed on an object. A comparison operator such as ‘<’ will not operate on the reference type object. Thus a conversion of object to a value type is required before the comparison is performed. This means to use a type, explicit casting required. For example, to convert to an integer, we write the following code:

int i = (int) x;
int j = (int) y;

Similarly, to check for string, we perform the following casting:

string str1 = (string) x;
string str2 = (string) y;

Second, when a value type is converted to a reference type (boxing), it involves an overhead. A conversion back to the value type from a reference type (unboxing) also involves an overhead. As a result of boxing and unboxing, there is a performance hit for an application. This is further magnified when Collections are used. Collections such as Stack, Queue, ArrayList operate on object type only. When an element is added to a collection, boxing takes places. Similarly, when a value is retrieved from the collection, unboxing is done. This means every time an element is added or retrieved from a collection, a performance overhead is involved.

The above issues can be catered through Generics. A C# 2.0 language feature, Generics introduced the concept of type parameters. Classes and methods can defer the definition of one or more type until runtime. With Generics, there is one implementation for all types. The type definition is performed when the method is invoked or an object is instantiated. Let us redefine our Compare method using Generics:

private T Compare <T> (T x, T y) where T : IComparable <T>
{
           return x.CompareTo (y) < 0 ? x : y;
}

A placeholder <T> appended to the method name points to a generic method. The placeholder <T> represents the type parameter to be provided when this method is used. The same placeholder is used for the input and output parameters. Note that since <T> is just a placeholder (and not a type), the CompareTo method cannot operate on it directly. For this reason, <T> implements the IComparable interface. We will look at the ‘where’ constraint shortly. For now we can use the above method for the comparison of different types as following:

int a = 20, b = 19;
int c = Compare <int> (a, b);

string str1 = "Zzzz", str2 = "Aaaa";
string str3 = Compare <string> (str1, str2);

Notice that there is no explicit casting required for the arguments. The method is invoked with the type parameter inplace of the placeholder and that’s it. The CLR is responsible for handling the rest.

The Generic concept also applies to classes and structures. Let us look at a Generic class:

public class UserAuthentication <T>
{
 private T myPassword;
             private string myUserID;

             public T Password
     {
  get { return myPassword; }
                          set { myPassword = value; }
     }

     public string UserID
     {
          get { return      myUserID; }
          set { myUserID    = value; }
     }
}

The above class creates a token for user authentication. Notice the placeholder <T> defined next to the class name. The password field is of the same type parameter. Similarly the Password property has a returns type of <T>. We can instantiated the above class with following code:

UserAuthentication <string> userAuth;
        
userAuth                 = new UserAuthentication <string> ();
userAuth.Password   = "Secret";
userAuth.UserID       = "User1";

UserAuthentication <int> userAuth;
        
userAuth                  = new UserAuthentication <int> ();
userAuth.Password    = 123456;
userAuth.UserID        = "User2";

.NET framework supports different generic collections under the System.Collections.Generic namespace. These generic collections include:

Stack <T> - a generic collection representing a Last-In-First-Out collection
Queue <T> - a generic collection representing a First-In-First-Out collection
List <T> - a generic collection of strongly type object list
Dictionary <K, V> - a generic collection of key-pair values

Let us see a generic List in action which accepts a strongly-typed parameter. We first define the strong-type Product followed by the generic list:

public class Product
{
     string productName;

     public Product (string pName)
     {
          productName = pName;
     }

     public string ProductDetails
     {
          get
          {
              return "Product-Name: " + productName;
          }
     }
}

public class ProductList <T> : IEnumerable <T> where T : Product
{
             List <T> productList = new List <T> ();

     public void AddProduct (T product)
     {
          productList.Add (product);
     }

             public T GetProduct (int index)
     {
          return productList [index];
     }
           
             IEnumerator <T> IEnumerable <T>.GetEnumerator ()
     {
          return productList.GetEnumerator ();
     }

              IEnumerator IEnumerable.GetEnumerator()
     {
          return productList.GetEnumerator ();
     }
}

We can now use the ProductList class to add and list products (a discussion of IEnumerable will follow shortly):

ProductList <Product> productList = new ProductList <Product> ();
        
productList.AddProduct (new Product ("Rice"));
productList.AddProduct (new Product ("Milk"));
productList.AddProduct (new Product ("Sugar"));

… = ((Product) productList.GetProduct (1)).ProductDetails;

Before I sum up the generics discussion, one last thing worth mentioning is constraints. If you have noticed in the ProductList class (and the Compare method), there is a use of ‘where’ constraint. A constraint is a condition applied on the type parameter. We can use constraints to treat only specific types. For this reason the constraint ‘where T : Product’ is added to the class definition. This way we create a generic list which only deals with Product objects. We can have the following different constraints attached to the generic type:

where T : class – type parameter is a reference type
where T : struct – type parameter is a value type
where T : new () – type parameter with a default constructor
where T : interface – type parameter implements an interface

Delegates

A delegate is an object which holds a reference to a method. When the delegate is called, the underlying method is invoked. This way a delegate behaves exactly like the referenced method. The method can either be static or an instance method. A delegate defines the method signature and any method with matching signature can be reference by the delegate. This makes it possible to change the reference to a different method programmatically and update the code in the methods without modifying the delegate. This simple concept of abstraction adds lots of power to the .NET Framework (A detailed discussion of delegates is beyond the scope of this post. A detailed post or two would cover delegates, events, asynchronous callback and threading in the future).

Working with delegates is a pretty simple in C#. Always keep in mind the method signature when defining a delegate. Let us look at the syntax of defining a delegate:

delegate result-type Name (parameters);

The delegate keyword is used as prefix to define the delegate. The result-type reflects the return type from the referenced method. The Name is the identifier of the delegate and the optional comma separated parameters are the input argument to the referenced method. The result-type and parameters define the signature of the delegate. Using the above syntax, we can create a delegate as following:

public delegate void Calculate (int value, int amount);

Any method with matching signature can be referenced by the above delegate. Let us define a method with the matching signature:

public class Accounts
{
          public void DebitAccount (int x, int y)
          {
                    int sum;
                    sum = ((x * 10) / y) * 2;
                    // use sum…
          }        
}

We can now instantiate and invoke the delegate by referencing the DebitAccount method as following:

Accounts objAccount = new Accounts ();
Calculate calc = new Calculate (objAccount.DebitAccount); // reference method
calc (2, 3); // call delegate

When we call the delegate, the DebitAccount method is invoked.

Multicasting is one of the features provided by delegates. A multicast delegate can reference more than one method at a time. When the delegate is called, all the referenced methods are invoked. The methods are invoked in the order in which they are referenced by the delegate. Let us modify the Accounts class by adding the following method to it:

public void CreditAccount (int x, int y)
{
           int average;
           average = ((x / 2) + 10) - y;
           // use average…
}

The calc delegate can now reference the above method using the compound assignment operator (+=):

calc += objAccount.CreditAccount;

Now if the call the delegate using calc (2, 3), both methods get invoked. Once you have used a delegate, the reference must be released. References can be removed using the compound subtraction statement or null value assignment as following:
calc -= objAccount.CreditAccount; // remove reference to CreditAccount
calc = null; // remove all references

Delegates are also used for Asynchronous Callbacks. When asp.net receives a request for a page, it assigns a thread from the thread-pool to the requested page. In a synchronous call, the page holds on to the thread for the duration of the request, blocking calls to the thread for new requests. This is acceptable for a short lived request but if the request is time-bound such as calling multiple web services or an I/O bound job, the delay is annoying.

Delegates help perform asynchronous tasks using Method Callback. With this technique, the delegate invokes the time-consuming method in a separate thread and the control returns immediately. The time-bound task executes in the background while we can continue with our processing. When the background job is finished, control is transferred to a callback method which can handle the result and update any control. Let me demonstrate this concept with an example. We first write the time-consuming process as following:

// the time consuming process
public bool LongProcess (int wait)
{
          // your lengthy task goes here 
          System.Threading.Thread.Sleep (wait); // just for demonstration
          return true; // return the result
}

Next we declare a delegate and use it to invoke the above method.

// define the delegate
public delegate bool LengthyProcessDelegate (int wait);

User clicks on a button to start the process:

// Use the delegate to start the lengthy process
protected void StartProcessing_Click (object sender, EventArgs e)
{
          LengthyProcessDelegate  lDelegate = new LengthyProcessDelegate (LongProcess);
          lDelegate.BeginInvoke (5000, new AsyncCallback (LongProcessCallback), lDelegate);
          for (int i = 0; i <= 50; i++)        
          { 
                    // do something 
          }
}

The above code first creates a new instance of the delegate which holds rerference to the time-consuming method. Next it calls the BeginInvoke method using the delegate. You must be wondering what this method is? Remember, when we declare a delegate, the compiler generates code similar to the following:

class LengthyProcessDelegate : System.MulticastDelegate 
{ 

             // synchronous execution
             public bool Invoke (int wait); 
 
            // asynchronous execution methods 
            public IAsyncResult BeginInvoke (int wait, 
                                               AsyncCallback callback, 
                                               object asyncState); 

            public bool EndInvoke (IAsyncResult result); 
} 

The Invoke method is used for sychronous calls. The other two methods, BeginInvoke and EndInvoke handle the asynchronous activity.

BeginInvoke method returns an instance of interface IAsyncResult. It accepts the same arguments as defined by the delegate plus two additional optional parameters. The first parameter is an instance of AsyncCallback (another delegate) which references the callback method. The second parameter is of type object which can be used to pass any information.

The EndInvoke method has the same return type as defined by the delegate. It accepts an instance of IAsyncResult. As mentioned above, BeginInvoke returns an instance of IAsyncResult. This instance is passed down to the callback method which is used by the EndInvoke method as parameter. It in turn returns the result of the time-consuming method which can be used for further processing. Let us look at the callback method in action:

// Callback method
public void LongProcessCallback (IAsyncResult result)
{
          LengthyProcessDelegate lDelegate= (LengthyProcessDelegate) result.AsyncState;
          bool returnValue = lDelegate.EndInvoke (result);
          // use returnvalue
}

As mentioned above, when BeginInvoke is called, the delegate invokes the callback method in a separate thread and the control returns to the program immediately. If you have noticed above, I have got a dummy loop after the call to BeginInvoke method. This is just to show you that the processing will continue and not wait for the time consuming process to complete.

delegates also provide a rich programming model to handle Events. An event lets an object notify the program when its state changes. Events allow objects to provide noification to be responded. This simple concept is very important for inter-process communication where the change of state of one object signals other objects to respond. A good example of events is a Graphical User Interface. The program transfers the control to an event handler when an event such as Button-Click is triggered by the user action. Another example would be an Accounts object raising an event when a transaction is made.

In C# events and delegates go hand-in-hand. Any object which triggers an event isn’t aware when the event is raised. This is left to a delegate which act as a bridge between the object and the event. Let us see this concept with a simple example. We begin with defining a delegate:

// define the delegate
public delegate void AccountDelegate (); // no input, output parameters

Next we define an Accounts class. This class has a Transaction property which fires an event when its value changes. The event is defined using the AccountDelegate. Since the delegate’s signature does not have any input or output parameter, the event handler for the event will have a similar signature. The event handling method OnTransactionOccur is defined as virtual which can be overridden by derived classes.

public class Accounts
{
         private int amount;
         // define the event
         public event AccountDelegate transactionComplete;

         public int Transaction
         {
                    get { return amount; }

                    set 
                    {
                              if (value <= 100)
                    amount--;
                              else
                                         amount++;

                              OnTransactionOccur (); // raise the event
                    }
          }

          protected virtual void OnTransactionOccur ()
          {
                  if (transactionComplete != null)
                        transactionComplete ();
          } 
}

We can now raise the event with the following code:

public void StartProcessing_Click (object sender, EventArgs e)
{
          Accounts account = new Accounts ();
          // register the event
          account.transactionComplete += new AccountDelegate (AccountEventHandler);

          account.Transaction = 100; // raise the event
}

First we instantiate an object of Accounts class. Next we register the event using the delegate with the event handler AccountEventHandler. We then set the Transaction property to raise the event. Remember the base class method OnTransactionOccur is actually responsible for raising the event. As soon as the event is raised, the control is transferred to the following event handler.

public static void AccountEventHandler ()
{
          // event handling code goes here
}

Enumerators

Enumeration, a powerful .NET concept, allows us to iterator through a collection of objects. In .NET, enumerators are based on the Iterator Pattern. Using this pattern, we can access elements of an aggregate (combination of many elements) object without revealing the inner working. The terms Enumerators and Iterators are used interchangably but .NET uses the term Enumerator.

A class must implement the IEnumerable interface to provide iteration. This interface exposes the following single method:

public interface IEnumerable
{
          IEnumerator GetEnumerator ();
}

The GetEnumerator method returns an object of IEnumerator interface. This object does the actual iteration on our collections. According to MSDN, Enumerators can be used to read the data in the collection, but they cannot be used to modify the underlying collection. Enumerator interface exposes the following methods:

public interface IEnumerator
{
          bool MoveNext();
          object Current{ get; }
          void Reset();
}

When we implement our own enumerators using the IEnumerator interface, the enumerator is positioned before the first element initially. To read the first (and subsequent) element, we use the MoveNext method. MoveNext method returns true until the end of the collection is reached. When MoveNext reaches the end of the collection, it returns false. To get the active element, we use the Current method. The Reset method positions the enumerator before the first element.

Let me demonstrate the above concept by a simple example. I begin by defining (beaten to death :-) Product class as following:

public class Product
{
          private string productID;
          private string productName;

          public Product (string id, string name)
          {
           productID   = id;
                    productName = name;
          }         

          public override string ToString()
          {
                     return String.Format ("Product details are ID: {0}, Name: {1}", productID, 
                                                     productName);
          }
}

Next we define our Custom Collection class which implements IEnumerable interface:

public class ProductCollection : IEnumerable
{
          private ArrayList productList;

          public ProductCollection ()
          {
                   productList = new ArrayList ();

                    productList.Add (new Product ("P1", "Tea"));
                    productList.Add (new Product ("P2", "Beverage"));
                    productList.Add (new Product ("P3", "Milk"));
          }
           
          public IEnumerator GetEnumerator ()
          {
                      return ((IEnumerable) productList).GetEnumerator ();
          }
}

I have used an arraylist to define a product collection. Since the arraylist already implements the IEnumerable interface, we can get hold of the its enumerator object by calling its respective GetEnumerator method. The enumerator object can be used with foreach loop to provide enumeration:

public void StartProcessing_Click (object sender, EventArgs e)
{
          ProductCollection collection = new ProductCollection ();

          foreach (Product p in collection)
                 // ListBox1.Items. Add (p.ToString ()); - ading to a listbox
}

The foreach statements simplifies the enumeration code for us. Under the hood, when we use the foreach loop, the compiler generates an initial call to GetEnumerator. It then uses MoveNext for each iteration to get the current item. Since the enumerator is positioned before the first element, the compiler doesn’t have to call the Reset method.

We can also create our own Enumerator class by implementing the IEnumerator interface. Let us modify our ProductCollection class with a nested class as following:

public class ProductEnumerator : IEnumerator
{
          private ProductCollection productCollection;
          private int index;

          public ProductEnumerator (ProductCollection collection)
          {
            productCollection = collection;
                 index = -1;
          }

          public void Reset()
          {
            index = -1;
          }

          public object Current
          {
            get
                 { return productCollection.productList[index]; }
          }

          public bool MoveNext()
          {
            index++;
                 if (index >= productCollection.productList.Count)
                               return false;
                    else
                               return true;
           }
}

Here is the tricky part. We used the GetEnumerator method of the ArrayList to get an enumerator object. We will modify that code to return an instance of our custom enumerator as following:

public IEnumerator GetEnumerator()
{
             return (IEnumerator) new ProductEnumerator (this);
}

One last thing worth mentioning are generic enumerators. These enumerators are used for a generic collection. The two interfaces are IEnumerable<T> and IEnumerator<T> found in System.Collections.Generic namespace. Both these interfaces inherit from their counterpart IEnumerable and IEnumerator. This means that generic enumerable objects are available both generically and non-generically. Let us first look at the IEnumerable<T> interface:

public interface IEnumerable<T> : IEnumerable
{
             IEnumerator<T> GetEnumerator();
}

IEnumerable<T> inherits from IEnumerable. This means any collection implementing IEnumerable<T> interface must define a generic and non-generic version of GetEnumerator method. So a generic collection class will have the following two implementations of GetEnumerator method:

public IEnumerator<T> GetEnumerator()
{
          return new Enumerator<T>(this);
}

IEnumerator IEnumerable.GetEnumerator()
{
          return new Enumerator<T>(this);
}

The same concept applies to IEnumerator<T>. This interface is defined as following:

public interface IEnumerator<T> : IDisposable, IEnumerator
{
           T Current { get; }
}

IEnumerator<T> interface implements IDisposable and IEnumerator interface. It has only one property. So any class implementing IEnumerator<T> inherits the rest of the members from IEnumerator and Idisposable interfaces.

Summary

In this post, we looked at some of the underlying concepts which help us better understand how LINQ works. Generics have introduced the concept of type parameter where the type definition is delayed till an object is instantiated. A place holder <T> defines a generic type. Generics apply to methods, classes and structures. We can apply constraints to our generic type to accept fixed type parameters. The .NET Framework ships with built in generic types such as Queue <T>, Stack <T> to reduce the development overhead.

Another feature important to understand LINQ is delgates. Delegates are objects which hold reference to methods. A call to the delegate invokes the reference method. Delegate can also hold reference to multiple methods. This property is known as multicating. Delegates are also used for asynchronous callbacks. Using a delegate, a callback method is attached to a long running process. After the process has finished, the delegate transfers control to the callback method which can retrieve the result and process it. Delegates also move hand-in-hand with events which notify us of a change. We can then write our event handlers to responsd to these changes.

We also looked at enumerators. Enumerators are used to iterate through the elements of an aggregate object. All enumerators implement IEnumerable and IEnumerator interfaces to provide iteration. The foreach comes in handy to iterate through a collection. Generic collections can also take advantage of enumerators by implementing IEnumerable <T> and IENumerator <T> interfaces.

When I sat down to write this post, I thought of explaining all the underlying concept in this post. But each topic is worth a separate post. For this reason, I will sum up the rest of the concept in the next post. Please do provide your feedback on this series and stay tuned for more…

LINQ Explained– Part 1

This is first installment of a multi-part tutorial series on Language Integrated Query or LINQ. In this series, my goal is to provide the readers with a detailed overview of LINQ. LINQ comes as a built-in feature with Visual Studio 2008 however; LINQ can also be used with Visual Studio 2005 by downloading the May 2006 CTP here.

What is LINQ

LINQ is a programming model which enables us to query and modify data independent of a data source. It is a set of extensions that adds native support for queries to the .NET Framework. With LINQ support, ‘Queries’ have become a first-class citizen within any .NET languages such as C# and VB.NET. By providing data abstraction over different data domains, LINQ provides a unified approach to manage data.

Why use LINQ

Today (and always :-) developers are responsible for managing data in their applications. The data belongs to different data domains and each domain comes with its unique set of rules to play with e.g. SQL is used for relational databases, XQuery/DOM are used to handle XML Documents and different Application Programming Interfaces (APIs) are used to manage Text files, Objects, Graphs, Registry, Active Directory etc. The developers are faced with the dilemma to master different data domains for the same purpose - to handle data. Wouldn’t it be nice to have a single set of rules to manage all our data requirements? This is where LINQ comes in handy. LINQ provides a unified programming model to manage data from different data sources. Hence with LINQ, we can invest our efforts in handling the business logic and not worrying about the syntax to manage data.

LINQ syntax and working

With LINQ, the notion of queries is now a built-in concept in the .NET Framework. The LINQ syntax (known as Query Expression) is a reminiscent of SQL. But this syntax is not limited to relational databases rather applies across all data domains under its umbrella. Following is a simple example of a LINQ Query which operates on a string array:

string [] fruits = { "apple", "banana", "orange", "pineapple", "carrot" };

var query =
             from fruit in fruits
             where (fruit == "orange" || fruit == "pineapple")
             select fruit; 

foreach (var fruit in query)
{
             ListBox1.Items.Add (fruit);
}

We will have a detailed look at LINQ syntax in the following posts. For now let us see what the above code does. We have an array of strings and a SQL-type query operates on this array. The query returns a subset of the array to an object of type var. The foreach-loop iterates through the object and displays the result. Simple isn’t it?

The worth noting point is that the same syntax above applies to a Relational database, DataSets, XML files or any other data domain. Our interface to handle the data remains the same but at the other end; the data domain can change depending on our requirements. This ability to have the same set of rules to access data across different domains is notable. I am sure, by now, you have started to see the strength of LINQ. The LINQ architecture depends on many .NET Framework features such as Generics, Delegates, Anonymous & Extension Methods etc. My next post will provide a detailed overview of these features.

LINQ comes in many flavors (LINQ Providers) to manage different data domains. Don’t confuse LINQ syntax with flavors. The syntax remains the same (with slight variation) across different providers. But the features may vary from one provider to the other e.g. the same LINQ syntax will fetch an Element/Node from an XML document but a DataRow from a database. Each LINQ Provider is responsible for converting the LINQ Expression to a form compatible with the underlying data source. LINQ has the following different flavors:

LINQ to Objects: Used to query in-memory collection of objects
LINQ to SQL: Handles data from SQL Server & SQL Server Compact databases
LINQ to Entities: Operates on object entities
LINQ to DataSet: Query data from DataSets
LINQ to XML: Handles data from XML Documents
Third Party: In the future we will see more providers written by third parties for different data sources.
The detailed discussion of these providers will be the topic of a future posts.

Summary

This installment provided an overview of Language Integrated Query. LINQ is a powerful technology which provides a unified programming model to manage data from different data sources. Queries now have native support in .NET Framework. LINQ syntax (Query Expression) resembles SQL syntax. There are several LINQ Providers for different data sources. Following are some useful links to know more about LINQ:

The LINQ Project
ScottGu on LINQ

In the next article, we will look at some of the C# Language features that are needed to understand to work with LINQ efficiently. So stay tuned…