Example: Using Jakarta Commons HttpClient for multithreaded HTTP retrieval

Public service announcement: The following code is not usable for real-world HTTP retrieval. If you write a spider or a news aggregator, please use gzip compression and conditional GETs. We now return you to our regularly scheduled blogging already in progress.

The Apache people have an HTTP client designed to deal with the shortcomings of the stuff you can find in the java.net packages. They have an example program in their svn repository of retrieving a given list of URLs in multithreaded fashion. Here’s a slightly simpler Scala adaptation which simultaneously retrieves the webpages for Google, Yahoo, and Sun:

import org.apache.commons.httpclient._
import org.apache.commons.httpclient.methods._

object mtget extends Application {
  // a list of uris to retrieve -- make a thread for each and start it
  List("http://www.google.com/", "http://www.yahoo.com/",
       "http://www.sun.com/") map (x => new Getter(x)) foreach (x => x start)

  class Getter(uri: String) extends Thread {
    val m = new GetMethod(uri)
    override def run() =
      try {
	m setFollowRedirects true
        (new HttpClient(new MultiThreadedHttpConnectionManager)) executeMethod m
        Console.println(m.getResponseBody.length + "b from " + uri)
      } catch {
	case e: Exception => Console.println("error: " + e)
      } finally {
        m.releaseConnection
        Console.println("connection released")
      }
  }
}

While my example is slightly simpler there are parts in the original Java program I don’t have to write or can write in a briefer fashion. For instance, code like this:

        Getter[] threads = new Getter[urisToGet.length];
        for (int i = 0; i < threads.length; i++) {
            threads[i] = new Getter(urisToGet[i]);
        }
        for (int j = 0; j < threads.length; j++) {
            threads[j].start();
        }

would become urisToGet map (x => new Getter(x)) foreach (x => x start) in Scala; there’s no use, after all, in creating variables if you’re not going to use them. I also don’t need to write boilerplate code for class variable declaration and a constructor that just initializes the class variables to the constructor parameters due to how you declare classes in Scala. Notice in the Scala code that I used a style of invoking methods comparable to Smalltalk. Instead of writing client.executeMethod(method), you could write client executeMethod method, whatever you prefer. It’s just syntactic sugar.

Also notice that I have fewer variables and less mutability. Removing unnecessary mutability (e.g., by declaring our GetMethod object to be an immutable val instead of a mutable var) reduces the number of places things can go wrong. The programming style Scala encourages, then, leads to fewer bugs. Also, we can have inner classes rather than having to declare things as static. Finally, the type inference system in Scala’s compiler lets us write code free of redundant type declarations.

The point of this post is that Scala lets you do real-world stuff in a pretty brief manner, and it compares favorably to Java. Scala is not a mere academic curiosity useless for getting real work done. Scala is useful for real-world programming, today. Say goodbye to for(int i = 0; i < N; i++); say hello to type inference and beautiful, brief code.