Planet Just-Scala

Scala Stream Fusion and Specialization

Jesper Nordenberg — Sat, 06 Mar 2010 09:39:00 -0800

Updated: Fixed code links and added view and stream benchmarks.

Inspired by the successful results of Haskell stream fusion (see Evolving Faster Haskell Programs (now with LLVM!) for some impressive optimizations) I was thinking if a similar concept is applicable to Scala collections. It turns out that with a combination of iterators and specialization it's possible to achieve similar optimizations in Scala.

The goal of stream fusion is essentially to optimize code like this:


 def scalaLibrarySum(a : Array[Int]) = a.map(i => i * 3 + 7).filter(i => (i % 10) == 0).foldLeft(0)(_ + _)

into code like this:


 def mapFilterSumLoop(a : Array[Int]) = {
 var i = 0
 var r = 0

 while (i < a.length) {
 val v = a(i) * 3 + 7

 if ((v % 10) == 0)
 r += v

 i += 1
 }

 r
 }

If you run the scalaLibrarySum method in Scala it will create two intermediate arrays with the results of the map and filter operations. This is totally unnecessary for this calculation as the functions passed to filter and map are side effect free and thus the result of the function applications can be performed lazily just before the result is needed in the fold operation. This is basically how the mapFilterSumLoop method works.

Besides creating intermediate arrays, boxing of primitive values must be avoided if we want to have any chance of competitive performance (the Haskell libraries contain specialized instances to avoid boxing). Fortunately Scala supports specialization of type parameters in version 2.8, which enables us to avoid boxing while still writing generic code. Unfortunately this feature seems to be quite buggy at the moment, just by playing around with a simple example I encountered two bugs (tickets #3148 and #3149). So, the code below contain some specialization done by hand. Hopefully these bugs will be fixed so that the code can be fully generalized.

The biggest difference compared to stream fusion in Haskell is that I use impure iterators in the Scala code. This is not as nice as the pure stream code used in Haskell, but it's a fact that Hotspot isn't nearly as good at optimizing pure functional code as GHC. Hotspot works best if fed imperative style loops.

Here's the definitions of the specialized functions and iterators I use in the benchmark below:


 // Specialized Function1
 trait Fn1[@specialized I, @specialized O] {
 def apply(a : I) : O
 }

 // Specialized Function2
 trait Fn2[@specialized I1, @specialized I2, @specialized O] {
 def apply(a1 : I1, a2 : I2) : O
 }

 // Specialized iterator
 trait SIterator[@specialized T] {
 def hasMore : Boolean
 def current : T
 def next()
 }

In addition to this I've defined array, filter and map iterators. Unfortunately these are not generic due to the problems with the specialize feature:


 class IntArrayIterator(a : Array[Int], var index : Int, endIndex : Int) extends SIterator[Int] {
 def next() = index += 1
 def current = a(index)
 def hasMore = index < endIndex
 }

 // Optimally this would be: class FilterIterator[@specialized T](iter : SIterator[T], pred : Fn1[T, Boolean]) extends SIterator[T]
 class FilterIterator(iter : SIterator[Int], pred : Fn1[Int, Boolean]) extends SIterator[Int] {
 def hasMore = iter.hasMore

 def next() = {
 iter.next()
 findNext()
 }

 def findNext() = {
 while (iter.hasMore && !pred(iter.current))
 iter.next()
 }

 def current = iter.current

 findNext()
 }

 // Optimally this would be: class MapIterator[@specialized U][@specialized T](iter : SIterator[T], fn : Fn1[T, U]) extends SIterator[U]
 class MapIterator(iter : SIterator[Int], fn : Fn1[Int, Int]) extends SIterator[Int] {
 def next() = iter.next()
 def current = fn(iter.current)
 def hasMore = iter.hasMore
 }

The fold function is straightforward and generic:


 def fold[@specialized T, @specialized U] (iter : SIterator[T], fn : Fn2[U, T, U], v : U) = {
 var r = v

 while (iter.hasMore) {
 r = fn(r, iter.current)
 iter.next()
 }

 r
 }

The map-filter-sum function can now be written using iterators:


 def mapFilterSum(a : Array[Int]) = {
 val filter = new Fn1[Int, Boolean] {def apply(a : Int) = (a % 10) == 0}
 val map = new Fn1[Int, Int] {def apply(a : Int) = a * 3 + 7}
 val s = new FilterIterator(new MapIterator(new IntArrayIterator(a, 0, a.length), map), filter)
 fold(s, new Fn2[Int, Int, Int] {def apply(a1 : Int, a2 : Int) = a1 + a2}, 0)
 }

The full iterator code can be found here. Compile the code using the latest Scala 2.8 build with the -Yspecialize flag. The optimize flag doesn't seem to have much effect on the performance.

I've benchmarked four different implementations of the map-filter-sum calculation:

The while loop shown above

The while loop split up into map, filter and fold functions with intermediate array results passed between them

The version using specialized iterators

The Scala library implementation shown above

Same as Scala library function but with a view instead

Same as Scala library function but with a stream instead

The benchmark is performed by taking the minimum execution time of 200 runs of each of the functions on an array of 1 million integers. Running the application with latest OpenJDK 7 (Hotspot version "build 17.0-b10") and the flags "-server -XX:CompileThreshold=100" I get the following results:


Loop: (4990,-423600172)
Loop with intermediate arrays: (6690,-423600172)
Specialized iterators: (5367,-423600172)
Scala array: (46444,-423600172)
Scala view: (39625,-423600172)
Scala stream: (63210,-423600172)

The first result value is the minimum execution time in microseconds, the second value is the result of the calculation. As you can see the method using specialized iterators is almost as fast as the single while loop. Hotspot has inlined all the iterator code, not bad! Using intermediate arrays is about 25% slower than specialized iterators. Using the Scala library is about 7-9 times slower! Clearly this bad result is a consequence of boxing taking place. Using a view is fastest here as it also avoids intermediate array creation.

The conclusion from this simple experiment is that it's certainly possible to write collection libraries with a nice high level interface and at the same time have excellent performance. When Scala specialization support is improved hopefully this power be available to all Scala programmers.

The full benchmark code can be found here.

Taking Advantage of Scala 2.8: Replacing the Builder

Erkki Lindpere — Fri, 05 Mar 2010 13:22:07 -0800

In Scala 2.8, using the builder pattern is no longer necessary (or the most optimal solution) in many cases, as Scala 2.8 adds support for named and default parameters.

I’ll give an example of this based on ScalaBox2D. In the Box2D object model, physics Bodies are defined mostly by a set of Fixtures, which in turn are defined by a Shape, density, friction, restitution and some other parameters.

In the Scala 2.7 version of ScalaBox2D, there were mutable fixture definitions, which were used by the engine to create the actual fixtures. The user code only worked with definitions, through an internal DSL that looked something like this (note: all code examples are simplified for clarity):

body { box(halfWidth, halfHeight) density 1 friction 0.3f restitution 0 computeMassFromShapes
}

For the DSL implementation, I used simple builders which left values to defaults if not specified:

def box(halfW: Float, halfH: Float) = new FixtureBuilder(FixtureDef(PolygonDef.box(halfW, halfH))) class FixtureBuilder(defn: FixtureDef) { def userData(userData: AnyRef) = { defn.userData = userData; this } def material(material: Material) = { defn.apply(material); this } def friction(friction: Float) = { defn.friction = friction; this } def restitution(restitution: Float) = { defn.restitution = restitution; this } def density(density: Float) = { defn.density = density; this } def filter(filter: FilterData) = { defn.filter = filter; this } def sensor(isSensor: Boolean) = { defn.isSensor = isSensor; this } def define = defn
}

The FixtureDefs are mutable mostly to simplify the builder. In Scala 2.8, I can use named and default parameters, and drop some more lines of code by not having builders at all. As a bonus, I can easily make the definitions immutable.

def fixtures(fd: FixtureDef*) {...}
val fixture = FixtureDef // a shorthand to the companion object of FixtureDef
val box = BoxDef // a shorthand to a BoxDef object that creates PolygonDefs case class FixtureDef( shapeDef: ShapeDef, /** The friction coefficient, usually in the range [0,1]. */ friction: Float = 0.2f, /** The restitution (elasticity) usually in the range [0,1]. */ restitution: Float = 0f, /** The density, usually in kg/m^2. */ density: Float = 0f, /** A sensor collects contact information but never generates a collision response. */ isSensor: Boolean = false, /** Contact filtering data. */ filter: FilterData = FilterData.Default, /** Use this to store application specific fixture data. */ userData: AnyRef = null
)

As you can see, the FixtureDef is now a case class with immutable parameters that have default values. Previously it looked very similar, but had only one parameter (ShapeDef) and all fields were mutable. The usage of the “DSL” now becomes a little bit more verbose (maybe I shouldn’t even call it a DSL any more) but I think it also becomes easier to understand for someone who knows the language but not the library, due to less moving parts and using built-in features instead of more code:

body { fixtures( fixture(box(halfWidth, halfHeight), density = 1, friction = 0.3f, restitution = 0) ) computeMassFromShapes
}

Deleting code while maintaining functionality always makes me glad and this is one of those cases. I guess there may be some more complex cases where builders may still work better, but for simple things like the above example, I really like the named and default arguments feature.

Note: moving ScalaBox2D to Scala 2.8 is still a work in progress for me and there may be some further changes to this “DSL” as well.

Named arguments

Henrik Huttunen — Sun, 13 Apr 2008 10:27:00 -0700

edit at 3rd of March, 2010: Scala will have named arguments in 2.8. Great! http://www.scala-lang.org/sid/1#

Scala unfortunately doesn't have named arguments. I would really like to have them, because they make code much clearer in some cases, and it's harder to make mistakes with them. That's why I'll show one way to simulate them.

First, let's create the function that can handle homemade named arguments:

 def f(x: {val name: String; val age: Int}) = {

f takes one argument, that contains the actual parameters we want. The type is shorthand for Any{ ... }.

Next we have imported the contents of x to be visible inside method f. This is possible due to Scala's handling of objects as first class modules.

 import x._

And now we can use the attributes as if they were f's real parameters:

 println(name + " is " + age + " years old")

How about from client's perspective? Well, client can just create a structural type that contains the needed attributes (actual arguments).

 f(new {val name = "Anthony"; val age = 5})

As you see, it's not really possible to make mistakes this way. But the calling has too much boilerplate, namely vals. So let's make it a bit better:

We create a case class for each actual parameter.

 case class Name(name : String)
 case class Age (age : Int)

Now we can make very clear function signature:

 def g(x: Name, y: Age) = {

Unfortunately we have to import all parameters one by one.

 import x._, y._
 println(name + " is " + age + " years old")

The client can call the method in the following way:

 g(Name("Tim"), Age(2))

That's it. It's annoying that we have to create case classes, but it's tolerable in important cases.

Here's the full code:


object Test extends Application{
 def f(x: {val name: String; val age: Int}) = {
 import x._
 println(name + " is " + age + " years old")
 }
 f(new {val name = "Anthony"; val age = 5})

 case class Name(name : String)
 case class Age (age : Int)

 def g(x: Name, y: Age) = {
 import x._, y._
 println(name + " is " + age + " years old")
 }
 g(Name("Tim"), Age(2))
}

Run the code and you get as output:
Anthony is 5 years old
Tim is 2 years old

edit:

On the client side, often one passes some constant values. If you call just like f(value1, value2, ... valueN), it's hard to know later when you see the code what the meaning for each value was; so you have to go see the API (if there's no tool help). Instead you have to define vals before the call:

val age = 8
val name = "Jim"
f(age, name)

But now there's the issue that you have to write the variables twice, and it would be nice to see immeaditely on the call what the value was.

Then:

f(new {val age = 8; val name = "Jim"})

isn't that bad at all. Neither the case class equivalent. If you have many arguments to pass, calling with single argument per line looks nice:

f(new { val age = 8
 val name = "Jim"
 val hobby = "blogging" })

Named arguments are especially useful when constructing objects, because it's then when you have to pass lots of vaguely related arguments.

Dependency Injection as Function Currying

Debasish — Sun, 28 Feb 2010 07:20:00 -0800

Dependency Injection is one of the techniques that I use regularly when I am programming in Java. It's a nice way of making an application decoupled from concrete implementations and localize object creation logic within specific bootstrapping modules. Be it in the form of Spring XML or Guice Modules, the idea is to keep it configurable so that specific components of your application can choose to work with specific implementations of an abstraction.

It so happens that these days possibly I have started looking at things a bit differently. I have been programming more in Scala and Clojure and being exposed to many of the functional paradigms that they encourage and espouse, it has stated manifesting in the way I think of programming. In this post I will look into dependency injection on a different note. At the end of it may be we will see that this is yet another instance of a pattern melding into the chores of a powerful language's idiomatic use.

In one of my projects I have a class whose constructor has some of its parameters injected and the others manually provided by the application. Guice has a nice extension that does this for you - AssistedInject. It writes the boilerplate stuff by generating an implementation of the factory. You just need to annotate the implementation class' constructor and the fields that aren't known to the injector. Here's an example from the Guice page ..

public class RealPayment implements Payment {
 @Inject
 public RealPayment(
 CreditService creditService, // injected
 AuthService authService, // injected
 @Assisted Date startDate, // caller to provide
 @Assisted Money amount); // aller to provide
 }
 ...
}

Then in the Guice module we bind a Provider<Factory> ..

bind(PaymentFactory.class).toProvider(
 FactoryProvider.newFactory(
 PaymentFactory.class, RealPayment.class));

The FactoryProvider maps the create() method's parameters to the corresponding @Assisted parameters in the implementation class' constructor. For the other constructor arguments, it asks the regular Injector to provide values.

So the basic issue that AssistedInject solves is to finalize (close) some of the parameters at the module level to be provided by the injector, while keeping the abstraction open for the rest to be provided by the caller.

On a functional note this sounds a lot like currying .. The best rationale for currying is to allow for partial application of functions, which does the same thing as above in offering a flexible means of keeping parts of your abstraction open for later pluggability.

Consider the above abstraction modeled as a case class in Scala ..

trait CreditService
trait AuthService

case class RealPayment(creditService: CreditService,
 authService: AuthService,
 startDate: Date,
 amount: Int)

One of the features of a Scala case class is that it generates a companion object automatically along with an apply method that enables you to invoke the class constructor as a function object ..

val rp = RealPayment( //..

is in fact a syntactic sugar for RealPayment.apply( //.. that gets called implicitly. But you know all that .. right ?

Now for a particular module , say I would like to finalize on PayPal as the CreditService implementation, so that the users don't have to pass this parameter repeatedly - just like the injector of your favorite dependency injection provider. I can do this as follows in a functional way and pass on a partially applied function to all users of the module ..

scala> case class PayPal(provider: String) extends CreditService
defined class PayPal

scala> val paypalPayment = RealPayment(PayPal("bar"), _: AuthService, _: Date, _: Int)
paypalPayment: (AuthService, java.util.Date, Int) => RealPayment = <function>

Note how the Scala interpreter now treats paypalPayment as a function from (AuthService, java.util.Date, Int) => RealPayment. The underscore acts as the placeholder that helps Scala create a new function object with only those parameters. In our case the new functional takes only three parameters for whom we used the placeholder syntax. From your application point of view what it means is that we have closed the abstraction partially by finalizing the provider for the CreditService implementation and left the rest of it open. Isn't this precisely what the Guice injector was doing above injecting some of the objects at module startup ?

Within the module I can now invoke paypalPayment with only the 3 parameters that are still open ..

scala> case class DefaultAuth(provider: String) extends AuthService
defined class DefaultAuth

scala> paypalPayment(DefaultAuth("foo"), java.util.Calendar.getInstance.getTime, 10000)
res0: RealPayment = RealPayment(PayPal(foo),DefaultAuth(foo),Sun Feb 28 15:22:01 IST 2010,10000)

Now suppose for some modules I would like to close the abstraction for the AuthService as well in addition to freezing PayPal as the CreditService. One alternative will be to define another abstraction as paypalPayment through partial application of RealPayment where we close both the parameters. A better option will be to reuse the paypalPayment abstraction and use explicit function currying. Like ..

scala> val paypalPaymentCurried = Function.curried(paypalPayment)
paypalPaymentCurried: (AuthService) => (java.util.Date) => (Int) => RealPayment = <function>

and closing it partially using the DefaultAuth implementation ..

scala> val paypalPaymentWithDefaultAuth = paypalPaymentCurried(DefaultAuth("foo"))
paypalPaymentWithDefaultAuth: (java.util.Date) => (Int) => RealPayment = <function>

The rest of the module can now treat this as an abstraction that uses PayPal for CreditService and DefaultAuth for AuthService. Like Guice we can have hierarchies of modules that injects these settings and publishes a more specialized abstraction to downstream clients.

Actor Styles... OO, Functional or Blended...

J. Suereth — Sun, 28 Feb 2010 07:12:00 -0800

I had the privilege of a free cup of coffee with David Pollak when I was in the bay area recently. He was describing his new goat-rodeo library when he showed me the interface for his "Worker" class (which is similar to an Actor class). He made a comment about "being annoyed with writing all those case statements" in actors, and so figured out a method using reflection and method overloading to define an actor that merely responds to all messages it receives. I'm sure there was more to it than this, but I believe he's partially solved an OO issue I complained about in an earlier blog about partial functions and inheritance. Let's take a bit to remember that solution to the issue.

The crux of the problem is the blend of OO and functional. Functions are composable. You can have one function take another and compose them all day long. Objects are also composable. One object can take another object and utilize it all day long. Methods (functions attached to objects) are not as composable. Scala traits give you means to compose with them, but all your composition is done at instantiation time (i.e. mixing in traits, inheritance, etc.). Now we come to the main issue. An Actor is an abstract class that you subclass. This leads one to compose behavior through traits (similar to other OO-style programs in Scala), but the method of doing so is a bit strange. We start with our "FinalMixinActor" class

trait BaseActor extends Actor {

 def makeMessageHandler() : PartialFunction[Any,Unit] = {
 case x => //Unhandled message, assume design flaw!
 error("Unknown message: " + x) 
 }
}
trait FinalActor extends BaseActor[T] {
 lazy val messageHandler = makeMessageHandler()
 override def act() {
 Actor.loop {
 react messageHandler
 }
 }
}

These two classes provide us with a way to compose actors. We make sure all behavior extends the BaseActor, and we make sure when creating an actor that "FinalActor" gets mixed in *last* in the inheritance linearization. This means that when FinalActor calls "makeMessageHandler", all the standard OO inheritance has a chance to kick and pull the behaviors you've composed. Let's see an example of composing an actor with two functionalities....

case class Ping()
case class Pong()

trait PingPongBehavior extends BaseActor {
 def makeMessageHandler() : PartialFunction[Any,Unit] = {
 val myBehavior : PartialFunction[Any,Unit] = {
 case Ping() => sender ! Pong()
 }
 return myBehavior orElse super.makeMessageHandler()
 }
}

This is our "PingPong" behavior actor for sending/receiving Ping/Pong messages. As you can see in the makeMessageHandler... we're manually converting functional composition into inheritance composition. It does give us flexibility if we'd like our behavior to run last. Let's create our second piece of behavior...

case class Shutdown()

trait RemoteControlBehavior extends BaseActor {
 def makeMessageHandler() : PartialFunction[Any,Unit] = {
 val myBehavior : PartialFunction[Any,Unit] = {
 case Shutdown() => Actor.exit
 }
 return myBehavior orElse super.makeMessageHandler()
 }
}

This is a simple actor that just lets us send a shutdown command to an actor instead of relying on the actor knowing when to shut itself down (or linking it). Not necessarily useful, but it will help illustrate our next point. Let's create an actor that mixes in both these functionalities.

val myActor = new PingPongBehavior with RemoteControlBehavior with FinalActor
myActor.start

Notice how we *have* to mix in FinalActor last so the composition of makeMessageHandler is done correctly. This also prevents other actors from adding behavior in their act methods....

Anyway, it's a decent method of allowing mixin behavior of actors. It doesn't have much type safety, but the Scala standard actors library actually makes you jump through some hoops to get type-safety anyway (plus all their samples are against Any... so it seems they tend to encourage this...)

Now for a look at how Goat-Rodeo helps solve this. First, Goat-Rodeo uses a type-safe message passing API, including knowing whether a message is allowed a response, and the type of that response. Let's take a look at a boiled down version of the twitter-clone sample on goat-rodeo's wiki. I've used elipses to ignore implementation details of Goat Rodeo:

class UserWorker(...) extends WorkerImpl[..., UserMsg](...) {
 /**
 * handle the Follow message
 */
 def doFollow(msg: Follow) {
 ...
 }
}

As you can see, it's just a method. Goat-Rodeo looks for methods with "do" or "handle" in the name and constructs a PartialFunction for the actor-behavior. The

Follow

class must be a subclass of

UserMsg

as Goat-Rodeo is also strongly typed. Because

doFollow

returns Unit, goat-rodeo knows that there is no return expected with this message. If you'd like to handle different messages, simple use method overloading. The best part, if we'd like to use OO inheritance to compose behavior, we can do so directly now...

class UserWorkerPlus(...) extends UserWorker(...) {
 /**
 * handle the Follow message
 */
 def doFollow(msg: Follow) {
 println("ZOMG!!!!!!!!!!!!!!!!!!")
 super.doFollow(msg)
 }
}

That should be a lot more comfortable to OO developers. With some type trickery you could even create the "management" behavior traits and mix them into "workers" as needed. It still leaves some things to be desired. e.g. when I compose partial functions in a purely functional sense, I can have the type-checker automatically infer what the new signature of a partial function should be after an orElse. With both of these methods, that is not possible. For the pure functional users, I'd recommend creating actors using something akin to this function (or you should just use scalaz, as their actors library is very well done):

import scala.actors.Actor
import scala.actors.Channel

class ChanneledActor[T](f : PartialFunction[T,Unit]) extends Actor {
 val typedChannel = new Channel[T](this)
 def act() {
 Actor.loop {
 typedChannel react f
 }
 }
}

def makeActor[T, U](f : PartialFunction[T,U]) : Channel[T] = {
 val a = new ChanneledActor(f andThen ( x => () )) //Ignore results...
 a.start
 a.typedChannel
}

This will let you compose partial functions in a functional style and let the type-checker figure out the signature. Finally when you create an Actor with the makeActor method, it will return the type-safe channel to send messages with. Here's an example interpreter session:

scala> makeActor[Int,Unit] { case x : Int => println("HAI") } 
res3: scala.actors.Channel[Int] = scala.actors.Channel@154f77b

scala> res3 ! "HAI"
<console>:10: error: type mismatch;
 found : java.lang.String("HAI")
 required: Int
 res3 ! "HAI"
 ^

scala> res3 ! 5 

scala> HAI

Anyway, if you truly desire a functional approach to actors, you should look into Scalaz, as they've done it right. If you desire a hybrid or an OO approach, I hope I've helped outline the details for you. I'm very interested to see how all these libraries evolve over time, and kudos to David Pollak for goat-rodeo.

SBT and Test Arguments

Jack Cough — Wed, 24 Feb 2010 09:01:00 -0800

If you want to pass arguments to tests in SBT, you've come to the right place. First know this - it only works in certain situations, it's a bit crufty, and its subject to change. But I'll try to keep this page up to date if it does change.

Another important note, if you want to pass arguments at the sbt command line, you can only do it for test-quick, and test-only (I think). It doesn't work for the default "test" command. This is a bit unfortunate, but there is a workaround (not a very good one, but at least it exists).

A final note: it works with ScalaTest and ScalaCheck. I haven't tested any of this with Specs, and not sure if Eric has implemented argument handling.

Passing Args to ScalaTest from the Command Line

Let's say you have this little ScalaTest class that uses both params and tags:


import org.scalatest.fixture.FixtureFunSuite
import org.scalatest.Tag

class WickedCoolTest extends FixtureFunSuite{
 type FixtureParam = Map[String,Any]
 override def withFixture(test: OneArgTest) {
 test(test.configMap)
 }
 test("1", Tag("dood1")){ conf => println("dood1: " + conf) }
 test("2", Tag("dood2")){ conf => println("dood2: " + conf) }
 test("3", Tag("dood3")){ conf => println("dood3: " + conf) }
}

To run all tests in in this class use:


test-only WickedCoolTest

To run only tests tagged with "dood1" use:


test-only WickedCoolTest -- -n dood1

To run tests tagged with dood1 or dood2, use:


test-only WickedCoolTest -- -n "dood1 dood2"

To run all tests except for tests tagged with dood2, run:


test-only WickedCoolTest -- -l dood2

To run all tests except for tests tagged with dood2 dood3, run:


test-only WickedCoolTest -- -l "dood2 dood3"

To pass configuration parameters to a test, use:


test-only WickedCoolTest -- -Danswer=42

To pass many config params, add more -D's:


test-only WickedCoolTest -- -Danswer=42 -DrealAnswer=54

Tags and parameters can be used in combination, like so:


test-only WickedCoolTest -- -n "dood dood2" -Dhey=you -Dm=f

This runs only tests tagged with dood or dood2 and produces the following output:

dood: Map(hey -> you, m -> f)
dood2: Map(hey -> you, m -> f)

Passing Args to ScalaCheck from the Command Line

I'll spare you the entire example and just get to the point here.

To set the minimum number of successful tests, use '-s'


> test-only *IntParallelArrayCheck -- -s 5000
...
[info] == scala.collection.parallel.mutable.IntParallelArrayCheck ==
[info] OK, passed 5000 tests.

The rest of the examples are very similar. Consult ScalaCheck itself for further documentation.

-s (minSuccessfulTests): Number of tests that must succeed in order to pass a property
-d (maxDiscardedTests): Number of tests that can be discarded before ScalaCheck stops testing a property
-n (minSize): Minimum data generation size
-x (maxSize): Maximum data generation size
-w (workers): Number of threads to execute in parallel for testing
-z (wrkSize): Amount of work each thread should do at a time

Default Arguments

Maybe you want to pass default arguments to your test framework of choice. That is, you want to pass the same arguments every time you run your tests (and still be able to pass more dynamically, if you wish). You can do this too, by adding elements to 'testOptions'. Doing so will take effect when you run sbt test, sbt test-only, sbt test-quick, etc.

Let's say you want the minimum number of ScalaCheck passing tests to be 5000 every time you run ScalaCheck. Here is how you do that:


override def testOptions = 
 super.testOptions ++ 
 Seq(TestArgument(TestFrameworks.ScalaCheck, "-s", "5000"))

Or maybe you only ever want to run your fast ScalaTest tests.


override def testOptions = 
 super.testOptions ++ 
 Seq(TestArgument(TestFrameworks.ScalaTest, "-n", "fast"))

Custom Test Tasks

Maybe you have some test that you run all the time. At the sbt command line, instead of saying > test-quick blah.blah.Blah, you just want to say: > blah.

You can do that too, and you can pass args to it dynamically, and you get the default arguments as well. You'll have to take the following code and put it into your sbt file. It's ugly, I know, but the results are nice.


lazy val blah = singleTestTask("blah.blah.Blah")
 
private def singleTestTask(className: String) = task { args =>
 defaultTestTask(TestFilter(_ == className) :: 
 testOptions.toList ::: ScalaTestArgs(args))
}

private def newScalaTestArg(l: String*) = 
 TestArgument(TestFrameworks.ScalaTest, l:_*)

private def ScalaTestArgs(args: Seq[String]): List[TestArgument] = {
 def KVArgs(args: Seq[String]): TestArgument = 
 newScalaTestArg(args.map("-D" + _):_*)
 def tagsFromArgs(tags: Seq[String]): List[TestArgument] = {
 if (tags.isEmpty) Nil else 
 List(newScalaTestArg("-n", tags.mkString(" ")))
 }
 val (kvs, tags) = args.partition(_.contains("="))
 KVArgs(kvs.toSeq) :: tagsFromArgs(tags.toSeq)
}

Subclasses

Finally, maybe you want to set up tasks like 'test-fast' and 'test-slow' which only run your fast and slow tests. Let's imagine that you've created a trait called blah.blah.SlowTest and all of your slow tests extend that trait. Tests that don't extend SlowTest are considered to be in the fast group. With the code below, at the sbt command line you'll be able to say > test-fast, and > test-slow.

Again, it's a bit ugly to put this stuff in your sbt project file, but it works for now, until I come up with a better plan :)


lazy val testSlow = 
 runSubclassesOf("org.nlogo.util.SlowTest")
lazy val testFast = 
 runEverythingButSubclassesOf("org.nlogo.util.SlowTest")

private def runSubclassesOf(className: String) = {
 val subclass: Boolean => Boolean = x => x
 subclassTest(className, subclass)
}
 
private def runEverythingButSubclassesOf(className: String) = {
 val notSubclass: Boolean => Boolean = x => ! x
 subclassTest(className, notSubclass)
}

private def subclassTest(className: String, 
 subclassCheck: Boolean => Boolean) = task { 
 args =>
 lazy val jars = 
 testClasspath.get.toList.map(_.asURL).toArray[java.net.URL]
 lazy val loader = 
 new java.net.URLClassLoader(jars,buildScalaInstance.loader)
 def clazz(name: String) = Class.forName(name, false, loader)
 lazy val superClass = clazz(className)
 def filter = 
 TestFilter(c => 
 subclassCheck(superClass.isAssignableFrom(clazz(c))))
 defaultTestTask
 (filter :: testOptions.toList ::: ScalaTestArgs(args))
}

Yeah...

Let me know if you have any issues with any of this, I'll be happy to help. I put it together pretty quickly. If there are any glaring errors or omissions, please tell.

DSL : Grow your syntax on top of a clean semantic model

Debasish — Sun, 21 Feb 2010 20:35:00 -0800

A DSL primarily has two components - a semantic model that abstracts the underlying domain and a linguistic abstraction on top that speaks the dialect of the user. The semantic model is the model of the domain where you can apply all the principles of DDD that Eric Evans espouses. And the linguistic abstraction is a thin veneer on top of the underlying model. The more well abstracted your model is, easier will be the construction of the layer on top of it. Here's a general architecture of a DSL engineering stack :-

It's interesting to observe that the two components of the stack evolve somewhat orthogonally.

The Semantic Model evolves Bottom Up

The semantic model usually evolves in a bottom up fashion - larger abstractions are formed from smaller abstractions using principles of composition. It can be through composition of traits or objects or it can be through composition of functions as well. How beautiful your compositions can be depends a lot on the language you use. But it's important that the semantic model also speaks the language of the domain.

Here's an example code snippet from my upcoming book DSLs In Action that models the business rules for a trading DSL. When you do a trade on a stock exchange you get charged a list of tax and fee components depending on the market where you execute the trade. The following snippet models a business rule using Scala that finds out the list of applicable tax/fee heads for a trade ..

class TaxFeeRulesImpl extends TaxFeeRules {
 override def forTrade(trade: Trade): List[TaxFee] = {
 (forHKG orElse 
 forSGP orElse 
 forAll)(trade.market)
 }
 
 val forHKG: PartialFunction[Market, List[TaxFee]] = { 
 case HKG => 
 // in real life these can come from a database
 List(TradeTax, Commission, Surcharge)
 }
 
 val forSGP: PartialFunction[Market, List[TaxFee]] = {
 case SGP => 
 List(TradeTax, Commission, Surcharge, VAT)
 }
 
 val forAll: PartialFunction[Market, List[TaxFee]] = {
 case _ => List(TradeTax, Commission)
 }

 //..
}

The method forTrade clearly expresses the business rule, which reads almost as expressive as the English version ..

"Get the Hong Kong specific list for trades executed on the Hong Kong market OR Get the Singapore specific list for trades executed on the Singapore market OR Get the most generic list valid for all other markets"

Note how Scala PartialFunction s can be chained together to give the above model an expressive yet succinct syntax.

The Language Interface evolves Top Down

Here you start with the domain user. What dialect does he use on the trading desk ? And then you try to build an interpreter around that which uses the services that the semantic model publishes. I call this thin layer of abstraction a DSL Facade that sits between your DSL script and the underlying domain model and acts as the glue.

It also depends a lot on the host language as to how you would like to implement the facade. With a language like Lisp, macros can come in very handy in designing an interpreter layer for the facade. And with macros you do bottom up programming, bending the host language to speak your dialect.

When you are developing an external DSL, the EBNF rules that you specify act as the DSL Facade for growing your syntax. Within the rules you can use foreign code embedding to interact with your semantic model.

In summary, when you design a DSL, the semantic model is as important as the dialect that it speaks. Having a well designed semantic model is an exercise in designing well-engineered abstractions. And as I mention in my book, the four qualities of good abstractions are minimalism, distillation, extensibility and composability.