/
Java
The Definite Guide to Terracotta by me and my colleagues; Ari, Geert, Alex, Orion and Taylor, is now available in the stores.
It is a very pragmatic and practical book, with tons of real-world examples and stories, so if you want to learn how to use Terracotta for real then go and order a copy.
In this post I will explain how you can build fault-tolerant systems using Scala Actors by arranging them in Supervisor hierarchies using a library for Scala Supervisors that I just released.
But first, let’s recap what Actors are and what makes them useful.
An actor is an abstraction that implements Message-Passing Concurrency. Actors have no shared state and are communicating by sending and receiving messages. This is a paradigm that provides a very different and much simpler concurrency model than Shared-State Concurrency (the scheme adopted by C, Java, C# etc.) and making it easier to avoid problems like deadlocks, live locks, thread starvation etc. This makes it possible to write code that is deterministic and side-effect-free, something that makes easier to write, test, understand and reason about. Each actor has a mailbox in which it receives incoming messages and can use pattern matching on the messages to decide if a message is interesting and which action to take. The most well known and successful implementation of actors can be found in the Erlang language (and the OTP platform) where it has been used to implement extremely fault tolerant (99.9999999% reliability - 9 nines) and massively concurrent systems (with hundreds of thousand simultaneous actors).
So what are Supervisor hierarchies? Let’s go to the source; http://www.erlang.org/doc/designprinciples/supprinc.html#5.
A supervisor is responsible for starting, stopping and monitoring its child processes. The basic idea of a supervisor is that it should keep its child processes alive by restarting them when necessary.
It has two different restart strategies; All-For-One and One-For-One. Best explained using some pictures (referenced from erlang.org):
OneForOne
AllForOne
Naturally, the library I have written for Scala is by no means as complete and hardened as Erlang’s, but it seems to do a decent job in providing the core functionality.
The implementation consists of two main abstractions; Supervisor and GenericServer.
The Supervisor manages hierarchies of Scala actors and provides fault-tolerance in terms of different restart semantics. The configuration and semantics is almost a 1-1 port of the Erlang Supervisor implementation, explained in the erlang.org doc referenced above. Read this document in order to understand how to configure the Supervisor properly.
The GenericServer (which subclasses the Actor class) is a trait that forms the base for a server to be managed by a Supervisor.
The GenericServer is wrapped by a GenericServerContainer instance providing a necessary indirection needed to be able to fully manage the life-cycle of the GenericServer in an easy way.
So, let’s try it out by writing a small example in which we create a couple of servers, configure them, use them in various ways, kill one of them, see it recover, hotswap its implementation etc.
(Sidenote: I have written about hotswapping actors before, however this library has taken this approach a but further and provides a more flexible and powerful way of achieving this. Thanks DPP.)
This walk-through will only cover some of the API, for more details look at the code or the tests.
1. Create our server messages
import scala.actors._ import scala.actors.Actor._ import com.jonasboner.supervisor._ import com.jonasboner.supervisor.Helpers._ sealed abstract class SampleMessage case object Ping extends SampleMessage case object Pong extends SampleMessage case object OneWay extends SampleMessage case object Die extends SampleMessage
2. Create a GenericServer
We do that by extending the GenericServer trait and override the body method.
class SampleServer extends GenericServer {
// This method implements the core server logic and naturally has to be overridden
override def body: PartialFunction[Any, Unit] = {
case Ping =>
println("Received Ping"); reply(Pong)
case OneWay =>
println("Received OneWay")
case Die =>
println("Received Die..dying...")
throw new RuntimeException("Received Die message")
}
}
GenericServer also has some callback life-cycle methods, such as init(..) and shutdown(..).
3. Wrap our SampleServer in a GenericServerContainer
Here we also give it a name to be able to refer to it later. We are creating two instances of the same server impl in order to try out multiple server restart in case of failure.
object sampleServer1 extends GenericServerContainer("sample1", () => new SampleServer)
object sampleServer2 extends GenericServerContainer("sample2", () => new SampleServer)
4. Create a Supervisor configuration
Here we create a SupervisorFactory that is configuring our servers. The configuration mimics the Erlang configuration and defines a general restart strategy for our Supervisor as well as a list of workers (servers) which for each we define a specific life-cycle.
object factory extends SupervisorFactory {
override protected def getSupervisorConfig: SupervisorConfig = {
SupervisorConfig(
RestartStrategy(AllForOne, 3, 10000),
Worker(
sampleServer1,
LifeCycle(Permanent, 1000)) ::
Worker(
sampleServer2,
LifeCycle(Permanent, 1000)) ::
Nil)
}
}
5. Create a new Supervisor
val supervisor = factory.newSupervisor
Output:
12:25:30.031 [Thread-2] DEBUG com.jonasboner.supervisor.Supervisor - Configuring supervisor:com.jonasboner.supervisor.Supervisor@860d49 12:25:30.046 [Thread-2] DEBUG com.jonasboner.supervisor.Supervisor - Linking actor [Main$SampleServer$1@1b9240e] to supervisor [com.jonasboner.supervisor.Supervisor@860d49] 12:25:30.062 [Thread-2] DEBUG com.jonasboner.supervisor.Supervisor - Linking actor [Main$SampleServer$1@1808199] to supervisor [com.jonasboner.supervisor.Supervisor@860d49] 12:25:30.062 [main] DEBUG Main$factory$2$ - Supervisor successfully configured
6. Start the Supervisor
This also starts the servers.
supervisor ! Start
Output:
12:25:30.078 [Thread-8] INFO com.jonasboner.supervisor.Supervisor - Starting server: Main$sampleServer2$2$@1479feb 12:25:30.078 [Thread-8] INFO com.jonasboner.supervisor.Supervisor - Starting server: Main$sampleServer1$2$@97a560
7. Try to communicate with the servers.
Here we try to send a couple one way asynchronous messages to our servers.
sampleServer1 ! OneWay
Try to get a reference to our sampleServer2 (by name) from the Supervisor before sending a message.
supervisor.getServer("sample2") match {
case Some(server2) => server2 ! OneWay
case None => println("server [sample2] could not be found")
}
Output:
Received OneWay Received OneWay
8. Send a message using a future
Try to send an asynchronous message - receive a future - and wait 100 ms (time-out) for the reply.
val future = sampleServer1 !! Ping
val reply1 = future.receiveWithin(100) match {
case Some(reply) =>
println("Received reply: " reply)
case None =>
println("Did not get a reply witin 100 ms")
}
Output:
Received Ping Received reply: Pong
9. Kill one of the servers
Try to send a message (Die) telling the server to kill itself (by throwing an exception).
sampleServer1 ! Die
Output:
Received Die..dying... 12:25:30.093 [Thread-8] ERROR c.j.supervisor.AllForOneStrategy - Server [Main$SampleServer$1@1b9240e] has failed due to [java.lang.RuntimeException: Received Die message] - scheduling restart - scheme: ALLFORONE. 12:25:30.093 [Thread-8] DEBUG Main$sampleServer2$2$ - Waiting 1000 milliseconds for the server to shut down before killing it. 12:25:30.093 [Thread-8] DEBUG Main$sampleServer2$2$ - Server [sample2] has been shut down cleanly. 12:25:30.093 [Thread-8] DEBUG c.j.supervisor.AllForOneStrategy - Restarting server [sample2] configured as PERMANENT. 12:25:30.093 [Thread-8] DEBUG com.jonasboner.supervisor.Supervisor - Linking actor [Main$SampleServer$1@166aa18] to supervisor [com.jonasboner.supervisor.Supervisor@860d49] 12:25:30.093 [Thread-8] DEBUG Main$sampleServer1$2$ - Waiting 1000 milliseconds for the server to shut down before killing it. 12:25:30.093 [main] DEBUG com.jonasboner.supervisor.Helpers$ - Future timed out while waiting for actor: Main$SampleServer$1@1b9240e Expected exception: java.lang.RuntimeException: Time-out 12:25:31.093 [Thread-8] DEBUG c.j.supervisor.AllForOneStrategy - Restarting server [sample1] configured as PERMANENT. 12:25:31.093 [Thread-8] DEBUG com.jonasboner.supervisor.Supervisor - Linking actor [Main$SampleServer$1@1968e23] to supervisor [com.jonasboner.supervisor.Supervisor@860d49]
10. Send an asyncronous message and wait on a future.
If this call times out, the error handler we define will be invoked - in this case throw an exception. It is likely that this call will time out since the server is in the middle of recovering from failure and we are on purpose defining a very short time-out to trigger this behavior.
val reply2 = try {
sampleServer1 !!! (Ping, throw new RuntimeException("Time-out"), 10)
} catch { case e => println("Expected exception: " e.toString); Pong }
The output of this call (due to the async nature of actors) is interleaved with the logging for the restart of the servers. As you can see the log below can be found in the middle of the restart output.
12:25:30.093 [main] DEBUG com.jonasboner.supervisor.Helpers$ - Future timed out while waiting for actor: Main$SampleServer$1@1b9240e Expected exception: java.lang.RuntimeException: Time-out
Server should be up again. Try the same call again
val reply3 = try {
sampleServer1 !!! (Ping, throw new RuntimeException("Time-out"), 1000)
} catch { case e => println("Expected exception: " e.toString); Pong }
Output:
Received Ping
Also check that server number 2 is up and healthy.
sampleServer2 ! Ping
Output:
Received Ping
11. Try to hotswap the server implementation
Here we are passing in a completely new implementation of the server logic (doesn’t look that different tough, but it can be any piece of scala pattern matching code) to the server’s hotswap method.
sampleServer1.hotswap(Some({
case Ping =>
println("Hotswapped Ping")
}))
12. Try the hotswapped server out
sampleServer1 ! Ping
Output:
Hotswapped Ping
13. Hotswap again
sampleServer1.hotswap(Some({
case Pong =>
println("Hotswapped again, now doing Pong")
reply(Ping)
}))
14. Send an asyncronous message that will wait on a future (using a different syntax/method).
Method returns an Option[T] which can be of two different types; Some(result) or None. If we receive Some(result) then we return the result, but if None is received then we invoke the error handler that we define in the getOrElse method. In this case print out an info message (but you could throw an exception or do whatever you like…) and return a default value (Ping).
val reply4 = (sampleServer1 !!! Pong).getOrElse({
println("Time out when sending Pong")
Ping
})
Output:
Hotswapped again, now doing Pong
Same invocation with pattern matching syntax.
val reply5 = sampleServer1 !!! Pong match {
case Some(result) => result
case None => println("Time out when sending Pong"); Ping
}
Output:
Hotswapped again, now doing Pong
15. Hotswap back to original implementation.
This is done by passing in None to the hotswap method.
sampleServer1.hotswap(None)
16. Test the final hotswap
sampleServer1 ! Ping
Output:
Received Ping
17. Shut down the supervisor and its server(s)
supervisor ! Stop
Output:
12:25:31.093 [Thread-6] INFO com.jonasboner.supervisor.Supervisor - Stopping server: Main$sampleServer2$2$@1479feb 12:25:31.093 [Thread-6] INFO com.jonasboner.supervisor.Supervisor - Stopping server: Main$sampleServer1$2$@97a560 12:25:31.093 [Thread-6] INFO com.jonasboner.supervisor.Supervisor - Stopping supervisor: com.jonasboner.supervisor.Supervisor@860d49
You can find this code in the sample.scala file in the root directory of the distribution. Run it by invoking:
scala -cp target/supervisor-0.3.jar:[dependency jars: slf4j and logback] sample.scala
Check out
The SCM system used is Git.
git clone git@github.com:jboner/scala-supervisor.git.Build it
The build system used is Maven 2.
scala-supervisor.mvn installThis will build the project, run all tests, create a jar and upload it to your local Maven repository ready for use.
Runtime dependencies
Automatically downloaded my Maven.
That’s all to it.
Have fun.
Scala is one those great languages that is scalable. With scalable I mean that it is the language that grows with the user, that it makes simple things easy and hard things possible. A language that is easy to get started and to become productive in, but at the same time a deep language with very powerful constructs and abstractions.
In this blog post I will try to highlight the power of Scala’s mixins and how you can use mixin composition to get AOP/interceptor-like style of programming.
First let’s define our service interface, modeled as a mixin (in this case without an implementation so similar to Java’s interface):
trait Stuff {
def doStuff
}
Now let’s define two different mixin “interceptors” that implement the service interface. The first one manages logging and the other one transaction demarcation (but for simplicity I am just using a dummy mock for TX stuff for now):
trait LoggableStuff extends Stuff {
abstract override def doStuff {
println("logging enter")
super.doStuff
println("logging exit")
}
}
trait TransactionalStuff extends Stuff {
abstract override def doStuff {
println("start TX")
try {
super.doStuff
println("commit TX")
} catch {
case e: Exception =>
println("rollback TX")
}
}
}
As you can see this example they both override the Stuff.doStuff method. If we look more closely we can see that it’s a pattern is:
The trick here is in the semantics of the call to super. Here Scala will invoke the next mixin in a stack of mixins, e.g. the same method in the “next” mixin that have been mixed in. Exactly what f.e. AspectJ does in its proceed(..) method and what Spring does in its interceptors.
But before we try this out, let’s create a concrete implementation of our Stuff service, called RealStuff:
class RealStuff {
def doStuff = println("doing real stuff")
}
Now we have everything we need, so let’s fire up the Scala REPL and create a component based on the RealStuff class and a mixin stack with support for logging and transactionality. Scala’s mixin composition can take place when we instantiate an instance, e.g. it allows us to mix in functionality into specific instances that object creation time for specific object instances.
First let’s create a plain RealStuff instance and run it:
scala> import stacks._ import stacks._ scala> val stuff = new RealStuff stuff: stacks.RealStuff = $anon$1@6732d42 scala> stuff.doStuff doing real stuff
Not too exciting, but let’s do it again and this time mix in the LoggableStuff mixin:
scala> val stuff2 = new RealStuff with LoggableStuff stuff2: stacks.RealStuff with stacks.LoggableStuff = $anon$1@1082d45 scala> stuff2.doStuff logging enter doing real stuff logging exit
As you can see the call to RealStuff.doStuff is intercepted and logging is added before we are invoking this method as well as after. Let’s now add the TransactionalStuff mixin:
scala> val stuff3 = new RealStuff with LoggableStuff with TransactionalStuff stuff3: stacks.RealStuff with stacks.LoggableStuff with stacks.TransactionalStuff = $anon$1@4512d65 scala> stuff3.doStuff start TX logging enter doing real stuff logging exit commit TX
As you can see, the semantics for this mixin stack is the exact same as you would get with stacking AspectJ aspects or Spring interceptors. Another interesting aspect is that the whole composition is statically compiled with all its benefits of compile time error detection, performance, potential tool support etc.
This approach is similar to Rickard Oberg’s idea on using the so-called Abstract Schema pattern for type-safe AOP in plain Java.
It is both simple and intuitive to change the order of the mixin “interceptors”, simply change the order in which they are applied to the target instance:
scala> val stuff4 = new RealStuff with TransactionalStuff with LoggableStuff stuff4: stacks.RealStuff with stacks.TransactionalStuff with stacks.LoggableStuff = $anon$1@a20232 scala> stuff4.doStuff logging enter start TX doing real stuff commit TX logging exit
Finally, just for fun, let’s a create a mixin that can retry failing operations. This particular one will catch any exception that the service might throw and retry it three times before giving up:
trait RetryStuff extends Stuff {
abstract override def doStuff {
var times = 0
var retry = true
while (retry) {
try {
super.doStuff
retry = false
} catch {
case e: Exception =>
if (times < 3) { // retry 3 times
times = 1
println("operation failed - retrying: " times)
} else {
retry = false
throw e
}
}
}
}
}
To test this behavior (as well as the rollback feature in the TransactionalStuff) we can change the RealStuff.getStuff method to throw an exception:
class RealStuff {
def doStuff {
println("doing real stuff")
throw new RuntimeException("expected")
}
}
Now we can try to add this mixin to the beginning of our our stack and run the service:
scala> val stuff5 = new RealStuff with RetryStuff with TransactionalStuff with LoggableStuff stuff5: stacks.RealStuff with stacks.RetryStuff with stacks.LoggableStuff with stacks.TransactionalStuff = $anon$1@a927d45 scala> stuff5.doStuff logging enter start TX doing real stuff operation failed - retrying: 1 doing real stuff operation failed - retrying: 2 doing real stuff operation failed - retrying: 3 rollback TX logging exit
Pretty neat, right?
That’s all for now. In the next post I will cover a bunch of ways to use Scala’s language primitives to do Dependency Injection (DI).
A month ago I wrote an introductory post about Scala Actors: HotSwap Code using Scala and Actors. For you who don’t know what it is I don’t want to start by reading the previous post let’s briefly recap what Actors are and what you can use them for.
An Actor is an abstraction that implements Message-Passing Concurrency. Actors have no shared state and are communicating by sending and receiving messages. This is a paradigm that provides a very different and much simple concurrency model than Shared-State Concurrency (the scheme adopted by C, Java, C# etc.) and is avoiding most of the latter one’s complexity and problems. This makes it possible to write code that is deterministic and side-effect-free, something that makes it easier to write, test, understand and reason about. Each Actor has a mailbox in which it receives incoming messages and normally uses pattern matching on the messages to decide if a message is interesting if action is needed.
Scala’s Actors are based on Doug Lea’s Fork/Join library and have for example been used very effectively in the excellent lift web framework to among other things to enable Comet style (push/streaming ajax) development. Actors allows us to, in a simple and uniformed way, parallelize applications using multiple threads, something that helps us take advantage of all the new dual/quad/… core or SMP machines that we are starting to get now days. But this also poses challenges; how can we make applications build on this “new” programming model highly available and how can we scale them out, if necessary. Would it not be cool if we could not only parallelize our application onto multiple threads but also onto multiple machines?
Note: Erlang, the most successful implementation of Actors to date, solves the challenges in building fault-tolerant and highly available systems in an elegant way using supervisor hierarchies. Nothing prevents an implement of this strategy in Scala Actors, all the primitives (like link, trap_exit etc.) already exists.
I have spent some time last weeks looking into if would make sense to utilize Terracotta to cluster the Scala Actors library to give a platform on which we can both scale Actors out in a distributed fashion and ensure full fault tolerance and high-availability. The result of this exercise have been successful and I’m happy to announce that they work very nice together. I will now spend the remainder of this post on walking you through a simple example in how to cluster a Scala Actor using Terracotta.
But before we do anything, let my point you to the SVN repository where you can fetch the Terracotta Integration Module (TIM) that I have implemented for Scala Actors. You can check it out anonymously by invoking:
svn co http://svn.terracotta.org/svn/forge/projects/tim-scala-actors-2.6.1/
When that is done, step into tim-scala-actors-2.6.1/trunk and invoke mvn install (the last command requires that you have Maven installed). it’ll take a while for Maven to download all its dependencies but after a while you will have a shiny new TIM for Scala Actors installed in your local Maven repository (usually in ~/.m2). The sample that we will discuss in the next sections is available in the src/samples/scala directory with the sample configuration in the src/samples/resources directory.
Now, let’s write a little Cart actor. This actor response to two different messages AddItem(item) and Tick. The former one adds an item to the Cart while the latter one triggers the Cart to print out its content (I’ll let you know why it’s called Tick in a second):
// Messages
case object Tick
case class AddItem(item: String)
class Cart extends Actor {
private[this] var items: List[String] = Nil
def act() {
loop {
react {
case AddItem(item) =>
items = item :: items
case Tick =>
println("Items: " items)
}
}
}
def ping = ActorPing.scheduleAtFixedRate(this, Tick, 0L, 5000L)
}
As you see the state is held by a Scala var, which holds onto a List (immutable). In react we wait for the next incoming message and if it is of type AddItem then we grab the item and append it to the list with all our items, but if the message is of type Tick we simply print the list of items out. Simple enough. But what is this method ping doing? It uses an object called ActorPing to schedule that a Tick should be sent to the Cart every 5 seconds (ActorPing is shamelessly stolen from Dave Pollak’s lift).
In order to cluster the Cart actor we have to write two things. First a hack, a simple configuration file in which declare which actors we want to cluster. This is something that later should be put into the regular tc-config.xml file, but for now we have to live with it. So let’s create a file with one single line, stating the fully qualified name of the Cart actor;
samples.Cart
We can either name this file clustered-scala-actors.conf and put it in root directory of the application or name it whatever we want and feed it to Terracotta using the -Dclustered.scala.actors.config=[path to the file] JVM property. Second, we have to write the regular Terracotta configuration file (tc-config.xml). Here we essentially have to define three things; the TIM for Scala Actors, locks to guard our mutable state and finally which classes should be included for bytecode instrumentation.
Starting with the TIM for Scala Actors. Here we define the version on the module as well as the URL to our Maven repository (in a short while we will put this jar in the Terracotta Maven repository and then you would not have to point out a local one).
Now we have to define the locks, which in Terracotta, also marks the transaction boundaries. The Cart has one mutable field (the var named items) that we need to ensure is guarded correctly and has transactional semantics. For each var Scala generates a setter and a getter. The getter is named the same as the field while the getter has the name suffixed with _$eq. That gives us the following lock definition:
We have to define a pair like this for each mutable user defined field in a clustered actor (not the standard one’s that are common for all Scala Actors, those are automatically defined).
It important to understand the TIM automatically clusters the Actor’s mailbox, which means that no messages will ever be lost - providing full fault-tolerance.
Finally we have to define the classes that we need to include for instrumentation. This naturally includes our application classes, e.g. the classes that are using our Cart actor in one way or the other. Those are picked out by the pattern like: 'samples.*'. We also have to include all the Scala runtime and library classes that we are referencing from the message is that we send. In our case that means the classes that are used to implement the List abstraction in Scala. Here is the full listing:
I could have included these (and many more) classes in the TIM, but since Terracotta adds a tiny bit of overhead to each class that it instruments I took the decision that it would be better to let the user explicitly define the classes that needs to be instrumented and leave the other ones alone. Since you can pretty much put any valid Scala data or abstraction in an actor message, it is very likely that you will have to declare some includes, else Terracotta will throw an exception (which is expected) with a message listing the XML snippet that you have to put in the tc-config.xml file. So don’t panic if things blow up.
Last but not least, we need to enable Terracotta in the Scala runtime (if you are planning to run the application in a Terracotta enabled application server, then you can skip this section - however I think it might still be useful to be able to try the application out in the Scala REPL). The simplest way of doing that is to do some minor changes to the scala command. First, let’s step down into the scala/bin directory and make a copy of the scala command called tc-scala, then scroll down all the way to the bottom. As you can see it is just a wrapper around the regular java command, which makes things pretty easy for us. We start by defining some environmental variables (here showing my local settings):
TC_SCALA_ACTORS_CONFIG_FILE=/Users/jonas/src/java/tc-forge/projects/tim-scala-actors-2.6.1/trunk/src/samples/resources/clustered-scala-actors.conf TC_CONFIG_FILE=/Users/jonas/src/java/tc-forge/projects/tim-scala-actors-2.6.1/trunk/src/samples/resources/tc-config.xml TC_INSTALL_DIR=/Users/jonas/src/java/tc/code/base/build/dist/terracotta-trunk-rev6814 TC_BOOT_JAR="$TC_INSTALL_DIR"/lib/dso-boot/dso-boot-hotspot_osx_150_13.jar TC_TIM_SCALA_ACTORS_JAR=/Users/jonas/.m2/repository/org/terracotta/modules/clustered-scala-actors-2.6.1/2.6.0-SNAPSHOT/clustered-scala-actors-2.6.1-2.6.0-SNAPSHOT.jar
When these variables have been defined we can replace the existing invocation of java with the following:
${JAVACMD:=java} ${JAVA_OPTS:=-Xmx256M -Xms256M}
-Xbootclasspath/p:"$TC_BOOT_JAR"
-Dtc.install-root="$TC_INSTALL_DIR"
-Dtc.config="$TC_CONFIG_FILE"
-Dclustered.scala.actors.config="$TC_SCALA_ACTORS_CONFIG_FILE"
-Dscala.home="$SCALA_HOME"
-Denv.classpath="$CLASSPATH"
-Denv.emacs="$EMACS"
-cp "$BOOT_CLASSPATH":"$EXTENSION_CLASSPATH":"$TC_TIM_SCALA_ACTORS_JAR"
scala.tools.nsc.MainGenericRunner "$@"
Enough hacking. Now let’s try it out. I think that the best way of learning new things in Scala is to use its REPL, so let’s start that up, this time with Terracotta enabled. But before we do that we have to start up the Terracotta server by stepping into the bin directory in the Terracotta installation and invoke:
$ ./start-tc-server.sh
Note: you need to grab Terracotta from SVN trunk to get the bits that work with the Scala TIM. See instructions on how to check out the sources and how to build it.
Now, we can start up the Terracotta enabled Scala REPL:
$ tc-scala 2008-01-25 07:42:11,643 INFO - Terracotta trunk-rev6814, as of 20080124-140101 (Revision 6814 by jonas@homer from trunk) 2008-01-25 07:42:12,136 INFO - Configuration loaded from the file at '/Users/jonas/src/java/tc-forge/projects/tim-scala-actors-2.6.1/trunk/src/samples/resources/tc-config.xml'. 2008-01-25 07:42:12,325 INFO - Log file: '/Users/jonas/terracotta/client-logs/scala/actors/20080125074212303/terracotta-client.log'. Parsing scala actors config file: /Users/jonas/src/java/tc-forge/projects/tim-scala-actors-2.6.1/trunk/src/samples/resources/clustered-scala-actors.conf Configuring clustering for Scala Actor: samples.Cart Welcome to Scala version 2.6.0-final. Type in expressions to have them evaluated. Type :help for more information. scala>
Here we can see that it has found and connected to the Terracotta server, found our clustered-scala-actors.conf config file and configured clustering for one Scala Actor; samples.Cart.
Let’s have some fun and start up another REPL in another terminal window. In each of these we do the following; import our classes, create a new Cart (Actor) and start up the Actor.
scala> import samples._ import samples._ scala> val cart = new Cart cart: samples.Cart = samples.Cart@81af82 scala> cart.start res0: scala.actors.Actor = samples.Cart@81af82 scala>
Now we have a distributed Actor just waiting to be fed with some messages. We don’t want to make it disappointed so let’s now add a bunch of bananas and apples to the Cart, and then feed it with a Tick message to make it print out the result:
scala> cart ! AddItem("bananas")
scala> cart ! AddItem("apples")
scala> cart ! Tick
scala> Items: List(apples, bananas)
scala>
Ok, so far no news. But comes the moment of truth, let’s take the other REPL and fire of a Tick:
scala> cart ! Tick scala> Items: List(apples, bananas) scala>
Yippee, it works. Now we can invoke the ping method to schedule a Tick (to print out status) every 5 seconds.
scala> cart.ping
res2: java.util.concurrent.ScheduledFuture = java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3a4388
scala> Items: List(apples, bananas)
scala> Items: List(apples, bananas)
scala> cart ! AddItem("football")
scala> Items: List(football, apples, bananas)
...
The Scala Actors TIM currently supports three different scopes; instance, class and custom. The scope is defined by appending a colon ‘:’ and the type of scope after the FQN of the Actor in the clustered-scala-actors.conf. If no scope is defined then the Actor is assumed to have scope instance. For example:
com.biz.Foo:custom com.biz.Bar:class com.biz.Baz:instance com.biz.Bam
The default scope named instance means that the Scala TIM is transparently intercepting the instantiation (f.e. new Cart) of all the Actors that you declare in the clustered-scala-actors.conf file. Each clustered Actor instance will have a unique identity across the cluster and each time this specific instance is created (f.e. when a new node joins the cluster) then the clustered instance with this specific identity will be handed out. The TIM distinguishes between actors of the same type but instantiated in different code paths. To take an example, let’s create one object ActorFactory with one single method create:
object ActorFactory {
def create: Actor = new MyActor
}
If we now have two classes Foo and Bar as follows:
class Foo {
val actor = ActorFactory.create
}
class Bar {
val actor = ActorFactory.create
}
Then Foo and Bar will have two distinct clustered Actors each with a unique but cluster-wide identity.
The class scope lets all Actors of a the same type share Actor instance, so each time an Actor of a specific type is created the same clustered one will be handed out.
Finally we have the custom scope. Which, as it sounds, allows custom user defined scoping.
If you want more control over scope and life-cycle of a specific Actor then you can define it to have custom scope in the clustered-scala-actors.conf file and create a factory in which you bind each Actor to whatever scope you wish. But now you have to create some data structure that is holding on to your Actors in the factory and explicitly define it to be a root in the tc-config.xml file. The factory might look something like this:
// Cart factory, allows mapping an instance to any ID
object Cart {
// This instance is the custom Terracotta root
private[this] val instances: Map[Any, Cart] = new HashMap
def newInstance(id: Any): Cart = {
instances.get(id) match {
case Some(cart) => cart
case None =>
val cart = new Cart
instances = (id -> cart)
cart
}
}
}
This means that we have to add some more configuration elements to our Terracotta configuration. First we need to add the root samples.Cart$.instances (Cart$ is the name of Scala’s compiled Cart companion object, all companion objects compiles to a class with the name of the original class suffixed with $):
Then we have to add locking for the Cart.newInstance(..) method and finally a whole bunch of new include statements for all the Scala types that are referenced by the scala.collection.mutable.HashMap that we used as root:
...... ... Cart_newInstance write * samples.Cart$.newInstance(..)
That’s pretty much all there’s to it. Check out the code, play with it and come back with feedback, bug reports, patches etc.
Enjoy.
In this post I will show you how you can do code hotswap in the same fashion as in Erlang using Scala and its Actors package.
An actor is an abstraction that implements Message-Passing Concurrency. Actors have no shared state and are communicating by sending and receiving messages. This is a paradigm that provides a very different and much simpler concurrency model than Shared-State Concurrency (the scheme adopted by C, Java, C# etc.) and is avoiding all of the latter one’s problems with deadlocks, live locks, race conditions, thread starvation etc. This makes it possible to write code that is deterministic and side-effect-free, something that makes easier to write, test, understand and reason about. Each actor has a mailbox in which it receives incoming messages and can use pattern matching on the messages to decide if a message is interesting and which action to take. The most well known and successful implementation of actors can be found in the Erlang language (and the OTP platform) where it has been used to implement extremely fault tolerant (99.9999999% reliability - 9 nines) and massively concurrent systems (with hundreds of thousand simultaneous actors).
Let’s start by writing a little server. We implement this in the form of a trait, which is Scala’s mixin construct. Traits allows you to build up your components using so-called mixin composition which is something that can give you a very high grade of reuse and flexibility. This trait only defines a single method named status which prints out info about the enclosing instance. Completely useless and not much for a server, but it will give you the idea. Then we subclass this mixin and define the ServerOne concrete server class (with the status method mixed in).
// servers
trait Server {
def status = println("current server: " this)
}
class ServerOne extends Server
Let’s instantiate the ServerOne class and see what the status method it will print out. Here we are doing it interactively through Scala’s REPL (read-eval-print-loop).
$ scala -cp . scala> val server = new ServerOne server: ServerOne = ServerOne@7be75d scala> server status current server: ServerOne@7be75d
Now, before we write the actor we have to define the messages it responds to. Here Scala is using something called case classes which are similar to normal classes but with some enhancements. First you can match on them, e.g. use pattern matching similar to the one found in Erlang. They also have some syntactic sugar, f.e. you can create them without using new, the compiler generates getters and setters for the constructor arguments, equality is not based on object id but on meaning/content (something that makes them ideal to use for value objects, but that is another story). We define two different messages; Status and HotSwap.
// triggers the status method case object Status // triggers hotswap - carries the new server to be hotswapped to case class HotSwap(s: Server)
Ok, now it is time for the actual actor. Actor is a base class in the Scala library and we can choose to either extend it explicitly or to create an anonymous actor through the actor {...} construct. When we subclass the actor we have to implement the method called act which is the callback method that is invoked when the actor is started up.
Scala comes with two different implementations; one that is based on Java threads in that each actor is getting its own thread, while the other one is based on events and very lightweight allowing tens of thousands of actors to run simultaneously. Here we will use the lightweight version (which is done by using the react method instead of the receive method for receiving messages).
The trick to do hotswap by using actors is to loop recursively and pass on the state in each recursive call. This is a very common idiom in functional programming. The beauty of it is that we do not update any mutual state but our execution is side effect free which makes it easier to test and reason about. In this case our state is the actual server. We start the loop by instantiating ServerOne. The pattern matching is happening in react statement in which we have three different cases (pattern matchers).
The first one matches our Status, when we receive this message we simply invoke the status method on our server and then taking another round in the loop passing along the server.
The second one matches our HotSwap message. It is here things are starting to get interesting. Now we can take the new replacement server (here called newServer), which is passed to us as an argument to the HotSwap message, and pass it in to the call to the loop method. Voila, we have updated our server at runtime. Now all subsequent messages will act on our new server instance.
This will work since the react method will in fact never return but infinitely recur. Infinite recursion would have been a problem in f.e. Java since each recursion would consume a new stack frame until we run out of memory. But recursion is one of the most powerful and commonly used idioms in functional programming and the Scala compiler optimizes tail-recursive algorithms and turns them into regular loops.
At the end we have also added a match-all pattern that does nothing, this is defined by the case _ => ... clause. Let’s take a look at the code.
class ServerActor extends Actor {
def act = {
println("starting server actor...")
loop(new ServerOne)
}
def loop(server: Server) {
react {
case Status =>
server.status
loop(server)
case HotSwap(newServer) =>
println("hot swapping code...")
loop(newServer)
case _ => loop(server)
}
}
}
Finally we will follow one of Scala’s idioms and create a companion object for our ServerActor class. In this object, which is a singleton but should be seen upon as a module for functions and immutable state, we define an immutable handle to an instantiated and started Actor.
Sidenote: the val holding our actor is initialized lazily when it is first accessed, and since we are starting up the actor in the initialization block of the val, the Actor will not be started until it is used.
// actor companion object
object ServerActor {
val actor = {
val a = new ServerActor
a.start; a
}
}
Let’s try to run it in the Scala REPL. The Scala function ! (pronounced bang) means “send a message”. So a ! msg means send message msg to actor a.
$ scala -cp .
scala> val actor = ServerActor.actor
starting actor...
actor: hotswap.ServerActor = hotswap.ServerActor@528ed7
scala> actor ! Status
current server: hotswap.ServerOne@226445
scala> class ServerTwo extends Server {
| override def status = println("hotswapped server: " this)
| }
defined class ServerTwo
scala> actor ! HotSwap(new ServerTwo)
hot swapping code...
scala> actor ! Status
hotswapped server: line5$object$$iw$$iw$$iw$ServerTwo@b556
Pretty cool, right?
This would be even more cool if Scala came with an SSH server that could provide this REPL remotely (like we have in Erlang OTP). Then we could connect to our application from the outside and change its behavior, fixing bugs, upgrade the server etc. Another solution would be to make use of the remote actors in the Scala distribution, but that is something for another post.
The last month I have been very busy writing on The Definitive Guide to Terracotta.
It is a collaborative effort from the Terracotta team each one writing on chapters in their area of expertise. It is a very practical and will teach you how to use Terracotta with the X framework/appserver or for a Y use-case, yet thorough and deep in concepts and theory.
It will be available sometime next year and if you’re interested in getting updates on the book can subscribe to: bookupdate AT terracottatech DOT com.
I had the pleasure of attending Oredev last month. This was a great conference. I have attended this conference three years in a row and it’s getting better and better every year.
This year they had a great speaker lineup and I was able to catch some very interesting talks. Among the most memorable ones were Dan North’s keynote about why “Best Practices” in software development are neither “Best” no “Practices”. I also attended Dan’s talk on BDD (Behavior-Driven
Development). Great talk. I find BDD very interesting, it feels like the natural extension of, or complement to TDD (Test-Drived Development) in that it focuses on getting complete concept coverage in the tests instead of code coverage (as in TDD).
Other great talks were Andy Hunt’s (Pragmatic Programmers) keynote and Erik Meijer’s talk on LINQ. The latter one was fun to watch, Erik (undeliberately it felt like) turned it more or less into a praise of Haskell; how they have stolen all the good stuff in LINQ from Haskell and that the world would be a better place if everyone just used Haskell.
My talk on Terracotta was able to attract quite a lot of attendees. One of the most interesting things was when I asked them, at the end of the talk, how many could see immediate need for something like Terracotta in their daily work roughly 80% raise their hands. I find this quite amazing and is actually something that have been consistent during last half year. I remember when I started asking this question, around 2 years ago, roughly 5-10 % raised their hands. This is quite a drastic change. Since I know that we were facing the same problems with scalability and HA a couple of years as we do now, I guess this is a sign of that the awareness of clustering, persistent and durable RAM and similar services has increased; that people have started to consider writing stateful applications with rich domain models - which implies another solution to HA and scalability than Oracle RAC and similar.
However, the best thing was that the conference had invited one of the best coffee shops in Malmo to serve all speakers unlimited amount of caffeine in the form of espresso, cafe latte, machiatto or whatever was asked for. I paid them 4-5 visits every day.
I had the pleasure of attending Joe Armstrong’s fantastic presentation at JAOO some weeks ago. It was by far the best talk at the whole conference, both in terms of content and presentation. Joe showed deep knowledge in both hardware and software, good teaching skills all wrapped up in a dry twisted sense of British humor - Joe was incredibly funny, I sometimes felt like had attended a stand-up comedian show.
Anyway, for all of you that missed Joe’s talk, here is a great interview with him discussing Erlang’s (and a functional programmer’s) view of the world, the problem with OO, shared state concurrency and much more. Not as funny as the talk, but very interesting.
Yesterday, Rich Hickey announced the birth of Clojure - a lisp dialect for the JVM.
After just a brief look, Clojure is perhaps the most interesting language in the (now fairly large) family of ‘dynamic languages for the JVM’. It brings the power of Lisp to the JVM but have made some design decisions that in some ways makes it more interesting than ANSI Common Lisp. Here are some of the things that I find particularly interesting:
Here are some tasters (from the forum).
Java integration:
(new java.util.Date) => Wed Oct 17 20:01:38 CEST 2007 (. (new java.util.Date) (getTime)) => 1192644138751 (.. System out (println "This is cool!")) This is cool!
Macros:
(defmacro time [form]
`(let [t0# (. System (currentTimeMillis))
res# ~form
t1# (. System (currentTimeMillis))]
(.. System out (println (strcat "Execution took "
(/ (- t1# t0#) 1000.0) " s")))
res#))
Usage:
(defn factorial [n]
(if (< n 2)
1
(* n (factorial (- n 1)))))
(time (factorial 1000))
=> Execution took 0.012 s
40…
It is still in beta but if you want to start playing around with it yourself, dive into the docs.
Dreaming on:
Stuff that I would like to see (in order to make it the ultimate playground) are among other things: message-passing concurrency (I don’t fully believe in STM…yet) and declarative pattern matching (from Erlang), implicit currying and laziness (as in Haskell), transparent distribution (as in Mozart/Oz and Erlang) and optional static typing. Some of these can be found in Qi, I just would love to see them on the JVM.
The second half of an interview that Xebia did with me some months ago have been published.
In this half which we discuss Terracotta in more detail, covering Network-Attached Memory and JVM-level clustering in depth (some stuff that we cover is slightly outdated, but most of it still holds).
You can find it here.
If you missed the first part you can read more it here (which gives more of an overview).
Xebia did an interview with me some months ago in which we discuss Terracotta, scale-out and high-availability for Java, JVM-level clustering, network-attached memory, distributed computing, concurrent programming, JEE best practices etc.
The interview is split up in two parts, the first one being more of an overview while the second one discusses things in more detail.
You can find the first part here.
Second part will be published in 2 weeks.
My and Geert Bevin’s article about how to cluster RIFE’s Web Continuations with Terracotta has just been published on Artima.
Here is the abstract:
In this article, we discuss how the RIFE Web framework helps you become productive and efficient in building conversational Web applications. Productivity with RIFE is in large part due to RIFE’s unique approach to Web development—its use of continuations for conversational logic, and complete integration of meta-programming to minimize boilerplate code.
We also introduce you to Terracotta and it’s JVM-level clustering technology, and show you how Terracotta and RIFE can work together to create an application stack that allows you to scale out and ensure high-availability for your applications, but without sacrificing simplicity and productivity. This means working with POJOs, and minimal boilerplate and infrastructure code.
It tries to not only explain but to show you in a pragmatic way how RIFE’s continuations and Terracotta is a perfect match with their common goal of power with simplicity. But don’t take my word for it, go on and read it yourself.
I was going through some of the older parts of my darcs repository the other day when I stumbled upon a pretty neat and powerful little remoting library that I wrote back in 2001. It is based on dynamic proxies and plain sockets, almost legacy programming techniques nowadays in a world of BCI (bytecode instrumentation), AOP, EJBs, Terracotta etc.
Anyway, I thought that the implementation is fairly interesting (and useful) and would make good blog post.
It basically implements a remote proxy, that can either be instantiated by the client or instantiated by the server and sent to the client. In either case, when methods are invoked upon the proxy then they are executed on the server. The communication is socket-based and the server is holding a resizable thread pool that can grow and shrink based on usage. The beauty of using dynamic proxies is that once the proxy has been instantiated, the RPC is transparent, e.g. pretty much the same as with RMI but without all the stub and skeleton mess. Is a very simple library, not meant as a full RMI replacement, but does its job pretty well. I have used it among other things for:
So without further ado, let’s dive into some code. First, and most importantly, would be to take a look at how the remote proxy can be used from a client’s perspective.
This simple unit test is testing (and highlights) two basic features.
public void testSimpleProxy() {
// 1)
// creates a new remote proxy for the POJOImpl1 class
// which maps to an instance of this class on the server
RemoteProxy proxy1 = RemoteProxy.createClientProxy(
new String[]{"test.POJO1"}, // interface(s)
"test.POJOImpl1", // implementation
"localhost", // server IP or hostname
6663 // server port
);
// retrieves the POJOImpl1 instance
POJO1 pojo1 = (POJO1) proxy1.getInstance();
// 2)
// invoke a method on the proxy (executed on the server)
assertEquals("POJO1 here", pojo1.test());
// 3)
// retrieve the proxy that is created on the server
RemoteProxy proxy2 = pojo1.getPOJO2();
// retrieves the POJOImpl2 instance
POJO2 pojo2 = (POJO2) proxy2.getInstance();
// 4)
// invoke a method on the proxy (executed on the server)
assertEquals("POJO2 here", pojo2.test());
// 5)
// close the proxies (close() must always be called)
proxy1.close();
proxy2.close();
}
That was easy. So much for the client side.
How can we now create the server that serves these two proxies? The only thing we have to do is to create a RemoteProxyServer by passing in the class loader that we want to use to instantiate our proxied objects as well as an implementation of the Invoker interface (which has one single method called invoke), an interface that gives you the possibility to invoke methods on your proxied objects any way you want. This example simply shows the most basic way of doing it:
// create a remote proxy server with a simple Invoker impl
RemoteProxyServer remoteProxyServer = new RemoteProxyServer(
classLoader,
return new Invoker() {
public Object invoke(String handle,
String methodName,
Class[] paramTypes,
Object[] args,
Object context) {
Object result;
try {
Object instance = RemoteProxy.getWrappedInstance(handle);
Method method = instance.getClass().getMethod(methodName, paramTypes);
result = method.invoke(instance, args);
} catch (Exception e) {
throw new WrappedRuntimeException(e);
}
return result;
}
};);
// start all server threads
remoteProxyServer.start();
Let’s now dive into the implementation of the server a little bit. When we invoke remoteProxyServer.start() then the server starts up X worker threads (managed by a thread pool). The work done by of one of these thread is roughly - in pseudo code:
Here are some code excerpts, highlighting the algorithm:
public void run() {
try {
m_socket.setTcpNoDelay(true);
m_socket.setSoTimeout(m_timeout);
m_in = new ObjectInputStream(m_socket.getInputStream());
m_out = new ObjectOutputStream(m_socket.getOutputStream());
} catch (IOException e) {
throw new WrappedRuntimeException(e);
}
while (m_running) {
try {
switch (m_in.read()) {
case Command.CREATE:
handleCreateCommand();
break;
case Command.INVOKE:
handleInvocationCommand();
break;
case Command.CLOSE:
m_running = false;
break;
default:
break;
}
} catch (Exception e) {
close();
throw new WrappedRuntimeException(e);
}
}
close();
}
private void handleCreateCommand()
throws IOException, ClassNotFoundException,
InstantiationException, IllegalAccessException {
String className = (String) m_in.readObject();
Class klass = Class.forName(className, false, m_loader);
Object instance = klass.newInstance();
// get a handle to the proxied instance
String handle = RemoteProxy.wrapInstance(instance);
m_out.writeObject(handle);
m_out.flush();
}
private void handleInvocationCommand()
throws IOException, ClassNotFoundException {
Object context = m_in.readObject();
String handle = (String) m_in.readObject();
String methodName = (String) m_in.readObject();
Class[] paramTypes = (Class[]) m_in.readObject();
Object[] args = (Object[]) m_in.readObject();
Object result = null;
try {
result = m_invoker.invoke(handle, methodName, paramTypes, args, context);
} catch (Exception e) {
result = e; // pass the exception to the client
}
m_out.writeObject(result);
m_out.flush();
}
If the client asks for a server created proxy, then the server would create it like this: RemoteProxy proxy = RemoteProxy.createServerProxy(myInstance, "localhost", 6663);, and write it to the object output stream - or even more simple have one of the already proxied objects create it (on the server) and return it (to the client).
This is pretty much the whole server. But if you’re still with me, I’m sure you’re eager to know what the RemoteProxy looks like. Well, here are some of the more interesting parts of its internals:
public class RemoteProxy implements InvocationHandler, Serializable {
... // constructors and field declarations are omitted
/**
* Creates a new proxy to a class. To be used on the client side to
* create a new proxy to an object.
*
* @param interfaces the name of the interfaces for the object to create the proxy for
* @param impl the name of the the impl class to create the proxy for
* @param address the address to connect to
* @param port the port to connect to
* @param ctx the context carrying the users principals and credentials
* @param loader the class loader to use for instantiating the proxy
* @return the new remote proxy instance
*/
public static RemoteProxy createClientProxy(String[] interfaces,
String impl,
String address,
int port,
Object context,
ClassLoader loader) {
return new RemoteProxy(interfaces, impl, address, port, context, loader);
}
... // some factory methods are omitted
/**
* Look up and retrives a proxy to an object from the server.
*
* @return the proxy instance
*/
public Object getInstance() {
if (m_proxy != null) {
return m_proxy;
}
if (m_loader == null) {
m_loader = Thread.currentThread().getContextClassLoader();
}
try {
m_socket = new Socket(InetAddress.getByName(m_address), m_port);
m_socket.setTcpNoDelay(true);
m_out = new ObjectOutputStream(m_socket.getOutputStream());
m_in = new ObjectInputStream(m_socket.getInputStream());
} catch (Exception e) {
throw new WrappedRuntimeException(e);
}
if (m_handle == null) {
// is a client side proxy
if (m_targetInterfaceNames == null) {
throw new IllegalStateException("interface class name can not be null");
}
if (m_targetImplName == null) {
throw new IllegalStateException("implementation class name can not be null");
}
try {
// create a new instance on the server and get the handle to it in return
m_out.write(Command.CREATE);
m_out.writeObject(m_targetImplName);
m_out.flush();
m_handle = (String) m_in.readObject();
m_targetInterfaces = new Class[m_targetInterfaceNames.length];
for (int i = 0; i < m_targetInterfaceNames.length; i ) {
try {
m_targetInterfaces[i] = Class.forName(
m_targetInterfaceNames[i], false, m_loader
);
} catch (ClassNotFoundException e) {
throw new WrappedRuntimeException(e);
}
}
} catch (Exception e) {
throw new WrappedRuntimeException(e);
}
}
// create and return a regular Java Dynamic Proxy
m_proxy = Proxy.newProxyInstance(m_loader, m_targetInterfaces, this);
return m_proxy;
}
/**
* This method is invoked automatically by the proxy. Should not be called directly.
*
* @param proxy the proxy instance that the method was invoked on
* @param method the Method instance corresponding to the interface method invoked
* on the proxy instance.
* @param args an array of objects containing the values of the arguments passed
* in the method invocation on the proxy instance.
* @return the value to return from the method invocation on the proxy instance.
*/
public Object invoke(Object proxy, Method method, Object[] args) {
try {
m_out.write(Command.INVOKE);
m_out.writeObject(m_context);
m_out.writeObject(m_handle);
m_out.writeObject(method.getName());
m_out.writeObject(method.getParameterTypes());
m_out.writeObject(args);
m_out.flush();
final Object response = m_in.readObject();
if (response instanceof Exception) {
throw (Exception) response;
}
return response;
} catch (Exception e) {
throw new WrappedRuntimeException(e);
}
}
/**
* Closes the proxy and the connection to the server.
*/
public void close() {
try {
m_out.write(Command.CLOSE);
m_out.flush();
m_out.close();
m_in.close();
m_socket.close();
} catch (IOException e) {
throw new WrappedRuntimeException(e);
}
}
}
That’s all there is to it.
Ok, perhaps not as powerful and flexible as Spring Remoting, EJBs, RMI or Terracotta’s Network-Attached Memory.
But plain sockets and DPs are not so bad either. Or is it just me that’s being a bit sentimental? At least it was pretty cool stuff back in 2001 
If you want to take a look at the code or are thinking of using it then you can check it out using this command (if you don’t have darcs installed you can get it here):
darcs get http://jonasboner.com/darcs/remoteproxy
Happy hacking.
I was in Rome, Italy some weeks ago. After strolling around in the old town for a couple hours I felt the need to grab a bite. So I stopped at a café, sat down and decided to order a pasta fettuccine and a glass of white.
The waiter, perhaps in his 50s, comes up to me and not only takes my order but treats me like a celebrity, like I was the only thing that mattered. It is hard to put the finger on it, but his charisma, the way he sets the table, folds the napkin, lays out the silverware, uses subtle gestures and facial expressions is close to perfection - truly a master of his craft. It a matter of minimalism and style, the way he pays attention to the tiniest detail. It really shines through that he takes pride in, and loves, his job.
We all have a lesson to learn here. If you feel stuck and trapped at work, not doing things you are passionate and care about, then I think you have two options:
Life is too short to spend 9-5 doing stuff you don’t love and is passionate about. Steve Jobs is puts it very well in his Stanford University Commencement speech (video - transcription):
You’ve got to find what you love….for the past 33 years, I have looked in the mirror every morning and asked myself: “If today were the last day of my life, would I want to do what I am about to do today?” And whenever the answer has been “No” for too many days in a row, I know I need to change something…almost everything - all external expectations, all pride, all fear of embarrassment or failure - these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.
Love what you do.
I recently saw an interview with Jan Guillou, one of the most successful and creative authors in Sweden. He had some interesting thoughts about creativity that I think applies to all pretty much an creative craft.
He is extremely structured and always does his writing in a predefined time box, when the time is up (triggered by an alarm clock or similar) he just stops typing, right in the middle of the sentence he was just typing.
I think there are two lessons learned here. I am myself a big fan of using time boxing to boost creativity and to force myself to eliminate the not so important stuff and focus on the essence, the things that will lead to actual progress. This holds both for writing, coding and meetings.
The second lesson is to stop right in the middle of an unfinished task. Doing that will help you to:
It reminds me of an advice Kent Beck gave in his book Test-Drived Development: In which he states that you should always stop coding with a failing test. This is the exact same idea; knowing what to start with when you come in to work the next morning, more easily loading in the “context into RAM” and forcing your self to become productive immediately.
Another, fairly controversial comment that I remember from the interview with Jan Guillou was:
Inspiration is for amateurs.
Although a bit arrogant, I think it is true and something that holds for most creative crafts. Inspiration is overrated, what matters is perseverance and creative talent (in that order). If you just start with whatever you are about to do, then inspiration will usually follow.
