SAND

Stability on SAND

Build a solid application over changing technology using Structs And Nodes Development (SAND).

By Eric Parker
March 2003

The past five years have seen a lot of API changes, and at least as many best practices for coding to them. Things have been changing rapidly, and there's not much slowdown in sight. That's a grim outlook if you need to stabilize a large scale development effort. How can you keep up with the latest and best without ripping up your application every 6 months?

One answer is to simplify. And we're not talking slight streamlining here, the idea is to take things all the way back to some of the most basic data structures and business logic declarations. By doing that, your application ends up as plain Java, and you address all your supporting technology by metacoding. That's what the Structs And Nodes Development (SAND) approach does.

The SAND approach

Remember structs from C? A SAND Struct is like that except it's a Java class with only protected data elements. Each data element has extra javadoc tags that explain things like valid ranges, invalid and default values, lengths and suchlike. Each Struct also declares what verb forms it supports, where verbs are things like Query, Collection, Update and History. The message forms corresponding to each struct are created automatically by the build/deployment environment (aka the "sandbox").

Here's an example of a simple Struct definition:

/**
 * A StudioItem is an instrument or piece of audio gear.
 *
 * @sand.structmessage persist
 * @sand.verbforms update query collection history
 * @sand.summaryfields name type
 */
public class StudioItemStruct
{

    /**
     * What we call this item
     *
     * @sand.stringlength 40 80
     * @invalid "STUDIOITEM_MISSING_NAME"
     */
    protected String name;


    /**
     * What kind of studio item this is
     *
     * @sand.enumint MISC 0 "miscellaneous"
     * @sand.enumint INSTRUMENT 1 "instrument"
     * @sand.enumint PROAUDIO 2 "pro audio"
     */
    protected int type;


    /**
     * The FMV of this item in $USD
     * 
     * @sand.metatype money
     * @sand.decimalize 2
     * @sand.range >= 0
     */
    protected int value;

}

A SAND Node is another Java class with only protected data elements, but it represents a piece of business logic processing. A SAND application is comprised of several nodes that talk to each other via messages. The term "node" is borrowed from graph theory, since when any portion of a SAND application is drawn out on a whiteboard it looks like a graph.

A node has two main components: its configuration parameters (represented by the data elements), and its I/O declarations. Each input and output, whether synchronous (analogous to a procedure call and accomplished via call/receive) or asynchronous (analogous to pub/sub messaging and accomplished via send/subscribe), is declared using special javadoc tags.

Here's an example of a simple Node definition:

/**
 * The AssetManager verifies updates to StudioItems, and adds up the total
 * value of all the items we have.  The total is logged and emitted as a
 * Stats message.  It is computed on startup and after each update.
 *
 * @sand.receive
 *  org.sandev.MusicStudio.sandmessages.StudioItemUpdate
 *  org.sandev.MusicStudio.sandmessages.StudioItemUpdate
 * @sand.call
 *  org.sandev.MusicStudio.sandmessages.StudioItemQuery
 *  org.sandev.MusicStudio.sandmessages.StudioItemCollection
 *  studioItemQuery
 * @sand.send
 *  org.sandev.basics.sandmessages.stats
 */
public class AssetManagerNodeDecl
{

    /**
     * The percentage amount to add to the FMV for insurance purposes
     * @sand.range >= 0
     */
    protected double insuranceMarkupPercentage;

}

The sandbox automatically writes all the code necessary for communications, so to complete your application code you only have to override the methods that handle the incoming messages and/or the methods that hook the control processing. You can query for additional data and/or call other nodes to help.

Your supporting code

After defining your application data structs and processing nodes, the next logical step is to consider the supporting code needed for it to run.

SAND divides supporting code into six major categories: Messaging, Persistency, UI, Control, Configuration, and Framework. The sandbox provides default implementations for all of these, which you override as required for your application. The default supporting technology for each category could vary a lot with the sandbox you are using, but here's how it works in SandBoss, an open source reference implementation built on top of the JBoss(tm) application server:

Messaging: SandBoss provides a DirectCallMessager, which translates messaging calls within the same process space into direct method invocations. Other Messager implementations extend this class to provide communications over specific protocols (web services, JMS), or specific communication products.
Persistency: SandBoss provides a default Persister which uses the HSQL database via JDBC. Other Persister implementations can leverage the same JDBC and SQL utilities, or leverage other technology.
UI: SandBoss provides fast XML serialization for all messages, and support for UI development at several levels through SANDForms. The default SANDForms implementation supports browser-based UIs using XHTML forms.
Control: SandBoss uses JMX for control, and generates MBeans for each node definition. The nodes on each server are automatically deployed in a tree off the SandRoot, and can be accessed via the JBoss(tm) control panel.
Configuration: SandBoss provides a standard configuration editor, which saves the entire deployment as a config.xml file in the deployment env directory.
Framework: SandBoss provides a dependency tracking build and deployment function using Jakarta Ant. The build produces a navigable web representation of all project documentation, including custom overview pages. SandBoss provides integrated support for logging, statistics, exception handling, and testing. There are built-in hook points to override, replace, or add processing.

Don't code, metacode

If you are implementing any supporting technology implementations, the preferred approach is to write a generator doclet. The build system provides access to all the struct and node definitions for the current deployment, so you have everything you need for code generation.

Using a generator guarantees consistent implementation. That prevents a ton of bugs, but the real value here is in maintenance: When a technology API changes, or the best practices change, you only have to change the generator. Your application source remains intact. That means you can keep up with API updates, compare alternative implementations, use vendor-specific extensions, and do other things that would be too risky if you were hand editing code.

This consistency also has a huge benefit when working on cross-context problems. Things like security, latency, failover, throughput, error handling, performance and so forth usually mean working horizontally across all the application code. Having a consistent implementation, and being able to change it in only one place, really helps get the work done.

Metacoding is incredibly powerful, and a central part of SAND.

Your deployment

With SAND, a deployment consists of three things:

your node instances,
your initial data,
identification of the servers you are running on.

These elements form the main components of a Configuration message. A deployment consists of a Configuration instance, which is serialized into a config.xml file and deployed onto each server.

The Configuration is edited using the configuration editor that comes with the sandbox. This is generated specifically for your application, so it follows the data model you've defined. At startup time for each server, the appropriate nodes for that server are started with the specified configuration parameters. Once all the nodes are started, the system verifies the integrity of the persistent store, and checks that the initial data is in fact present. If the data is missing, then it is automatically added. This ensures a consistent starting point for the application.

Synchronization of configuration changes made while the application is running is left to each application. The typical approach is to use ConfigurationUpdate messages sent between ConfigManager nodes on each server. Evolution of the system over time can be tracked via changes to the underlying XML file.

Testing

Because a Configuration completely describes the initial state of the processing nodes, and the starting data, it forms an excellent foundation for testing. To define a test, a Configuration is created which includes a MessageDriverNode instance, which is in turn configured to load one or more TestScripts.

When the sandbox starts up, it loads each test Configuration it finds in the test directory (located off of $SAND_DEPLOY). By managing which Configurations and TestScripts are copied into this directory, it is possible to manage a regression suite as part of the build processing.

Because any given collection of nodes doesn't know if incoming messages are coming from a MessageDriverNode or elsewhere, it is possible to set up test scenarios for the vast majority of application functionality. The same mechanism can be used for unit, regression, milestone, smoke and other testing.

Conclusion

The SAND approach allows you to build a stable application over changing APIs. By working from a canonical application definition, it's possible to develop your supporting technology interfaces and application source separately, taking full advantage of the latest technology without negative impact. The open source SandBoss project provides a default sandbox implementation.

In the future SAND may be applied to other languages beyond Java, or potentially become a meta-language itself. However it's likely that it will stay fairly settled for a while, being tempered and proven with the SandBoss reference implementation. The goal would be to build a development standard, and encourage the development of othe sandboxes.

Let's build some apps.

Links:

Eric is a software architect. He can be reached at eric@sandservices.com

JBoss is a trademark of Marc Fleury