Comparison of the high end BPM/SOA product offerings

After a long week of working excessing hours on a BPM project coming to the end of its development cycle, it was nice to spend a day just catching up on emails, drinking large amounts of coffee and reading a few new BPM/SOA articles.  One such article I found quite interesting and thought I’d share.  AMR Research have issued this report on SOA/BPM products in the higher end market share and offer a great comparison of features and architecture by asking each vendor to demo running a common process.  It’s a great birds eye view of the big vendors and how the BPM/SOA tools they offer integrate with their application servers, ESB’s and registry features of existing products.  My only issue was that it appears to be very non Micrsoft with Biztalk Server being left out of the picture… it has an solid foundation in IIS, MSMQ and .NET and is an SOA server afterall with its EAI/B2B/BPM capabilities? – The report itself is not exactly fresh from the shelves (issued in 2007) but still brings interesting and valid findings to the table.

Anyway, a good report by AMR (which I think may have been leaked as you’d normally have to log into AMR’s web site for report access.

check it out here.

Jargon Explained (Quickly!) : Frameworks, Methodologies, Models.

Initial disclaimer (When I say models, I’m talking about the type you draw up, not the other type… sorry chaps)

So not too long ago, someone asked me what I know about the ‘Zachman Methodology’ and the ‘UML Framework’?! – It got me thinking that all of these enterprise architecture terms and their respective ‘technologies’ can confuse a person; what exactly is a methodology or a framework and what approaches are classified as what?

Each time some new group of companies comes up with a new approach to software / process modelling, IT project management and their position in the overall enterprise architecture model it gets even more confusing for the man in the street.  Acronyms such as RUP, UML, TOGAF and SCRUM (well, this one isn’t really an acronym) can be found on CV’s the world over and are closely tied with IT infrastructure and software development projects.  What I’m attempting to offer in this quick article is some separation and understanding of some of the common frameworks, notations and methodologies relating to both business process and software development in an enterprise. I don’t claim to know them all, but I’ve experienced ZACHMAN, AGILE and UML via projects I’ve worked on which sparked my interest in the more pragmatic and structured view of BPM/EAI and Software planning and development.

What’s what?

What we mean by Framework, Methodology and Models/Notation. I’ve nicked what I feel is the most comprehensive definitions from one or two well known ‘wiki’ ;-) sites (plus my additional comments):

Framework : A basic structure of support. Basic indeed!? – In terms of traditional building architecture, the framework is the mechanical structure of the building.  In software design its more of a conceptional foundation for solving a particular software problem.  The .NET framework for example is named so as it forms the basic class structure and rules that allow software to be rapidly built and executed in a secure way on the Windows platform.  Zachman is the framework for the architecture of an enterprise (more on this in future articles).

Model:  A diagram representation of a physical or conceptual ‘real word’ object. Frameworks (structures) of all kinds can be broken down and described in detail.  Physical or conceptual idea’s are best described through images with descriptive notation or ‘Metadata’ (data describing data).  In traditional building architecture, blue prints are the model used to convey what conceptually needs to be built – it contains detailed information about sizes, materials and relative positioning (the model’s metadata).  In terms of the structure of the world, maps provide the model to getting around and understanding distances etc and in software development, models describe the classes, objects and information that make up the desired software solution. It’s important to note that models not only describe the framework / structure of something but the way parts of that thing interacts (the behaviour).  Creation of models (or modelling) pays a large part in the planning of anything and in terms of an enterprise, the planning of business processes and software development amongst others.  UML (Unified Modelling Language) is one of the more well respected, widely used and popular modelling styles (albeit not without its share of criticism) and is used to model software behaviour and structure and business processes.

Methodology: A system of principles, practices, and procedures applied to a specific branch of knowledge. In terms of enterprise architecture, this describes the principles, practices and rules to delivering an enterprise project which ultimately supports the enterprise architecture model and the business model in general.  It’s an approach on how parts of the enterprise architecture should be planned, developed and monitored.  All projects tend to follow a particular type of ‘methodology’ that best fits with the desired results. Agile and Six Sigma are applied methodologies for software development and business process projects.

Metastorm BPM : Corrupt saving

I’ve noticed on quite a few occasions, on different systems that when the Metastorm designer issues a runtime exception it can corrupt the saving of your Metastorm procedure if you opt to save your xep post exception.  I’ve had a 300 kb file be reduced to a 240 kb file on a save and unwittingly I have published this reduced version of the procedure many times before noticing that Metastorm has ’stripped’ out parts of my procedure (usually the forms).

Let me explain some more…

I’ve been working on a project for the last few months and the xep file is around the 300 kb mark at the moment.  I’ve noticed on a few occasions that when I save the procedure file, the Metastorm Designer fails to save the full procedure and the resulting xep is heavily reduced in size.  The main problem is that as I develop new functionality and deploy new versions of the xep, I’m unaware that I’m working with a corrupt procedure and that half of the forms I had designed previously had been stripped out of the file and my published work was in fact a reduced version of the system I am designing.  Once I’m aware of the problem, I can obviously back track to previous deployed versions of the procedure by retrieving those published xep’s, but of course this means I lose work I have spent valuable time developing and have to bring the file up to date with lost changes.

After a few noted occurrences, the common pattern seems to be when the Metastorm designer throws an exception of some sort.  If I attempt to save my procedure or library after the exception has been thrown, the designer is sometimes unable to save the xep in full and so appears to only save half  (watch the save bar closely when this occurs… it will get to 50% of the way and disappear).

My conclusion is that my scenario may have something to do with the problem.  I run Metastorm on a Windows Server 2003 VM and save state a lot of the time meaning that the Metastorm designer stays open pretty much all of the time.  The fact that the designer is open all of the time appears to be a contributing factor to the programs exceptions and so the corrupt saving of procedures.  My advice is to close the designer regularly.  I tend to find that designer runtime exceptions only occur when the application has been resident in memory for a long period of time and so to avoid the problem of losing your hard work in the form of corrupt saves, close the designer regularly.

I find that another best practice is to save daily instances of the procedure in separate directories.  Naming each directory in a date format (01122009) works well for me.

One final point on this… The Metastorm designer still throws quite a few run time errors and considering we are on v7 of the designer, this is really not good.  I’ve read a few articles and forum posts about version 9 and the overall view is that it appears to be a good upgrade on v7… Let’s hope the designer has been worked on!

Metastorm BPM : Locks when updating and refreshing database data in a grid

I continue to see problems with Metastorm / SQL Server when updating and viewing data in form grids.  Particularly in large procedures where database call levels can be high.  All Metastorm procedures need to pull data from a SQL server at some point in the workflow (not including native calls to the Metastorm database) and so this is quite an annoyance.  The problem I see surrounds updating, inserting or deleting data from a table and then refreshing the grid to reflect such changes. Database locks appear to occur (in my case on the KEY) resulting in a ‘frozen’ Metastorm form.

To illustrate the problem I create a grid of telephone numbers and look to the table dbo.TelephoneNos. Next I add a text field and a button to the form that will capture the phone number. The text field will capture the number and the button will run the %ExecSQL to insert the telephone information, finally the grid is marked as having dependants so will refresh based on the button press.

The issue occurs when the button is pressed. The sql is executed and the grid will refresh once the button has completed its actions however the INSERT and the SELECT (selecting the data from dbo.TelephoneNos to the grid) hit the database at the same time and SQL server appears to decide which order to run the processes in. In some instances the SELECT occurs first, keeping the INSERT at a wait status whilst the SELECT holds the KEY of the table. This lock stays in place restricting the INSERT until it times out, releases the KEY lock and allows the INSERT to proceed. At this point Metastorm reports the time out back to the client in the form of the helpful message ‘Root element is missing’ (possibly an invalid XML response with no root element).

The resolve for this (albeit not ideal) is to implement a dirty read on the table. Implement the sql NOLOCK hint in your grids table property = TelephoneNos WITH(NOLOCK) which instructs SQL to read uncommitted data allowing the INSERT to proceed and lock the table. Now I’m not proposing using this hint is good database design, all good DBA’s know this isn’t best practice but this approach may help in this instance and when the data being read is not critical information.

Metastorm BPM : Clearing service dependency on SQL Server

I carried out a Metastorm BPM 6.x to 7.6 upgrade today and also migrated the database from Sql Server 2000 on the local VM over to 2005 on a physical box (keeping the Metastorm install on the VM).  After changing the Metastorm DSN to look to the new 2005 server I wanted to remove the local dependency on SQL 2000 in order to quit the engine and free up resources, without having the Metastorm service quit.  Regardless of the Metastorm install looking elsewhere for its data, the Metastorm service still held dependancy on the local MSSQLSERVER install.

Unfortunately there’s no way to release the dependency without stopping the services, but knowing where to go to ditch the dependency is a good thing. Open up regedit and stumble on down to:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Metastorm Process Engine.  In there will be a  ’Depend On Service’ multi-string value. Remove the MSSqlServer entry (btw a Control Set is a single windows configuration set that holds information on drivers and services. Each time you boot a clone control set is created… this is how you are able to roll back to ‘Last Known Good Configuration’).

Because the service needs to re-read this reg value, the Metastorm service does need to be restarted. This shouldn’t cause to much havoc to your production environment, but the amount of times I’ve dealt with a non start on a ‘quick’ service reboot, chancing it seems daft… wait until office hours are over.

Reg Edit Dependency

UML : The 4+1 View of a System

UML Title

A little about UML

UML is a Standard Modelling Language for system design. It allows for the visual modelling of business processes, software and systems and is made up of:

- Notation (Symbols, Connectors, Notes and Values)
- Diagrams (The visual make up of a system).

UML is a set of specifications managed by the Object Management Group (www.omg.org). UML is extensible and it has been designed to be flexible and scalable (UML can be used to model a huge system or a quick business process sketch). UML can be used to model the following:

- Business Processes
- Application Structure
- Software behaviour (workflow)
- Model data structures
- Sketch out general structured idea’s
- Communication workflow
- Deployment workflow

The technology evolved from object orientated system design using classes and objects as the basis for components of the workflows. This concept is not only applied to programming however, business processes are made up of components that can be represented as objects. Lets cover some Object Orientation basics…

What is a system?

A related set of objects and instructions that together provide a working process.

What is an object?

Something (a component) that exists in the context of a system. An object is an instance of a class. A repeat product of a prototype / template idea (Class).

What is a Class?

Classes are templates that can create objects.
Classes have attributes (properties) and Methods (how it operates) associated with them that allow object instances to use.
Classes can inherit properties and methods from other classes, this is called class inheritence.

Relationships : Object Orientation

Created Objects do not always exist in isolation. They have relationships with other objects (ARE ASSOCIATED). For example, a speaker object and cd player object are working in associated to provide you with music. In the example of a fish bowl, The bowl object has a relationship with each individual fish object.

Class:Basketball Player
SubClasses:Center, Forward, Guard
Methods: Dribble the ball, take a 3 point shot.
Properties:Black Shorts, White Shoes.

Each object created from the subclasses hold a relationship with other objects in that they work together as a basketball team.

The 4+1 approach to modelling systems

When looking at an entire system as a whole, it can seem complicated at first and a good starting point if to create a 4+1 model diagram by breaking it down into 5 parts or Views. This model ensures you have considered and documented these 5 high level important aspects of any system. This approach breaks down the modelling of any system into the following views (inclusive of a use case view):

UML - 4 plus 1 System View

Each segment of the above offers its own perspective on the system architecture. Each of these views are clearly offering different information and so each view has a different UML diagram associated with it.  As you can see its made up of 4+1 parts:

Logical View : The logical objects (parts) of the system.
Process View: The process (workflow) of operation of these objects.
Physical View: The physical hardware and software the systems runs on.
Development View : The packages, run-time environments and class libraries used.
Use Case View : What the system does (functionality) from the perspective of a user (scenarios). Used in combination with requirement / specification documents.

Lets break these 5 system views down some more:

Logical View (The Systems Objects)

This view shows the components (objects) of the system as well as their interactions / relationships.  So effectively is the object model layout. It comprises the classes and objects within the system. UML diagrams that illustrate a systems logical view include :

- Class Diagrams (Most common UML diagram)
- State Diagrams
- Object Diagrams
- Sequence Diagrams
- Communication Diagrams

Process View (The Systems Workflow)

The process view shows the processes / workflow rules of a system and how those processes communicate with each other. It explores ‘what needs to happen’ in a system. UML diagrams that illustrate a systems process view include :

- Activity Diagram

Physical View (The System Environment)

This view shows the systems execution environment (hardware / software platforms). UML diagrams that illustrate a systems physical view include :

- Deployment Diagram

Development View (The Systems Software Hierarchy)

This view shows the parts of a system used in its operation (its layers of operation). It gives a building block view of the system. For example Packages Used, Execution Environments (i.e JVM or .NET), Class Libraries and Sub systems utilized. UML Diagrams that illustrate a systems process view include :

- Component Diagrams
- Package Diagrams

Use Case View (What the System is supposed to do)

This view offers an outside users perspective of the system. It captures the goals of the system and looks at the desired functionality of the system. Used along with use cases (user requirements / specification documents that the business analysts generally write) to understand the systems required use / functionality. UML diagrams that illustrate a systems process view include :

- Use Case Diagrams

VB.NET : System IO

In the second part of my VB.NET notes I cover system input / output and the available classes for reading and writing file system streams from the System.IO namespace.  This document is the second in an 8 part VB.NET Basics course, so if you’ve not already seen our VB.NET Types document, check it out here.

VB_System_IO

Metastorm BPM : Client Side Scripting Basics

Metastorm BPM allows for client side scripting using JScript and VBScript.  Since at the moment, IE is the only fully supported browser for the web client, I tend to utilize JScript as the language of choice (it’s a personal preference as I lean towards the C# like syntax).  Client side scripting can assist in form validation, loading dynamic text into labels, working with local Active X Objects and of course DOM object manipulation as well as a number of other useful client side ‘tweaks’.  You can obviously go mad with client scripting and get as creative as you need, but always remember that some techie somewhere will need to support your deployed Metastorm project once its been put into production and a script heavy process on either side (server or client) can become a support nightmare.

Client Scripting

So what do your client side scripts have access to? Client scripts execute on, well… the client so variables or functions are not available to the scripts, however the values of fields on the currently loaded form are available (since they are within the scope of the client, i.e. the data that the client browser is currently hosting).  Grids and fields on the form are holding values away from the Metastorm engine and so these values are accessible by the client.  Client side methods provide access to these ‘temporary’ field values, what follows is a quick summary of the methods you can access in your scripts:

eworkSetField(“<field name>”,”<attribute>”,”<value>”) – Sets the value of a named field

eworkGetField(“<field name>”,”<attribute>”) – Gets the value of a named field

eworkGetCell(“<name of grid>”,”<column index>”,”<row index>”) – Gets the value of the specified cell

eworkSetCell(“<name of grid>”,”<column index>”,”<row index>”,”<value>”) - Sets the cells value

The above allows you to pull the values of metastorm fields into your scripts for calculation or other manipulation and then set those values back to the client fields.  The client side scripts are called from the client extensions of buttons, grids and forms and support the client events OnLoad, OnBlur, OnClick, OnFocus and OnCellBlur. In other words, client side scripts can be executed from button clicks, entering a field, exiting a field, loading a form, entering a grid cell and exiting a grid cell.

OnSubmit / OnCancel

You are able to specify a client side script, usually on form load that hides or shows the Submit and Cancel buttons of a standard Metastorm BPM form. This is a very simple client side function that looks something like the following (the same applies for eworkShowSubmit) :

function hideCancel() {

eworkShowCancel(0) //This is a boolean parameter

}

On your forms ‘OnLoad’ event you can simply reference this parameterless script, like so (in client extensions) : OnLoad=hideCancel() & Language=JScript

On Submit

Whenever you create a client side script that is to be executed from an ‘OnSubmit’ event, you will want to return a boolean value as to whether the submit should be processed by the Metastorm BPM Engine.  Lets take a simple validation script, if the script validates the forms field values correctly, then a true value should be returned and the form will submit, else the script should return false, meaning that the submission will not occur, giving the user an opportunity to correct field values (an alert() should be used as a prompt to the user regarding what exactly needs correcting on the form).  An important point to note is that the ‘OnCancel’ event will ignore true or false returns as a client script cannot prevent the form from being cancelled.

The Document Object Model

Since Metastorm forms are represented in the browser, fields are subject to the document object model, therefore they are subject to standard DOM and JScript manipulation.  Lets take the document.getElementById method for example, this can be used to reference a metastorm field object within the client, as follows:

var MyData = document.getElementById(“myMetastormTxtField”).value //returns the value of the object being referenced, in this case a forms text field.

For fields such as a label caption, that references an html div (i.e. writing <div id=”MyDiv”></Div>) you can populate the innerHTML property like so:

document.getElementById(“MyDiv”).innerHTML = “<b>My HTML Content</b>”

Most HTML fields have a common set of properties and methods that can be used in client side scripting. Common field properties, that apply to text, check boxes, memo etc:

field.value – sets or returns the value of the field object
field.size - sets or returns the size in characters of the field
field.name - sets or returns the name of the field (document.getElementByName used to refer to the field)
field.id - sets or returns the name of the field (document.getElementById used to refer to the field)
field.innerHTML - gets or sets the string HTML content of an element (normally used for the DIV object).

… and common methods:

field.click() – simulates a button or checkbox mouse click
field.blur() - removes focus from the field
field.focus() - sets focus on the field
field.select() - selects the contents of the field (e.g. text contents of a text field of comments field)

Style

Lastly, I want to touch on style, since you may wish to manipulate a Metastorm form fields style characteristics. The style object is applied to other document objects in order to apply style characteristics. They are accessible by referencing the object the style is applied to:

document.getElementById(‘MyTextField’).style.property = ‘value’

Examples:

document.getElementById(‘MyTextField’).style.color = ‘red’;
document.getElementById(‘MyTextField’).style.font=”italic bold 12px arial,serif”;
document.getElementById(‘MyTextField’).style..textAlign=”right”;
document.getElementById(‘MyTextField’).style.visibility = “hidden”;

As you can see, client side manipulation can be useful to your Metastorm design needs, but be warned that making your procedure overly complex with client side scripts when Metastorm is quite capable of providing certain functionality is a bad move.

Six Sigma : The Birds Eye View

Six Sigma is one of those buzz words that company management teams tend to throw around. It may sound like high level tosh, but its actually one (if not THE) most respected methodology for improving business processes with the aim of increasing business revenue and competitive advantage. There are lots of useful examples of companies who have deployed a six sigma project and have their processing running at six sigma. So what’s it all about? In this article I attempt to give a 1000 foot view of what it is and try and explain what it can do for an organizations processes. Remember though that this isn’t nearly enough information to understand it completely. There are courses for becoming practitioners at certain belt levels and it can all become pretty costly, however recorded results and ROI have recognised that this methodology works.

So what is it?

Six sigma is an applied methodology (organized collections of methods/activities) for improving business process performance. It can also be seen as a problem solving methodology for minimising the amount of errors in a business process out of a possible opportunity for errors. It started in the manufacturing sector with its creator Bill Smith working at Motorola back in 1986. A process that performs at six sigma is extremely efficient and experiences only 3.4 errors per million error opportunities. That’s quite a target to achieve, so efficient that the statistical analysis of a process is required to locate every possible point of defect, that’s why a six sigma project generally produces substantial improvements in process and why its known as the THE best process improvement methodology.

Since a six sigma deployment project aims to gain such efficient processes, it effectively maximises the value of the process output (remember a business process is just one or more inputs, a collection of interelated activities and a value added output). A six sigma deployment project will use a plethora of different tools to analyse business process data at a detailed level. What I hope to convey in more detail in future articles is the stages of a six sigma based project, the roles and the tools for each stage of the project. Finally, the key point to remember is that EVERY organization needs to improve in order to obtain the competitive edge (against other improving competitors).

The project generally has two levels to make it work. Managerial and Technical. The technical roles will be responsible for the physical transformation of a process based on the information gathered. The Managerial level is where the project starts, it must have managerial buy in and be driven from the top of the organization (or business area), not hugely different from most business critical projects.

Management: Management must understand that the project will provide them much improved ROI (one of its biggest focuses) and also its customer focus (since for most businesses, the output of certain process may benefit the customer directly). Finally, management should be taking accountability for the project and leading it, including organizing the structure of deployment, the strategies and tools that are to be used and who will take on which roles (belts in black, green and yellow).

Technical: The technical portion is the definition (D), analytical study and measurement (M and A) of the ‘as is’ process and the improvement (I) and control (C) of the ‘to be’ process (DMIAC).

Variation

Each part of a process output (i.e a service, a product or transaction) has characteristics that are generated as a result of the process. It’s looking at these characteristics that results in ‘component’ improvements. Improving the individual characteristics of each small component part of a bigger product, service or transaction is what results in a larger scale improvement in the end result. Each time a business process runs, its component parts (and their characteristics) don’t generate exactly the same output each time and together with all other parts of the process this can result in defects in the processes end result. Each ‘difference’ in component processing is known as variation.

Let’s take the production of a teddy bear, each bear could roll off of the conveyor belt in a slightly different form to the previous. This variation could be a result of variations in the way the bear’s nose is
attached. Understanding the variations is the key to change. As component defects multiply, the percentage of overall error is large (especially if one component of the process is dependant on another, compounding defects). Variation happens naturally as a result of cause and effects, its a case of managing or controlling that variation to produce really efficient processes. One important point to note is that variation can change over long periods of time, therefore short term variation and long term variation should been seen as different (variation may increase as a higher rate over time for example).

The Sigma Scale

The scale is the measure of how well a component characteristic performs against its requirement. The higher the sigma score the more capable the characteristic. If a component for example, lets say our bears nose assigning machine has a defect rate of 31%, then it is operating at two sigma. The goal of course is to up the operational accuracy and lower defects to a 0.0000019% defect rate, or for the process to performance at six sigma.

Six Sigma Chart

The Six Sigma Equasion.

Y = f(X) + E

or in simple terms, Process Output (Product, Service etc) = Component Function (Functions Inputs) + Defect Variable. Simpler again, a certain set of process inputs is transformed by a process component and together with its known variations generates a process output (with or without defects dependant on the variation variable). The DMAIC process can improve

DMAIC Methodology

Define, Measure, Analyse, Improve and Control are the technical systematic methods of the problem solving process. The DMAIC stages can improve any type of process in any organization, increasing  its efficiency and effectiveness. The ‘Define’ stage deals with setting the context and project objectives. ‘Measure’ is the stage for getting the baseline performance for the process to be improved. ‘Analyze’ covers using a selection of tools to understand the cause and effect relationships in the process. ‘Improve’ is of course the roll out of modifications to the process and finally ‘Control’ is the stage at which plans are put in place to ensure that improved processes are sustained.

The DMAIC methodology is performed by trained practitioners and ‘tollgates’ are met. As one tollgate is completed, the next one can be tackled.

Six Sigma DMAIC

As I mention, performing these stages can include a lot of detail along with a lot of available tools to perform them. I’m not getting into that here as its far to much to go into, but there are lots of great Six Sigma resources on the web along with a great little iphone application that covers the tools and methods available to each of the stages in te DMAIC methodology. Pop Six Sigma into itunes for more information. There are also certification programmes for each of the belt levels that train individuals the tools and methods involved in the DMAIC process. Read a lot about six sigma and then get involved in a six sigma based project.

Metastorm BPM : User Action from a read only form

Initiating a user action from a form can be a useful and efficient design tip.  It is quite possible to launch a user action from a button, but only within the same workflow and from a read only view.  The reason the folder needs to be in read only mode is because when the folder is in ‘action’ mode, the folder row is locked on the database.  To my knowledge and testing, I’m unable to open just any old folder action (so an action from a different workflow within the same procedure file).  For launching an action from a button, here’s how its done:

1) Create the following client side function:

function OpenAction(actionName)
{

var IBaseOpener = new esOpener(“folder_action”);
var IChildOpener = esFolderActionFormOpener(IBaseOpener);
IChildOpener.openWindow(“”,”Metastorm BPM Server”,document.getElementById(“FolderID”).value,document.getElementById(“FolderID”).value,actionName,”");

}

2) Add a button to your form and set its action to ‘client operation’.

3) Add a button press script call – OnClick=OpenAction (“NextAction”)&Language=JScript (you’ll see that we are passing the name of the action we want to kick start).

This is tested as working in Metastorm 7.6. Remember that the user clicking the button must have role access to performing this action, otherwise errors will occur.

One final point on this subject is around the eServiceName of your Metastorm BPM server. The code above assumes the out of the box service name of ‘Metastorm BPM Server’… If you or a colleague in IT know something about Metastorm and have been mucking about in the web config file, you’ll know that you can change the service name.  Just be aware that if you have Dev, Test and Live Metastorm servers with different service names, you’ll need to use the eServiceName variable when implementing the above code.