Their product ActiveVOS is one of the first to support BPMN 2.0 diagrams, but they use it to create BPEL. Such a use was actually anticipated by the developers of BPMN 2.0 – IBM, Oracle, and SAP – all of which have been shipping BPEL engines for some time. So that’s not such a big deal, and Michael doesn’t really need to defend doing it. Saying it’s a “simpler” way to go than executable BPMN 2.0, however, is a stretch.

Simpler for whom? When you sort it all out, he’s actually saying BPEL is simpler for an engine vendor because it doesn’t have overlapping or alternative constructs like BPMN does. No one would argue with that, but who cares? No engine vendor is going to support every possible BPMN 2.0 element and attribute called out in the metamodel. And I’m not saying just in the first release. Not ever. In that sense, BPMN 2.0 is not a self-contained execution language like BPEL is.

However, for process modelers, and even for executable process designers, there’s no way BPEL execution makes BPMN modeling “simpler.” That’s because the subset of BPMN supported with BPEL execution excludes the very features that make BPMN attractive to business-oriented modelers in the first place: things like freeform looping back to a previous step in the flow. BPEL is inherently block oriented, like a computer program, while BPMN is inherently graph oriented, like a flowchart. 

In the BPEL mapping section of BPMN 2.0, the spec describes “basic mapping” of block-oriented BPMN. Then follows a section on “extended mapping” where the BPMN does not conform to BPEL’s natural block structure. The spec essentially says here engine vendors are on their own, noting “in many cases there is no preferred single mapping of a particular block, but rather, multiple WS-BPEL patterns are possible to map that block to.” 

Freeform loopbacks are a prime example. They typically describe what BPEL calls “interleaved loop” structures. To map these to BPEL, the BPMN spec says the following:

The looping section of the Process, where the loops first merge back (upstream) into the flow until all paths have merged back to Normal Flow, shall be separated from the main WSBPEL process into a set of derived processes that will spawn each other until all the looping conditions are satisfied.The section of the process that is removed will be replaced by a (one-way) invoke to spawn the process, followed by a receive to accept the message that the looping sections have completed and the main process can continue. 

In other words, just to support freeform routing in the diagram, a single BPMN process must be split into multiple BPEL processes that signal to each other through messages. Is this simpler? No I don’t think so.

Over the past year, engineers at Oracle and IBM have been making changes to their process engines precisely to break out of the BPEL box, not just to better handle BPMN constructs like OR gateways, boundary events, and sequence flow loopbacks, but ad-hoc behavior as well. If BPEL were adequate to execute processes the way business wants to model them, it would have become the BPM runtime standard. It hasn’t.

Like Lombardi in Teamworks 7, Appian 6 is emphasizing new capabilities that enable “massive reuse”, design team collaboration, and one-click deployment. This is natural as the human-centric end of the BPMS market evolves from isolated projects to “enterprise” BPM programs. Of all the human-centric vendors, Appian seems most at ease with complete business empowerment.

All of Appian’s design and runtime administration tools are browser-based, and Appian 6 now supports import, export, and publishing of all application components – flows, forms, rules, etc. – in one click. In order to support today’s multi-application BPM environments, Appian 6 has also adapted the portal to provide tab-selectable application-specific views.

Appian FORUM appears to be the successor to the promised online “marketplace” for Appian Anywhere, the company’s cloud-based offering. Available to customers of both Appian Enterprise and Appian Anywhere, the new FORUM provides a library of downloadable application templates, process templates, and reusable tasks called Smart Services, developed by customers, partners, and Appian itself. Templates include Employee Onboarding, Spend Request, and IT Service Management. Appian-developed templates are free, while partner-developed are fee-based.

Appian’s new professional services framework is built around a its own BPM maturity model that aims to move customer self-sufficiency from the isolated project level to the enterprise program level. Like Lombardi’s similar offering, it takes a fairly narrow product-centric approach to “BPM in the large.” At the same time, Appian appears to be expanding its relationship with Mega, a major player in the BPA space. That could help to enlarge Appian’s view of the BPM market to include the majority of “BPM” users that are still at the modeling and analysis stage and not ready to automate anything just yet.

BPMS Watch readers know I am a big fan of OMG’s Business Process Modeling Notation (BPMN) 2.0, which has passed its first approval hurdle and is now in the Finalization Task Force stage. A major reason is that for the first time, BPMN has standardized the schema for XML interchange of process models. That means you will be able to create a BPMN model in one tool with confidence you can open it in a different tool. I think that’s what every user expects from a “standard,” but BPMN never had it until v2.0. But there is one part of the standard that the team messed up big time: Diagram Interchange (DI), meaning the graphical layout of the shapes and symbols.

Prior to BPMN 2.0, the de facto interchange format for BPMN models has been XPDL, an existing standard from the Workflow Management Coalition (WfMC) modified to handle BPMN. Each node in XPDL represents both an element in the BPMN semantic model and its corresponding visual element – including position, size, and other graphical details – in the diagram. But BPMN 2.0 separates the semantic model from its diagrammatic representation. That’s in a separate model – the DI model – that sits alongside the semantic model in the XML interchange document.

That’s actually a good thing. It allows multiple diagrams (views, pages, etc.) to represent the same semantic element in different ways, such as different levels of detail. For example, if you have a hierarchical diagram of a complex process model, with a top-level view showing the entire process at very coarse-grained detail and many child-level views showing increasing levels of detail, BPMN 2.0 maintains a single consistent semantic model for the process, referenced by all the various views. That is good for maintaining the integrity of the model. In contrast, with XPDL each view of a particular model element represents a different XML element in the export, so if a subprocess or pool appears on multiple pages of the model, it is up to the tool – or the modeler – to maintain consistency between their definitions.

But OMG did not want to make DI, the graphical model, BPMN-specific. They wanted it to support other types of diagrams as well, particularly UML, something completely different but also owned by OMG. And they didn’t want to define the format in an XML schema document (XSD), the universal standard used by XML tools, SOA, and BPM. They wanted to define it in XMI (XML Model Interchange), OMG’s own standard for interchanging UML models. The problem with DI as specified in the BPMN 2.0 draft is that BPMN-specific diagram structure is not defined in a format usable by XML tools. Instead, it is defined in a separate “library,” which you could think of as a cheat sheet off to the side to remind you of the particular attributes allowed or required for BPMN-specific shapes.

If you think that is a strange way to define what is still nominally a notation (i.e. graphics) standard, you are not alone.

The problem is not that it is impossible to define a BPMN 2.0 model using the graphics as specified by DI, just that it will be too difficult to do so in practice. Unless you can represent DI as a true BPMN-specific schema, tool implementers cannot use the rich array of XML tools available to validate, transform, and test their XML exports. Graphical model interchange is difficult enough in XPDL, where the graphics come pre-attached to their semantic elements. A few BPMN 1.x tool vendors – ITP Commerce, Fujitsu, Global360, eClarus, TIBCO, BizAGI, and Lombardi – have achieved some level of BPMN interoperability using XPDL 2.1, but it requires tweaks to the XPDL using XSLT, the standard language for XML transformation. It would be extremely difficult to create those tweaks with DI – and they no doubt will still be needed with BPMN 2.0 – since tools that create, execute, and debug XSLT all need a proper schema. OMG needs to try again on the DI part.

It is no great feat to turn DI into a proper BPMN-specific schema. I did it myself (BpmnDi.xsd) in a few hours. I also created an XSLT that maps XPDL 2.1 to BPMN 2.0, using BpmnDi as the graphics model. If you are interested in such things, you can download from http://www.bpmnstyle.com on the Tools menu page. If your BPMN tool outputs XPDL 2.1, you can try it out on your own models. [So far only the XPDL "standard"-level conformance is supported.] If you are interested in interoperability between BPMN tools, please make your sentiments known to OMG. These implementation “details” are what the FTF phase is all about.

A long time ago, when we wasted time talking about the relative merits of BPEL vs XPDL, Keith Swenson used to go on and on about the need to port the diagram graphics in addition to the semantic model. (BPEL has no graphics model, but today that seems almost besides the point.) I argued at the time that graphics portability was hopeless anyway, since each tool uses different graphics libraries, fonts, connection points, etc. You could never get the graphics to port exactly and remain editable.

The truth, as it turns out, is something in between. Portable BPMN graphics is really an approximation of the layout, capturing the arrangement of nodes in swimlanes and connector bendpoints, but not necessarily fonts, precise connection points, etc. Those are still determined by each tool. This is more or less in line with user expectations, and XPDL can deliver on this… in principle.

Upon further investigation of the Blueprint-ITP issue, I found out that ITP ignores the graphics info in the Blueprint XPDL and reconstructs the diagram from the semantic model (sans lanes). The engineer said it was too complicated to explain, as he headed off for vacation. So I had to figure it out on my own.

Turns out there are several related issues. One is coordinate systems. “Standard” XPDL measures node xy coordinates as position of the top left corner relative to the top left corner of the page. Yes, XPDL expects diagrams to have pages, but they are not required. Blueprint has no pages.

Second is scale. There is a PixelsPerMillimeter scale attribute in XPDL, but tools can omit or ignore it. Scale turns out to be important because most of the graphics in the diagram are fixed in size.

Third is standardization in XPDL. The spec says, per Robert Shapiro, my XPDL guru:

X and Y coordinates of node’s upper left corner (bounding box). Tool specific and depends on ToolId. Usual implementation based on upper left corner of page being (0, 0) and all coordinates >= 0.

The operative words here are tool-specific and depends on ToolId. You can’t have real portability when key elements are tool-specific. It’s more like EDI: you can integrate, but it takes side agreements to do so.

In the end, it seems that ITP only imports the graphics when it recognizes its own ToolId. OK, I can forge that. And it measures coordinates of node centers relative to bottom left corner of page. OK, I can do the math. And it uses the PageId attribute. That’s a bit harder, since Blueprint is pageless. When it constructs the graphics from the imported Blueprint XPDL, ITP creates pages for the expanded subprocesses. This is the hierarchical style I teach in BPMessentials. I’ll figure that out somehow…

I set about writing an XSLT transform that would map the Blueprint export XPDL into something ITP can import, including graphics. I played around to arrive at a scale factor that made the imports look OK. If I had more time, maybe I could figure out how to create multi-page hierarchical models in the XSLT, but to start I just put everything in one page — which is what the Blueprint export actually provides.

Here are the results. This is a fragment of the sample Hiring process on the Blueprint site:

Here is how it looks imported from the raw XPDL. Note the lanes have been lost and pages for the subprocesses have been added.

Here is the import after my XSLT transformation:

The lanes are back! Actually, I had to do two things first in Visio. Send the expanded subprocess shapes to the back layer in order to see the connectors, and change the page size (fixed at A4 in the XSLT) to fit the drawing. Note the connectors do not touch the activities in exactly the same place as in the Blueprint original, but the general layout is preserved. 

And here’s a cool thing you can do in the ITP tool: convert the expanded-inline subprocesses into collapsed subprocesses with the expansion on another page of the diagram, as in the hierarchical style. Here is what that looks like:

I’m not much of a programmer, but it seems to work. If you are interested in the xslt, you can download it here. I used the xsl:function tag, so it requires an XSLT 2.0 engine, such as XMLSpy (the engine is free from Altova). It probably could be rewritten in XSLT 1.0 without that.

After months of my nagging Lombardi about the need for this, it popped up like a surprise gift in the July Blueprint release. You might think they always had this. Not really. Notwithstanding Forrester’s swooning over Blueprint’s BPDM export feature, that never really worked. The one that does work is based on XPDL 2.1, which has emerged as the interchange format of choice for BPMN 1.x. Robert Shapiro and I are working on the portability conformance levels in XPDL 2.2, which supports BPMN 2.0, and we welcome Lombardi’s commitment to BPMN model portability between tools. 

 There is a small problem getting lanes to interchange properly with Process Modeler for Visio. I am trying to get ITP to debug that for me and then I will post the minor xslt tweak needed to go between Blueprint and Process Modeler for Visio. This is key for me as I am committed to BPMN training on both tool platforms. Kudos to Lombardi for making it happen.

In these tough times, even the most change-resistant organizations are re-examining whether past practice should continue to govern standard operating procedures. Government and airlines, for example, spring to mind. Last week, I saw further evidence of this in delivering a BPMN training class to one of the many Federal agencies involved in financial regulation. I was surprised to find that most in the class were experienced process modelers already. Many had prior BPMN experience, for some including BPMN-based BPM Suites. The processes of greatest interest concerned internal policies and procedures: hiring and employee onboarding, granting security clearances, etc.

On the flight home, another pleasant surprise: wi-fi and AC power at my seat in coach! For about the same price as one of those nasty “snack-paks” they offer, you can get in-flight Internet service coast-to-coast. While other airlines solemnly invoke policies and procedures to ban cell phones, GPS devices, and even powered headphones below 10,000 feet, Virgin America has figured out that giving business travelers an extra full workday might be good for customer satisfaction.

On the flight home, I noticed something familiar in the latest column from my favorite business journalist, Steven Pearlstein of the Washington Post. Headlined “The Productivity Revolution Trickles Into Government,” the column relates how a top government priority, $80 Billion in loan guarantees for clean and renewable energy, did not expect to achieve approval of its first guarantee until the end of 2010, a full five years after Congressional authorization. Lots of rules, you see. Policies and procedures.

Not good enough, said Obama’s new Energy secretary Steven Chu on taking office in early 2009. He challenged his staff with the simple question, “What would it take to get the approvals done in three months?” Well. If you ask it that way… It turned out the secretary had the authority to waive, modify, or expedite half of the procedures – no one had bothered to ask – and the first loan guarantee was issued to a solar panel manufacturer in March 2009.

The real problem was no one was thinking about the end-to-end process. As Pearlstein relates,

[Chu's consultant] assembled staff members from all the offices involved and vested them, collectively, with end-to-end responsibility for the approval process. Up to that point, people had been focused only on their own narrow tasks, and when they were finished they tossed the application over the wall to the next office. Once they started working as a team, however, they found that some of the work… could be done simultaneously rather than sequentially.

While he could not have provided a better summary of BPM, Pearlstein frames the story in the more familiar narrative of “productivity,” government plodding versus the “efficiency” of the private sector. But this is not really about productivity, which measures output of work per unit of labor or capital applied. This is about cycle time, which is more often hampered by the structure of the end-to-end process rather than resource efficiency: unnecessary steps, excessively sequential processing, and no KPIs or governance at the end-to-end, customer-facing level.

Other recent government “productivity” gains Pearlstein cites are similar: DoD civilian security clearances reduced from 440 days to 40 days. IRS new hire process reduced from 120 days to 15. This tells me that while the principles of BPM are taking root in government just as they have in the private sector, the media still does not understand the role of process. They don’t yet get the BPM story.

That story begins with understanding the end-to-end process as a single “thing,” and measuring performance from that end-to-end, customer-facing perspective. That means, in turn, you need to be able to document the process end-to-end in a concise but readily understandable way, so you can clearly visualize the handoffs, exception paths, rework loops, and other sources of delay and stalemate.

Once you can visualize the process in this way, it becomes much easier to tackle process improvement. What would it take to do it faster, with less cost, or with higher quality? Why do we do this step here? In fact, why do we do it at all? Aimed at a process diagram laid out on the table rather than at the responsible person or organization, such questions lose the sting of a personal attack. They put the focus on the process rather than on the resources performing it. That’s a plus. Sure, implementing some of the proposed improvements may involve some office politics, but it no longer requires a miracle.

It all starts with documenting the as-is process, in a clear and sharable way. BPMN, the industry standard for process modeling, is great for that, and government is eagerly adopting it. But using BPMN effectively to create process models that stand on their own, clear from the diagram alone, requires a methodology and a consistent style. You can get that in my new book, BPMN Method and Style. Even better, you can learn it in a two-day class at BPM Institute, in Washington DC September 23-24 or in New York November 3-4. So whether you regulate Wall Street or work on Wall Street, here’s a way to jump-start your organization’s process improvement efforts.

As BPM begins to expand beyond isolated projects to mainstream programs at the division or enterprise level, there is a need to engage a far greater number of business people in the effort. That’s not easy, and achieving it is going to require significant change in the way BPM is practiced.

The most important role for business is probably documenting current-state business processes and analyzing them for possible improvement. But conventional practices in this area are inefficient and inherently small-scale.

One typical scenario involves a team of business analysts or business architects mapping various processes using Visio or Powerpoint. Doing this requires no special tools or training, but the resulting process maps are not easily shared without elaborate accompanying explanation or documentation. The reason is each modeler makes up his or her own conventions about what the shapes and symbols mean. There is no defined methodology. The sphere of understanding is thus limited to the individual modeler. This scenario doesn’t scale, and you can’t base a BPM program on it.

In another typical scenario, a process improvement consultant comes in and works with a small project team, with the goal of fixing some process problem in the business, or at least mapping out a plan of action. The consultant’s emphasis is typically on a facilitated methodology, not on tools. In fact, the “tools” employed are frequently yellow stickies on the wall and paper-based tables and charts. While this is fine for fixing isolated problems, it does not scale to program-level BPM. Again the sphere of shared understanding of the process, as-is and to-be, is limited, in this case to a small radius around the facilitator. And paper-based tools? Come on, this is the twenty-first century!

At the other end of the spectrum, a small group of business architects or enterprise architects may use repository-based Business Process Analysis suites to model the business at the division or enterprise level. Besides process, these suites model data, strategies and goals, KPIs, policies and rules, organizations and roles, services and systems, and so on. The tools are great for looking at the company’s core end-to-end processes and aligning them for consistency, data integration, and governance at the enterprise level. But they rarely get down into the weeds of the problems affecting process performance and quality at ground level. They are really about architecture, not process improvement. The tools are proprietary, very expensive – over $10K per seat – and require weeks of training in the methodology. For this reason alone, the approach does not scale.

Each of these approaches brings something important to engaging the business on a larger scale: software tools that are easily accessible, business-friendly, and inexpensive; a way to clearly express problem areas, including exceptions, in the process model; a well-defined methodology for translating process understanding into diagrams that can stand on their own; and a repository to facilitate model sharing, cross-referencing, versioning, and reuse. The trick is combining them.

Pieces of the solution are coming into focus. BPMN is one key ingredient, as it provides a business-friendly visual language for process modeling, able to express exception-handling and other process details, with predefined meanings for the shapes and symbols. Thus the diagrams, if created correctly, can stand on their own, shareable across the enterprise without accompanying explanation. BPMN is a standard, and tools are inexpensive, in some cases free.
A model repository is another. This is a database of process models supporting versioning and governance, allowing collaborative editing by team members without sacrificing authorization and access control.

Making these tools easily accessible across the business has been a challenge in the past. Now a number of tool vendors, including IBM, Lombardi, and SoftwareAG, are making repository-based modeling, including BPMN, available through cloud computing. Typically these are vendor-hosted sites that allow users to set up collaborative BPM team spaces on the web for a low monthly subscription. While the previous BPM era emphasized downloadable desktop tools for business users, the new trend is toward browser-based tools hosted on the web by the tool vendor.

With the tools in place, you still have the problem of using them effectively. BPMN proudly has no methodology, and its full palette of shapes and symbols is admittedly overwhelming. Very few people, in fact, either in business or IT, know how to use BPMN effectively. To encourage business engagement, one tool vendor strategy is to reduce the modeling palette to a minimal working set, essentially those carried over from traditional flowcharting, and make up the lost expressiveness through freeform commenting features. The idea here is that to engage business more fully, we need to lower the tooling barriers, making BPM so intuitive that anyone can do it.
I would call that the predominant view right now, but I don’t think it’s the right view. Certainly it’s not the complete answer. The reason is that in order to capture the exceptions at the root of process improvement, and in order to collaborate with IT in the development of improved to-be process implementations, business can’t rely on what it already knows about process modeling, because it doesn’t know enough. It needs to learn to think about process in a more disciplined, structured way – more like IT, but from a non-technical business perspective. It needs to understand what the BPMN shapes and symbols really mean, and how to use them effectively to communicate through the diagram. And it needs a methodology, a step-by-step cookbook for going from an empty page to a complete diagram.

In other words, instead of just dumbing down the tools to the level of untrained users, we could also educate and train business people to use BPMN as it was meant to be used. I’ve been doing that for over two years, and I’ve come to see through that experience what is probably obvious to many of you already: business users are not all the same. They don’t all have the same skills and they are not all trying to do the same thing with process modeling. What is essential detail to one group may be down in the weeds for another.

Consequently, I believe the right approach is to view BPMN as a hierarchy of levels. Level 1, meant for any business person, uses a limited palette and is intuitive to untrained users. Level 2, aimed at business analysts and business architects, uses a wider palette to capture nuances of process behavior, including event and exception handling. Level 2 is that long-sought common process language shared by business and IT, something absolutely needed to truly empower business in BPM. Level 3 is using BPMN to describe executable process implementations, really just for IT.

The hard part is making each level a refinement of the previous one, not a do-over. That means imposing some structure and discipline in the methodology of Level 1, so that the Level 2 modeler doesn’t just throw out the Level 1 effort but instead just tweaks it. The end result is one model, viewable at different levels of detail by different groups of users for different purposes, not separate models for each purpose.

That idea is the basis of my new book, BPMN Method and Style, and my new BPMInstitute class Process Modeling with BPMN, offered September 23-24 in Washington DC. If you want to learn how to effectively engage the business in BPM across your company, come to that event.

Last Wednesday I got a full scale indoctrination into the agile software development methodology called Kanban, loosly based on the Toyota Production System (TPS) mechanism with the same name. Toyota uses the kanban as a mechanism to allow for just the right amount of parts to be ordered and to be delivered just in time (JIT) in order to avoid overproduction and waste in the production line. Kanban Software Development Methodology (KSDM) brings the same lean ideas to a development team.

The Kanban track at QCon, the SD conference in San Francisco, had a line of speakers coordinated to give a thorough explanation: David Anderson, Jeff Patton, Henrik Kniberg, Chris Shinkle, and David Laribee. What I found impressed me as an approach that may answer some of the problems I have been having in moving teams to agile development.

My first thought was that it should not be called “Kanban” since there is no kanban card: neither physical nor virtual nor electronic. Instead the focus is on “leveling” (also known in TPS as “heijunka”) which is a way of keeping the amount of work in each category steady. Kanban is the mechanism in TPS for achieving this leveling, but in KSDM achieves this leveling without a Kanban token itself. I won’t dwell any more on the name, because it is in widespread use, and the ideas attached to the name as so important.

Resources

Being new to the topic, I refer you right away to resources with a more thorough treatment of the subject:

• LimitedWIPSociety – a focal point for this philosophy. “WIP” means “Work In Progress”.
• The kanbandev group at Yahoo has discussions on these topics.
• Kanban Applied to Software Development: from Agile to Lean – a good article from Kenji Hiranabe at InfoQ that lays things out in good detail.
• The Kanban Primer: A Cultural Evolution in Software – a good overview article from David Anderson
• Taiichi Ohno Reinterpreted – my post from a few weeks ago visiting the fundamentals of Toyota Production System

My goal will be to simply cover my impression of what is different or unique about this approach to agile software development — there is a lot more to it than I can cover here.

Key Concepts

The key to any Lean (with a capital L) method is to eliminate waste. In software, work that is partially complete is waste. This partially complete work is known as “Work In Progress” or WIP. There is a necessity to have a certain amount of WIP, but the point is to minimize that to the degree possible. You minimize WIP, and as a result that work gets accomplished quickly. It only stands to reason: with a fixed amount of workers, reducing the number of things being worked on will allow them to spend more time on those fewer things, and get them to a completed state more quickly.

Let me emphasize how important this is. Development projects that go for 8 months with no visible results are dangerous on many levels. By taking such a big bite of work at once, you commit the entire department to a direction without knowing whether it is going to work. There is no way to gauge the progress of the team, except in vague status measures which can be manipulated by incompetent players to hide reality. KSDM is about nibbling away at the work. Each small unit is completed and made customer ready BEFORE the next unit is started. In my reading of Taiichi Ohno, this production in small continuous batches, instead of huge batches, is the essential ingredient of Just In Time manufacturing.

The focus is on “flow” of development activities. Focus on flow does two thing: first it obviously help to continuously get things completed. The other thing is that it helps to expose trouble. Whenever there is a disruption in the flow, when the flow does not work quite right, it is an indication of a problem that needs addressing. If works starts to pile up at one step, this is an indication that you should focus effort on that step and find out what is not working well.

How it Works

This seems in many ways contrary to the punctuated approach emphasized by Scrum and other strict iterative development. This apparent difference is an illusion caused simply by scale. KSDM allows for finer grained control. Scrum is a fairly course grained approach, e.g. all work gets done in a three week cycle. If you look at the larger scale, over the year, Scrum is providing a steady flow of features into the product. The 3 week iteration is a mechanism to assure that there is not a huge huge backlog of incomplete work. All the work, of the complete development cycle must be started and finished in that one sprint, which can be difficult if you have people that specialize in certain phases of development.

KSDM takes this control to a different level. You break the work process in a series different activities (phases). You then set a limit of how many job units you will allow at any phase. A simple rule of thumb: you can have a few more work units as you have people doing that job, so that each has one thing to work on at a time, and a small cache of completed jobs. The people in a given phase will do their work on a job unit to completion, so that it is ready for the next phase of work.

This is where the somewhat brilliant key idea behind KSDM comes to play. The completed job does NOT free the person for working on another job, until that job is pulled into the following phase and work is started there. If work is piling up at a particular phase, those people are NOT ALLOWED to work ahead. As Taiichi Ohno makes so clear, that working ahead is waste. Instead of working ahead, they can look around to see what is wrong.

“I feel a disturbance in the flow…” – Obi Wan Kenobi on software development

To put this in concrete terms, consider a process which involves (1) detailed design, (2) coding, (3) testing, and (4) documenting. Each of these stages you place a limit on the number of jobs, and for the sake of example lets say that limit is 4. Say for example that the coders have finished coding on their four job units, and are ready to take a new one. But the testing is backed up for some reason, still working on the last four job units, and are not ready to take a new job. The developers are not allowed to pull in a fifth job unit. There is no point in coding up more features when the earlier features are not getting tested or documented. It is also possible that because the developers are not pulling jobs from design, that the design phase becomes filled up with completed tasks. When work backs up in this way, one should go and figure out why testing is stuck. Maybe the real problem is that the documentation is blocking test. Whatever it is, the primary job of the entire team is to identify the problem with the flow, and fix it. Do not simply work ahead accumulating a huge pile of work for “someone else”. Instead, focus on the big goal, which is to get features completed to a customer ready state as quickly as possible.

Reflection

This idea of setting limits on the number of discrete jobs that can exist at any given phase at a time is so simple, and yet so profound, that I want to format these paragraph all in bold italics. It allows people to specialize into different role, to focus on their particular work, while the mechanism assures that the primary goal is not being lost. Some teams can take a feature (a story actually, a small separable part of a feature) from design to completely implemented, tested and document in only four days.

Kanban works in a car factory because the time to complete a particular job is well known. A Corolla comes off the line every 97 seconds. The factory is set up to produce a car’s worth of parts every 97 seconds as well. The parts are highly repeatable, so if it takes more than 97 seconds to build four doors, then you need either multiple steps, or parallelism, whatever makes the most sense. Once you have figured out exactly how long it takes to build a door it can be done again and again. Software is not so repeatable. Every software job is different and unique. How does that work with Kanban? The interesting thing: it does not turn out to matter. You break features into stories that are approximately the same size, but if one story takes three times the effort as another story, no problem. With software the job does not need to be timed to fit into 97 second slots. We can be flexible in the time dimension: one story takes one day, and another takes three days. What is important is that that story, no matter how long it takes, is completed before another is pulled in by that resource. With this understanding, my biggest concern was eliminated.

Clearly the method requires that features be broken down into fine grained stories. If you did not do this, you run the risk of “starving the line”. That is, one phase is filled with long running jobs, and the following phase has completed all their work. One area to explore is when the feature designer claims that the feature can not be broken down into small stories. There are surely techniques to address this, including job classification and prioritization, minimum marketable features, and other details which I can’t cover in this post.

Finally, it appears to me that a teams with a long practice of waterfall development might be able to evolve into this method. There is no big disruptive change. Just make the process visible, and limit the amount of work that is in progress at any given time. This allows Taiichi Ohno’s concept of “autonomism” to take over, and team can self organize to become more efficient. This make a lot of sense.

Attacking back at the Spammers

Some of my friends and acquaintances know that I am have been experimenting with a new scheme to control spam email. Like many people, I have had to abandon email addresses in the past due to over-abundance of spam. When you open a new email address, there is no spam. But as you continue to use the box, eventually the knowledge that you are actually using a particular email address gets out. Once your email address becomes known to the spammers there is no sure way to get them to forget it.

A verified email address has value, and lists of email addresses are traded and bartered in the spam underworld. Even a non-verified but potentially valid email address has value. Sending a piece of spam does not cost much, but it has a cost. Sending to all possible email addresses (start with aaaaaaaa@aaaaa.com, then aaaaaaab@aaaaa.com, etc) is not viable. Spammers want an email address that is known or at least likely to work.

You could say that a particular email address has value because it is so much rarer than a completely random email address. A list of 1000 addresses that reaches 1000 people is the same value as a list of 100,000 addresses that reaches 1000 people. From a spammer’s perspective, you would like every email address to be valid (reach a person) forever, and have no addresses that go to dead ends. When people change their email address, there is a cost (albeit small) to the spammer, because the old email address become invalid. My goal then is to frustrate spammers by filling their lists with invalid email addresses.

The Answer

The value of my email address is due to its relative rareness. I can decrease the value of a particular email address, by increasing the number of email addresses I use. The idea is this: I can use thousands of email addresses out of a pool of billions of possible address. I can use a unique email address for (almost) every occasion. All of the email addresses deliver mail to me. Imagine the extreme: I print my business cards with a unique email address on every card. Anyone who uses the address has no problem sending email to me.

It is possible that in the course of normal email interchange, an email message with that address on it, gets posted to some sort of web page (e.g. email forum archive) and the spammers pick it up from there. The anti-spam feature is that whenever I start to receive spam on a particular email address, I turn off (disable) the address.

What if a legitimate party was using that address? What if that is the email address I gave to my mom to use to contact me? This would block email from her as well. Part of the scheme has to be keeping a record of who I have given the address to. When I turn that email address off, I go back to the legitimate person that I gave it to (e.g. mom), and give them a new email address to use.

You might be thinking correctly that this is onerous to have to tell people to use a different address. But keep this in mind: if I have given unique addresses to each of my hundreds of correspondents, then all of those addresses except this one remain unaffected. In the past, I have had to abandon entire email inboxes to ALL correspondents, and send them ALL a new address. Since there is no way I can remember all of them, I undoubtedly lose a many along the way. The need to abandon an email address is rare in general, and contacting one person to switch is painful, but far better than contacting all your contacts.

Version 1.0

About 6 months ago I put in place a plan to experiment with. It turns out that the XPDL.ORG site, which I help run, has unlimited free email forwarding. So what I did was create new cryptic email addresses, and forward them all to my regular email inbox. For example:

kds_Why54TzrvyZfgzNqqerf@xpdl.org

Every time I signed up for some sort of online service or account, I would create a new forwarding address. I created a private wiki page where I recorded the cryptic address, who or what I gave the address to, and when I did that. The idea is that if I ever have to turn that forwarding off, I can get back in touch with whomever I gave it to.

The email address must be long so that it can not be guessed. For example if I just use “keith1″, “keith2″, etc. it would be too easy for the spammers to guess other valid email addresses. This could cause me to have to turn off many many addresses inconveniently. If I make the address long and cryptic, then it is very very hard to guess other legitimate addresses, making those addresses relatively safe.

Most of these email addresses are entered into online forms, and used by those services, without anyone actually having to read them, or type them, so it really does not matter how long and complex the email address is.

It is not perfect…

What about “from” address on email? On my standard email, I created a new cryptic address as my “from” address every month. It does not matter how long or complicated an email address is when people simply use the “reply” button. Cycling every month is not perfect because if someone puts that email in their address book, and it also gets on a spam list, I might turn it off, and I don’t know who using that address, so I don’t have any way to let them know a newer address to use. Creating a new unique address for every email might be better, because this decreases the chance that someone would hang on to an address that also got on a spam list, but that causes other difficulties.

Some services require you to log in using your email address. If you really want to keep your “real” address private, then you have no choice but to give them and use the cryptic on to log in. Typing that long and meaningless address is a pain, so in those cases I have to create an address that is easier to remember and type, which unfortunately decreases its security.

Because you are using many email addresses simultaneously, it is possible to start getting multiple copies of a message. For example, if a message comes to you using address “a”, and you reply to it using address “b”, then both addresses become part of the ongoing email address. Some email in-boxes are smart enough to eliminate the duplicate, but not all are.

Every time you sign up for a mailing list, you use a unique cryptic email address, but again this can cause message multiplication when the message is addresses to multiple lists which have different email addresses for you.

In the six months that I have been doing this, I have not had the opportunity yet to turn off an email address. This is because it takes time to get on those spam lists, so as far as I know, none of my “new” email addresses that are less than 6 months old are on any lists yet. So it is really too early to tell. It is also true that going to the admin interface and creating a new cryptic email address, recording what I am using it for, and then using that in the sign up form, makes signing up for any service quite a bit more trouble. Sometimes I am too lazy, and just go ahead and use the fixed address because it is easier.

Version 2.0

I just found out about a new service called otherinbox.com. This is the service that I have been looking for, and it is aimed at exactly this problem. (Scott Francis: you mentioned this service to me a while ago, but it took me this long to investigate.)

You get an account ($20/year – trial accounts are free) and it gives you an infinite number of email addresses which all go to you. It has all the capabilities described above, including the ability to block an address at any time in the future. You can record notes about a particular address to remind you of who you gave the address to, and when.

There is one particular improvement over my old scheme: you don’t have to set up the address in advance. When signing up for an account at Barnes and Noble I can create a suitably cryptic address on the fly, and it will automatically create the inbox for that address without extra work from me. Usually such services start by sending an email for you verify that you own the address, so I can go to the otherinbox.com, find that new email address, and set the address to be forwarded. This is so much nicer to do later instead of having to do it before you sign up, particularly when you are not on line. You can even make up an address while filling in a paper form with a pencil, and eventually the account is created for you — if needed.

They have a lot of other features for filtering and such. Email can be forwarded, or picked up directly from their web or IMAP interface. If you want, you can let the email pile up there, and receive only a digest of the email once a day. That might be really handy with some of the email lists I am on.

Contemplations

Perhaps this seems like a lot of trouble, to have to set up and manage a bunch of different email addresses so that you can have the option to cut one off if necessary. To be honest: it is lot of trouble. OtherInBox looks like a lot less trouble than my initial way, but it is still more trouble than just being able to give out a single address forever. Some of the need for this might go away if we had widespread cryptographic signing of email messages so we could know who the email came from, but there are many forces working against that. Signed messages would not help in a mailing list situation when you are exchanging messages with people you do not know. There are some possibilities that social software will offer some benefits in this area, once they have matured a bit more. So for now given the current infrastructure, this looks like the best hope for combating spam.

Down-to-Earth Contracts that Keep the Cloud Aloft – A look at the basic interoperability requirements when communicating with the Cloud, Keith Swenson & Jacques Durand, Nov 2009

Working together with Jacques Durand, a colleague and expert in the B2B exchange standards space, we put together this article exploring how many of the same standards and agreements necessary today will also be necessary for applications deployed to the cloud. Just published!

Fujitsu made a couple of press releases last week, announcing two things: a new release of Interstage BPM, Version 11, and our Cloud BPM offering. This post just contains links to articles on the subject of these announcements.

• Product Review: Interstage BPM V11 – Column 2
• Fujitsu Interstage BPM in the Cloud – Column 2
• Fujitsu BPM Cloud Challenges IBM, Oracle in Cloud Computing – Channel Insider
• Fujitsu fields BPM on cloud computing platform – Search SOA
• Fujitsu Announces Free Access to Enhanced Cloud BPM Platform – Enterprise Systems
• Fujitsu Cracks BPM Auto Discovery Code – CTO Edge
• Auto-Discovery Comes to BPM – IT Business Edge
• Fujitsu releases new version of Interstage BPM Captures hybrid collaborative process patterns – CBR
• Fujitsu Interstage BPM Version 11, Lets Businesses Proactively “Sense and Respond” to Change – SOA World
• Fujitsu Announces Free Access to its Enhanced Cloud BPM Platform for Solution Providers and Enterprise Teams – EBizQ
• Process intelligence tools reduce guesswork, increase payout of BPM – Search CIO Midmarket
• Process intelligence tools reduce guesswork, increase payout of BPM – a writeup on the use of Process Discovery at ESI.

While there are many small features in the Version 11 release, the two main ones are a significantly extended capabilities in Dynamic BPM and extended tenant management capabilities. The latter feature helps to support the extended cloud BPM offering which includes a complete BPM design, development, and run-time capability which is free for small teams.

This button in this situation produces an error report … therefor the button should be disabled.

I question this line of reasoning. I have observed this reasoning used at all levels, from programmers, to UI designers, to Product managers, and even to customers (users) themselves. The goal seems to be “protect the user from error messages”. Some people naively think that a perfect user interface is one in which the user never sees an error message.

“Good usability” means that a user can use a product without requiring significant training. It should be expert friendly as well as novice friendly. This means that the product itself helps to train the novice user. An error message is an opportunity to learn.

Intelligent people learn from mistakes. My dad used to say that “if you are not making mistakes you are not learning anything.” Imagine, if you can, a school designed around the idea of preventing students from making mistakes. Imagine a math teacher who instead of grading a test, would correct all the mistakes or otherwise prevent students from having incorrect answers. Clearly, the student would learn very little. Students who already knew all the concepts would do fine, but those attending to pick up concepts would be out of luck. It would be “expert friendly” but not “novice friendly”. Allowing students to make mistakes (safely) and learn from them is a widely accepted pedagogical principle.

How odd it is that many software designers attempt to eliminate the occurrence of error reports. Error messages are assumed to be “bad”. Usually it is explained: “the user should not have to see an error message.” So they disable controls. I have covered this in the past with posts Please Don’t Disable My Menu Options! and More on Disabling UI Controls. Because of this misguided design principle, the user interface is not as good at training new users. It seems that such software is designed to be “expert friendly” but not “novice friendly”.

In defense of these people, many software packages have extremely poor error reporting interfaces. Often the software “accuses” the person of making a mistake, blaming them for doing something wrong. This makes error reports unnecessarily distasteful.

An error message should instead be presented as an opportunity to learn something about the program. Like any good tutor, the information is presented at the time that you need it, at the time that you try a particular approach. The error message should contain useful information about why that particular action was not applicable at the moment. From this, the user would learn about the software, learn what was possible, and what is not possible.

Instead of trying to “protect” the user from error messages, we should be aiming for the exact opposite extreme: copious error messages. Instead of disabling actions, the actions should be enabled, and should explain why that action might not be used at that moment, and when it can be used. It is harder to make this kind of user interface, but worth it. It is far more novice friendly because it helps them learn the product as they use it.

Taiichi Ohno is credited with the creation of the Toyota just-in-time production system, and his book “Toyota Production System: Beyond Large Scale Production” is a surprisingly good read even today when many of these principles are considered well established.

My interest was in understanding how this philosophy applies to Agile/Lean Software Development.  I was pointed this way by Tom and Mary Poppendieck’s excellent book on the same subject, as well as an insightful paper from Fujitsu on the subject of Innovation in Software Development Process by Introducing Toyota Production System (pdf). As I am reading I am surprised at how perfectly the advice from Mr. Ohno fits to the advice I would give to a software team for Lean/Agile software development

Mapping Concepts to Software

Before the advice applies, you need to understand how software maps to 20th century automobile manufacturing. Ohno’s central theme is to eliminate waste.

One form of waste is the consuming of parts (which cost money) into incomplete products (which can not be sold yet). The most wasteful form of mass productions would be one where you build 1000 car chassis, then you put 4000 wheels on them, then you put 1000 engines in them. All the time you have this inventory of 1000 incomplete cars until they are finally finished. He would prefer that there be only a single incomplete car at a time. The reality is that it takes time to put the parts together, and so there is an assembly line with a number of cars being worked on at the same time, and the real goal is a very steady even stream of not only the finished cars, but also all the parts flowing into the process. The total accumulation of incomplete cars, as well as the accumulation of incomplete parts, adds to the amount of waste, and should be eliminated to the degree possible. If you have a significant amount of over production, then there is additional waste from the need to store the extra parts and handle them multiple times.

For software, the bytes that form the lines of code cost nothing. You should not think of lines of code as being the parts of the system, and you should not think about the source code as being the output of the process. This focus on the source code as the “product” is one of the biggest misconception in the software industry. Reality is less tangible. The “parts” that are consumed are simply the “person-hours” that are invested in discovering and developing a feature. This includes time from not only programmers, but all the various roles on the team. The output is best described as “satisfied customers”. While slightly inaccurate, this phase is meant to remind us that it is not enough to burn a CD with the program on it, but the final sale is for installed software which is successfully used to solve customer problems. If you have done the job right, they will be “satisfied”.

Just as in a car factory, you do not want to build 1000 cars in parallel, similarly in software you do not want to create 1000 features in parallel. As you start to build a car, the waste builds as you add more and more parts, but the waste disappears when the car is completed and ready for shipment. The uncompleted cars are a liability on the organization, and building cars in parallel multiplies the amount of waste. Similarly, once your start designing a feature, and start investing person-hours, the waste builds in direct proportion. Finally, when the new feature is actually in a “customer-ready” shape, that waste disappears. The started-but-uncompleted features are a liability. Features designed but not coded is a waste of time. Code that is written but not tested is a real liability because there is now a hidden amount of work that must be done to correct problems in the coding. Completed features in the code, but can not be installed are still a waste. Building a bunch of features in parallel multiplies the waste.

Careful readers will notice that the above discussion is slightly simplified. Even a perfect factory would have cars in an incomplete state, and a perfect software team would have some feature in an incomplete state. It is wrong to call all items in transit “waste”, rather it is waste only if it is more than the minimum necessary. We want to minimize the number of incomplete features, not eliminate them entirely.

While you can envision a software development team as a factory that produces “features”, you need to be very careful. When building 1000 cars/day, all of the various steps are highly repeatable. The time for each step can be measured to a high precision, and the exact costs of parts can be predicted. For software development no such predictability exists. You start a software feature with no reliable estimate of the cost. This does not change the goal of elimination of waste. Even though you don’t know ahead of time the amount of waste, it still makes sense to runs thing in such a way as to minimize waste in whatever amount occurs.

Quotes from the Book

Understanding the mapping, we can now take quotes directly from the book (including page numbers), and translate them in to meaning for software development teams.

The loom stopped instantly if any one of the warp or weft threads broke. Because a device that could distinguish between normal and abnormal conditions was build into the machine, defective products were not produced. – p6

Automated testing and continuous testing are a central pillar of Agile development. Ohno is talking here about looms that were perfected by Toyoda Sakichi from 1910 to 1926. Amazing that the concept of continuous automated tests was in use a century ago.

Stopping the machine when there is trouble forces awareness on everyone. – p7

When a test fails, a software team should drop whatever it is doing and address the problem of the test. This does force awareness on everyone.

…the workers themselves should push the stop button to halt production if any abnormality appears. – p7

Stopping a production line is expensive, but here we see that Ohno understand that the expense of an unfixed problem is far greater. The biggest enemy of software quality is ignoring test failures. Once I joined a team where the building of the product produced thousands of warnings from the compiler. The developers felt that warnings were harmless, but they are not. First thing I had them do was fix the code that caused the harmless warnings, so we could then see if any important warnings came out. Allowing anyone to check in source with a broken test propagates those broken tests, and that causes a blindness to the real problems in the code.

Repeating why five times, like this, can help uncover the root of the problem and correct it. -p17

So much of software is written to get around problems at other layers of software. Code bases often seem like a huge elaborate house of cards. Compensating code is distributed throughout to hold it together. One should ask: “why is that routine written this way”, “why does that support routine return this result”, “why is this value needed”, etc. Addressing the problem at the lowest level will eliminate the need to multiple copies of the compensating code, and usually reduce the potential for bugs.

We regard only work that is needed as real work, and define the rest as waste. … we must make only the amount needed. -p19

Here Ohno explains an important concept that overproduction is waste as well. Well known in software methodology as YAGNI. Don’t implement anything until you need it for an actual end-user use case. Contrary to this, programmers often pride themselves on “complete” implementations of interior object classes, commonly adding a host of getters and setters which are not (yet) needed by anyone, but they anticipate will be needed at some time in the future. This additional coding is precisely the kind of overproduction that Ohno is speaking of: not only does it waste the programmer’s time to implement it, but the excess lines of code increase the code review and maintenance burden in the way exactly analogous to the need to store the extra parts in a car factory.

I used to tell production workers one of my favorite stories about a boat rowed by eight men. One rower might feel he is stronger than the next and row twice as hard. This extra effort upsets the boat’s process and moves it off course. -p24

Lessons on teamwork are always useful, and this points out the danger of measuring anything intermediate to “customer satisfaction”. The goal in rowing is to get the entire boat to the end line as quickly as possible, and a focus on anything else, such as individual force on the oar can distract from this goal. In software, getting a workable feature to the final customer is the goal, and measuring anything else, such a number of lines produced, or bugs found at a particular phase, can distort the process in unacceptable directions.

The operating method of the Toyota production system is kanban. This piece of paper carries information that be divided into three categories: (1) pickup information, (2) transfer information, and (3) production information. -p27

Kanban was a way of decentralizing the control of production. The factory is oriented on pull-based production, and the kanban is a way to communicate the detailed need for production, in order to avoid over (or under) production. The kanban might be considered to be analogous to storycards that are used to communicate details of a customer use case. This is only the top level, because kanban is used to communicate within the factory within any producing/consuming relationship, and for the most part this is NOT done in a software team. Sometimes a bug report can be used to allow one team to make a request for enhancement on another team, and to track the progress. In this case the bug report is analogous to the kanban. But caution should be used in the analogy: remember that software parts are unpredictable by nature. The very regularity of car parts, and the known quantity and cost of the parts, makes it very easy to implement specific requests for production. In software, however, every part is unique and custom made for the situation, and so a much richer, more interactive discussion is needed. Kanban should not be implemented literally in a software development team to replace internal communications. Still, the idea that performance level is tuned according to the need to produce a given use case is very important.

I have good reason for emphasizing the role of top management in discussing the first rule of kanban. …management commitment and strong support are essential to the successful application of this first rule. -p31

Here he is talking about moving control of production from the central planners, and replacing that with the more “autonomic” method of Kanban. It is interesting to see how this must have been very uncomfortable for upper management to “trust” that the people on the floor would regulate things correctly. It is the same with moving software development to an agile approach, where individuals on the team self-organize to get the current sprint done. The waterfall approach gave more control to external authority, and also provided a better way to measure progress. Changing to a lean software approach means that this control and to some extent this ability to report progress is eliminated. This top management support is essential.

Unless one completely grasps this method of doing work so that things will flow, it is impossible to go right into the kanban system when the time comes. -p33

The key word here is “flow”. In a factory the flow of physical parts is paramount. In software, the flow of feature ideas is paramount. The flow of continual builds, and regular sprints, with customer quality code available at all times is the key. Stockpiling a huge number of features for implementation in an 8 month project is precisely like the mass production of cars that was beginning to show signs of inefficiency in the 60’s and 70’s. Mass production creates waste. Waste is eliminated by a steady flow of product releases, each quickly generated from the last. This is the central theme of the book, and it is the central theme of Lean Software Development.

In the beginning, everyone resisted kanban because is seemed to contradict conventional wisdom. … To make kanban understood throughout the company, we have to involve everyone. If the manager of the production department understood it while the workers did not, kanban would not have worked. -p35

How interesting that Mr. Ohno had so much pushback. He admits else where that it was non-intuitive, but remember that conventional wisdom is based exclusively on what you used to do. We see the same with software method, including the need to include everyone. If you train part of a team on the method, you will surely fail, (and I have direct personal experience with this).

While traditional planned mass-production system does not respond easily to change, the Toyota production system is very elastic and can take the difficult conditions imposed by diverse marked demands and digest them. – -p37

This is the promise of Lean/Agile software development: the continuous flow of small features allows for much more rapid response to market changes. It is ironic that cars with all their physicality can be developed in an agile way, while software which due its lack of physicality aught to be easy to change, but is still developed with a waterfall model making it impervious to change.

Production Leveling: It had been a long accepted production fact that continuous punching with one die in the press beings the cost down. It was considered common sense to product in the largest lots possible and punch continuously without stopping the press. -p38

In the 1940’s, Toyota’s die changes took two to three hours … By the late 1960s it was down to a mere three minutes. -p39

The idea that large production runs can reduce cost is well established, and is exactly counter to the idea of producing only as many of a thing that you need. Not deterred by this, Toyota figured out how to eliminate much of the waste in small production runs. Ohno had faith that die changes could be made faster, and without that faith it would never have been accomplished.

The same thinking perpetuates waterfall software development: the established “fact” that there is overhead in checking in and testing code, therefor, if you have to touch the code you should implement as many features at once as possible. If you have the orientation that testing is long and difficult, and you do it only once a quarter, then the idea of doing it once a day is simply impossible to consider, and quick development sprints equally impossible. However, if you understand the advantage of continuous flow of features, then you automate your tests, and make them so you can run them every day. If you have faith that tests can be automated and run quickly, then you have a chance of realizing this potential.

To ensure that we have 100% defect-free products, we must set up a syste that automatically informs us if any process generates defective products -p41

Automated testing, continuous builds, with automated runs of tests. Require each programmer to run the tests before checking in, don’t check in if the test does not run. This is Agile/Lean software development, and Ohno is telling us about it in the 70’s.

Rule three of kanban prohibits picking up or producing goods without a kanban. Overproduction is automatically checked, even if someone wants to make more. -p43

If software is developed only to meed the needs of a specific use case, and the use case is attached to the check in, we will eliminate over coding of things not needed for the use case.

If one sticks to the idea that, once set, a plan should not be changed, a business cannot exist for long. Sticking to a plan once it is set up is like putting the human body in a cast. It is not healthy. -p46

This puts a dagger in the heart of the idea that plans should be perfectly formed ahead of time, and this is particularly true in software where nothing is predictable. Instead, a lean/agile approach to software allows plans to be flexible over time. Very important, and an idea I frankly did not expect to come from a car manufacturing giant.

The plant should be a place where such judgments can be made by workers autonomously. -p45

Again a common theme in lean/agile software development approaches, but not what I expected to hear from a manufacturing giant. In this case he describes the business organization as having an “Autonomic Nervous System”.

Similarly we want information only when we need it. Too much information induces us to procese ahead, and can cause mix-up in sequence. In business, excess information must be suppressed. -p50

The kanban allows the information to be carried with the products/parts, one of the reasons why kanban works. It is interesting to see the bias against too much information. He talks about automatic fine tuning in this quote:

as long as we cannot accurately predict the future, our actions should change to suit changing situations. -p52

Again we see flexibility and adaptibility as a core value, the antithesis of putting together a perfect plan and then executing long term against that plan. Instead, adapt as you go along.

In any manufacturing situation, we frequently see people working ahead. Instead of waiting, the worker works on the next job, so the waiting is hidden. If this situation is repeated, inventory begins to accumulate at the end of the production line or between lines. -p59

Here we touch upon a very pernicious area of waste in software products. If the schedule is relatively fixed, when a programmer finds that an implementation went well and they get done early, they often turn to implementing an extra pet project they have been intending to implement. It is routine for development team to report to product management that few extra features have been included “for free”. The worst cases of these I have seen have actually caused delays in the product because once implemented we found flaws in the approach, but it was too much work to pull out the functionality, so instead extra work went into fixing it. Extra work is not extra value, it is instead increased waste.

Making large lots of a single part — that is, punching out a large quantity of parts without a die change — is a common sense production rule even today. … The Toyota system takes the reverse course. Our production slogan is “small lot sized and quick setups.” -p95

In this chapter he compares Ford production system to Toyota system in order to emphasize the differences. The parallel of the Ford system to waterfall software method is strong, while the Toyota system is like Lean/Agile software development. The biggest single driver of continued use of waterfall is the idea that creating a bunch of features at once will reduce overall cost. There is a feeling that there is an “overhead” to the cycle that can not be avoided, and therefor do as much in a single cycle is the most efficient way. Toyota’s concept of flow and production leveling is just as radical as the idea of continuous builds and fast sprint feature cycles. Both are are not intuitive, and have to be seen to believe that it is possible. It should be obvious, however, that large lot sizes will make changing the product more difficult and maybe nearly impossible. Thus it is the method of manufacturing that causes the inability to respond.

For automation to be effective, we must implement a system in which the machines sense the occurrence of an abnormality and stop themselves. In other words, we must give the automated machines a human touch — enough intelligence to make them autonomated and achieve “worker saving” rather than “labor saving”. -p113

In software development, of course, machines are replaced with automated script and programming that does steps of the process, and these must include tests at every step. Ohno’s statement here is strong: the automation without the sensing causes more waste than it is worth. Thus tests built into the automated process, or possibly into the product itself, becomes most important in automating builds and steps in the software cycle.

Conclusions

In 117 pages he sets out a clear description of how Toyota has accomplished the climb to be the world’s biggest car maker, and is a must read for anyone interested in industrial production strategy.

Then, if you understand the mapping to software development, you find that he sets out an equally clear description for Lean/Agile software development. If Ohno was alive today, I am convinced he would have been an eloquent proponent for developing software in short iterations, with continuous builds, continuous testing, immediate response to build breakage, and guided by feature burn-down. In other words, a proponent of Agile Software Development.

You know that book on how everything important is learned in Kindergarten? Along that same line, before I got into Kindergarten, my mother taught me to that if I put my toys away, I will be able to find them again later. I am sure there was a lot of crying and whining involved, but like most people I eventually got the point.

Fast forward to the adult world. How many times have you heard these questions:

• Where is the latest spreadsheet?
• Does this document have the latest changes in it?
• Are your changes in this document?
• Can you send the copy of the file that contains all the latest updates?

This situation is caused by the worst scourge of our time: the addiction to email.  Specifically the attempt to use email as a document management system.

The solution is so simple: find a place to put the document, where everyone can access it. Put it there, and always update it there. If you always get the document from there you will always have the latest copy. If the always put updates there, everyone else will always get the change. This is so obvious that I am sure most of you reading this will feel slightly insulted that I am spelling it out in such detail.

Why is it then so difficult to teach people this behavior?

Part of it is simply short sighted thinking: it takes extra effort to put your toys away, and the benefit of putting them away might be a long time from now (like tomorrow). Just like it is easier to leave your toys sitting in the middle of the room, when someone requests a document, it is simply easier to just email it to them. Attach it as an attachment, and off it goes, problem solved … for the moment. It takes quite a bit longer view to realize that (1) others may come back and request the document again, and (2) even that person is going to in the future want updated versions. Why put out extra effort now, in order to save that work which will come in the future (like tomorrow).

Some of it is blindness to the dynamics: documents change over time, but for some reason we always act as this version I am sending now is the final version never to be modified again. I laugh every time I see a document with a title that includes “FINAL” in the name. It is even funnier when the title has multiple repetitions of FINAL. How does this blind spot persist when at the same time we all have such difficulty locating the “updated” copy.

Some of it is selfishness: I received an email this week from the finance department detailing exactly how to handle a particular kind of financial transaction. From the detail of the procedure involved, it was obvious that the entire company was not expected to memorize the exact procedure. They apparently expected everyone in the company to carefully file this important information away in a place that you can find it if the need ever arises. Instead of taking the time to organize the information in a single place where everyone can get it, they push this task out to everyone in the company. Multiplying the real effort to the company by many fold. Clearly it would be better for everyone to integrate this into a corporate knowledge base, organized so I can find it if I needed, at that time that I need it. But that kind of foresighted thinking is rare when it is just so easy to send an email.

First Step is Acknowledgment of the Problem

Let say that everyone realizes that email attachments are the problem, what can we use instead. There are some problems with current document management systems (DMS).

Access : I can send an email to whomever I want, and attach the document. But putting a document into a DMS, how can I guarantee that the intended recipient can access the document? Most internal corporate networks have become byzantine mazes as the result of attempt to control who can access what through the use of blocking certain addresses at certain routers. I have heard this complaint from many people at many organizations. The blocks are put into place in the name of safety, and getting thing unblocked is nearly impossible since it is virtually impossible to prove that such a change is still safe. Desktop computer A is cut off from desktop computer B because a virus might propogate along that path. How can you prove that virus will not? You can’t.

Access Control: If you are lucky enough to find a DMS that everyone can access, how do you make sure that only the intended recipient(s) can access it? All DMS have fine grained access control, but it is usually a large amount of trouble to change it because of the detail that you have to work at. Addressing an email message is easy, especially if you use “Reply”. Assigning users to access a document is easy, but there is no analog to “reply”. Few DMS offer a way for people to request a document from those who own it. Most DMS don’t even allow people to be aware of documents they don’t have access to, which would be necessary to make the request in the first place.

Transitive Access Control: If I send a document to Joe, he can forward it to Mary. If I give right to Joe to access a document, most DMS systems will NOT allow Joe to give access to Mary. It wouldn’t be “control” otherwise, would it? But to allow a DMS to replace email, this is exactly what is needed. In the Process Leaves system, we implemented such ability to “forward” the rights to another person, but most users find this so surprising and unexpected that they never use it.

The solution is to make a shared “room” where all the toys can be shared equally within a group. That is the solution that many approaches have taken, and it is not difficult. But someone still has to set up the room in advance, in anticipation of the need to share, and most people will not take this step. It is just easier to send the documents as an attachment and force the work onto everyone else. In groups that I work with, even making the room available to people, they rarely get used.

Any ideas on how to break the email addiction? Any tricks you know of in getting people to use “places” to put documents? I made a similar plea last year in “Page First, Then E-Mail, Please” about the email message itself. Otherwise, I am afraid, it is a bit like teaching teenagers the benefits of a tidy room — in other-words: a lost cause.

I could call this post “Removing the Risk from Lean Process Improvement” because it starts with the assumption that you want to improve your processes using Lean principles, but you want guidance on how to apply those principles most effectively.

Soooo much discussion of Lean last week at the Forrester Forum and the Gartner BPM Summit. Who can argue against Lean? It is after all a focus on providing more value with less waste. Lean is a focus on eliminating waste, the original sevens wastes identified by Toyota, as well as elimination of anything that does not provide value to the customer. We all want to get rid of waste and inefficiency.

How do you identify the waste in your business process? This is harder than you might think. Sure, you probably have many anecdotes about cases that went terribly wrong, but how much did that really cost? How common is it? After all, one extreme case does not signify a trend. What you need is a quantitative measure of how much your current process inefficiencies are costing you.

This is exactly what Electro Scientific Industries, Inc. (ESI) wanted to do. They knew they had process inefficiencies for two reasons: they were having inventory problems and they were having a hard time responding quickly to customers. But what was causing this? ESI embarked on a Process Mining exercise using Fujitsu’s Automated Process Discovery. What they found astounded them. What they expected to be a 5 to 7 step process, with maybe a dozen variant paths, turned out to be far more complex in reality. They analyzed 12,000 cases that had occurred in the past 3 years, and they found 1300 distinct process paths. They found actual evidence for hundreds of cases of severe repetition, including one case where the order was changed 120 times! They were able to identify errant paths, and tell precisely how many cases had gone that way. They found specific process steps with unusually long average transition time, and could measure exactly how many cases were effected by this. All of this information was pulled out of their existing SAP system without the need to install any new software in their environment.

This is exactly what you need for a Lean Process Improvement project. Greg Mueller who headed up the project at ESI, was able to show precise measures of the cost of process inefficiency to upper management. This resulted in ample support for initiatives to fix them. Peter Schooff did a podcast with Greg on exactly how they went about this. As Greg says: “Process Discovery found 17 different ways to eliminated waste from their business process, any one of which would have more than made up for the effort of doing the discovery.” Also, see the blog post from Sandy Kemsley covering this use case.

Being able to get precise measures of waste that can be eliminated is critical, even more amazing is how quickly this evidence can be gathered. Some of the most important evidence was exposed on the second day of investigation. The data gathering was completed in the first two days. Analysis continued with what is best described as a “learning” phase: gaining a better and better understanding of what was really going on in the organization. The team would slice the hypercube different ways, isolating teams from each other to discover process patterns in different offices or for different situations. That data collection is still available today for immediate access so that when someone comes up with an innovative idea to eliminate waste, they can go back to the historical data and estimate the amount of savings that will generate. The analysis never ends, but within two weeks they were able to identify 17 specific projects to eliminate waste from the business process, along with evidence based estimates of the savings to be made.

For those of you planning Lean Process Improvement projects, visualize sitting there, one week from today, with specific evidence based assessments of where your biggest process inefficiencies were, and how much they were costing you? How much would that be worth? Would that help you get funding and support for your project? Sure it would. You owe it to yourself to at least investigate what a process mining session might be able to do.

Fujitsu has been offering this service for about a year. The biggest barrier is that is it just so hard to believe that the results are possible. It seems too good to be true. I admit, if I had not seen the results myself, I would be skeptical. But I have seen the results (1, 2) which are nothing short of amazing. Anyone considering process improvement will be wise to learn about them as well.  Informing yourself up-front with quantitative measures of the waste in your current processes, will eliminate much of the risk in a Lean Business Process Improvement program.

There might be three distinct kinds of process support necessary:

1) System Centric Processes
2) Human Centric Processes
3) Knowledge Worker Processes

System centric processes are fairly well defined: it is a kind of software engineering for very complex distributed systems. These are designed to completely automate the information flow so that no person has to be involved.

Human Centric Processes are those well defined processes which involve people. Since people are not “invoked” the way that software services are, there are a lot of extra capabilities that are needed in order to interface successfully with the asynchronous nature of personal interactions.

Are Knowledge Worker Processes distinct type of process, or just an extreme case? The unique thing about knowledge work is that it is unpredictable. Human Centric Processes can still be formal processes that are all predefined. Knowledge work, by it very nature, can not be predicted in advance. There is no fixed process for this kind of work. The work is “impromptu” in that the sequence of steps is figured out at the time that the work is done.

Some have called the technology to support knowledge workers “Case Management“. Especially in the medical field where you have experts who examine the “case” (the collection of documents and reports compiled for a given case) and then decide what to do next. (See CSMA.) Case management is also the name used in the legal field. See particularly work at National Center for State Courts which is promoting an idea of Configurable Case Management. Some also call this area Dynamic BPM. Some call it “Unstructured BPM”. A better term might be Situational Process Management.

Where exactly is the boundary, if any, between Human Process and Knowledge Worker Process? Clearly KWP need extra support at run time to allow processes to proceed in ways that were not predicted in advance.

Thought Leader Summit Workshop

WfMC will be holding a Thought Leader Summit on this subject on Nov 3 in Maidenhead England. Please find additional information at:

http://wfmc.org/november-member-meeting.html
http://www.xpdl.org/nugen/p/lpqyjklyf/public.htm

The purpose of the workshop is to clarify the requirements of “the kind of process support which is needed in order to support knowledge workers”. What is and is not included in this kind of a product / technology? This is the essence of this discussion hosted by WfMC. Henk de Man from Cordys will be there. Dana Khoyi from Global 360 will be there. John Hoogland from Pallas Athena will be there. Max Pucher from Isis Papyrus will be there. Many other thought leaders including Justin Brunt, Nathaniel Palmer, Keith Swenson (myself), Robert Shapiro, are expected to attend in order to get to the bottom of this highly relevant issue.

Goals: Determine the best name; create a short and long definition of the capabilities; get agreement on a list of requirements that such technology must have to be considered in the category; draw up a “Reference Model” or “Notional Architecture” for the category; follow through with a site for market education. It is a tall order. To those coming: be prepared to roll up your sleeves.

The summit will continue on Nov 4 with a workshop on BPMN 2.0 portability. Will BPMN diagrams be exchangeable between tools? Will BPMN 1.2 diagram be portable to BPMN 2.0? Answers to these questions will be worked on as a continuation of the fine work of the BPMN 2.0 / XPDL 2.2 working group. This group has already completed a proposed BPMN2.0 serialization schema, prototype transform from XPDL2.1 to proposed BPMN2.0 serialization, and prototype transform from BPMN2.0 serialization to XPDL2.1. This workshop will continue with discussions of BPMN 1.2 Status and use today, how fast people are migrating to 2.0, diagram interchange conformance classes, tools for validation, BPMN 2.0 Finalization Task Force Status, concerns over the direction, and other working group activities.

I collect nuggets of wisdom on various topics. Recently I have been going over the topic of Agile software development; what really matters? Below is a list of 26 key principles to guide an agile software development team.

• Get case 1 fully working before starting case 2. Another way of saying this to use a kitchen metaphor is: “Serve the current meal before starting to cook the next“. The biggest problem with software development is to start a bunch of things in parallel, because inevitably work will include something that is later discarded, meaning wasted work. Work on one case (or a small number of cases depending upon team size); get that fully functional; get the tests running; write the documentation; check it all in as a finished piece of work, before you start on the next case.
• Never break the build. Pretty obvious, but must be included in any list of software development advice. A programmer who is taking all the proper precautions to test before checking in will never break the build. If the build is broken, it is always because someone took a shortcut.
• Never implement a routine before it is needed in a use case. When implementing a particular class, you should have a particular use case in mind, and you should implement only the methods required for that use case. You might think about the potential for other capabilities on a class, and you might document this in a comment, but implementation should wait until it is actually needed in a use case.
• Never add a data member before it is needed in a use case. Exactly like above except with regard to class data members. It may seem obvious to the programmer that a “customer” record will need a “ship to address”, but that ship to address should not be implemented until you have a use case which specifically needs it.
• Don’t be afraid to make a decision; don’t be afraid to change an earlier decision. Agile development is about quickly responding to uncertainty. Early in development you do not have complete information. You should delay decisions as long as possible, but there comes a time that a decision is needed to move forward. You can not hold up decision until that information comes in. Instead, make the best decision you can on the available information. Later, when new information arrives, don’t be afraid to change that decision. (Some dinosaurs call this flip-flopping, but I call it reacting to a changing environment.)
• Continually learn how to improve quality. This job never ends, so you should expect to be constantly on the look out for things that could be improved, and collect examples of ways that quality problems were identified and addressed.
• Measure, measure, measure. Agile development helps address the problem of uncertainty about the future, but there should be no uncertainty about the past. Tests should be continually running. Performance of every run should be measured and recorded.
• Design around people, not systems. Too often developers get sidetracked into designing for technical opportunities. Never lose sight of the ultimate purpose of the software, and that is to help people get work done.
• Tests are part of the product. Many developers and managers consider the product to be what you ship to the customer, and everything else less important. The tests should be considered an actual part of the product, worthy of full consideration during design, and even, in many cases, delivered with the product to the customer. (This latter part is controversial, but a built-in self-test as part of a software delivery takes inconsequential space, and yet provide tremendous benefit when needed.)
• Write the test before the code. The test itself can be used to clarify the design for exactly what is needed. Many times there are flaws in the design approach which are discovered when working through the test cases. Think how much time would be saved to work through those cases before coding. But: write the test for case1, and code for case 1, before starting case 2.
• Eliminate Waste. Frankly, another ubiquitous platitude which must be included in any list of development principles because it is so important. There is no end to the job of looking for waste where it exists and eliminating it. Eliminate anything that does not add value to the actual customer. If you can not identify the value to the customer, then you probably don’t need it.
• Build a culture of immediate response to build breakage. Understand that when the build is broken, it effect everyone in the project, and so there is nothing more important than making sure that the central core code is building and testing properly. I have seen teams that allowed broken tests to persist for months because is was someone else’s job. Everyone suffered, but nobody acted. Instead, there needs to be widespread recognition that a little work will pay back in a big way over the team.
• Every team members needs to understand the needs of the customer. Large complex projects must be broken into separate teams and further divided for handing out to developers, but this should never be done to the extent that people lose sight of the desires and goals of the actual users of the final product.
• Keep related definitions together. Structure the code so that highly related things are located together, possibly within one class. This is a standard OO design principle of encapsulation. Ideally, all the code outside the class will not need to know the details of the internal workings. Some developers delight in spreading details across multiple files in order to organize in different way: such as to keep all the same data types together, or to organize alphabetically. For example, putting all the constants in one class in a different package from where they are being used adds unnecessarily to the complexity of the program. The guiding rule should be to group by relatedness with the result being to hide complexity.
• Always run the tests before checking in. This guideline will help you satisfy the “never break the build” guideline.
• Premature optimization is the root of all evil. A quote from Don Knuth which rings true today. Code should be written well to avoid needless waste at the micro level, but optimization beyond the individual method level should wait until testing within the entire program with a stress test bases on an actual end user use case. Intuition of what is important for overall performance is almost always wrong when based only on a static understanding of the code. Instead, measure the behavior of the complete system, to identify the 1% of the code that really makes a different in performance, and focus on that.
• Minimize backlog of uncompleted coding tasks. When a developer starts to work on a use case, there is a cost associated with all the code that has been modified, but not completed and tested. Holding uncompleted changes for days or weeks adds up to a significant risk of waste due to rework. Consider three tasks estimated to take 1 day each. Starting all three at one time, and working in parallel for three days involves an accumulation of 9 “units” of cost. But doing each task sequentially, completing each task before starting the next, involves only 3 “units” of cost. This is not intuitive. Our intuition tells us that while we are in there, we might as well do three things at once, before buttoning the work up. But software is not like physical construction. Short, quick, and complete jobs not only cause less cognitive load, but also reduce the chance that uncompleted work will conflict with another person’s uncompleted work.
• Never overgeneralize functionality. This is also know as “YAGNI – You Aren’t Going to Need It” . While coding a particular class, programmers like to think with a small tweak this class might be used for several other purposes. This is fine if those purposes are required by the current use case, but usually the programmer is thinking about uses which have not been invented yet, and which may in fact never be needed. (This subject always reminds me of the classic Saturday Night Live skit about the product which was both a floor wax, and a dessert topping.)
• Never use 3 lines when 2 lines would do. Succinctness in code pays every time someone else has to read it. But don’t shrink the code to the point of being difficult to read. Smaller, well written code can easier to maintain and easier to spot errors in, than verbose, ornately written code. Always simplify as much as possible, but no more.
• Never ever measure code by counting lines. The number of lines needed to do a particular task varies greatly from programmer to programmer, and from style to style. The number of lines of code does not tell you much of anything about the completeness or the quality of the code. Code quality can vary by a factor of 200, and this overwhelms any usefulness of the count of the lines. Count instead functioning use cases.
• Continually re-design and re-factor. Apply this cautiously because some code is brittle and difficult to change, but in general you should not be afraid to change the code to match the real use case. A data member may have been an integer in the past, but when a use case requires it to be a floating point don’t be afraid to change it, only be sure to complete everything: tests, documentation, installer.
• Delete dead code. There is a tendency to let “sleeping dogs lie” when it comes to large blocks of code that is not well understood. One example is adding a new method to a class to replace another, quite often the developer will leave the old method there “just in case”. Some effort should be expended to check to see if that method is needed, and delete it if there is no evidence that it is needed. The worst offense is commenting out blocks of code, and leaving that commented code around. Commented code should be deleted as soon as you know that the tests run, and certainly before checking it in. It is easy to add code at any time, it is hard to delete code at any time. Therefor, at a particular time that you have a good idea that something might not be needed, and small extra effort to verify this and eliminate the code will make the codebase more maintainable.
• Don’t invent new languages. Programmers love to make text files that drive functionality in way configurable at run-time. There are no end of configuration files to be able to change the behavior of the program without recompiling. The advent of XML is driving a chain of specialized custom “scripting languages” that allow functionality to be “programmed” by the end user without having to compile. The flaw in this reasoning is that the precise definition of the behavior of the operations almost never well defined outside of the context of a particular implementation, and these types of scripting languages are mainly useful only to people who have an intimate knowledge of the internal working of the code body in question. Thus, real end users without detailed internal knowledge can never possibly know what is necessary to anticipate the effect of complex combinations of commands. Scripting languages have a use, and can not be eliminated, but the designer must take a very very conservative approach and use existing languages as far as possible, and avoid inventing new ones.
• Do not create a design until you are ready to implement and test the implementation. You should have some overall idea of where you are going, and a overview of the system architecture that will be aimed for, but no detailed design, no detailed description of functional implementation should be written down until the development iteration that will allow that design to be implemented and tests. The detailed design should cover only as much as is needed to handle the current use case. The biggest cause of waste in software development is time spend designing things that are not needed or need to be redesigned because of some mistaken assumptions that the design is based on.
• Software is Plastic. Unlike physical manufacturing, software can be changed in significant ways very easily. In fact there is plenty of evidence that software is easier to change than the design specifications that describe the software. Furthermore, software communicates the design more effectively than the specification. Therefor, you should spend the time to implement the design directly, so that customers can see the details of the design. If you miss and have the change the design, it is easier to change the software than it would be to change the spec. But most important, your information about what the customers wants is far better after they have seen the code running.
• Take the time to code a complete description of the problem in the code that detects and reports exceptional situations. Programmers are often very lazy and throw exceptions with superficial descriptions of what is wrong. Thinking that they are the only people who will ever see the problem, and they will remember the meaning of the problem from the vague description included. But in fact more time is wasted in customer support situations because of inaccurate or incomplete error reports than any other cause. Write every error message is if you are explaining the situation to someone who just walked into the room and has no experience with the code. The customer, and the customer support team, after all, have no experience with the code.

These are presented in no particular order. I welcome comments on principles that I have left out, or (if this is the case) principles that you disagree with.

EuGENia is a front-end for GMF that enables developers to generate a fully functional GMF editor by attaching a few high-level annotations to the Ecore metamodel. The original aim of EuGENia was to lower the entrance barrier for new GMF users and enable people to quickly and easily develop the first version of their editor.

However, after the initial excitement of (at last) being able to get a working GMF editor with minimal effort and no cryptic error messages, all users (including ourselves) used to come to a dead-end. EuGENia could generate an editor that looked 90% similar to what we wanted but when we started polishing the .gmfgraph, .gmftool, .gmfmap and .gmfgen models manually to get to 100%, it meant we couldn’t use EuGENia any more as subsequent invocations of the tool would overwrite our manual changes. Obviously, providing support in EuGENia for all the options that GMF provides wouldn’t be reasonable either as this would progressively make EuGENia as complex as GMF itself.

Our first thought was to try merging (instead of overwriting) the generated with the existing GMF models, but due to the complexity and inter-weaving of these models we’ve found no sensible way of doing this in an automated way. Therefore, we’ve come up with a different solution.

Behind the scenes, EuGENia runs set of model-to-model transformations written in EOL which generate the necessary GMF models from the annotated Ecore metamodel. In order to accommodate the need for persistent customizations, we’re now allowing developers to specify their own little EOL scripts next to the annotated Ecore metamodel, which are responsible for customizing the generated GMF models.

Let’s go straight into a short example. We define this minimal flowchart metamodel using Emfatic.

@namespace(uri="flowchart", prefix="flowchart")
package flowchart; @gmf.diagram(foo="bar")
class Flowchart { val Node[*] nodes; val Transition[*] transitions;
} @gmf.node(label="name", label.icon="false")
abstract class Node { attr String name; ref Transition[*]#source outgoing; ref Transition[*]#target incoming;
} @gmf.link(label="name", source="source", target="target", target.decoration="arrow")
class Transition { attr String name; ref Node#outgoing source; ref Node#incoming target;
} class Subflow extends Flowchart, Node{ } @gmf.node(figure= "org.eclipse...examples.flowchart.diagram.figures.SquareFigure")
class Action extends Node { } @gmf.node(figure= "org.eclipse...examples.flowchart.diagram.figures.DiamondFigure")
class Decision extends Node { }

We then use EuGENia to generate a GMF editor out of it and here is what we get:

However, what we’d really like to get is this (spot the differences?)

To achieve this we need to tweak a few things in the flowchart.gmfgraph model:

We can either do these changes manually after every invocation of EuGENia (boring), or specify the changes in the following ECore2GMF.eol script next to our metamodel.

ECore2GMF.eol:

-- Set text of transitionLabel to empty string
var transitionLabel = GmfGraph!Label. selectOne(l|l.name='TransitionLabelLabel');
transitionLabel.text = ''; -- Add bold font to actionLabel
var actionLabel = GmfGraph!Label. selectOne(l|l.name='ActionLabelFigure');
actionLabel.font = new GmfGraph!BasicFont;
actionLabel.font.style = GmfGraph!FontStyle#BOLD; -- Add italic font to actionLabel
var decisionLabel = GmfGraph!Label. selectOne(l|l.name='DecisionLabelFigure');
decisionLabel.font = new GmfGraph!BasicFont;
decisionLabel.font.style = GmfGraph!FontStyle#ITALIC;

Now, each time we click Eugenia->Generate GMF tool, graph and map models, EuGENia will perform the built-in transformation and then it will also run our custom script on the generated models so that we can get the tailored output we need. Similarly, by specifying a transformation named FixGMFGen.eol we can customize the generated .gmfgen model (in this example we add a dependency to the figures plugin).

Except for enabling developers to now use EuGENia throughout the GMF editor development process, this extension also provides a good evolution mechanism for EuGENia itself. Reoccurring scripts will be good candidates for inclusion in the form of high-level annotations, that will be natively supported by future versions of the tool.

The complete source code of this example is available (projects named org.eclipse.epsilon.eugenia.examples.flowchart.*) in the SVN. You can also find another example, as well as more technical details about this extension here.

Apart from adding support for managing inconsistent EMF models with HUTN, and fixing several bugs, version 0.8.5 of Epsilon also includes a few new and noteworthy features.

Use of = both for assignment and for comparison

As EOL builds on OCL, it (re)uses = for comparison (e.g. if (a = b) {…}) and := for assignment (e.g. a := b;). As Louis reported here, this has been a source of confusion and well-camouflaged errors because when a mistyped a = b; assignment statement is executed, it will do nothing, but will not complain either. To deal with this we decided to follow the old VB6 paradigm and support = both for assignment (in addition to the := that of course still works) and for comparison. Therefore, since 0.8.5 the following code runs just fine:

var x = 2;
(x*x).println(); -- prints 4

Support for additional options in EuGENia

We’ve added support for several more options to EuGENia (the front-end provided by Epsilon for simplifying the development of GMF editors), such as for specifying if the model and the diagram are to be stored in the single file, for defining the extensions of the model and diagram files etc. A complete list of the supported options is available here. Kudos to Michael Moser for putting in these requests for enhancement in the bugzilla.

Support for float and double numbers in EOL

The EOL Real type now provides support both for Double and for Float numbers (only floats were supported up to 0.8.4). Kudos to Aleksander Bandelj for raising this issue in the bugzilla.

ANT tasks for EMF models

The new epsilon.emf.loadModel task has been added to the list of Epsilon ANT tasks as an alternative for the generic – but more cumbersome – epsilon.loadModel task. Using the new task, a sample UML model can be loaded as follows:

<project default="main"> <target name="main"> <epsilon.emf.loadModel name="MyUMLModel" modelfile="sample.uml" metamodeluri="http://www.eclipse.org/uml2/2.1.0/UML" read="true" store="false" /> <epsilon.eol> MyUMLModel!Class.all.size().println(); <model ref="MyUMLModel"/> </epsilon.eol> </target>
</project>

We’ve also added two more EMF-specific tasks. The epsilon.emf.register task, which registers the EPackages of an Ecore file to the EPackage registry:

<project default="main"> <target name="main"> <epsilon.emf.register file="sample.ecore" /> </target>
</project>

and the epsilon.emf.loadRegisteredMetamodel which can load a registered EPackage as a model that can be then accessed by other Epsilon tasks:

<project default="main"> <target name="main"> <epsilon.emf.loadRegisteredMetamodel name="UMLM2" metamodeluri="http://www.eclipse.org/uml2/2.1.0/UML" /> <epsilon.eol> UMLM2!EClass.all.size().println(); -- Prints 246 <model ref="UMLM2"/> </epsilon.eol> </target>
</project>

We are working on more enhancements, such as support for multiple iterators in first order logic operations (kudos to Edd Turner), and Java-ifications for the upcoming 0.8.6 which will be made available in late June. Stay tuned!

A model and metamodel are consistent when the metamodel specifies every concept used in the model definition. A metamodel can evolve (be adapted by a developer), which can cause inconsistency. We’ve recently added a tool to Epsilon, the HUTN / XMI bridge, that seeks to aid developers in manually managing inconsistency.

HUTN, an OMG standard, specifies a human-usable, textual notation for MOF-based models. Epsilon provides an implementation of HUTN for Ecore-based models. HUTN can be thought of as an alternative concrete syntax to XMI. While XMI is optimised for use by machines, HUTN is optimised for use by humans. We discuss HUTN in more detail in this paper at MoDELS 2008.

We have recently extended our HUTN implementation to include an XMI bridge, which allows us to generate HUTN source code from EMF models. Furthermore, the HUTN / XMI bridge can be used to report inconsistencies between a model and its metamodel, and to resolve those inconsistencies using a human-usable notation.

Example

To demonstrate the HUTN / XMI bridge, here’s a short example. Below is an exemplar metamodel, along with a sample model:

Channel B has ConnectionPoint both?! as its reader and as its writer.

Suppose we now wish to make distinct ConnectionPoints that are used for writing and ConnectionPoints that are used for reading. We may elect to change our metamodel, as shown in the diagram below:When we try to load our model with EMF, we receive a single error, stating, for example, that “The class ‘ConnectionPoint’ is not a valid classifier.” By contrast, the HUTN/XMI bridge is able to produce a complete list of the inconsistencies:

(Produced by the Check Consistency option, available by right-clicking a .model file).

To aid developers in resolving consistency, the HUTN / XMI bridge can generate HUTN from a model’s XMI, even when that model is inconsistent with its metamodel:

Once there are no further inconsistencies, HUTN can be used to generate a model that is now consistent with its metamodel.

Related Work

The HUTN / XMI bridge is a tool for manually managing inconsistency between models and metamodels. There are situations in which an automatic approach is more suitable. For example, perhaps you have a large number of inconsistent models. In those situations, we’d recommend using a different tool, such as COPE. You may also be interested in Antonio Cicchetti’s work; Cicchetti advocates a fundamentally different approach to the one used in COPE.

Installation

If you’d like to try the HUTN / XMI bridge, it’s available from the Epsilon SVN repository (access instructions). It will also appear in the upcoming 0.7.5 HUTN release*. We’ll be adding new features and fixing problems per user requests – do drop by the Epsilon newsgroup if you have questions, problems or suggestions.

* [Update 27/04/09] The HUTN XMI Bridge is included in HUTN 0.7.5, which was released today, as part of Epsilon 0.8.5.

Test-Driven Development (TDD) is an approach to software development that advocates writing tests before implementation. TDD has many advantages when compared to other forms of development, in which tests are often an after-thought. Proponents argue that TDD produces clean interfaces, minimal implementations and a suite of tests that can be used to guard against regressions. (See “TDD by Example,” Beck and, to some extent, “Refactoring,” Fowler).

In the past, we’ve been reluctant to construct automated unit tests for Epsilon, largely because we lacked structures and processes for constructing and checking models in the context of a testing framework, such as jUnit.

Since developing HUTN, we’ve been discovering/inventing/hallucinating patterns for constructing and checking models that allowed us to construct tests that were readable and malleable. This week, we wanted to make an enhancement to EOL, and, using these patterns, I was able to do so in a (more) test-driven manner.

EOL defines distinct operators for comparing expressions and for performing assignment. The concrete syntax is = for the former and := for the latter. We’ve found that we’ve had a few reports of code like the following:

var a : Integer;
a = 2;

EOL compares a and 2 in the second line, rather than assigning a to 2.

We decided that we could change EOL to rewrite statements whose first operator was = to an equivalent statement whose first operator was :=

I started by defining some tests that exercised the EOL execution engine with the old behaviour and with the new behaviour:

private static final EolEvaluator evaluator = new EolEvaluator(); @Test
public void assignment() { evaluator.execute("var a; a := 1 + 2;"); assertEquals(new EolInteger(3), evaluator.evaluate("a"));
} @Test
public void equalityActsAsAssignment() { evaluator.execute("var a; a = 1 + 2;"); assertEquals(new EolInteger(3), evaluator.evaluate("a"));
}

I’m using the EolEvaluator class, which decorates the EOL execution engine and can be used to execute small snippets of EOL code.

The first test passed, and the second test failed as expected. Typically, we’d now define some unit tests on the unit that needs to change to accommodate those changes, but I’ll leave that as an exercise for your imagination.

Crucially, because I’d used TDD, I felt comfortable editing a piece of code that I’d not developed, and never touched before – I had tests that verified (to some extent) my work. Plus, we’ve got tests that guard against regression in this part of EOL.

The Epsilon website already includes several screencasts and examples that demonstrate the tools and languages it provides. The preview release of the Google App Engine for Java has made it possible to go one step further and allow people to actually write and execute EOL (the core language of Epsilon) programs straight from their browser without needing to download or install anything.

http://www.eclipse.org/gmt/epsilon/live includes an editor where one can write and execute EOL scripts, and a console that captures their output. Apart from playing with the basic features of EOL (variables, control flow structures etc.) one can also query/modify a live EMF model (which for simplicity is Ecore itself). To help you get started, the page also provides a number of ready-to-run scripts as a starting point for further exploration.

The EPackage registry (EPackage.Registry.INSTANCE) contains references to all registered ECore EPackages in EMF. To visualise the contents of the registry, we have implemented the following EPackage Registry view.

Using this view, one can browse through the EClasses contained in each registered EPackage, discover the super/sub types of each EClass, and navigate through its features and operations. The view provides options to show/hide derived features, operations, inherited features and opposite references, supports quick navigation from a feature to it’s type (double-click), and integrates with the Properties view.

If you’d like to give it a spin, it comes as a part of Epsilon 0.8.3 (or later) and can be installed via the update site.

The Epsilon Validation Language (EVL) is a model validation language that provides additional features to OCL, such as guards, constraint dependency management, user interaction, access to Java objects, and the ability to specify a number of fixes for each constraint, that the user can invoke if the constraint fails in order to repair the model. We’ve been recently working on integrating EVL with GMF so that failed constraints can appear as error/warning markers in GMF editors, and fixes can be invoked using the standard Quick Fix mechanism. An example follows:

The following screenshot shows a file system model in a GMF-based editor (constructed here).

We now specify the following EVL constraints and bind them to the editor (using the instructions provided here):

context File { constraint HasName { check : self.name.isDefined() message : 'Unnamed ' + self.eClass().name + ' not allowed' } } context Folder { critique NameStartsWithCapital { guard : self.satisfies('HasName') check : self.name.firstToUpperCase() = self.name message : 'Folder ' + self.name + ' should start with an upper-case letter' fix { title : 'Rename to ' + self.name.firstToUpperCase() do { self.name := self.name.firstToUpperCase(); } } } } context Sync { constraint MustLinkSame { check : self.source.eClass() = self.target.eClass() message : 'Cannot synchronize a ' + self.source.eClass().name + ' with a ' + self.target.eClass().name fix { title : 'Synchronize with another ' + self.source.eClass().name do { var target := UserInput.choose('Select target', _Model.getAllOfType(self.source.eClass().name)); if (target.isDefined()) self.target := target; } } } }

Then, we evaluate the constraints by clicking Diagram->Validate, and our editor and Problems view now look like this:

We can now right click on the warning and invoke quick fix we have defined for the NameStartsWithCapital critique.

By clicking Finish, the fix is executed and the warning disappears.

A complete reference of the syntax and semantics of EVL can be found in Chapter 6 of the Epsilon Book (PDF). Also, instructions for binding EVL constraints with an existing GMF editor can be found here. Finally, a Flash screencast of this example is available here.

Last week we released v0.7.1 of our Human-Usable Textual Notation implementation. It’s available as part of Epsilon 0.8.1. In this release, we’ve aligned our implementation more closely to the OMG specification, and added some new features. Here’s a summary of the changes:

Validation

An error is now produced when enumerations, data types and abstract classes are specified as the type of an object.

An error is now produced for objects that do not specify the value of attributes with a lower bound greater than zero.

Adjectives

Positive adjectives are no longer prefixed with the # symbol. For instance:

DogPackage { male Dog "Fido" {} // old syntax was #male Dog
}

Adjectives prefixed with the # symbol will cause a deprecation warning.

Enumeration Values

Support for enumeration values in association instances has been added. The syntax is:

DogPackage { Dog "Lassie" { breed: Collie // Enumeration value name: "Lassie" // String value }
}

Classifier-Level Attributes

Support for classifier-level attributes has been added. The syntax is:

DogPackage { Dog.breed: Collie; // All dogs in this package are Collies by default Dog "Lassie" {} Dog "Jessie" {} Dog "Scooby" { breed: GreatDane }
}

Classifier-level attribute values take precedence over any default value rules specified in a HUTN configuration model.

Associations: Block and Infix Notations

Support for association blocks has been added. Assuming that the Family class has a reference called “pets” accepting objects of type Dog, the syntax is:

DogPackage { Family "The Smiths" {} Dog "Rover" {} pets { Family "The Smiths" Dog "Rover" // Refers to the existing Pet object, defined above Family "The Smiths" Dog "Fido" // Created automatically by the association block }
}

Similarly, associations may be specified using an infix notation. The example above could be written:

DogPackage { Family "The Smiths" {} Dog "Rover" {} Family "The Smiths" pets Dog "Rover"; Family "The Smiths" pets Dog "Fido";
}

We hope you find the improved validation and more flexible syntax useful. You can update your installation using the Eclipse update manager, or follow the instructions on the Epsilon download page.

A little off-topic but hopefully useful. Recently, I needed to find all the .java files in my workspace that did not contain a particular phrase (“Eclipse Public License”) to find out which files I had forgot to add copyright notices to. I tried to do this with regular expressions but couldn’t seem to make it work. So I looked into the Eclipse search plugin and finally managed to scribble a negative text search engine i.e. an engine that returns all the files that do not contain a particular phrase. Instructions on how to install and use the plugin are available here.

GMF is a great piece of software in terms of features – and not only because it’s built atop GEF and EMF. Nevertheless, (here it comes) it also requires significant effort to learn, the error messages it produces during validation are often cryptic, the gmfmap wizard typically gets it all wrong, and all put together just make it too hard for the inexperienced user.

To reduce the effort required to build a basic editor we’ve implemented a tool, called EuGENia, which can derive the gmfgraph, gmftool and gmfmap models that are required to generate a fully functional GMF editor using only the ECore metamodel itself, augmented with a few GMF-specific annotations (I know that annotations pollute metamodels but they’re irresistibly convenient).

The Filesystem editor example

In this example we’ll construct a GMF editor for a filesystem metamodel. To generate a basic GMF editor from the metamodel, all we need to know is that:

• The root of the diagram is FileSystem
• File and all of its subclasses (Drive, Folder, Shortcut etc) are nodes with a label that reflects the value of their name attribute
• Shortcut.target is a link
• Sync is also a link class with Sync.source as the one end, Sync.target as the other and lastSync as the label

To capture the above, plus a bit of formatting information, we add the following @gmf.* annotations to the metamodel (in Emfatic):

@namespace(uri="filesystem", prefix="filesystem")
@gmf
package filesystem; @gmf.diagram
class Filesystem { val Drive[*] drives; val Sync[*] syncs;
} class Drive extends Folder { } class Folder extends File { @gmf.compartment val File[*] contents;
} class Shortcut extends File { @gmf.link(target.decoration="arrow", style="dash") ref File target;
} @gmf.link(label="lastSync", source="source", target="target", style="dot", width="2")
class Sync { ref File source; ref File target; attr String lastSync;
} @gmf.node(label = "name")
class File { attr String name;
}

Through the process demonstrated in this screencast EuGENia can generate the following editor without the user needing to touch any of the .gmfgraph, .gmfmap or .gmftool models manually.

Obviously EuGENia doesn’t support all the options the .gmfgraph and .gmfmap models do but only a small subset of commonly needed features. A complete list of the annotations EuGENia supports as well as instructions for obtaining it is available here.

Follow up: EuGENia: Polishing your GMF editor