Software Engineering, Object-Oriented and Patterns Design

SOA Manifesto

2009-10-28T11:15:00.004+08:00

The SOA Manifesto is a fanfare to architects everywhere, its Signatories like a triumphant procession heralding the pragmatic delivery of business value to enterprises via the provision of IT services.

Service orientation is a paradigm that frames what you do. Service-oriented architecture (SOA) is a type of architecture that results from applying service orientation. We have been applying service orientation to help organizations consistently deliver sustainable business value, with increased agility and cost effectiveness, in line with changing business needs. Through our work we have come to prioritize:
Business value over technical strategy
Strategic goals over project-specific benefits
Intrinsic interoperability over custom integration
Shared services over specific-purpose implementations
Flexibility over optimization
Evolutionary refinement over pursuit of initial perfection
That is, while we value the items on the right, we value the items on the left more.

It was a bit of a toss-up whether to write this blog here or under Innovation and Business since the impact of enterprise architecture is across both the business and IT segments of the enterprise - the very issue addressed by myself and several colleagues in Is EA a general business term, or an IT term and Is Enterprise Architecture a philosophy or an artefact?

However I want to reprise just my own comments from the latter discussion:

I reckon enterprise architecture is the outcome of practising enterprise architecture; like design is the outcome of, well, 'doing the design and constructing the pieces.' Enterprise architecture means either or both of 'enterprise business architecture' or 'enterprise IT architecture.' We usually leave off 'business' or 'IT' as implied but there is a non-subtle difference.

The business architecture is a constraint on the IT architecture whether or not it reflects as-is or to-be business strategy. The IT architecture, on the other hand, we can shape and mold, just as creative an exercise as it is methodical. Either way, our IT terminology and outcomes should/must match the expectations of business stakeholders, corporate and IT governance, strategy and objectives.

As an exercise, using TOGAF or Zachman, for example, adds structure ('constraints give us freedom') however the creation and evolution of enterprise architecture is necessarily ontological, that is, associated with the existence of entities and the relationships between them. The philosophy, architecture practice and implementation are inseparable facets of enterprise architecture reflecting the underlying nature of complex, emergent systems.

We must think carefully about what we are trying to achieve, about alignment of IT strategy with organisational strategy, culture, values and objectives. As architects, we must ensure the technical solutions we provide are cohesive in themselves as well as delivering business value.

The SOA Manifesto and associated Guiding Principles, when followed, should help us to deliver on this promise.

Code generation and Godel's genie

2009-09-26T20:14:00.003+08:00

Many of the advantages gained by generating executable code from models are matched by various problems. At any particular level of abstraction, analysis and requirements languages like UML, Z and OCL are wonderful for specifying, with varying levels of precision, the constraints for the next, more-detailed model.

Model-driven architecture and code generation involve executing a series of transformations which use as input the higher level of abstraction to produce the more detailed architecture layer as output. The usual practice is to elaborate additional details of implementation in the more-detailed layer: The logical layer is used to generate the physical layer; the platform-independent model is used to generate the platform-specific model.

The problem with generating executable code is that the model language has been morphed into an implementation meta-language. When there is no room for elaboration because the model contains all the necessary details then the model is itself the implementation. The problem of verification has to be generalised from using the model to verify the implementation to one of verifying the model itself.

Godel's incompleteness theorem essentially says, in this context, that in-principle we cannot verify the model or its elaborated implementation. Hofstadter might say the meta-genie has been let out of the bottle. We verify models by inspection, consistency checking and theorem proving, when possible; likewise we verify implementations by inspection and the closest thing we have to theorem proving, i.e. testing.

The key assumption of testing is that we can verify the implementation against a subset of test vectors representing equivalence sets across the input domain. The key for code generation is that the set of valid assumptions embodied in the model is retained by an isomorphic transformation from the model into the implementation. Model consistency checking does not help if the model is inconsistent or any of its assumptions have been invalidated by the process of elaboration.

Agile Adoption: How to stuff it up

2009-07-01T09:53:00.015+08:00

ThoughtWorks ran a thoroughly enjoyable and rather informative pair of sessions cheekily titled Agile Adoption: How to stuff it up in Perth on Tuesday 30 June 2009. My hearty thanks go out to Karin Verloop for the invitation to attend the Quarterly Technology Briefing over a light breakfast at The Parmelia Hilton, and the Executive Round table Luncheon at The Globe.

Martin Fowler, Chief Scientist at ThoughtWorks and one of the original signatories of the Agile Manifesto, was unable to attend but Andy Marks, who specialises in enterprise-level adoption of Lean and Agile concepts, ably led the earlier discussion and along with Keith Dodds, Director of Sales & Marketing, facilitated discussion over the round table luncheon.

The Quarterly Technology Briefing was introduced by Karin Verloop and handed off to Andy Marks, who said he has been looking after the largest agile adoption in Suncorp. The Agile Manifesto states the common ground of lightweight approaches, eg. Scrum and XP. The notable software craftsmen who drafted the manifesto in 2001 at the Utah meeting decided to call it 'Agile' instead of the other possibilities mooted because they felt the word closely reflected their approach.

"It is not necessary to change. Survival is optional." said W. Edward Deming, the original thought leader in continuous quality improvement.

There are about six root cause issues to cover in this session. ASWEC 2009 on the Gold Coast recognised that "Agile is the new mainstream."

Andy credited Mike Cohen with the common adoption patterns for Agile:

Scale - from start small to all-in.
Interest focus - from teaching practices to iterations.
Visibility - from stealth mode to public display.

Not addressed was the trivial pilot anti-pattern. It's best to start with a high risk, high value of success to engage the business.

Here's a basic recipe for stuffing it up:

Underestimate the change.
Assume minimal cultural impact.
Undervalue experience.
Learning finishes at school.
Let others fail first.
De-emphasise technical design.

The people will break first. Small group comes to realisation that agile works or is worth looking into; by executive decree - why are we doing this? User story is statement of customer requirement, ideally can be implemented in 1-2 days; binary status reporting 0/100% completed.

As opposed to projects reporting green, green, green,... then red one month before delivery - 6 months late; instead early indication, eg. green, green, amber, red... needs a cultural change.

"He who rejects change is the architect of decay," Harold Wilson.

On organisational culture: "It is very evident that all culture has developed as an initial reaction against adaptation, " Beatrice Winkle. [Can someone help me with this literary allusion?]

IT is tailor made to embrace change, especially software developers who can be [ahem] obsessive compulsive (OCD) and show signs of attention deficit disorder (ADD). Making new connections, changing existing connections, 'lusting after change' - you want these people in you team with Agile adoption.

Standup meeting - what I did yesterday, what I am doing today, any blockers. Co-located team, customer embedded, tolerance for failure.

"Failure is the opportunity to begin again more intelligently," Moshe Arens.

Often rewarded publicly for 'success' - Herculean efforts, 'by hell or high water'; other projects ('failures?') which have taken risks, lessons are not advertised.

The inspirational ravine of effectiveness-versus-time has a dip as a team moves through learning, change, transition. It is guaranteed to kill adoption if you stomp on people in the ravine.

Outliers by Malcolm Gladwell speaks of the conditions to achieve high level performance - his studies show 10,000h to achieve high-level aptitude, or about 5y at 8h/day. (Note that a journey man plumber, not master, is achieved after 5y.)

"Experience is the name every one gives to their mistakes, " Oscar Wilde.

It is a mistake to de-emphasise technical design. The Manifesto implies paying continuous attention to technical excellence and good design enhances agility. [I myself speak of a culture of quality and merciless refactoring.]

The best architectures emerge from Agile design. Don't over-invest in design ('big upfront design'); however the pendulum may swing too far the other way.

Customer focus is important because the impact of technical debt is that the entire organisation, says Kent Beck, can be brought to a standstill under the debt load of unconsolidated implementation. Debt load is lack of technical quality, i.e. refactoring, patterns.

The next step:

Conduct readiness review.
Get experience.
Plan in Agile way.
Identify a quick win.
Budget to include learning.
Publicly reward effort to improve and acknowledge lessons learned.

The discussion at the Executive Roundtable Luncheon was interesting and stimulating. As a marketing tools, Agile is clearly finding its way into the hearts and minds of practitioners and IT leaders alike however there seems to a concomitant forgetfulness of supporting principles and practices.

In particular, the role of project managers and the governance structures provided by approaches like Prince2, PMBOK and CMMI go largely unremarked, except for a lingering negativity towards such approaches.

[Chatham House Rule applies to the following recollections.]

Surprisingly to me, the organisation with the highest statutory compliance obligation and hence a satisfactory management system was the least inclined towards a traditional project management approach.

To some extent I think their position is a misunderstanding about the purpose of documentation and the intent of the expression in the Agile Manifesto to value, "Working software over comprehensive documentation." The purpose of documentation is knowledge capture and sharing, and they should be so lucky to have it!

The issue of testing in an Agile environment was raised. Like most things Agile, even formal testing using the traditional techniques fits the description. Test specifications can almost entirely replace user and functional requirements because that is precisely what is evaluated during user acceptance, functional and system testing.

Inadequate testing will deliver a fragile product into production every time. Worse still, it will be very difficult to run regression tests, both manual and automated, in order to identify new defects.

Requirements management was the crux of another question: Are changes to requirements assessed and approved by the business first? This is a tricky one because in an Agile approach, a customer representative is supposed to be available and one of the problems is that often they are not empowered to make decisions, or lack the specialist knowledge of a subject matter expert.

Of course, this is the reason for functional requirements but I agree that a user story, written from scratch or enveloping one or few cohesive requirements, can be given to a developer in an Agile team to take 1-2 days.

The Agile business analyst has a similar role to the traditional business analyst and that is to interpret the customer requirements and to act as the customer proxy to the development team. It is not so Agile to interpose someone between the customer and the developer but for complex requirements this is fairly necessary. It was also noted that user stories could be expanded into proper use cases.

Business rules came up for discussion as well and how they can be captured: in user stories? For simple or few rules this may be fine but otherwise it is begging for disaster to scatter business rules across informal user stories and throughout the code base. A little bit of abstraction may be called for using design patterns, eg. policy or delegate, or use a full-blown business rules engine - an architectural decision. The cost of rework can be high unless this decision is made upfront.

Agile in the context of enterprise software development was discussed, involving the integration of legacy and commercial-off-the-shelf (COTS) software and systems. Interface contracts between systems, common messaging formats, architecture of enterprise service bus (ESB) are design elements usually associated with big upfront design (BUFD) but are the only things that matter when developing distributed systems.

It was noted that stubs and mock objects enable the creation of unit tests before launching into full-on integration testing.

Does Agile apply to COTS only development? Yes was the answer, mainly to do with requirements gathering - well, maybe but I'm not sure. The business requirements and scope should be the same regardless of software approach because a project can only be initiated once a problem or gap has been identified by the business.

(The detailed functional requirements, use cases or user stories, in general, are allowed to be a moving target - part of the reason is that many 'discovered requirements' are unknown at the outset.)

It makes a lot of sense to write the evaluation criteria as a test specification so that each function can be ticked-off and compared during the evaluation phase. Then, in place of a traditional software construction there will be several iterations of 'construction by configuration' where the user stories are met at each closure.

Hire these guys if you want to learn to do software development well. The people at ThoughtWorks really know what they are doing and if the marketing hype about Agile gets you down just ignore it.

The values behind the Agile Manifesto are the key to their approach which I would happily classify as Agile-Lean because unnecessary processes and requirements are discarded by applying common sense to software development.

Future trends in software engineering

2009-03-03T09:18:00.010+09:00

About a year after the panel convened at ASWEC 2008, my words on future trends in software engineering are still pertinent to ongoing work I am undertaking in software engineering and service-oriented architecture.

Model-driven architecture (MDA)

not old-style computer-aided software engineering (CASE), i.e. discredited round-trip engineering
capture different levels of abstraction; elaborating lower-levels, constrained by higher-levels

Software factories

component-based libraries and frameworks
meta-object facility (MOF) for store and registry of component meta-data
dependency-injection, eg. Spring factory
mocks objects to complement, not replace, unit and integration testing using fixtures

Service-oriented architecture (SOA)

continuing growth and refinement of ontology-based software engineering
return to not-quite components, i.e. facade for service interfaces that are explicit
growth of resource description framework (RDF), web ontology language (OWL) - semantic-description languages and extensions (hard problem!)
realisation that C++, Java, C#, perl, python, yacc/bison (of course) incorporate support for domain-specific languages (DSLs) by extension, genericity and composition
convergence of object-oriented analysis and design with design and enterprise patterns
combine patterns and domain ontology to give DSL independent of technology and platforms - "form follows function"

Non-trivial systems

software and systems engineering merge into concrete discipline
integration architecture, distributed and real-time computing are increasingly seen as facets of same problem
enterprise security, single sign on, unified data views, portals - service and system mashups at user and data levels
two-level development: 1) service and business levels based on business/logical processes and composite services; 2) implementations that conform to service and interface contracts can be in any technology, i.e. custom software, commercial-off-the-shelf (COTS) or free and open source software (FOSS)
well-defined registry interface - "think global over enterprise, act local" every system and application adheres to enterprise guidelines
configurable and adaptable - constrained flexibility due to absolute need to confidently test known configurations

Cross discipline

computer science and software engineering, psychology, management
unified analysis, richer toolkit of analytics and heuristics
mature processes, peer review and accreditation

Messaging Completes a Full Circle

2008-10-11T18:47:00.018+08:00

"riverrun, past Eve and Adam's, from swerve of shore to bend of bay, brings us by a commodius vicus of recirculation back to Howth Castle and Environs." James Joyce, Finnegans Wake

In the beginning there was RPC or Remote Procedure Call, later mostly standardised as Distributed Computing Environment or DCE-RPC and reappearing as Microsoft's MSRPC. XDR is a protocol for the description and encoding of data and is useful for transferring data between different computer architectures - roughly analogous in purpose to Abstract Syntax Notation (ASN). Often used for C programming but with bindings to numerous languages on many platforms.

Time passed and Corba or Common Object Request Broker Architecture appeared, less well known in modern incarnations as the most-widely deployed interoperability standard, IIOP (Internet Inter Orb Protocol), also underlying RMI-IIOP and EJB-IIOP in the Java stable, and deployed with every web browser, most mobile phones and other devices. Corba and IDL (Interface Definition Language) allows many languages and platforms to interoperate transparently using objects often written in C++, Java or C# but possibly in almost any language. Regular claims that Corba is dead are premature and show ignorance or denial of reality.

Various message queuing technologies soon appeared on the computing horizon to start the trend towards loosely-coupled systems. IBM MQ Series, more recently called WebSphere MQ, Microsoft Message Queue (MSMQ) and Java Messaging Service (JMS) offer interoperability options that may cross platforms depending upon vendor support. WebSphere MQ is offered by IBM at a fee on virtually every platform imaginable; MSMQ is a Microsoft-centric technology; and JMS is a community-defined standard with many cross-platform implementations.

Bringing us to recent times where the buzz word for loosely-coupled systems is SOA or Service Oriented Architecture, often built upon web services. Over the past few years, the WS-standard stack or web service protocols, including WSDL (web service definition), SOAP (message protocol) and UDDI (discovery) that leverage XML, have appeared. Unsurprisingly they closely mirror the Corba stack, using XSD (XML schema definitions) as the new IDL.

Web services define function interfaces like RPC XDR, not object interfaces like Corba IDL, so have we just completed a full circle and returned back to where we started?

XML compared to IDL, XML first:

<message name="SayHelloRequest"> <part name="firstName" type="xsd:string"/> </message> <message name="SayHelloResponse"> <part name="greeting" type="xsd:string"/> </message> <porttype name="Hello_PortType"> <operation name="sayHello"/> <input message="SayHelloRequest"/> <output message="SayHelloResponse"/> </porttype>

Now IDL:

struct SayHelloRequest { string firstName; }; struct SayHelloResponse { string greeting; }; program Hello_Server { version Hello_Version { SayHelloResponse sayHello(SayHelloRequest) = 1; } = 1; } = 0x20000001;

Part of the answer is that SOA or Service-Oriented Architecture is one characterised by:

Loosely coupled services.
Platform-agnostic interfaces.
Dynamic discovery and invocation.

SOA is an approach to system integration centred around business processes.

Another part of the puzzle is found in BPM or Business Process Management, which embodies:

The concept of choreographing work items through a multi-step process or orchestration.
The work items are identified and tracked as they move through each step, with either specified people or applications processing the information.
The process flow is determined by choreography with the applications themselves playing no role in determining where the messages are sent.

MDA or Model Driven Architecture, as defined by the OMG or Object Management Group, fills in the technological process of managing the transition between the artifacts generated by the business and those used by IT:

Zachman's framework for enterprise architecture is the final piece of the puzzle for slotting it all together:

So there you have it, we have finally gained the hard-won knowledge accumulated through experience that the business should drive IT systems development (fancy that) and that models have greater and longer-lasting value than technology implementations (who would have guessed).

Now quieten down and listen to me, messaging was actually a dead end and the next big thing in enterprise integration and software development is...

Recipe for Getting and Staying Ahead in CSSE

2008-06-01T15:23:00.002+08:00

I am a bit embarrassed about not posting here for so long however the reason is that I have been busy with ASWEC 2008. A while ago, I was honoured to be invited to present a seminar in CSSE at UWA, as follows:

Goal of CS and SE degrees

2007-05-26T16:58:00.005+08:00

The goal of CS and SE degrees are respectively to turn out Computer Scientists and Software Engineers. Sometimes it is forgotten that graduates are much more than merely programmers insofar as studies at university can spend an inordinate amount of time immersed in the minutiae of programming.

As part of a recent discussion about programming languages following in part from my earlier articles on the topic and a stimulating presentation at a SE Forum, I read an article entitled An Objective Comparison of Languages for Teaching Introductory Programming (Mannila and de Raadt). To put it in a nutshell, I would contest the word Objective in the title and I think their assumptions about what makes a valuable teaching language and their analysis of the candidate languages against those criteria are both flawed. Some of their criteria, for example, The language is suitable for teaching with reference to Wirth's Pascal and Modula languages are self-serving and circular tautologies.

On the other hand, issues of strong typing (eg. constness), static type safety (i.e. compile-time type checking) and support for reuse via generics, mixins, delegation and inheritance are glossed over or ignored completely. These being some of the very charactistics that make programming languages useful as opposed to such criteria as The language can be used to apply physical analogies in reference to Papert's Logo and Alice that purports to teach object-oriented programming concepts by enabling the student to manipulate objects via their properties and methods in a microworld.

It is as if CS should be taught at university the same way that mathematics are taught through the progression of primary, secondary and tertiary education followed by practice - in the footsteps of Piaget's childhood development of cognition. It trivialises the purpose and mode of tertiary education to suggest that concepts already known should be retaught. The context of the learning environment must be taken into account but teaching object-oriented programming as defining an ontology of classes modelled on the real world - the best description of the thought process being found in Booch's seminal work - is very different to crawling through an object-world sandbox before learning to walk on crippled programming languages designed for teaching.

I think that Bloom's taxonomy or a similar kind of learning path is more appropriate with the tertiary student starting from a higher level from the outset. For example, comprehension is quite distinct from synthesis as evidence of high-level cognitive skills. The same applies for method based reading or code inspections when the inspector can perhaps read and comprehend the code but may lack the technical ability to synthesise the equivalent result in any programming language. Call this a desktop review and arguably one of little value.

Piaget's theories of cognitive development are not applicable to programming because the adult learner is already at an operational stage of competence with abstractions, for example, mathematics. Therefore easy-to-use and simple syntax perhaps are not valid criteria for selection of teaching or industry programming language. Koenig and Moo argue persuasively that C++ is an ideal language for teaching and for industry practice. The study cited uses few if any of the features that set C++ above and C++-like languages including Java and C# apart from the other languages in the feature comparison.

The flavour of the past decade was for C++ and Unix to be maligned as out of step with modern practice however the collective wisdom on Unix has shifted back to one based on common sense; C++ soon to follow in supporting several programming paradigms including functional and generic programming in addition to object-oriented programming. The modern programming languages like Java and C# are becoming more like C++ in feature richness just as so-called modern operating systems are becoming more Unix-like in the elegant simplicity of its design.

Precision teaching in the behavioural formalism of Skinner is perhaps of greater relevance where the reinforcement of outcomes is desirable, again reflecting on industry programming practice and code inspections in the sense of Fagan and Weinberg. Many of the best known text books and reference books available on programming build from an elementary foundation, adding advanced concepts and implementation details according to the authors taste.

Frequently we refer to language features that are not strictly necessary, for example, operator overloading as syntactic sugar. Similarly, the syntax of the language is for all intents and purposes irrelevant since an alternative and equivalent syntax could be substituted. Tritely some argue that their preferred syntax is superior but this is merely an expression of the very-human tendency to believe one's own opinion as to the superiority or elegance of form to be better than another's favoured form.

In teaching and industry practice I do not believe the choice of programming language has as significant an impact as other factors. More important is the way the language is taught, the manner of reading or inspection and other supporting practices, eg. design. I have found that many people who are focused on programming believe that programs are an outcome in themselves however in most settings the business outcome is the only consideration outside of the programming team itself.

Since all of the general-purpose programming languages are Turing complete then business considerations of maintainability and total cost of ownership, for example, mean that project management, enterprise architecture, business processes and issues of service contracts are the business drivers and the implementation mechanism is not directly relevant.

CS and SE degrees programmes must include significant mathematics components. The coverage of material and degree of difficulty depends on the goals of each school however the fundamentals of logic, set theory, statistics, matrix algebra, sequences and series must always be included. Not every graduate needs to know much or perhaps anything about multivariate calculus of a complex variable, number theory, or advanced numerical methods.

Some of this material will probably be covered adequately in a unit on algorithms or maybe even robotics. Probably not to the same level of rigour as if the material were taught in a mainstream unit taught by the mathematics department or even one customised for the needs of CS and SE.

The objectives claimed from teaching simplified programming languages specially designed for this purpose are questionable. The example earlier in this article where the programming language Alice is used to teach object-oriented concepts captures the fatuous assumption that the students in such course do not learn basic mathematics. Set theory, for example, captures and defines the concept of class and object together with axioms, mappings, functions and properties. Likewise a course on algorithms should build on or, if you please, specialise rather than replace or reinvent a unit on numerical methods.(*)

To summarise, we should treat our students as adults not as children to be molly-coddled!

(*) I could launch into a discussion about the seemingly unknowing teaching of kinetics, kinematics and dynamics in physics, mathematics and engineering units simultaneously with equivalent concepts in fluid dynamics and electrostatics.

For example, the curl and div of vector potential fields in Navier-Stokes and Maxwells equations, or more simply the moments V=I.R and F=M.A where V=voltage, I=current; F=force vector, M=mass, A=acceleration vector=second derivative of vector displacement; .=scalar multiplication or vector dot product.

But I won't. :-)

Matching the Process to the Project

2007-04-29T15:48:00.000+08:00

Gathering the Right Requirements
(Julie Blake, Business Analyst, Object Training)

Since returning I have been happily telling all and sundry that with this presentation, Julie was awarded the best Industry Track Paper for the conference - she's a BA and her presentation nailed the requirements process. Julie delivered case studies about two projects, contrasting the approaches taken and identifying lessons learned.

Project 1:

300 use cases.
6 experienced BAs.
4-5 months.
Success - happy client.

Project 2:

100 use cases. (different level of detail; about same size ~300 under more analysis)
40 new BAs.
8 months+.
Still going - trying Agile.

Both projects were mentored.
Project 1:

BAs experienced.
Customised templates and tasks.
Fixed time frames.
Clear, succinct high-level requirements.
Clearly defined boundaries.
Clearly defined interactions with external systems.

Project 2:

Inexperienced BAs (from business or SW developers) - little or no training or coaching.
'Project office' customised roadmap and templates.
Ongoing activity.
High-level requirements too detailed.
Cloudy project boundaries.
New external systems were constantly discovered: 50-70 kinda involved.

Lessons learned:

Don't spend too much time on detailed requirements up front.
Team responsible for customising templates.
Time spent on requirements eats into development time.
Requirements and development should be done in parallel -> Agile.
Avoid analysis paralysis.

When do you stop?

Requires thought.
Do developers have enough to go on with?
Can business sign off?
Have the complex parts of the solution [problem?] been captured adequately.

Essential requirements iterations:

Scope; Context; Capability.
Detail; Quantifiable Risk; Buildability.
Resolution; Completion; Testability.

Deadlines slipping - why?

Too much detail up front?
Problems getting signoff?
Don't know what to do next?

Process may help - what to do next.

Customise the process:

Every project should customise for project needs.
Process chosen should be flexible enough for customisation.
Agile is not always the answer.

Forces:

Agile for high rate of change. <--
and risk. <--
--> Plan driven for depth of business user knowledge.
--> and project complexity.

How do I customise?

Key - experienced practitioners, not third party.
Activities to touch lightly and in detail.
Experience and coaching are required.
Process does not turn bad BAs into good BAs.
Need support for coaching to customise process and activities for BAs, developers and testers.

Experience makes the difference.

Q. Methodology?
A. UML, use cases, activity diagrams, sometimes sequence diagrams. Not casual. Any consistent methodology. Same methodology for each phase - more detail and elaboration.
Q. What size?
A. Is anybody going to need what I am writing? Break into small, bite-size chunks - subject areas.

Requirements engineering is important - vital.
Business needs to have, to state key project requirements that are clear [objectives!].
Detail of dozens/hundreds of use cases can change - in detail - does not affect key outcomes of project.

Panel on Standards and Accreditation

2007-04-29T15:31:00.000+08:00

Panel on Standards, Accreditation and Joint EA/ACS Board on Software Engineering
(Prof. Leon Sterling, University of Melbourne, Chair Joint EA/ACS Board)

Joint EA/ACS Board has existed since 2001 - low key role in shaping professionals involved in software engineering. Cooperate rather than compete - three members from each plus academic and industry representatives, alternating Chair.

Bob Hart: Accreditation from ACS and Joint Board - primary role to cooperate on accreditation; should we have licencing and certification for SE and registration issues; role in running ASWEC (was alternating between EA and ACS). ~5 meetings per year - review of guidelines for accreditation - shapes accreditation visit, '... to the engineering [verb; gerund] of software.'

SWEBOK as example of knowledge base.
Best practice.
Code of ethics.
Team project.
Clear course aims - not CS extended - articulate vision of graduates.

ISO standard on certification of SE professionals, 11 March 2007 - Bob Hart, Manager of Professional Standards and Development, ACS:

SEPs work across international borders.
Need for portability of professional status (Washington Accord for engineers; not all jurisdictions can meet).
Cohort aspect of engineering degrees.
Certify on basis of examination but no ongoing requirement in some jurisdictions.
Common framework for evaluating certification schemes.
ISO 17024:2003 standard for conformity and accreditation as a certifying body.
Required to describe the type of SEP that certification covers: Title; tasks undertaken; level of accountability and responsibility; skills and competence, knowledge and cognitive levels; other assisting information to identify types of SEP.

Knowledge and skills:

SWEBOK as comparison.
Cognitive skills (eg. Blooms taxonomy).
Other knowledge and skills not in SWEBOK.
Generic professional capabilities.
Appropriate standards knowledge.
Domain (industry, application, &etc) knowledge and skills.

Evaluation - competencies, KAs, examination, reports, interviews, employer assessments, external examiners accreditation and QA.
Code of Ethics - minimum set of ethics; compliance; complaints and disciplinary processes.
Maintenance of Certification - period of certification and request for renewal; signed compliance statement from SEP; renewed commitment to code.
Continuing Professional Education - requirement for CPE; what and how much; compliance and sanctions.

How Open are we to Open Source?

2007-04-29T13:52:00.001+08:00

(Michael Coates, Object Consulting - Java J2EE on Windows and Unix. Focus on solution architecture - Consulting, Development, Training and Support.)

Java development - relies on open source software on every project he works on including development environment and frameworks - in commercial and government. Java - lots of open source; Microsoft.Net - filling gaps especially Java developers desiring specific features.

Examples:

Eclipse, Netbeans, xplanner.
Ruby - mainly DSL for build processes.
Some Linux and database deployment.
Java language itself is becoming open source.
JBoss, Tomcat - rare in givernment and enterprise.
Few business, enterprise applications to match Office, Notes, SAP.

Australian National Archives has three open source projects on Source Forge. Infomedix - five major hospitals has systems based on Linux plus MySQL commercial licences. University departments - eg. 350 students onsite - open source everything, high reliability (uptime). Lesser GPL or Apache licence for reuse of libraries.

Licencing considerations:

As a platform? ef. JBoss, Tomcat, Linux.
As a library? eg. log4j, Spring, nHibernate.
Modifying, borrowing from, eatending lib or product? eg. customer CMS.
Internal solution.
Developed on behalf of third party.
Plan to distribute for free or for a cost.
How to get new licence approved?

Barriers to adoption - Support:

Forums, wiki pages.
Community versus Enterprise licences.
How does it compare to support for current solutions? Vary between the development team and the business users.
Where do we get training? Available for mature projects.

Barriers to adoptions - Incumbent vendors:

No one every got fired for buying...
Cost of change for retraining, deployment. Need to have a significant reason to change.
Bargaining.

Yet er're still using Open Source:

Cost effective - existing skills, low cost to try.
Open standards - avoid vendor lock-in; interop, duture proofing.
Reduced time to market - particularly for development teams.
Best of breed.

Trend towards greater focus on business domain components over technical domain, eg. model-driven architecture (MDA) for business patterns. You cannot sell infrastructure to business - OS projects focus on techies (devel tools, libs, OS). For wide adoption provide visible business value (CMS, CRM, ERP). Government agencies making open source available to wider community.

Architectural Knowledge Sharing

2007-04-29T12:15:00.001+08:00

Prerequisites for Successful Architectural Knowledge (AK) Sharing
(Rik Farenhorst, Patricia Lago and Hans van Vliet)

Architectural knowledge not shared eventually dissipates. Sharing:

Prevents loss of crucial architectural knowledge.
Enables exchange of experiences.
Enables reuse of expertise.
Assists to train juniors.

Not so popular because:

Costs time and effort.
Lack of perceived short-term benefits.

People need to be motivated - incentives - to make them aware of intrinsic benefits and to induce people to share AK. Incentives for AK sharing:

Establishment of social ties - leading to increased transparency and trust.
More efficient decision making.
Knowledge internalisation - ownership of explicit knowledge; control, stored in own mind.

Study at large software development organisations identified four angles that need to be activated:

Context and environment.
Description.
Decisiveness.
Stakeholders roles and responsibilities.

Observed issues with AK sharing:

Lack of consistency between architecture and design documents.
Communication overhead between stakeholders - not properly-documented decisions.
No explicit collaboration with maintenance teams.
No feedback from developers to architects - architects may lack up-to-date technology knowledge.
No up-tp-date knowledge from development teams in repository.
No up-to-date knowledge from main customer in repository - reference architecture used by organisation.

Prerequisites for AK sharing:

Alignment between design artifacts -> [create] higher-quality knowledge -> [incentive] knowledge internalisation.
Traceability between architectural decisions and descriptions -> [as above].
Architect fulfil a central role -> minimise competition -> establishment of social ties.
Central architectural knowledge repository -> creates feedback loop -> more efficient decision making.

Goal: Effective tool support for AK sharing - close alignment to architecting process; lightweight and appealing to users. Solution: AK sharing platform that features explicit focus on collaboration, hybrid AK model (codification and personalisation), reuse of best practices, expertise, easy adding and editing. For AK that is hard to make explicit: Who has what AK? - 'yellow pages' and store unstructured information - blogs, discussion groups - searching and enriching this information.

ASWEC 2007

2007-04-22T10:12:00.001+08:00

ASWEC 2007 was held at the RACV Club in Melbourne from 11-13 April with tutorials on Tuesday 10 April and was a great success. Expect several followup articles covering various sessions and presentations that I attended. Welcome by Prof. Doug Grant, conference General Chair - who gave thanks to authors, sponsors, Steering Committee and organisers.

Doug noted that it has been 39 years since the NATO conference (1968) held on German city of Garmisch, coined the term 'Software Engineering.' The theme of this years conference is an example of reuse, "Taming Complexity with Research and Practice.' Since that time much has changed: Thomas Freedman, 'The World is Flat' explores globalisation; ASWEC is a global conference that is located in Australia.

The charter of ASWEC is to communicate and share problems, solutions, to form partnerships. There may be impact from the Research Quality Framework (RQF) which seeks to assess the quality of research in each discipline. Research publications and local conferences appear to be disadvantaged by these metrics. Those is academia must influence attendance of conferences and members of university advisory committees should do likewise.

Competent graduates are the ones who can communicate well; they are snapped up by industry. Persuading young people to study in our universities - ICT - not for lack of places, we need ideas to reenergise and reengage young people in these disciplines.

Prof. Jun Han - few words about the program; research and industry track papers; keynotes; tutorials (yesterday). 100 submissions; half local, half overseas; selected 39 - reviewed by at least 3 (sometimes 4 to resolve issues). Selected 19 industry presentations. Mixed program of industry and academic papers by topic.

The keynote was delivered by Prof. Oscar Nierstraag, head of the Software Composite Groups - making systems more adaptive to changing requirements, visiting professor at the Sorbonne, coauthor of Software Reengineering Patterns - speaking about modelling change as a first-class entity. Kinds of change include configuration, refactoring, new functionality, run-time adaption, projects include: Piccola, Traits, Classboxes, Persephone and Geppette, Object Flow Analysis, Moose, Changeboxes to encapsulate change. Software inevitably changes but program languages and development environments inhibit change rather than support it. Symptoms include assumption of global consistency, static type systems, design patterns - covered in 'On the Revival of Dynamic Languages.'

Piccolo. Problem - reconfig application (without changing eveything). Solution - composition language - application = components + script. Separates stable components from flexible configuration. Scripts plug together (not wire) components and define style of composition for plugins. Lessons - first-class namespaces enable flexible composition, eg. sandboxes; conceptual gap between core and target models (components, plugins and styles versus agents, forms and channels).

Traits. Problem - refactoring commonality (without introducing fragility). Solution - factor commonalities into traits (mixins and multiple inheritance are fragile); resolve conflict locally. Traits are parameterised behaviours that provide and require methods - no state (assumption may be weakened) - class = superclass + traits + glue.

Classboxes: Controlling visibility in class extensions. Problem - changing a running system without anticipated hooks. Solution - refactor on demand - partial, behavioural reflection. First idea - ByteSurgeon to transform Smalltalk code.

Persephone. Reflection methods - higher level - byte code plus abstract syntax tree (AST).

Geppetto. Behavioural reflection on demand. Lessons - reflectoin enables analysis; reflection need not be expensive; should be scoped to relevant contexts (eg. Lists in application context).

Moose. Extensible metamodel repository - navigation, metrics, querying, grouping - tools for enabling research. Java, Colol, C++ - XMI - import to enable model-driven development, explicit and available at runtime. Enables "ever more fine-grined changes" without artificial separation between source code and compilation (eg. Smalltalk and Eiffel).

Certainly an interesting and stimulating talk by the keynote speaker. My own notes and several pointed questions and remarks raise the question, "Is this software engineering?" Some of these techniques approach the mainstream insofar as dynamic interception can be implemented with run-time proxies using reflection; the Spring framework allows for wired configuration at run-time; the new keyword in .Net localises the context for a method extension; Corba using the dynamic invocation interface.

The issue of regression testing, verification and validation to give confidence in the run-time is similar to proxy interceptors, SQL triggers and aspect-oriented programming. It is ironic that low-level techniques are arising to allegedly solve problems with the object-oriented approach to design but aspect-oriented programming, while described as orthogonal to object design, is at a lower level of abstraction.

When asked about bugs related to threading, locks, logging, transactions and similar, I usually say that the design is flawed - no amount of hunting will track down the elusive bug if it isn't there in the first place. Go back to analysis and design to isolate the parts of the system that are variable from those that are general - see, for example, Coplien's Commonality-Variability principle. Software engineering is more than just programming.

Commonality and Variability

2007-02-16T23:47:00.000+09:00

Following the same thread towards commonality-variablity analysis, a practical and common-sense approach to designing abtract and concrete classes by distinguishing common behaviour of a class-of-types from their distinct variable behaviour from each other within that class-of-types.

I was just refreshing my memory about commonality-variablity analysis (see James O. Coplien, et al) and I noticed the first reference D.L. Parnas et al., "The Modular Structure Of Complex Systems," IEEE Trans. Software Eng., Mar. 1985, pp.408-417.

Prof. David L. Parnas was the keynote speaker the Australian Software Engineering Conference I recently attended in Brisbane. An inspirational speaker, he is "one of the grandmasters of software engineering." (see). "Software Fundamentals: Collected Papers by David L. Parnas is a practical guide to key software engineering concepts that belongs in the library of every software professional." (ibid; see also).

I was privileged to have the opportunity to sit down and chat with Prof. Parnas on several occasions over the course of the conference and found him to be a gentleman and still passionate about software engineering, licencing and education or professionals in our field. One of the recurrent topics is the content and quality of University education.

Recently, I had the opportunity to give a seminar in CS at UWA about Java, C++, type-safety and emergent design, in part motivated by discussions with David and also stimulated by thoughts from Douglas Hofstadter's fabulous Godel, Escher, Bach: An Eternal Golden Braid.

You haven't read GEB? I heartily recommend it as a glorious and stimulating mental roller coaster for all readers.

Which programming language to teach?

2007-02-16T23:32:00.001+09:00

Having just stumbled across these comments in a personal email I sent to a colleague about what programming languages to teach, it seems a good idea to reporoduce it here. Not intended to start a flame war but rather to offer a commercial perspective on the relevance of the various languages in undergraduate teaching at university. My reply to his question is shaped by my thoughts having just returned from ASWEC 2005, held in Brisbane, Queensland.

I have strong feelings about the structure of undergraduate and postgraduate courses at university. I had discussions with several people at the conference, including world expert Prof. Parnas, one of the keynote speakers, and several others about problems with course content, mainly the tension between science, IT, CS, fundamentals, engineering accreditation, resources and so on. An enduring feature of these discussions is what languages to teach.

I think UWA has traditionally had a good balance however the recent lack of strong C/C++ teaching has been a disadvantage while Haskell, and functional languages, are closer to much commercial reality when code generation and style-sheet processing (like grammars, parsing and rules application eg. make, ant, xslt, yacc/lex) are involved. Lisp, prolog and their ilk are yet another kettle-of-umm-fish.

My personal feeling is that Smalltalk and Eiffel have limited applicability in commercial applications. My main reasons are that they lack flexibility and sufficient object-oriented constructs, are overly rigid and, as a result, never achieved commercial acceptance. I will state, however, that Smalltalk concepts are endemic through the community and have found a place in other languages and common practice, C# delegates and C++ Boost function prototypes being modern implementations of Smalltalk message passing based on method signatures rather than, for C++ and others previously, where an explicit, common base class has been required for polymorphism and dynamic dispatch to take place.

Teaching of these languages, or similar, must be mandatory because otherwise there is no sense of history and development, for one thing, and they also provide a platform for describing improvements (not always the case though) through language evolution. As Isaac Newton said, "If I have been able to see further, it was only because I stood on the shoulders of giants."

My reasons for and against Smalltalk and Eiffel, neither of which I have used in earnest, follow. Sigh. Single inheritance means that natural object hierarchies cannot be built. Same objection against Java and C# managed components. Leads to code duplication and in some cases jumping-through-hoops to achieve satisfactory design. For example, p->impl-ing a class to implement a sort-of decorator which delegates to the another class. Certainly we prefer composition to inheritance however the mixin is incredibly useful and, importantly, leads to a very simple design, most important for understanding, and implementation, less important except directly impacts on maintenance costs over lifetime of component.

Similarly for const-correctness in C++, lacking in the others, where this additional contract improves code robustness. We use the const idiom in Java, the read-write interface extending the read-only interface (Aro <- Arw) in order to get similar, but without type safety. Eiffel programming by contract is a design principle and has some arguments to its merit, expressing invariants in code, however the mechanism is rigid and hence brittle. I argue the same for Ada.

Specifically, it is unnecessary, even wrong, for whole families of constraints to be captured in source code and caught at compile time. Static type safety is, I believe, fundamental to building robust software. However, pre and post conditions are run-time constraints so they are of less benefit than contraints imposed by the "interface contract" of a base class or interface. Early failure is as desirable as late binding and lazy initialisation.

An open question on requirements capture is occupying my thinking, and I should be generating a paper on equivalence between typographical, textual requirements, formal specification (eg. Z or object Z), use cases and invariants asserted in test cases. Trying to capture all of the same information is more than one layer of abstraction is of questionable value because it appears that instead of a higher-level language being used for analysis, requiring a tranformative step and further elaboration (eg. detailed design) it is assumed that direct translation can be used. Which is pointless.

Consider that if the specification language can be translated to an implementation then it is just another isomorphic implementation language. Oddly enough, that is how in the limit I tend to describe all of the modern programming languages - as syntactic variations of Algol. Maybe that's a bit extreme. Include Lisp: Treat modules and classes as metaconstructs. Maybe not. I treat comments as part of the interface contract. Test-driven development exercises these constraints as well.

So there you have it. Teach all of the programming languages. Keep them in context. Stand on the shoulders of giants.

Pragmatic Software Design

2006-12-14T22:47:00.001+09:00

Is there such a thing as minimal design? In my experience there is good design, bad design or no design at all. The easy throw-away line that minimal design and coding is the way to go just does not wash with any rational test. The problem is how to identify and eliminate poorly designed bits of code while recognising and keeping well-designed pieces of code.

How do we identify bad code? The easiest way is to follow a set of rules and match the code against those rules. Design antipatterns are a good place to start. Martin Fowler and friends in Refactoring speak of code smells and have a taxonomic breakdown of the smells that emanate from bad code and the steps to be taken to rectify the problem.

Can we recognise good code? If we start with design patterns, enterprise patterns, language-oriented idioms and set the goal of explicitly naming the patterns and idioms in our code then we have a metric of pattern density, or as I like to call it, the emergence of pattern poetry.

The emergence of structure from goal-seeking behaviour among autonomous agents is a characteristic of all distributed systems including software developers. Some might call it, with apologies to John Vlissides, pattern hatching.

What can be done to rectify poor code? The baby steps of refactoring are a good starting point for improving code structure in the small. The detail of implementation can be directly improved by making incremental changes to the code. In order to ensure the continuation of correctness in the behaviour of the code it is necessary to have unit tests that exercise each unit of code and confirms its invariants are satisfied as expected.

Test-Driven Development in concert with Mock Objects for unit testing help to ensure that the expected behaviour of each unit can be modelled, tested and implemented. One of the corollary conditions is that the contract for each unit is expressed as an interface or abstract base class so that production and mock implementations can be provided for units to test against.

Unit testing, glass box or white box testing is concerned with assuring the internal behaviour of each component is correct. Integration testing or black box testing is distinguished as concerned with validating the external behaviour of various integrated sets of components against the expectations of the customer. Acceptance tests and integration tests are synonyms for tests that are specified by the customer or are derived from the user requirements, otherwise known as the system definition.

It is perfectly clear to most developers that unit testing is sufficient to ensure that components will successfully integrate and interoperate together in order to fulfil the purpose for which they were designed. Usually, each component has been assigned a number of specific functional requirements, use cases or user stories during requirements analysis and negotiation, iteration planning or the planning game. The problem is this crystal clear conclusion is completely wrong.

A number of units of code cannot just be stitched together and expected to work as anticipated. Each component or unit has its own invariants, constraints and boundary conditions. Some of these match globally across all of the other components while other are distinct constraints. The picture is one of trying to stitch together a large area of fabric from different sized and shaped pieces that each have their own size, shape, material and thickness.

For the mathematically minded, the situation is analogous to a analytic function in complex space in the sense of satisfying the Cauchy-Riemann equations for the existence of higher-order derivatives and so demonstrating smooth continuity.

Most system partitionings into subsystems and components result in units of code that are analogous to distinct regions of complex space where each one of the functions may be locally analytic. However there are mathematical difficulties at the boundaries of each of these regions trying to stitch the functions together.

The problem cannot be remedied by small, incremental, baby steps to refactor the code. Fowler's large refactorings cannot be done in baby staps at all. Continuous integration that requires code to compile against all the most recent versions in the source repository of other components is not possible.

Such a backward-looking constraint causes the design of the code to remain in a local minimum in the phase space of design possibilities with little or no chance or breaking out into a quality design.

Evidence for progress is a continually breaking build because an advance or change is made to one or more component interfaces causing compilation failure, or change in one or more component implementations causing unit or integration test failures.

The need for unit and integration tests is paramount for the possibility of successfully integrating the diverse components. And for interface contracts to allow the unit and integration tests to be coded in the first place using mock objects.

The result is confidence in the code, the unit tests and validation of acceptance and integration tests. This is a pragmatic approach and there is nothing minimal about it.

Modern Object-Oriented Programming

2006-12-10T20:23:00.000+09:00

The most popular object-oriented programming (OOP) languages of C++, Java and C# are very similar to each other and indeed to their progenitors of Smalltalk, Simula, Algol and C. Similarly, the programming skills employed are similar regardless of the development language that has been chosen. Each of the modern languages shares structured programming constructs like if-then-else, for-next loops, or equivalents, and so on, with each other and their older cousins. With each other they share constructs that can be used to build classes, employ polymorphism and specify type safety within the bounds of the domain-specific language (DSL) that extends the base language for the problem domain in which we are working.

That there are so many commonalities between the three modern languages is unsurprising since C++ was the first programming language to cohesively tie together object-oriented concepts and Java copied, sometimes insensitively, from C++. Without doubt C# is a copy of Java with a few extensions along the lines of syntactic sugar (eg. properties and indexers, perhaps itself a workaround for the absence of operator overloading). Even more telling is the fact that Java and C# betray their own origins by adopting the very features that they initially dropped from C++ and had previously been decried by the language designers as bad features. The features that were dropped from Java when otherwise copying C++ syntax include multiple inheritance, operator overloading and templates, otherwise known as generics. C# included operator overloading in its first incarnation as a throw-back to its supposed C++ lineage but copies the Java path, perhaps set by Smalltalk, in disallowing multiple inheritance and generics. These and other features have crept their way into Java and C# however it is arguable whether programmers are more productive, not withstanding the marketting of the respective sponsor organisations for each language.

The exploration of the details behind the key constructs of C++, the equivalent code in Java and C#, the extensions and omissions in those languages, and the revised C++ standard, i.e. C++0x and Technical Report 1 (TR1), following over a decade of stability, are beyond the scope of the present discussion. The points of note are that the languages are very similar and appear, at first glance, then second and third blush, to borrow insensitively from C++, clearly their motivation. Java differs quite markedly in some syntax while maintaining the same class flavour while the more-recent C# seems, to invoke evolutionary atavism, to be a throwback to C++ with a few enriching extensions. All three are, for all intents and purposes, equivalent for purpose of business programming, among others, and being general-purpose programming languages any one can in general implement with relative ease the same solution as in any of the other two.

The building blocks of complex software are groups of components that number several classes grouped together into a library, jar file or assembly. The opposing forces of decoupling and cohesion determine which classes and components to group together into a single unit of release. Robert Martin's Principles of Object-Oriented Design speaks of the Common Closure Principle (CCP) and Common Reuse Principle (CRP) for grouping classes together; the Dependency Inversion Principle (DIP) and Interface Segregation Principle (ISP) for classes to depend on abstractions and client-specific, or narrow, rather than fat interfaces.

Beyond design patterns at a higher level of abstraction one may speak of analysis patterns and enterprise patterns or more generally to speak of application and framework architecture. The elements of architecture include frameworks of reusable components that are idiomatic for each programming language so that C++, Java and C# libraries may be unique in design to take advantage of each language benefits. Examples include OpenGL and Open Scene Graph (OSG) in C++; Spring for Inversion of Control (IoC) or dependency injection in Java; COM+ and Enterprise Services in C#, the natural language of .NET. Other frameworks may be similar or identical for all of these, including Corba-IIOP, SOAP/SOA, ESB and Brokers at a high level; Hibernate persistence framework, xUnit test (i.e. CppUnit, JUnit and NUnit) and logging frameworks (i.e. log4cpp, log4j and log4net) at a lower level.

Similar to analysis patterns that transform into a myriad of design patterns, each of which can be implemented in any language and reflect the customary idiom, Corba-IIOP and DCE-RPC turn up as Corba in C++, RMI-IIOP in Java and .NET Remoting in C#.NET; message queuing WebSphere MQ, Java Message Queue (JMQ) and MSMQ are platform and language independent to a greater and lesser extent. By their nature, integration technologies need to span various technologies and platforms in order to serve their purpose. Adaptive Communications Environment (ACE), TAO Corba ORB (Object Request Broker) in C++; perhaps Internet Communications Engine (ICE) with bindings to these and other platforms is the most cohesive of all. The same can be said for transaction models that seem much alike from IBM CICS, BEA Tuxedo, Java EE, Microsoft MTS/COM+ and .NET Enterprise components.

The modern concept of an application server embodies many of these concepts in various combinations.

Peer Review and Problem Solving

2006-11-12T12:34:00.000+08:00

Peer review, open communication and collaboration are probably the hardest nuts to crack and the foundation of successful team. Some years ago, a colleague of mine suggested that the increased communication encouraged by OO design and interfaces is more important than the OO itself as a determinant of success.

Curiously, she has a background in communications and media before completing a postgrad computing degree. Some of the most successful people I know in this field have three year degrees or a background outside of computing. Not to mention the fathers of the field like Dijkstra, Barry Boehm and David Parnas, whom I had the pleasure of meeting at ASWEC 2005 in Brisbane two years ago.

My current reading extends to Karl Popper (falsability and scientific progress), Kuhn-Popper "debate" on established order in science versus recurrent critique, where the philosophy is stretching my own thinking beyond its usual bounds (Bertrand Russell, Piaget in the stack of books).

I encourage peers to read and try to understand books in problem solving and Hofstadter's GEB, for instance, in order to extend themselves intellectually and to learn new approaches to problem solving (eg. induction, depth first, work backwards) to add to their own personal toolkit of problem solving techniques.

Gerald Weinberg introduced the concept of egoless programming in The Psychology of Computer Programming, originally published in 1971 and re-released in 1998. Weinberg recognized that people tie much of their perceived self-worth to their work.

Lessons and Licensing

2006-11-12T10:57:00.000+08:00

I had the great pleasure of meeting and having several extended conversations with the renowned Prof. David Parnas who I met at the Australian Software Engineering Conference where he was guest speaker in Brisbane, 2005.

The chats I had with Prof. Parnas reinforced the unchanging nature and lessons of software development. He talks about loose coupling between modules, software engineering as a profession and attention to fundamentals. We both have strong, personal interests in the quality of undergraduate education, postgraduate and continuing professional education and licensing.

In Australia, there is ongoing discussion between the Australian Computer Society and the ITEE College of Engineers Australia about licensing arrangements. The likely outcome is hopefully a joint board that offers national accreditation, similar to those that operate in Canada and Ireland where Prof. Parnas is himself involved in these activities.

He must have influenced more than I had realised because I have been hammering on these points ever since!

Note that while I am a competent engineer my professional standing does not compare to David Parnas who is widely regarded as a pioneer in software engineering.

What is Software Engineering?

2006-11-12T10:49:00.000+08:00

Half the battle in any discussion is nailing down what is the same and what is different. It is easy to disagree about pretty much everything while having lots of common ground, or conversely to agree in words while having significant underlying differences.

The Guide to the Software Engineering Body of Knowledge serves as a foundation for the accreditation of university curricula, the licensing and certification of practicing software engineers, and raising software engineering toward a professional status.

Lessons learned and sometimes forgotten include Coplien's Commonality Variability Analysis (*). David Parnas warns about forgetting the past and being doomed to repeat the same mistakes. Tom Demarco, Barry Boehm, Fred Brooks, Edward Yourdon, in addition to Grady Booch, Martin Fowler, Robert Martin and others, have much to teach us through their writing.

It is important that software engineers recognise the context in which they work:

Management. In particular, risk management, mitigation and minimisation.
Metrics. Formal and informal. How to measure success at each step of the way.
Planning, estimation and reporting. Keeping stakeholders informed, onboard and happy.

(*) My good friend and colleague Gian introduced me to Commonality-Variablity and taught me a lot about interface and class design, among other lessons in life.

Refactoring

2006-11-11T22:10:00.000+08:00

Some people may disagree however I believe that if you read Is Design Dead? and Refactoring with Martin Fowler the fair reader should get the idea that Martin Fowler supports some upfront design.

The absence of some design thought up front and lack of visible design effort before refactoring is that the programming effort may be just adhoc coding and rework rather than directed refactoring. The mitigating practice in XP is pair programming. But what about those not doing XP?

Any in-depth discussion about interface contracts, abstraction and encapsulation, Liskov and open-closed principles, state and invariants with regard to elements of design can be more than a little surreal. These are fundamental object-oriented design principles that some post-modern developers choose to ignore based on their understanding of Agile development.

Martin Fowler and Kent Beck allow for top-down, architectural refactorings in addition to smaller, finer-grained bottom-up refactorings. Most of the smaller refactorings can go either way while the bigger refactorings have bigger cost-benefit tradeoffs.

Baby steps do not always cut it.

Refactoring, Chapter 12: Big Refactorings (from Amazon summary):

Kent Beck co-wrote this chapter with Mr. Fowler. They discuss what they call the 4 Big Refactorings: Tease Apart Inheritance, Convert Procedural Design to Objects, Separate Domain from Presentation, and Extract Hierarchy. These refactorings are of a more all-encompassing type than the smaller individual refactorings from the preceding chapters. The co-authors do a great job at putting in a nutshell what would normally take very long explanations.

The problem with taking only baby steps during refactoring is getting stuck in a local minimum within the phase space of possible designs and having no way of getting out. The question in reality is to ask when the Big Refactorings turn into simply a code rewrite.

Simple Agile

2006-11-11T21:59:00.000+08:00

Continuing the previous exchange in the form of a Socratic dialogue

Why are you talking about Karate and JKD?

The ability to draw analogies sets sentient beings apart. I was using Karate and JKD (if it exists) as similar creative and technical analogies for software engineering. Life in general. Primary school; secondary school; tertiary education; post-education independence is another which fits this context. Fundamentals followed by application, plus understanding, then extension.

I conclude that you now completely agree with my points, which is very fine since I believe I have spoken absolutely sensibly. Please confirm this.

You indeed have spoken sensibly. I neither expect to agree with all of your points, nor you with mine. Nor could it be so if we are thinking and developing independently without following a common dogma.

We agree with Fowler and Kent's opinions as I have quoted them. We both agree that a simple agile approach to software engineering is the most likely approach.

Not clear to the first assertion. We certainly disagree on interpretation of Fowler's words, among others. The "simple agile approach" needs a little more elaboration and definition if you want to pick-and-choose parts rather than specify to which agile approach (eg. scrum, xp) you are referring.

We both agree that a very disciplined approach is necessary, and that this discipline need only be applied to a very few rules.

Again, we disagree on intent and degree. How much discipline is too much or too little?

Pay attention to the current circumstances, decide your next small step using the information to hand, always trying to improve things.

Here we agree. I think. Perhaps we have different definitions of small step. For me, three days is a small step compared to an iteration or closure over three or four weeks.

We agree that if something appears difficult, hard, convoluted, and complex it is not well understood. We agree that if something is simple it is understandable and communicable (even when encapsulating a great deal of complexity).

Here we are in agreement. No qualification.

Design-Code Strange Loop

2006-11-11T21:47:00.000+08:00

Partly inspired by Hofstadter's GEB, this exchange loosely follows the form of a Socratic dialogue, probably best known as the form of Plato's Republic.

Hmmm I have some major concerns with these statements. First of all, there is no separation between design and implementation (coding). Each step of coding is detailed design. Each part of design is influenced by the code.

Ron Jeffries and I both agree with you that the way should be as you say it is.

XP does support up-front design--it's just that it recommends that design always proceed in the presence of code. The more effort we spend on design without feedback, the more likely that our design is incorrect. Design is done during the planning game, in addition to being done constantly throughout development. XP by no means prohibits us from sketching out UML diagrams prior to beginning an iteration, a task, or a test case. But we learn as we repeat, and our need to physically sketc diminishes as our understanding of good OO design and the system increases.

Abstractions and ideas. A body of actual code contains and includes its working existant design. They are one and cannot be separated.

I disagree with you about how well the code expresses its intent, hence the value and need to focus on OO and UML.

Quoting Ron Jeffries again:

[1] The design is treated similarly. The main place for the design to be is in the code. An Agile project is producing a number of "living documents", the most important of which is the program itself, which is produced in a growing, fully integrated form, from the beginning of the project right on until the end. [2] Working together, the team communicates design essentials through conversation supported by pictures and documents as needed.

My insertions of the numbers into Jeffries' words: [1] is essentially a restatement of what you have said; [2] is a restatement of what I have said.

I add that developer-to-developer communication and collaboration is essential for success but that it is inadequate for the customer. I also emphasise that the "supported by pictures and documents as needed" is contentious and applies equally if not more so to design as to the code.

As soon as the first line of code is written you have software. It has a design, it has functionality.

Indeed.

Any statement which lowers the supposed worth of code to "least-important" is fatally flawed in my view.

As you noted above, the code is inseparable from the design. I suggest that the design is actually a specification for the code, as is the test suite, as are the requirements; demonstrably we can implement the same design in more than one way.

The worth in monetary terms is in the contract. Okay, that is a bit cheeky! But the customer does not care about the code whatever we do. The customer cares about what it does for them.

You appear to be saying the scaffolds on the building site are more important than the building itself. Surely the entire point is to deliver a working system (working code) to a customer.

The designs and specifications are more important than the building itself is a statement I would unhesitatingly make. Any qualification would be for each stakeholder; the builder thinks the permits are more important, the subcontractors want to finish by 3pm, the kids want a backyard, Mom wants a kitchen and Dad a shed, okay, a study.

Be Like Water

2006-11-11T21:36:00.000+08:00

Sensei: The Bruce Lee movie I allude to is Game of Death, where Bruce's character finally defeats Kareem Abdul-Jabbar because he realises there are no set forms, no correct response.

"Don't get set into one form, adapt it and build your own, and let it grow, be like water. When water gets into a cup, it becomes the cup, when it goes into a teapot, it becomes the teapot. It can flow, or it can crash. Be water my friend. Adapt!" Bruce Lee.

Student: Similar to OOD, what is the purpose of XP? Why is it used and followed? Because it is believed by some to aid in reaching the final goal of workingsoftware. Again, the practices of XP are not the point in and of themselves.

Sensei: At one extreme, any formalism can provide a crutch and a barrier to progress. Just as Lee regarded the practice of traditional style like Wing Chun as dogmatic. For most practitioners, and indeed for Jeet Kune Do (JKD) as taught by Lee to his latter-day students, it was vital to master the fundamentals.

Student: The purpose of OOD is simply to:

Manage complexity.
Divide the problem.
Hide details.
Model the real world.

Sensei: These concepts predate OO-practice in software development by decades and are vprobably fundamental to most fields, like mathematical and econometric modelling, financial and marketting planning and execution.

Student: If we do not follow XP exactly then my question would be: so what? As with coding and design, a decision to follow XP more closely would be made if it was thought it could actually *help*.

Sensei: Like in Karate where the three Ks, Kihon (fundamentals), Kumite (free-fighting) and Kata (forms) are all necessary, perhaps OO principles, XP and formal processes, like Kihon and Kata, are key to Kumite, the free-flowing practise you espouse.

Bad Tests are a Joke

2006-11-11T21:25:00.000+08:00

Bad tests resemble the Microsoft joke about their help desk giving perfectly correct but useless information. Joke reproduced below to help our readers find the humour in this otherwise serious subject.

The Turing Theorem has something to say about this, too. Can anyone tell me what?

My Helicopter Is Lost

A helicopter was flying around above Seattle yesterday when an electrical malfunction disabled all of the aircraft's electronic navigation and communications equipment. Due to the clouds and haze, the pilot could not determine the helicopter's position and course to steer to the airport.

The pilot saw a tall building, flew toward it, circled, drew a handwritten sign, and held it in the helicopter's window. The pilot's sign said "WHERE AM I?" in large letters. People in the tall building quickly responded to the aircraft, drew a large sign, and held it in a building window.

Their sign said "YOU ARE IN A HELICOPTER OVER SEATTLE." The pilot smiled, waved, looked at his map, determined the course to steer to SEATAC airport, and landed safely.

After they were on the ground, the co-pilot asked the pilot how the "YOU ARE IN A HELICOPTER" sign helped determine their position.

The pilot responded "I knew that had to be the MICROSOFT building because, similar to their help-lines, they gave me a technically correct but completely useless answer."

Happy Path Tests

2006-11-11T18:16:00.000+08:00

Someone posted an article to the relevant company internal news group at which I was consulting, referring to JUnit Antipatterns and happy path testing. Another colleague replied and started off by saying, "You'll find that he only uses the term 'bad test' in the conclusion."

He continues by saying the author states that "Writing good tests is the hard part," implying that many of the easy tests may be bad tests and goes on to show illustrative examples. He could define 'bad test' as 'not(good test)' I suppose.

After referring to the section Easy Tests my colleague quotes the article, "The result is lots of passing tests that don't exercise the system, which leads to a misrepresentation of the code's health," and this is a bad outcome which need not be spelled out.

He goes on to say that he can only assume that the author was referring to tests that don't work. e.g. 1000 assert(true)'s. "That is, it's hard to say whether he's considering a 'happy path' test to be a *bad* test. You can't be serious. You did read the examples and make such a trivial assertion(asserting(1000 assert(true)'s))) as satire of some sort, right?"

Well no, actually.

The author is quite explicit and says, "Actually, happy path tests aren't an antipattern. The antipattern is when the developer stops at happy path tests."

After a couple of examples showing why "happy path tests" are necessary but are not themselves sufficient for adequate testing, the author states, "The point is, it isn't difficult to get a test to pass coincidentally," and this is a bad outcome as well.

I disagree with the assertion that "One single test of one component that tests the 'Happy Path' gives a massive, massive gain in confidence in the system. Even if it only tests 5% of the system in one ideal case it actually *proves* something about the behaviour of the system, where before you knew absolutely nothing."

Such tests prove nothing and in fact are worse than having no tests because such a view engenders a false sense of security and confidence in the system that is misplaced. One counter-example, quoted directly from the article:

One happy path test verifies that Factorial.eval(3) returns 6.


public class Factorial {
  public int eval(int _num) {
      return 6;
  }
}

How's that for a false positive? If you've never encountered test-driven development, you might object that no one would write such a simple-minded implementation. One of the practices of test-driven development (TDD) is to write the tests first, then "do the simplest thing that could possibly work" -- in this case, return 6.

In answer to my other colleague I choose to borrow from the legal profession, "If the facts are on your side, bang on the facts. If the law is on your side, bang on the law. If neither the facts nor the law is on your side, bang on the table."

Good tests demonstrate a requirement (user or discovered, user-actor or controller-actor) of the system, usually asserting on an invariant of the class. I disagree that tests are especially useful for construction and other static tests, and these are taken care of by the compiler and linker anyway, by default giving us the underhyped benefit of static-type safety.

The place for static setup is in, you guessed it, setUp(), so the test fixture is not polluted by such detail.

So what do we test in the fixture itself? Dynamic behaviour. Specifically, scenarios of unit tests, or bits of user stories, to test that the software works as expected. From the junit cookstour, "We found that the most challenging application of JUnit was testing its own behavior."

I can certainly believe it.