MIP | Agile Data Warehousing

Agile and Architecture

Phil Considine — Tue, 30 Nov 2010 01:09:43 +0000

IT architecture spans generations of technologies that in and of themselves, may not be complex, but become so when combined. The role of Architect is one that tries to bring order to the Tower of Babel that pervades technology infrastructures. Like many other IT issues, this involves a high degree of regulation.

One of the failings of the Agile manifesto and principles is that the “as is” state of practices and technologies is ignored. Therefore Agile DW practitioners are confronted with the problems of developing in an innovative way and conforming to what seem to be archaic rules of engagement with the corporate technology and software infrastructures. Agile is easy in an environment where there are no competing rules or dependencies. When faced with the complexities of infrastructure and data integration, Agile practitioners need to adapt.

So what can we do when architects tell us that we MUST do things that impact our ability to deliver business value?
First the operating environment. There are no agile technology products. Agile is a philosophy driven methodology and is therefore independent of technologies. However, the problem is there are many anti-agile practices entrenched in the corporate IT environment. These range from “Corporate Data Models” down to SOE operating environments that preclude Agile “dependence free” practices.

The issue is that all complex systems have dependencies. IT technologies are usually robust in, and of themselves but in combination are quite fragile.
For example, just because a corporate data model exists, it doesn’t necessarily follow that it reflects the real world state of corporate data. In fact that is usually not the case. Nor is any corporate data model truly reflective of business needs. That is because the business changes faster than the model. So it’s rather fanciful to demand that the real world should match the model rather than the opposite case.

It is also rather silly to expect a free reign development philosophy to work effectively in a multigenerational IT environment. The idea that small groups of developers can dispense with specialisation and build effective and efficient systems is folly.
For the agile movement to succeed in larger organisations, it needs to develop practices that meet the realistic needs of the IT organisation. The cost effectiveness of developing systems iteratively is well proven, but agile needs to adapt better to the real world where risk is defined as upsetting the status quo.
To the corporate world, agile is little more than a new way of prototyping that results in continuing beta releases. This would be a real problem were it true, it is not. However the perception that it is must be addressed. The recent troubles with a well-planned but flawed upgrade in one of our Australian Banks shows that errors occur no matter how good the planning.
Small targets (largely the current focus of Agile) are tactical and therefore reinforce the view that it is not a legitimate competitor for waterfall.

For agile to succeed in the corporate world, it needs to adapt to the reality of the multigenerational IT environment.
While we have spent a lot of time and effort resolving many of the Agile problems of dealing with real world data management and integration, we still encounter these attitudes. So our agile methodology is, like agile itself, is continually evolving as lessons learned are incorporated. We therefore have a competitive advantage in that we can cogently argue the case for agile – it costs less, and delivers more and better information sooner.
When we all can argue that approach convincingly, Agile will come of age.

For MIP it represents a competitive advantage, but only over competitive waterfall approaches to the same situation. We can and do use the knowledge we have gained and we do win many more clients than we lose. We do come up against the Architecture/Risk/SOW/CMMI/ITIL etc. arguments against Agile. But over time we have accumulated the counter arguments. In some cases the entrenched interests will insist we conform to “standard methodologies”. We do! We still deliver faster, though not as fast as we could, we still cost less and we still deliver higher quality, more functional systems – we just try not to focus on the frustrating inefficiencies.

Agile cannot succeed in isolation, so we need more IT people to take up the challenge and THINK.

Improving Business Satisfaction

Phil Considine — Wed, 06 Oct 2010 05:56:24 +0000

What do business people dislike most about IT projects? Is it the cost? Is it the adaptability? Is it the time it takes to go through the process? The long wait for delivery? The fact that IT projects fail to deliver business value? Is it “IT / Business Alignment” ?
My experience is that it all has to do with the value IT delivers. Value is a vexed issue because it means different things to different people. As IT is now really a functional service provider, it measures its value in terms of uptime and throughput .
Business survives on innovation and that means it values anything that contributes to better delivery of its product. The term ‘better’ connotes any or all of faster, cheaper, more effective. In other words- Value. While IT can and largely has automated business processes, it has been less successful at supporting business innovation – especially with regard to the facilitation of business decision making. As all business decisions are based on the concept of obtaining greater value for effort, IT’s path to re-establishment is less thought about “alignment” and more about how to create business value.
It’s well documented that large scale EDW/BI projects have a poor record of meeting business needs. Only about one in five are acknowledged by business as being of ANY value.
In a recent discussion with a public sector client, we looked at how large projects are procured. Long lead times, large often overly complex requirements, long bid processes and strict contractual terms. This supposedly mitigates risk! It may, so far as a “level playing field” procurement approach but it kills off any thoughts on the part of bidders regarding innovation.
For example, if you can rapidly develop a working system to confirm business needs, you will find out in the first few weeks of a project what you REALLY need to deliver. All too often that is not what is specified in the contracts. In fact CHANGE is specifically excluded in contracts because of the risk to the provider. So in most cases suppliers bid for work that, while signed off and understood by the authors as representing what they want, by the time it is developed no longer represents a valuable solution.
That is why Agile approaches to EDW/BI development will prevail. As business SEEs the results of IT becoming innovative again, it will buy into the concept. Agile reconnects IT with Business in a partnership that delivers value sooner at lower levels of risk and cost.

Managing Agile Projects

Phil Considine — Thu, 09 Sep 2010 10:31:51 +0000

Agile and micromanagement are incompatible concepts. Agile teams function best when given responsibility and have freedom to self-manage to agreed outcomes.

Such approaches are anathematic to many managers who see any decision as their responsibility. When Agile teams are micromanaged in such a way the development is adversely impacted and the more likely result is delay, disruption and distraction.

The problem managers face is how much control do they give up?

Unless they are expert developers and monitor the code line count, quality and relevance of each developer in the team, the answer is none.

My contention here is that leaders make better agile managers than, well – managers.

The first principle of leadership is Know Thyself.

That is a powerful phrase because it carries the implications of knowing your strengths, weaknesses and limits. Most micromanagers see themselves as vulnerable to criticism by their peers and therefore they manage introspectively downwards rather than upwards and outwards.

This means they subtly or otherwise, get their charges to do their bidding regardless of learned, technical or other advice and rarely listen to contrary argument.

Invariably teams, when following strict directions, will stop innovating, stop committing and stop feeling responsible for their work.

This either means the micro manager will learn quickly and adapt or the project will fail. Usually the latter.

Leaders create an environment where their teams can succeed. They will manage above and below their teams to smooth out any impediments to delivering working software. Leadership engenders mutual respect and trust and these are much more likely to ensure the team works to deliver to commitments.

Teams prefer leaders who are firm, friendly and fair. Leaders who have Integrity, are great communicators are innovative, intelligent and decisive are likely to get the most from their teams.

Test Driven Development

Phil Considine — Fri, 13 Aug 2010 00:07:28 +0000

A key practice of Agile is Test Driven Development. Agile Best Practice is that you:

Write a test
Run the test
Write code to pass the test
a. If it passes go to 4
b. If it fails go to 3
Run all tests to ensure the test doesn’t break any prior test.
Embed the logic in code.
Run all tests to ensure the test doesn’t break any prior test.
If that works then check in the code.

Now I admit, that is a great idea when you are writing code. But with data it’s another thing altogether.

One of the problems we face in developing Corporate Information Systems, is data. In fact data is so important, that other than a regular supply of coffee, I can’t think of anything else that CIS developers would rate as more critical to their livelihood and wellbeing.

Data however comes with its own rules and problems and, as it is the fundamental enabler of an Information System, developers need to ensure that it means and represents the same thing when it finally arrives at the BI system as it did in its source system.

If it doesn’t then your Information System does not work!

So we don’t write tests of code, we write scripts that execute queries against the data we get from source and make sure the results are consistent with those we get when it reaches the BI layer.

Using a single tool (RED) to integrate the models and data means that, if the results are consistent the system is consistent, the business rules applied are consistent and therefore the end to end data integration is successful.

In most cases the code we develop in the Information System is around data queries, integration and persistence. So if the data returns consistent results, the tools we use can be relied upon to repeat that for the end to end provisioning of the Information Product.

Indeed as most if not all code is produced by a tool (in our case WhereScape RED) it is pointless testing code. You are in fact testing the tools RED and the underlying RDBMS that execute the stored procedures. If a tool is chosen for the development there is no sense in continuously testing its functionality.
Data, rather than code, matters in Information Systems. Using TDD to prove data is much more effective and efficient than testing code. Few data warehouse projects are lost because of SQL and Database functionality issues, most fail because of data problems.

Self Organising Teams

Phil Considine — Wed, 28 Jul 2010 03:11:36 +0000

It’s easy to see why IT traditionalists are not easily convinced about self-organising teams. After all the key agile principles run exactly counter to accepted practice, and Agile has implications for the career minded IT professional.
Self-selecting and self-organising teams are an anathema to accepted approaches to creating project teams. So you will face opposition.
The key to Agile teams is motivation. One of the most effective ways of motivating programmers and business is to add responsibility and a little authority to their job description.
The thing that strikes me most with our Agile approach is how many FEWER people we need to get the job done. In fact we often complain that the team is too large!!
Fewer resources and lower costs for better results.
That’s Agility IT, Motivation

Iterations

Phil Considine — Wed, 14 Jul 2010 03:13:49 +0000

We don’t use Sprints.
Sprints are difficult at the best of times for the Corporate Information Systems developer. First of all a fixed timeframe means that the deliverable is compromised. This is especially true of an end to end development – what we call a swim-lane – that includes the integration of data from Source to EDW to Star Schema to BI tool to Information Product.
Sprints work in AGILE OLTP systems because the deliverable is small and contained. Usually it is a set of data input and processing features built around capturing and processing a transaction in support of a particular business process. So a small increment in functionality is not going to be particularly challenging.
In the CIS the deliverable is Data rather than Feature oriented. That has a raft of implications. We, as developers, have to guarantee the semantic consistency of data from source through to the BI Information Product.
That influences our estimation of the Stories and the Tasks associated with their completion.
Our approach to this problem is to use variable length Iterations, each of which is determined by the team’s estimate of the time to deliver the increment of working software to satisfy a business need. We don’t constrain the team to a fixed two week sprint because they could not possibly meet the deadline in a steady and controlled manner.
So our Iterations typically range from 2 to 6 weeks. The initial iteration tends to be short because we want rapid feedback from the Product Owner. Usually it is measured in days. Once we get feedback we find the middle iterations require 4 to 6 weeks as the end to end integration issues bite. Once the end user is happy the latter iterations, UAT and Pre Production take one or two weeks. Typically that means we can build and deploy and end to end new requirement in as little as four and a maximum of ten weeks.
For us working software is the driver rather than fixed time boxing.

Agile Starts with a Team

Phil Considine — Thu, 01 Jul 2010 03:14:03 +0000

When we go on site we generally tell customers that we want to work in teams of five to nine people, including their business product owner, a developer or two, a data analyst perhaps and three to five of our people. Of course such a team can’t build the whole data warehouse, all the data marts and the BI analysis cubes as well as handle all the ELT in the system.

But four or five such teams can.

The trick here is to work out how to make this happen.

We know we can build a data mart pretty quickly, so we get the high priority business areas to nominate their high priority needs (stories). We get the team to envision the data mart, and then get the user to nominate the ones that will convince him to invest resources to build the system out. The team then works out the tasks involved the time needed to build them and nominates the Show and Tell meeting data and time.
Usually that is only a few days or weeks away. This is a great way to convince the business that they can have information in business timeframes.

For me that’s the core of the Agile approach.