A key practice of Agile is Test Driven Development. Agile Best Practice is that you:

1. Write a test
2. Run the test
3. Write code to pass the test
   a. If it passes go to 4
   b. If it fails go to 3
4. Run all tests to ensure the test doesn’t break any prior test.
5. Embed the logic in code.
6. Run all tests to ensure the test doesn’t break any prior test.
7. If that works then check in the code.

Now I admit, that is a great idea when you are writing code. But with data it’s another thing altogether.

One of the problems we face in developing Corporate Information Systems, is data. In fact data is so important, that other than a regular supply of coffee, I can’t think of anything else that CIS developers would rate as more critical to their livelihood and wellbeing.

Data however comes with its own rules and problems and, as it is the fundamental enabler of an Information System, developers need to ensure that it means and represents the same thing when it finally arrives at the BI system as it did in its source system.

If it doesn’t then your Information System does not work!

So we don’t write tests of code, we write scripts that execute queries against the data we get from source and make sure the results are consistent with those we get when it reaches the BI layer.

Using a single tool (RED) to integrate the models and data means that, if the results are consistent the system is consistent, the business rules applied are consistent and therefore the end to end data integration is successful.

In most cases the code we develop in the Information System is around data queries, integration and persistence. So if the data returns consistent results, the tools we use can be relied upon to repeat that for the end to end provisioning of the Information Product.

Indeed as most if not all code is produced by a tool (in our case WhereScape RED) it is pointless testing code. You are in fact testing the tools RED and the underlying RDBMS that execute the stored procedures. If a tool is chosen for the development there is no sense in continuously testing its functionality.
Data, rather than code, matters in Information Systems. Using TDD to prove data is much more effective and efficient than testing code.  Few data warehouse projects are lost because of SQL and Database functionality issues, most fail because of data problems.