Data integrity is at the core of a successful data warehouse
Data integrity involves putting the necessary rules and data structures in place to ensure that the company’s data is consolidated, integrated and accurate. It’s a guarantee that the data is consistent, and gives the company a single version of ‘the truth’.
If your data is not correct then the information you base your business decisions on may be false, and that can have a severe negative impact on the profitability of the organisation. Data integrity means complete and accurate data that makes it possible to formulate predictions and make well-informed decisions.
The reality for most organisations, however, is that their data resides in a number of unrelated systems and databases. For example, one system may handle enterprise resource management, another financial data and yet another human resources systems.
Data warehousing unifies data lodged in diverse formats in different systems so as to enable the business to develop reliable, integrated reporting and analysis – a single version of ‘the truth’.
Proper business and technical analysis processes leading up to the modelling of the data are critical to ensuring its accuracy and its successful application in the business. This is the most important element in determining whether the data warehouse will be successful. You need to ask the right people – both executives and end users – what they want out of the data warehouse. Frequently, different people have different views of the data and different perceptions of which data is correct, and these conflicts must be resolved.
Data warehousing should be able to overcome these issues, not by redesigning the data input systems, but by consolidating the data so as to make it appear consistent in order to support relevant reporting and analysis.
It’s not essential to wait until all data is accurate before embarking on implementing a data warehouse, but it is essential to include data clean-up as part of the overall programme.
Cleaning data must be a joint effort between business management and the IT department. Both need to ensure that standards are enforced, particularly in regard to data capture, access and modification. Only in this way will data achieve its value as an asset that can be measured in rands-and-cents terms.
A key aspect to bear in mind is that it’s important to focus at the outset on what information you want to get out of the system. This is the only way to ensure that what you put into the data warehouse is appropriate. You can obtain a single view of the customer, for example, only by defining the key performance indicators you require – is it purchases, region, credit days outstanding?
You have to define and design the system to generate the information you require, particularly if you are working towards implementing a business intelligence system at a later stage.
The implementation of a data warehouse allows the organisation to apply universal business rules on a single source of data, rather than trying to apply the same standards across each disparate data source.
The biggest risk is that the goalposts seem to shift all the time, particularly as the business’ focus changes. The solution must be flexible and robust enough so it’s possible to make changes as you go along. If you follow sound design principles in modelling and building the database, this should not be difficult. You have to be patient and keep refining requirements. It’s an ongoing process, and there’s no golden rule. But you do need to make sure at the outset that your project planning time schedule is realistic.
To ensure that the database is an accurate reflection of the universe it is modelling, it must have entity, referential and domain integrity. This is done through the proper use of primary and foreign keys and automated auditing and balancing of data. This enables the consolidation of all data sources to ensure that data is complete, correct and stable, as well as fast and reusable for different purposes.
Data integrity is the bedrock of the data warehouse and the foundation of business intelligence, leading to better and more accurate decision-making.