Data Quality Solution for the Non-Household Water Retail Market in England

This video demonstrates the Entalysis data quality solution for the non-household water retail market in England.

In April 2017, the England non-household water retail market opened for competition, enabling customers to choose their retailer. The regional water companies separated their non-household retail and wholesale operations, and new retailers entered the market. Retailers and wholesalers became independent and no longer seamlessly shared property, services and occupier data.

However, this data is required for market operation and settlement. Market participants must be able to define and communicate their respective properties, services or occupiers to ensure the correct financial settlement between retailers and wholesalers. This requires a data model, a central IT system and a set of transactions to enable updates and notifications.

The data model is complex and comprises hundreds of data items that define each aspect of a property, service or occupier and their relationships. There are 1.2 million non-household customers in the market and their water and wastewater service configurations are complicated and often unique. With endless configurations across a large customer base and significant legacy data quality issues, this is a large task for every wholesaler and retailer.

However, even when a problem is found, deeper investigation is required before cleanse can begin to cross-check related data items and determine the exact reason for the failure. Furthermore, properties with trade effluent or meter-related rule failures often need mapping out on paper to understand all the relationships. All this makes the cleanse process involved and time-consuming.

The Entalysis data quality solution does all this automatically and also generates the transactions to bulk cleanse CMOS via the medium volume interface or your internal IT system via a SQL script. In summary, the Entalysis solution:

• Applies a complete data quality ruleset to your market data, testing for completeness, validity and accuracy. Analyses CMOS MDS alone or CMOS MDS and internal IT system mirror MDS together if available (full or partial view).
• Records the rule failures against the relevant SPID, DPID, meter or calculated discharge for analysis both vertically by data item and horizontally along the entire SPID, DPID, meter, etc.
• Provides a one-page view of all a SPID's data items and their data quality status in both CMOS and your internal IT system, including a full description of the reason for the rule's failure so cleanse agents understand why it failed and what they need to do to cleanse it.
• Displays a visual representation of the relationships between a SPID pair's SPIDs, DPIDs, meters and calculated discharges to understand the SPID's structure.
• Analyses rule failures by type, transaction and data owner (wholesaler, other wholesaler, retailer), and automatically generates the manual cleanse packs and the auto-cleanse corrective transaction scripts to cleanse CMOS and/or the internal IT system in bulk.
• Records the data quality rule failure occurrence and fix date by object (SPID, DPID, meter, etc.), so wholesaler, other wholesaler and retailer data quality errors and their durations can be analysed.

Ultimately, this solution automatically generates the investigation reports, cleanse packs and corrective transactions that the cleanse agents and data scientists would otherwise have to perform manually. The end-to-end cleanse process becomes simpler, clearer and faster.

Glossary of terms used in the video:

• MOSL - Market Operator Services Limited, the market operator of England's non-household water retail market.
• CMOS - Central Market Operating System, MOSL's core IT system that handles market transactions with wholesalers and retailers.
• SPID - Supply Point (Identifier), the representation of an eligible premise within the market consuming a water or wastewater service; a premise consuming both water and wastewater will have a separately tradeable SPID for each. This ranges from an individual office or shop within a building to a collection of neighbouring (contiguous) buildings like a university campus.
• DPID - Discharge Point (Identifier), the representation of a sewerage connection with consent to discharge trade effluent (part of a sewerage SPID).
• Calculated Discharge - an unmeasured (estimated) trade effluent discharge (part of a DPID).
• Market Data Set (MDS) - the full set of data items for all SPIDs, meters, meter reads, DPIDs, etc. in the market, filtered for each market participant.
• Medium volume interface (MVI) - a CMOS transaction interface allowing the upload of text files containing multiple transactions.
• Auto-cleanse – for the given data item and associated transaction, the master data in the source system can be safely used to cleanse the destination (following) system.
• SQL - Structured Query Language, program code to manipulate and read data in a database.

Next, please watch the video of the Entalysis Leakage Solution for the water industry.