Supplier master data comes first in many organizations. If the same supplier exists more than once, a duplicate invoice check that only searches within one supplier number will not detect a second payment. At the same time, purchasing volumes are split across several records. This weakens the basis for framework agreements and supplier evaluations.
Customer master data has a direct impact on revenue and customer experience. A customer kept in two records receives duplicate mail, contradictory offers or an incorrect credit limit. A unique customer view is a prerequisite for sales analytics and AI-supported forecasts.
Material master data causes hidden costs. If the same part is stored under several numbers, it is reordered even though stock is available. In maintenance, this can in the worst case lead to downtime because the required part cannot be found in the system.
Legal entity and shareholding data is a special case. Here, distinguishing between a true duplicate and a legally independent entity is particularly critical. An incorrectly merged entity distorts group structures, consolidation and reporting obligations, for example to the transparency register.
This leads to a simple rule: clean up duplicates first where money, compliance or security are affected. You should prevent duplicates in all domains, because prevention costs less than any later cleanup.
The 4 root causes of duplicates in master data
Duplicates rarely result from the carelessness of individuals. They are the result of structural gaps. If you know the causes, you can target your duplicate check where it has the greatest impact.
Root cause 1: Many entry points, no central authority
In a typical enterprise landscape, ERP, CRM, e-procurement, ticketing systems and specialist applications create master data independently of each other. Each system checks, if at all, only against its own dataset. A supplier created in parallel by purchasing and accounting then exists twice. Nobody notices this across systems.
Root cause 2: No shared definition of “identity”
Is the subsidiary in Poland the same customer as the parent company in Germany? Is a material with a new revision a duplicate or a separate entity? As long as business units answer these questions differently, no software can make reliable decisions. The result is either overlooked duplicates or incorrectly merged records.
Root cause 3: Data entry without checks at the source
Many entry screens only check mandatory fields. A search for similar, existing records either does not take place or only works on exact matches. Anyone who types “Mueller” will not find “Müller” and creates a new record. Under time pressure, creating a new record is simply the faster option.
Root cause 4: Migrations, acquisitions and bulk imports
With every ERP migration, for example to SAP S/4HANA, every acquisition and every import of trade fair or marketing data, two data worlds collide. Without upfront duplicate matching, duplicates flow into the target system by the thousands. There, they are linked to transactions and much harder to clean up.
Not a search problem, but a question of identity
Most organizations treat duplicates as a technical search problem: find the right tool, start a cleansing run, done. This view falls short.
The algorithm can only calculate how similar two records are. Whether “similar” also means “identical” is a business decision. A similarity score of 92% between two companies with the same name can be a true duplicate or two legally separate entities in different countries. An algorithm that merges automatically here causes more damage than the duplicate itself.
The trade magazine it-daily aptly describes the division of labor: AI finds and compares records, humans decide. After all, orders, sales orders and postings are linked to master data. What happens to these dependent objects must be defined before a single record is merged.
Successful duplicate checks therefore do not start with a tool, but with three decisions:
Identity rules: Which attributes make an entity unique in each data domain?
Decision rights: Who confirms possible duplicates, and who decides in the event of a conflict?
Survivorship rules: Which value wins when merging, and what happens to the references?
These decisions are part of your data governance. Our article on the data governance framework shows how to establish the roles and responsibilities for this.
6 steps to a sustainable duplicate check
The following framework has proven itself in enterprise environments. It combines one-time duplicate cleansing with permanent prevention.
Step 1: Define duplicates for each data domain
Define together with the business units when two records describe the same entity. The criteria differ by domain:
Business partners (customers, suppliers): name, address, VAT ID, commercial register number, LEI, bank details
Materials and products: manufacturer part number, GTIN/EAN, standardized short description, characteristic sets
Legal entities: company name, legal form, register court, register number, registered office
People: name, date of birth, personnel number, email
Also define which attributes are unique identifiers (such as the VAT ID) and which only provide indications (such as the name).
Step 2: Profile and standardize data
Matching only works on comparable data. First analyze the dataset: fill rates, formats, spelling variants. Then standardize before comparing:
Harmonize legal forms (“Gesellschaft mit beschränkter Haftung” → “GmbH”)
Normalize addresses (“St.” → “Street”, validate postal codes)
Align umlauts, upper and lower case, special characters and spaces
Bring phone numbers and IDs into a uniform format
This step alone significantly reduces the number of overlooked fuzzy duplicates. It is also the foundation for any later real-time check.
Step 3: Combine matching strategies
No single method finds all duplicates. A combination has proven effective:
Deterministic matching compares unique keys exactly, such as the VAT ID or GTIN. It is fast and precise.
Fuzzy matching calculates the similarity of strings, for example using the Levenshtein distance. It counts how many insertions, deletions or substitutions are needed to turn one term into the other.
Phonetic methods detect names that sound alike. For German-language data, the Cologne phonetics (Kölner Phonetik) is more relevant than the English Soundex, because it is tailored to German pronunciation and spelling rules.
Weighted scoring combines several fields into an overall score. A matching VAT ID carries more weight than a similar name.
For large data volumes, blocking is also useful. Here, records are only compared within meaningful groups, for example those with the same postal code. This keeps duplicate matching performant.
Step 4: Define thresholds and manual review
Translate the Fellegi-Sunter principle into clear rules. An upper threshold defines certain duplicates, which are automatically proposed for merging. A lower threshold defines certain non-duplicates. Everything in between goes to a review queue for the responsible data stewards.
Start conservatively with a high upper threshold. Evaluate the manual decisions regularly and adjust. This increases the level of automation without increasing incorrect merges.
Step 5: Implement match & merge with survivorship rules
Merging turns several duplicates into one consolidated record. Survivorship rules define for each attribute which value is adopted:
Source priority: The leading system wins, for example the ERP for bank data.
Recency: The most recently changed value wins, for example for contact data.
Completeness: A filled value wins over an empty field.
Validation: The verified value wins, for example a confirmed register number.
Just as important: all references to the resolved duplicates must be re-linked to the surviving record. This affects open orders, contracts, invoices and hierarchies. A merge without reference mapping creates orphaned documents. Document every merge in an audit-proof way so that it remains traceable and reversible if needed.
The result is the Golden Record: the one trustworthy record per business entity. Our article Golden Record Master Data explains what makes a Golden Record and how it becomes the foundation for AI.
Step 6: Prevent duplicates at the source and measure them
The most effective lever is a duplicate check at the point of data entry. Before a new record is saved, the system uses fuzzy search to look for similar entries and displays matches. The user then decides: use the existing record or create a new one with justification.
In addition, every process should include:
Central creation processes with approval for critical domains. Our article on workflow management for master data shows how to design such processes.
Regular batch checks for data coming in via interfaces or imports.
Monitoring of the duplicate rate per domain as a fixed KPI in the data quality dashboard.