November 20, 2026

Duplicate Check: How to Systematically Eliminate and Permanently Prevent Duplicate Records

Gernot Lepuschitz

By Gernot Lepuschitz

Chief Technology Officer

18 min read

Share this post

The same supplier appears three times in the ERP, the same customer twice in the CRM, and each system considers its own version correct. A duplicate check only solves this if it is more than a one-time cleansing run. This guide shows how to reliably detect duplicates in master data, merge them without losing data, and stop them from being created at the source. Includes a comparison of methods, a step-by-step framework and an FAQ.

Key takeaways

  • Duplicate checks cover all methods used to detect, evaluate, merge and prevent duplicate records for the same business entity.

  • Exact comparisons are not enough. Most duplicates are “fuzzy”, meaning they are spelled differently. You need fuzzy matching, phonetic methods and clear thresholds.

  • According to Salesforce, data and analytics leaders estimate that around a quarter of their organizational data is untrustworthy. Duplicates are a major driver of this.

  • The real work begins after detection: match & merge with survivorship rules and re-linking all dependent transactions.

  • Only a duplicate check at the point of data entry has a lasting effect. Any cleansing without prevention will be due again within a few months.

  • For AI projects, the topic is business-critical: Gartner predicts that organizations will abandon 60% of AI projects without AI-ready data through 2026.

  • Preventing duplicates permanently requires a business-aligned definition of what “the same entity” means. This is a governance question, not a tool question.

What is a duplicate check?

A duplicate check is a systematic process that identifies, evaluates and resolves duplicate or multiply created records in a dataset. A duplicate exists when two or more records describe the same real-world entity: the same customer, the same supplier, the same material or the same legal entity. The goal of a duplicate check is one unique, trustworthy record per business entity.

From a business perspective, there are two types of duplicates:

  • Exact duplicates are identical or nearly identical, for example after a double import. An exact field comparison finds them.

  • Fuzzy duplicates differ in spelling, format or completeness. “Müller Maschinenbau GmbH, Main St. 5” and “Mueller Maschinenbau, Main Street 5” are an example. They require fuzzy search (fuzzy matching).

A complete duplicate check comprises four phases. Duplicate detection finds candidate pairs. Duplicate matching evaluates how likely it is that they refer to the same entity. Duplicate cleansing merges confirmed duplicates. Prevention stops new duplicates from being created.

The methodological foundation is older than many people think. As early as 1969, Ivan Fellegi and Alan Sunter described the principle that still applies today in their theory of record linkage in the Journal of the American Statistical Association: record pairs are classified as a match, a possible match or a non-match. Possible matches go to manual review. Modern MDM platforms implement exactly this principle, complemented by machine learning and workflows.

It is important to distinguish this from data quality as a whole. Duplicates violate the quality dimension of uniqueness. You can read how to measure and manage all dimensions in our guide to improving data quality.

Why is a duplicate check relevant for organizations?

Duplicates seem inconspicuous. Taken together, they distort KPIs, cause operational errors and undermine trust in data. Five data points show the scale:

Metric

Value

Source

Share of data that data leaders consider untrustworthy

26%

Salesforce State of Data and Analytics, Nov. 2025

Organizations naming data quality as their top data integrity challenge

64%

Precisely/Drexel LeBow, 2025 Outlook

Organizations without (or unsure about) the right data management practices for AI

63%

Gartner press release, Feb. 2025

Data collections rated “acceptable” in a study with 75 executives

3%

Harvard Business Review, Sept. 2017

Estimated cost of bad data (industry benchmark)

15–25% of revenue

MIT Sloan Management Review, Redman 2017

Trust and decision-making

According to the State of Data and Analytics report by Salesforce, data and analytics leaders estimate that 26% of their organizational data is untrustworthy. The study surveyed 3,800 analytics and IT decision-makers in 18 countries. The same study shows that 84% believe their data strategy needs a complete overhaul for their AI goals to become achievable.

Data integrity as an ongoing issue

In the 2025 Outlook: Data Integrity Trends and Insights by Precisely and Drexel University, 64% of respondents name data quality as their biggest challenge, up from 50% in 2023. According to the same study, 67% of respondents do not fully trust the data used for decision-making.

AI readiness

In a press release on the lack of AI-ready data, Gartner predicts that through 2026, organizations will abandon 60% of AI projects that lack AI-ready data. The accompanying statement by analyst Roxane Edjlali is clear: if the data has issues, it is not ready for AI. A customer who appears three times in the dataset is exactly that kind of issue.

Compliance

The GDPR requires in Art. 5(1)(d) GDPR (principle of accuracy) that personal data be accurate and kept up to date. Anyone who keeps one person in three records can hardly fulfill a request for access under Art. 15 GDPR or an erasure request completely. For high-risk AI, Art. 10 of the EU AI Act requires training, validation and testing data that are, to the best extent possible, free of errors and complete.

Duplicates in master data: where they cause the most damage

Not every duplicate costs the same. A duplicate newsletter contact is annoying. A duplicate supplier with two bank accounts is a financial risk. When prioritizing, it is therefore worth looking at the individual master data domains.

Data domain

Typical duplicate example

Business impact

Suppliers

The same supplier under two vendor numbers

Risk of duplicate payments, distorted purchasing volumes, weaker negotiating position

Customers

The same customer in the CRM and the ERP with different spellings

Multiple outreach, incorrect credit limits, incomplete customer view

Materials

The same spare part under several material numbers

Excess inventory, tied-up capital, incorrect orders

Legal entities

The same shareholding in two group systems

Incorrect consolidation, risks in register filings and audits

Employees and identities

One person with two user accounts

Orphaned access rights, security and audit risks

Supplier master data comes first in many organizations. If the same supplier exists more than once, a duplicate invoice check that only searches within one supplier number will not detect a second payment. At the same time, purchasing volumes are split across several records. This weakens the basis for framework agreements and supplier evaluations.

Customer master data has a direct impact on revenue and customer experience. A customer kept in two records receives duplicate mail, contradictory offers or an incorrect credit limit. A unique customer view is a prerequisite for sales analytics and AI-supported forecasts.

Material master data causes hidden costs. If the same part is stored under several numbers, it is reordered even though stock is available. In maintenance, this can in the worst case lead to downtime because the required part cannot be found in the system.

Legal entity and shareholding data is a special case. Here, distinguishing between a true duplicate and a legally independent entity is particularly critical. An incorrectly merged entity distorts group structures, consolidation and reporting obligations, for example to the transparency register.

This leads to a simple rule: clean up duplicates first where money, compliance or security are affected. You should prevent duplicates in all domains, because prevention costs less than any later cleanup.

The 4 root causes of duplicates in master data

Duplicates rarely result from the carelessness of individuals. They are the result of structural gaps. If you know the causes, you can target your duplicate check where it has the greatest impact.

Root cause 1: Many entry points, no central authority

In a typical enterprise landscape, ERP, CRM, e-procurement, ticketing systems and specialist applications create master data independently of each other. Each system checks, if at all, only against its own dataset. A supplier created in parallel by purchasing and accounting then exists twice. Nobody notices this across systems.

Root cause 2: No shared definition of “identity”

Is the subsidiary in Poland the same customer as the parent company in Germany? Is a material with a new revision a duplicate or a separate entity? As long as business units answer these questions differently, no software can make reliable decisions. The result is either overlooked duplicates or incorrectly merged records.

Root cause 3: Data entry without checks at the source

Many entry screens only check mandatory fields. A search for similar, existing records either does not take place or only works on exact matches. Anyone who types “Mueller” will not find “Müller” and creates a new record. Under time pressure, creating a new record is simply the faster option.

Root cause 4: Migrations, acquisitions and bulk imports

With every ERP migration, for example to SAP S/4HANA, every acquisition and every import of trade fair or marketing data, two data worlds collide. Without upfront duplicate matching, duplicates flow into the target system by the thousands. There, they are linked to transactions and much harder to clean up.

Not a search problem, but a question of identity

Most organizations treat duplicates as a technical search problem: find the right tool, start a cleansing run, done. This view falls short.

The algorithm can only calculate how similar two records are. Whether “similar” also means “identical” is a business decision. A similarity score of 92% between two companies with the same name can be a true duplicate or two legally separate entities in different countries. An algorithm that merges automatically here causes more damage than the duplicate itself.

The trade magazine it-daily aptly describes the division of labor: AI finds and compares records, humans decide. After all, orders, sales orders and postings are linked to master data. What happens to these dependent objects must be defined before a single record is merged.

Successful duplicate checks therefore do not start with a tool, but with three decisions:

  • Identity rules: Which attributes make an entity unique in each data domain?

  • Decision rights: Who confirms possible duplicates, and who decides in the event of a conflict?

  • Survivorship rules: Which value wins when merging, and what happens to the references?

These decisions are part of your data governance. Our article on the data governance framework shows how to establish the roles and responsibilities for this.

6 steps to a sustainable duplicate check

The following framework has proven itself in enterprise environments. It combines one-time duplicate cleansing with permanent prevention.

Step 1: Define duplicates for each data domain

Define together with the business units when two records describe the same entity. The criteria differ by domain:

  • Business partners (customers, suppliers): name, address, VAT ID, commercial register number, LEI, bank details

  • Materials and products: manufacturer part number, GTIN/EAN, standardized short description, characteristic sets

  • Legal entities: company name, legal form, register court, register number, registered office

  • People: name, date of birth, personnel number, email

Also define which attributes are unique identifiers (such as the VAT ID) and which only provide indications (such as the name).

Step 2: Profile and standardize data

Matching only works on comparable data. First analyze the dataset: fill rates, formats, spelling variants. Then standardize before comparing:

  • Harmonize legal forms (“Gesellschaft mit beschränkter Haftung” → “GmbH”)

  • Normalize addresses (“St.” → “Street”, validate postal codes)

  • Align umlauts, upper and lower case, special characters and spaces

  • Bring phone numbers and IDs into a uniform format

This step alone significantly reduces the number of overlooked fuzzy duplicates. It is also the foundation for any later real-time check.

Step 3: Combine matching strategies

No single method finds all duplicates. A combination has proven effective:

  • Deterministic matching compares unique keys exactly, such as the VAT ID or GTIN. It is fast and precise.

  • Fuzzy matching calculates the similarity of strings, for example using the Levenshtein distance. It counts how many insertions, deletions or substitutions are needed to turn one term into the other.

  • Phonetic methods detect names that sound alike. For German-language data, the Cologne phonetics (Kölner Phonetik) is more relevant than the English Soundex, because it is tailored to German pronunciation and spelling rules.

  • Weighted scoring combines several fields into an overall score. A matching VAT ID carries more weight than a similar name.

For large data volumes, blocking is also useful. Here, records are only compared within meaningful groups, for example those with the same postal code. This keeps duplicate matching performant.

Step 4: Define thresholds and manual review

Translate the Fellegi-Sunter principle into clear rules. An upper threshold defines certain duplicates, which are automatically proposed for merging. A lower threshold defines certain non-duplicates. Everything in between goes to a review queue for the responsible data stewards.

Start conservatively with a high upper threshold. Evaluate the manual decisions regularly and adjust. This increases the level of automation without increasing incorrect merges.

Step 5: Implement match & merge with survivorship rules

Merging turns several duplicates into one consolidated record. Survivorship rules define for each attribute which value is adopted:

  • Source priority: The leading system wins, for example the ERP for bank data.

  • Recency: The most recently changed value wins, for example for contact data.

  • Completeness: A filled value wins over an empty field.

  • Validation: The verified value wins, for example a confirmed register number.

Just as important: all references to the resolved duplicates must be re-linked to the surviving record. This affects open orders, contracts, invoices and hierarchies. A merge without reference mapping creates orphaned documents. Document every merge in an audit-proof way so that it remains traceable and reversible if needed.

The result is the Golden Record: the one trustworthy record per business entity. Our article Golden Record Master Data explains what makes a Golden Record and how it becomes the foundation for AI.

Step 6: Prevent duplicates at the source and measure them

The most effective lever is a duplicate check at the point of data entry. Before a new record is saved, the system uses fuzzy search to look for similar entries and displays matches. The user then decides: use the existing record or create a new one with justification.

In addition, every process should include:

  • Central creation processes with approval for critical domains. Our article on workflow management for master data shows how to design such processes.

  • Regular batch checks for data coming in via interfaces or imports.

  • Monitoring of the duplicate rate per domain as a fixed KPI in the data quality dashboard.

How solid is your data foundation?

Duplicates are often the most visible symptom of a deeper problem. Goldright’s AI Data Foundation Check assesses your data foundation across five dimensions: data availability, data quality, governance, integration and AI readiness. You get a structured self-assessment that brings business units and IT to the same table.

  • Evaluation score for all data dimensions

  • Ready-to-use roadmap template

  • 100% free

Best practices: what really works in duplicate cleansing

Clear patterns emerge from enterprise projects. The following practices determine whether a duplicate check has a one-off or a lasting effect.

Start with one domain

Start where duplicates cause the greatest economic damage. This is often supplier master data, because duplicates there encourage duplicate payments and distorted purchasing analyses. Initial measurable successes build support for the next domain.

Quantify the business case

Before you start, measure the duplicate rate on a sample. Harvard Business Review describes a simple method for this: review the last 100 records and count the error-free ones. This Friday Afternoon Measurement method by Redman and colleagues delivers a reliable baseline within a few hours.

Make business units the owners

The rules for when two records are identical must come from the business units. Data stewards from purchasing, sales and finance confirm possible duplicates. IT provides the platform and automation.

Standardize first, then match

Clean normalization improves match quality more than a more complex algorithm. Invest time in step 2 before fine-tuning thresholds.

Log merge decisions

Every merge must be traceable: who, when, and on what basis. This is essential for audit and compliance and makes it possible to reverse incorrect merges.

Use AI in a targeted way

Machine learning models can learn from confirmed and rejected duplicate pairs and improve thresholds. Humans keep the final say on unclear cases. This reflects the principle that AI only works reliably on verified data.

Comparison: methods and approaches for duplicate checks

The choice of method determines which duplicates you find and which you miss. The following overview compares the common methods.

Method

How it works

Finds

Strengths

Limitations

Typical use

Exact matching

Comparison of identical field values

Exact duplicates

Fast, no false positives

Misses every spelling variant

Unique IDs (VAT ID, GTIN)

Fuzzy matching

Similarity measure, e.g. Levenshtein distance

Typos and format errors

Finds fuzzy duplicates

Requires thresholds, false positives possible

Names, addresses, descriptions

Phonetic methods

Encoding by sound, e.g. Cologne phonetics

Names that sound alike

Strong for specific languages

Only useful for names

Person and company names

Weighted scoring

Combines several fields into one score

Complex duplicates

Reflects business logic

Complex configuration

Business partners, materials

Machine learning

Model learns from confirmed pairs

Patterns beyond fixed rules

Keeps learning continuously

Needs training data, explainability

Large, heterogeneous datasets

Beyond the method, the organizational approach determines success:

Criterion

One-time cleansing

Tool in a single system

Duplicate check in MDM

Scope

One dataset, one time

One system, ongoing

All connected systems

Prevention when creating records

No

Only within its own system

Yes, across systems

References and dependent documents

Manual

Partial

Managed via mapping

Governance and approvals

Project-based

Limited

Roles, workflows, audit trail

Sustainability

Low, duplicates return

Medium

High

Suitable for

Migrations, initial cleansing

Isolated use cases

Enterprise landscapes

KPIs: how to measure the success of your duplicate check

What is not measured deteriorates unnoticed. For CDOs and Heads of Data Analytics, a few clearly defined KPIs are therefore more important than an extensive dashboard. The following five KPIs fully cover the duplicate check, from dataset quality to prevention.

KPI

Calculation

What it shows

Duplicate rate

Number of confirmed duplicates ÷ total number of records per domain

State of the dataset, baseline for the business case

Prevention rate

Duplicates prevented at record creation ÷ all creation attempts with a match

Effectiveness of the check at the source

Match precision

Confirmed duplicates ÷ all proposed duplicates

Quality of matching rules and thresholds

Review queue throughput time

Average time from proposal to decision

Effectiveness of governance

Reversal rate

Reversed merges ÷ all merges

Early warning signal for overly aggressive matching

The duplicate rate is the starting point. Measure it separately for each domain, because an average across all master data hides the critical areas. The prevention rate shows whether the real-time check at record creation is working. If it rises, fewer new duplicates are created.

Match precision and reversal rate belong together. High precision combined with a low reversal rate indicates well-calibrated thresholds. If the reversal rate rises, matching is set too aggressively. Finally, the review queue throughput time shows whether data stewards have sufficient capacity and authority.

Report these figures monthly and link them to a business metric, such as avoided duplicate payments or reduced inventory. This turns the duplicate check from an IT topic into a manageable contribution to value creation.

Case study: supplier master data after a system consolidation

Initial situation

After several acquisitions, an internationally operating industrial company had consolidated three ERP instances into one system. The supplier master data was migrated without upfront duplicate matching. Shortly afterwards, accounts payable raised more and more queries: invoices were assigned to different supplier numbers, and purchasing volumes per supplier were no longer reliable.

Challenge

A pure name comparison produced too many false positives. National subsidiaries of the same group with the same name are legally independent and could not be merged. At the same time, many duplicates were linked to open orders and framework agreements.

Approach

Together with purchasing and finance, the project team first defined the identity rules: VAT ID and bank details as strong attributes, name and address as weighted indications, legal form and country of registration as exclusion criteria. After standardizing legal forms and addresses, a combined matching process with deterministic, fuzzy and phonetic rules was run. Data stewards reviewed possible duplicates in an approval workflow. Confirmed pairs were merged using survivorship rules, and all open documents were automatically re-linked.

Result and prevention

Since the project, a real-time duplicate check has been checking every new supplier record against the entire dataset. The duplicate rate is anchored as a KPI in the monthly data quality reporting.

Key lesson

The breakthrough did not come from a better algorithm, but from a shared definition of when two suppliers are actually the same.

Pitfalls: what often goes wrong with duplicate checks

Overly aggressive matching

Low thresholds find more duplicates, but also merge records that do not belong together. An incorrect merge is more expensive than an overlooked duplicate, because it mixes up documents and histories.

Ignoring group structures

The same name does not mean the same legal entity. Parent companies and subsidiaries, branches and national companies need a hierarchy instead of a merge. In legal entity management in particular, this distinction is essential.

Merging without reference mapping

Anyone who deletes or blocks records without re-linking orders, contracts and open items creates orphaned documents and new errors in downstream processes.

Duplicate cleansing as a one-off project

Without checks at record creation, the number of duplicates immediately starts growing again. At the latest with the next import or acquisition, everything starts all over again.

Checking only for exact matches

A duplicate check that only compares identical values finds the easy cases. The costly fuzzy duplicates remain undetected and create a false sense of security.

Lack of accountability

If nobody is responsible for deciding on possible duplicates, the review queue grows and is eventually ignored. Appoint data stewards with a clear mandate and time budget.

Duplicates are a symptom. Have you found the root cause?

Where in your system landscape are duplicate records created, and how AI-ready is your data really? The AI Data Foundation Check gives you a well-founded assessment of where you stand in a short time. Or talk directly to our experts about how a duplicate check with Goldright would be implemented in your landscape.

Conclusion: a duplicate check is an ongoing task with a clear result

Duplicates arise wherever several systems and people create master data. They distort decisions, cause operational errors and jeopardize compliance and AI initiatives. An effective duplicate check therefore does not treat them as a one-time cleansing run, but as an integral part of master data management.

The path there is clearly structured. It starts with a business definition of when two records describe the same entity. This is followed by standardization, a combination of exact matching, fuzzy matching and phonetic methods, and clear thresholds for manual review. Match & merge with survivorship rules and clean reference mapping creates the Golden Record. Checking every new record keeps it clean.

For CDOs and data analytics leaders, the duplicate check is therefore a concrete lever: it makes data quality measurable, builds trust in KPIs and lays the foundation for AI. Anyone who wants to manage duplicates in master data across systems needs a central authority that bundles identity rules, workflows and checks. Our article Master Data Management: Definition and Practice explains how this works in principle.

Your next step: Use the AI Data Foundation Check to find out where your data foundation stands today. Or schedule a conversation in which we show you what implementing a cross-system duplicate check with Goldright would look like in your landscape.

Frequently Asked Questions

A duplicate check is a process that identifies, evaluates and resolves duplicate records referring to the same real-world entity within a dataset. It covers duplicate detection, duplicate matching, duplicate cleansing and prevention. The goal is one unique record per customer, supplier, material or legal entity.
Duplicate records mainly come from multiple independent points of data entry, no similarity search when new records are created, data migrations, acquisitions and bulk imports. Spelling variations add to the problem, for example with umlauts, abbreviations or legal forms.
Exact duplicates are identical or nearly identical records that an exact comparison will find. Fuzzy duplicates differ in spelling, format or completeness. Only fuzzy matching or phonetic methods can detect them reliably.
Fuzzy matching, also called fuzzy search, measures how similar two strings are instead of checking whether they are exactly equal. A common measure is the Levenshtein distance, which counts the number of character edits needed to turn one string into the other. This way, “Meier” and “Maier” or “Main St.” and “Main Street” are recognized as similar.
Match & merge is the two-step process of duplicate cleansing. Matching identifies records that belong together. Merging then uses survivorship rules to consolidate them into a single record: the Golden Record.
You prevent duplicates permanently by checking every new record against the entire dataset in real time. This includes central creation processes with approval steps, standardized input formats, regular batch checks for data coming in through interfaces, and ongoing monitoring of the duplicate rate.
Clear-cut cases can be automated, for example when the VAT ID matches. Borderline cases should be reviewed by data stewards. Machine learning can gradually raise the level of automation, but humans should keep the final say on unclear cases.
A migration is the ideal time for duplicate cleansing. Duplicates should be detected and merged before the data is transferred to the target system. After that, they are linked to transactions and much harder to clean up. Ongoing checks whenever a record is created keep the new system from filling up with duplicates again.
Key metrics are the duplicate rate per data domain, the prevention rate when records are created, match precision, review queue throughput time and the reversal rate of incorrect merges.
AI models inherit the errors in their training and context data. Duplicates distort customer counts, revenue per business partner and forecasts. Gartner predicts that through 2026, organizations will abandon 60% of AI projects that lack AI-ready data. A clean duplicate check is therefore part of any AI readiness effort.
Data cleansing is the umbrella term for all measures that correct incorrect, incomplete or inconsistent data. A duplicate check is one part of it. It focuses only on uniqueness, meaning duplicate records.