Glossary

What is data deduplication?

Data deduplication is the process of identifying and resolving records that unintentionally represent the same entity more than once. For a product catalogue, resolution may involve retaining a preferred record, linking source identifiers or merging compatible information. It requires an explicit identity decision so that legitimate variants, packaging levels and supplier offers are not removed as duplicates.

Confirm identity before resolving records

Duplicate detection can use exact identifiers, combinations of attributes or similarity-based matching. Those methods produce evidence for a decision; a similar name alone is not sufficient. Decide what counts as the same entity within the catalogue and how uncertain matches will be reviewed.

Before a merge, examine linked images, documents, category assignments and downstream references. The record with more populated fields is not automatically the most trustworthy one.

Example: a repeated manufacturer part

A distributor receives the same manufacturer part through two supplier files. The catalogue already contains the product, but a later import creates another product record. Review confirms the brand, part number and sellable unit match.

The team can retain the chosen catalogue record, preserve both supplier relationships and transfer any verified unique information. It should also update references that pointed to the retired record and record how the identity decision was reached. A pack of ten remains separate if it is a different sellable item.

Deduplication and matching

Product matching finds records that may describe the same product. Deduplication acts on confirmed unwanted duplication. It does not require deleting every repeated value or combining all commercial offers. Keep a recoverable record of changes so an incorrect merge can be investigated and reversed where necessary.

Related terms

Sources and further reading

Product Data Services

Explore practical support for organising, improving and maintaining product information.