Find Duplicates

Finds duplicate records in a list of data models or JSON objects, using field rules from a JSON configuration, and returns the unique list, the duplicates and how they relate.

The Find Duplicates action looks for duplicate records in a list of data models (or JSON objects), for example the same customer entered twice with a different spelling. You decide which fields must match and how (exact, fuzzy, sound-alike…). The action returns three lists: the records without duplicates, the duplicates it removed, and which record each duplicate belongs to. To find duplicates in a simple list of text or numbers, use DeDuplicate.

Add the action

In the Process Designer toolbox, open List and pick Find Duplicates (or type Find Duplicates in the search box). Then click Configure to open the configuration panel.

The Find Duplicates action properties with the Configure button

Inputs and outputs

Field	Direction	Type	Required	What to put / what you get
List to process	Input	List of data models / JSON objects	Yes	The records to check. Each record needs a unique ID field.
Duplicate identification JSON configuration	Input	Text (JSON)	Yes	The rules, see below.
Duplicates found	Output	List of data models / JSON objects	No	The records identified as duplicates of an earlier record.
Duplicates correlation	Output	List of objects	No	One entry per kept record that has duplicates: `{"MainID":"1","CorrelatedIDs":["3","4"]}` (IDs as text).
Deduplicate list	Output	List of data models / JSON objects	Yes	The input list without the duplicates. The first record of each group is kept.

The JSON configuration

{
  "IdField": "id",
  "FieldMatches": [
    { "FieldName": "name",  "MatchType": "Fuzzy", "FuzzyAlgorithm": "Levenstein", "FuzzyThreshold": 0.8 },
    { "FieldName": "email", "MatchType": "Exact" }
  ],
  "GeneralIgnoredTerms": [" ltd", " srl"],
  "AlternateFields": [
    { "FieldName": "email", "FieldAlternateNames": ["secondaryEmail"] }
  ],
  "Sort": [
    { "Field": "name", "OrderType": "Asc" }
  ]
}

Key	Required	Meaning
`IdField`	Yes	The field that identifies each record (for example `id`). Its values must be unique; they are used in Duplicates correlation.
`FieldMatches`	Yes	The rules (at least one). Two records are duplicates only when all rules match (AND). Each rule has `FieldName`, `MatchType` and the options of that match type.
`FieldMatches[].MatchType`	Yes	`Exact`, `Contains`, `Similar`, `Fuzzy`, `Soundex` or `SimilarWordMatch`. They work as described in Simple Filter, but compare one record's field with the other record's field.
`FieldMatches[].IgnoredTerms`	No	Words or characters removed from this field before comparing, for example `[" ltd", ","]`.
`GeneralIgnoredTerms`	No	Terms removed from every field before comparing; added to each rule's `IgnoredTerms`.
`AlternateFields`	No	For a rule on `FieldName`, also compare the first record's field with these fields of the other record; the rule matches if any of them matches. One direction only.
`Sort`	No	Sorts Deduplicate list and Duplicates found: a list of `{"Field": "...", "OrderType": "Asc"}` or `"Desc"`, applied in order. Without it, the original order is kept.

Fuzzy uses FuzzyAlgorithm and FuzzyThreshold (0–1); Soundex uses SoundexDistance (0–4, default 4); SimilarWordMatch uses SimilarFirstNwords / SimilarLastNwords. The algorithm names are listed in Simple Filter.

Example

List to process:

[
  {"id":1,"name":"Leia Winnett","email":"lwinnett2@lycos.com","dept":"Support"},
  {"id":2,"name":"Blair Streak","email":"bstreak4@yellowpages.com","dept":"R&D"},
  {"id":3,"name":"leia winnett","email":"LWinnett2@lycos.com","dept":"Support"},
  {"id":4,"name":"Leia Winnet","email":"other@x.com","dept":"Support"},
  {"id":5,"name":"Blair Streak","email":"bstreak4@yellowpages.com","dept":"R&D"}
]

FieldMatches	Deduplicate list (ids)	Duplicates found (ids)	Duplicates correlation
`name` Exact and `email` Exact	`1`, `2`, `4`	`3`, `5`	`[{"MainID":"1","CorrelatedIDs":["3"]},{"MainID":"2","CorrelatedIDs":["5"]}]`
`name` Fuzzy, `Levenstein`, `0.8`	`1`, `2`	`3`, `4`, `5`	`[{"MainID":"1","CorrelatedIDs":["3","4"]},{"MainID":"2","CorrelatedIDs":["5"]}]`

Record 3 is an exact duplicate of 1 because the comparison ignores upper/lower case.

Good to know

Values are compared in lower case, with double spaces reduced and accents removed. Numbers are compared as text.
Ignored terms are removed exactly as written, spaces included. With "IgnoredTerms": ["ltd"], Acme Ltd becomes acme (with a trailing space) and is not an Exact match for Acme. Write " ltd" (with the leading space), or use Contains.
Sort compares values as text, so 120 comes before 45.
Records are compared in pairs, so the time grows quickly with the list size; on very large lists split the work or narrow the list with Filter first.
A list with one record is returned as it is, with no duplicates.
The step fails when the list is empty, when IdField or FieldMatches is missing, when the JSON is invalid, when the list holds simple values instead of records, or when a record lacks a field named in a rule (the error then reads Object reference not set to an instance of an object.).

DeDuplicate (simple lists), Filter, Sort Data Model, Similarity Search
Step-by-step tutorial with a video: Duplicate identification