DeDuplicate

Finds duplicate values in a simple list (text or numbers) and returns the unique values, the duplicates and how they relate.

The DeDuplicate action finds duplicates in a simple list of text or numbers — for example the same name written as Jack and jack. It returns the list without duplicates, the duplicates it removed, and which value each duplicate belongs to. By default values are compared exactly (ignoring case); a JSON configuration lets you use fuzzy, sound-alike or other comparisons. For lists of data models, use Find Duplicates.

Add the action

In the Process Designer toolbox, open List and pick DeDuplicate (or type DeDuplicate in the search box). Then click Configure to open the configuration panel.

The DeDuplicate action properties with the Configure button

Inputs and outputs

Field	Direction	Type	Required	What to put / what you get
List to process	Input	List of text or numbers	Yes	The values to check.
Duplicate identification JSON configuration	Input	Text (JSON)	No	The comparison rule. Leave it empty for `{"MatchType":"Exact"}`.
Duplicates found	Output	List of text	No	The values identified as duplicates of an earlier value.
Duplicates correlation	Output	List of objects	No	One entry per kept value that has duplicates: `{"MainID":"eric","CorrelatedIDs":["eRic","Eric"]}`.
Deduplicate list	Output	List of text	Yes	The list without duplicates. The first occurrence is kept.

The JSON configuration

{
  "MatchType": "Exact",
  "IgnoredTerms": [" ltd"],
  "Sort": "Asc"
}

Key	Required	Meaning
`MatchType`	Yes (when you give a configuration)	`Exact`, `Contains`, `Similar`, `Fuzzy`, `Soundex` or `SimilarWordMatch`. They work as described in Simple Filter, but compare the values of the list with each other.
`IgnoredTerms`	No	Words or characters removed from every value before comparing, exactly as written (spaces included).
`Sort`	No	`Asc` or `Desc`: sorts Deduplicate list and Duplicates found as text. Without it, the original order is kept.
`FuzzyAlgorithm`, `FuzzyThreshold`	Only for `Fuzzy`	Algorithm name (see Simple Filter) and minimum score from `0` to `1`.
`SoundexDistance`	Only for `Soundex`	`0` to `4`, default `4`.
`SimilarFirstNwords`, `SimilarLastNwords`	Only for `SimilarWordMatch`	How many first / last words must be equal.

Example

List to process	Configuration	Deduplicate list	Duplicates found
`["Jack","jack","John","john","eric","eRic","Eric","487",487,"48-7"]`	(empty)	`["Jack","John","eric","487","48-7"]`	`["jack","john","eRic","Eric","487"]`
`["b","a","B","c"]`	`{"MatchType":"Exact","Sort":"Desc"}`	`["c","b","a"]`	`["B"]`
`["John-Doe","John Doe","376-323-1111","323-1111"]`	`{"MatchType":"Similar"}`	`["John-Doe","376-323-1111"]`	`["John Doe","323-1111"]`
`["Robert","Rupert","Smith","Smyth"]`	`{"MatchType":"Soundex","SoundexDistance":4}`	`["Robert","Smith"]`	`["Rupert","Smyth"]`
`["Acme Ltd","Acme","Other"]`	`{"MatchType":"Exact","IgnoredTerms":[" ltd"]}`	`["Acme Ltd","Other"]`	`["Acme"]`

For the first row, Duplicates correlation is [{"MainID":"Jack","CorrelatedIDs":["jack"]},{"MainID":"John","CorrelatedIDs":["john"]},{"MainID":"eric","CorrelatedIDs":["eRic","Eric"]},{"MainID":"487","CorrelatedIDs":["487"]}].

Good to know

Values are compared in lower case, with double spaces reduced and accents removed.
The outputs are always lists of text: [1,2,1,3.5] gives the deduplicated list ["1","2","3.5"]. The number 487 and the text "487" are duplicates.
IgnoredTerms are removed exactly as written: write " ltd" with the leading space so that Acme Ltd becomes acme.
The step fails when the list is empty (The input list does not have any element!) or holds data models (The input list is a Data Model! This action can only use simple lists.), or when the configuration is invalid JSON or has no MatchType.

Find Duplicates (lists of data models), Simple Filter, Concatenate, Similarity Search
Step-by-step tutorial: Deduplicate list