Lesson 15 · 3. Deep Dive Concepts · Last updated · By Dheeraj Sharma

How to Use Remove Duplicates Node in n8n | Clean Your Data Fast

Learn How to Use Remove Duplicates Node in n8n | Clean Your Data Fast in this comprehensive n8n tutorial. Step-by-step guide with video, code examples, and expert tips from the n8n Zero to Hero course.

Table of Contents

Overview

This lesson is part of the n8n AI Automation - Zero to Hero course, Section: 3. Deep Dive Concepts.

Watch the video above for the full tutorial, or read the written guide below.

What does the Remove Duplicates node do in n8n?

The Remove Duplicates node filters out repeated items in your workflow data, offering three operations: remove items repeated within the current input, remove items processed in a previous execution, or clear the duplication history entirely. Without it, downstream nodes receive repeated records, which can trigger multiple emails to the same customer, duplicate orders in your database, or skewed dashboard reports. It is a core data-integrity node in any business automation pipeline.

How to configure the Remove Duplicates node to deduplicate by a specific field

The Remove Duplicates node's most critical setting is the Comparison field, which controls what counts as a duplicate. Three options are available: All Fields (every field must match for a row to be flagged), All Fields Except (useful when you want to exclude one field and compare everything else), and Selected Fields (compare only the fields you explicitly choose). In the lesson workflow, 41 order records contained repeated order IDs across three entries for the same order. Selecting Selected Fields and dragging in the order_id field reduced the 41 records to 13 unique orders.

Once deduplication is set, open the Options panel, click Add Field, and enable Remove Other Fields. This strips every field except the ones used in comparison, so only the 13 unique order IDs pass forward. That keeps the payload lean before it hits the Aggregate node, which then bundles all IDs into one item for a single email or Slack message.

The correct workflow architecture is one Remove Duplicates node per branch, not a shared node after merging branches. In the lesson, merging all four switch branches (pending, processing, cancelled, refunded) into one Remove Duplicates node caused the downstream Aggregate and email nodes to execute four times, sending four separate messages. The fix was to place individual Remove Duplicates nodes on the pending and processing branches, and to use a Merge node before a single Remove Duplicates node on the combined cancelled-and-refunded branch, since both shared the same final actions.

Key Takeaways

  • The Remove Duplicates node offers three operations: deduplicate within the current input, deduplicate against previous executions, or clear duplication history.
  • The Comparison setting determines what a duplicate is. Selected Fields is the right choice when you only need to compare one or two fields (such as order_id) rather than entire rows.
  • Enabling Remove Other Fields in Options discards non-compared fields immediately, keeping the data payload minimal before it reaches downstream nodes.
  • Each branch in a Switch node should have its own Remove Duplicates node. Merging branches before deduplication causes every downstream node to execute once per branch, generating multiple emails or messages.
  • Skipping deduplication in a real workflow produces visible business errors: the lesson email showed the same order IDs repeated multiple times before the node was added, then collapsed to 13 unique IDs after.

Next Steps

Continue your n8n journey with the full n8n AI Automation - Zero to Hero course.

Frequently Asked Questions

The Remove Duplicates node removes repeated records from your workflow data to maintain data integrity. It prevents business errors such as sending multiple emails to the same customer, writing duplicate orders to a database, or skewing report numbers. It operates on either the current input batch or across previous workflow executions.

The Remove Duplicates node compares records using one of three modes. 'All Fields' flags a record as duplicate only if every field matches another row, so records that differ on any field pass through. 'All Fields Except' lets you exclude one field and compare everything else, which is useful when one column like a timestamp is always unique. 'Selected Fields' compares only the fields you explicitly choose, such as order_id, and is the right pick when you want to deduplicate on a single identifier regardless of what other fields contain.

The Remove Duplicates node was placed after a point where all four Switch branches (pending, processing, cancelled, refunded) were merged into one path. Because n8n executes top to bottom and each branch fed the shared node separately, every downstream node ran four times, producing four messages. The fix was to give each branch its own Remove Duplicates node so each branch stays isolated all the way to its final action.

The 'Remove Other Fields' option, found under the node's Options panel, strips every field from each item except the ones selected for comparison. In the lesson, enabling it after selecting 'order_id' meant only the 13 unique order IDs passed to the Aggregate node, removing all other columns from the payload and keeping the downstream data lean.

Dheeraj Sharma

Dheeraj Sharma

AI Systems Builder
Creator of the n8n Zero to Hero course (42 lessons, 31+ hours). I help solopreneurs build AI systems that grow revenue without growing workload.

Get the n8n Mastery Bundle

All workflows, cheat sheets, and premium resources from the entire course in one package.

Get Premium Resources