uniq Command — Removing Duplicate Lines in Linux¶

The uniq command is used to identify, remove, and count duplicate lines in text files. It is commonly used after the sort command to clean data, generate reports, analyze logs, and process large datasets. Every Linux Administrator, DevOps Engineer, and Cloud Engineer should know how to use uniq effectively.

Learning Path¶

Linux Mastery → Module 3: Text Processing → Lesson 5

Difficulty: Beginner → Intermediate

Reading Time: 25 Minutes

Course Progress

Course: Linux Mastery

Module: Text Processing

Lesson: 5 of 18

What You'll Learn¶

After completing this lesson, you'll be able to:

Understand the uniq command
Remove duplicate lines
Count duplicate entries
Display only duplicate lines
Display only unique lines
Combine uniq with sort
Analyze production logs and reports

Prerequisites¶

Before starting this lesson, complete:

Module 1 – Linux Fundamentals
Module 2 – Linux Command Line Essentials
Module 3 Lessons 1–4

Why Learn uniq?¶

Imagine you have a log file containing repeated error messages.

ERROR

ERROR

WARNING

INFO

INFO

INFO

Instead of manually identifying duplicates:

sort logfile.txt | uniq

Output:

ERROR

INFO

WARNING

This is one of the most common Linux data-cleaning operations.

What is uniq?¶

The uniq command filters or reports repeated lines in a text file.

Important: uniq only works on adjacent duplicate lines.

For reliable results, always sort the file first.

Syntax:

uniq [OPTIONS] FILE

Sample File¶

Create a file.

cat > fruits.txt

Contents:

Apple

Apple

Banana

Banana

Banana

Orange

Orange

Mango

Press:

Ctrl + D

Remove Duplicate Lines¶

uniq fruits.txt

Output:

Apple

Banana

Orange

Mango

Why sort is Important¶

Consider:

Apple

Banana

Apple

Orange

Command:

uniq fruits.txt

Output:

Apple

Banana

Apple

Orange

The duplicate Apple is not removed because the duplicate lines are not adjacent.

Correct approach:

sort fruits.txt | uniq

Output:

Apple

Banana

Orange

Count Duplicate Lines¶

sort fruits.txt | uniq -c

Output:

2 Apple

3 Banana

1 Mango

2 Orange

The -c option prefixes each line with the number of occurrences.

Display Only Duplicate Lines¶

sort fruits.txt | uniq -d

Output:

Apple

Banana

Orange

Only repeated values are displayed.

Display Only Unique Lines¶

sort fruits.txt | uniq -u

Output:

Mango

Only lines that appear once are displayed.

Ignore Case¶

Create:

Apple

APPLE

apple

Banana

Command:

sort -f fruits.txt | uniq -i

Treats uppercase and lowercase letters as identical.

Skip Characters¶

Suppose:

001 Alice

002 Alice

003 Bob

Ignore the first four characters.

uniq -s 4 users.txt

Useful for fixed-width reports.

Skip Fields¶

Example:

1 Alice

2 Alice

3 Bob

Command:

uniq -f 1 users.txt

Ignores the first field while comparing lines.

Combining with Pipes¶

Remove duplicate users.

cut -d ":" -f1 /etc/passwd | sort | uniq

Count duplicate IP addresses.

cat access.log | cut -d " " -f1 | sort | uniq -c

Display unique departments.

cut -d "," -f2 employees.csv | sort | uniq

Common uniq Options¶

Option	Description
`-c`	Count duplicate lines
`-d`	Display only duplicates
`-u`	Display only unique lines
`-i`	Ignore case
`-f`	Skip fields
`-s`	Skip characters

Real Production Examples¶

Count failed login attempts.

grep "Failed password" /var/log/auth.log | sort | uniq -c

Find duplicate IP addresses.

cat access.log | cut -d " " -f1 | sort | uniq -c

Display unique Docker images.

docker images | awk '{print $1}' | sort | uniq

Count Kubernetes namespaces.

kubectl get namespaces | sort | uniq

Generate unique usernames.

cut -d ":" -f1 /etc/passwd | sort | uniq

Production Perspective¶

The uniq command is commonly used for:

Cleaning reports
Removing duplicate records
Counting repeated log entries
Security log analysis
Audit reporting
Data preprocessing before scripting

It is almost always used together with sort.

Hands-on Lab¶

Task 1¶

Create:

cat > servers.txt

Contents:

server01

server02

server01

server03

server02

server04

Task 2¶

Sort the file.

sort servers.txt

Task 3¶

Remove duplicates.

sort servers.txt | uniq

Task 4¶

Count duplicates.

sort servers.txt | uniq -c

Task 5¶

Display duplicates only.

sort servers.txt | uniq -d

Task 6¶

Display unique entries only.

sort servers.txt | uniq -u

Task 7¶

Count unique usernames.

cut -d ":" -f1 /etc/passwd | sort | uniq | wc -l

Task 8¶

Count duplicate shells.

cut -d ":" -f7 /etc/passwd | sort | uniq -c

Command Deep Dive¶

Command	Purpose	Production Example
`uniq`	Remove adjacent duplicates	Reports
`uniq -c`	Count duplicates	Log analysis
`uniq -d`	Show duplicates	Security audits
`uniq -u`	Show unique values	Data validation
`uniq -i`	Ignore case	User reports
`uniq -f`	Skip fields	Fixed-format reports

Production Troubleshooting Scenario¶

Scenario

A web server's access log contains repeated client IP addresses.

Tasks:

Display unique client IPs.
Count how many requests came from each IP.
Display only duplicate IP addresses.
Identify IPs that appear only once.

Solutions:

cut -d " " -f1 access.log | sort | uniq

cut -d " " -f1 access.log | sort | uniq -c

cut -d " " -f1 access.log | sort | uniq -d

cut -d " " -f1 access.log | sort | uniq -u

Mini Challenge¶

Create:

employees.txt

Contents:

Alice

Bob

Alice

Charlie

David

Charlie

Alice

Eve

Perform the following:

Sort the file.
Remove duplicate names.
Count each name.
Display duplicate names only.
Display names that appear only once.
Count the total number of unique employees.

Best Practices¶

Always sort data before using uniq.
Use uniq -c to generate quick frequency reports.
Combine uniq with cut and grep for advanced analysis.
Use uniq -u to identify unique records.
Use uniq -d to detect duplicate entries.

Common Mistakes¶

❌ Using uniq on unsorted data.

✅ Incorrect:

uniq file.txt

Correct:

sort file.txt | uniq

❌ Assuming uniq removes all duplicates automatically.

✅ It only removes adjacent duplicate lines.

❌ Forgetting to sort before counting duplicates.

✅ Always use:

sort file.txt | uniq -c

Interview Questions¶

Beginner¶

What is the purpose of the uniq command?
Why is sort commonly used before uniq?
What does uniq -c do?
What is the difference between uniq -d and uniq -u?

Intermediate¶

Explain why uniq only removes adjacent duplicates.
How do you count duplicate IP addresses in a log file?
What does uniq -i do?
When would you use uniq -f?

Architect Level¶

How would you analyze duplicate log entries in a production environment?
Why is uniq useful for security and audit reporting?
How would you combine cut, sort, and uniq to generate a summary report?

Summary¶

In this lesson, you learned:

Removing duplicate lines
Counting duplicate entries
Displaying only duplicate or unique lines
Combining uniq with sort
Production log analysis
Data cleaning techniques

The uniq command is a simple but powerful utility for cleaning and summarizing data. When combined with sort, it becomes an essential tool for Linux administration and text processing.

Key Takeaways¶

uniq removes adjacent duplicate lines.
Always sort data before using uniq.
uniq -c counts occurrences.
uniq -d displays only duplicate lines.
uniq -u displays only unique lines.
uniq is commonly used for log analysis, reporting, and data cleanup.

What's Next?¶

tr Command — Translating and Transforming Text in Linux

In the next lesson, you'll learn:

Character translation
Converting lowercase to uppercase
Replacing characters
Deleting characters
Squeezing repeated characters
Real-world text transformation examples