current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 18 down vote favorite
6

Is there a unix command that can check if any two lines in a file are the same?

For e.g. Consider a file sentences.txt

This is sentence X
This is sentence Y
This is sentence Z
This is sentence X
This is sentence A
This is sentence B

We see that the sentence

This is sentence X

is repeated.

Is there any command that can quickly detect this, so that I can perhaps execute it like this - 

$ cat sentences.txt | thecommand
Line 1:This is sentence X
Line 4:This is sentence X
share|improve this question
add comment

3 Answers

up vote 35 down vote accepted

Here is one way to get the exact output you're looking for:

$ grep -nFx "$(sort sentences.txt | uniq -d)" sentences.txt 
1:This is sentence X
4:This is sentence X

Explanation:

The inner $(sort sentences.txt | uniq -d) lists each line that occurs more than once. The outer grep -nFx looks again in sentences.txt for exact -x matches to any of these lines -F and prepends their line number -n

share|improve this answer
 
Your edit just barely beat me from posting the exact same answer. +1 –  casey Feb 5 at 18:40
 
+1 for giving the exact output in a really nice way! –  aularon Feb 5 at 18:42
 
Thanks! This is a really informative answer. –  CodeBlue Feb 5 at 19:26
1  
@CodeBlue - yes. It's called Command Substitution –  grebneke Feb 5 at 19:29
5  
sort sentences.txt | uniq -d | grep -nFxf - sentences.txt would be a little more efficient and would avoid potential arg list too long problems. –  Stephane Chazelas Feb 6 at 9:34
show 2 more comments
up vote 8 down vote

Not exactly what you want, but you can try combining sort and uniq -c -d:

aularon@aularon-laptop:~$ cat input
This is sentence X
This is sentence Y
This is sentence Z
This is sentence X
This is sentence A
This is sentence B

aularon@aularon-laptop:~$ sort input | uniq -cd
      2 This is sentence X
aularon@aularon-laptop:~$

2 here is the number of duplications found for the line, from man uniq:

   -c, --count
          prefix lines by the number of occurrences

   -d, --repeated
          only print duplicate lines
share|improve this answer
add comment
up vote 5 down vote

IF the file contents fit in memory awk is good for this. The standard one-liner in comp.lang.awk (I can't search an instance from this machine but there's several every month) to just detect there is duplication is awk 'n[$0]++' which counts the occurrences of each line value and prints any occurrence(s) other than the first, because the default action is print $0.

To show all occurrences including the first, in your format, but possibly in mixed order when more than one value is duplicated, gets a little more finicky:

awk <sentences.txt ' !($0 in n) {n[$0]=NR;next} \
    n[$0] {n[$0]=0; print "Line "n[$0]":"$0} \
    {print "Line "NR":"$0} '

Shown in multiple lines for clarity, you usually run together in real use. If you do this often you can put the awk script in a file with awk -f, or of course the whole thing in a shell script. Like most simple awk this can be done very similarly with perl -n[a].

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.