current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 5 down vote favorite

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy?

Exemple

Input file: 

A    1,2,3,45,1,8,2,3
B    5,6,6,6,6,6,2,3,7

Expected output:

A    1,2,3,45,8
B    5,6,2,3,7
share|improve this question

6 Answers 6

up vote 2 down vote accepted

Another perl solution:

perl -anle '                                                                    
    print "$F[0] ", join ",", grep {!$seen{$_}++} split ",",$F[1];              
    %seen=();                                                                   
' file
A 1,2,3,45,8
B 5,6,2,3,7
share|improve this answer
up vote 5 down vote

A sed one:

sed '
  s/[^[:blank:]]\{1,\}/,&,/g;:1
  s/\(\(,[^,[:blank:]]*\)\(,[^,[:blank:]]*\)*\)\2,/\1,/;t1
  s/,\([^[:blank:]]*\),/\1/g'

(it processes all the fields that contain , characters and preserves spacing)

share|improve this answer
    
why didn't I notice this 17 hours ago? –  mikeserv Aug 6 at 3:32
up vote 3 down vote

With perl:

perl -MList::MoreUtils=uniq -pe 's{\S*,\S*}{join ",", uniq split ",", $&}ge'

(it processes all the fields that contain , characters and preserves spacing)

share|improve this answer
up vote 2 down vote

I guess this is like Stephane's though it is a little different. Anyway, I took the trouble to write it. I based it on this thing I did before here (where I also explain it a lot better)...

sed ':t
s/\([^,]*\),\(.*\1\)/ \2/;tt
s/  */,/g;s/,/ /;s/,$//' <<\DATA
A 1,2,3,45,1,8,2,3,
B 5,6,6,6,6,6,2,3,7
DATA

OUTPUT

A 45,1,8,2,3
B 5,6,2,3,7
share|improve this answer
up vote 1 down vote

For completeness, a solution in awk:

BEGIN {
    FS = "[ \t,]+";
    OFS = ",";
}

{
    delete seen;
    for(i = 2; i <= NF; i++) {
        if($i in seen) {
            $i = "";
        }
        seen[$i] = 1;
    }
    sub(",","\t");   #separate first field with a tab
    gsub(",,+",","); #squeeze empty fields
    sub(",$","");    #remove trailing comma, if any
    print;
}
share|improve this answer
up vote 1 down vote 
awk '{n=split($2, a, ","); $2=a[1];
  for(i=2; i<=n; i++)
    {$2 = ($2 ~ "(^|,)" a[i] "($|,)") ? $2 : ($2 "," a[i])}}1' OFS='\t' file
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.