How to create an array of unique elements from a string/array in bash?

If you are using zsh:

$ array=(1 2 3 2 1)
$ echo ${(u)array[@]}
1 2 3

or (if KSH_ARRAYS option is not set) even

$ echo ${(u)array}
1 2 3

For an array with arbitrary values, it's quite tricky with bash as it doesn't have a builtin operator for that.

bash however happens not to support storing NUL characters in its variables, so you can make use of that to pass that to other commands:

The equivalent of zsh's:

new_array=("${(@u}array}")

on a recent GNU system, could be:

eval "new_array=($(
  printf "%s\0" "${array[@]}" |
    LC_ALL=C sort -zu |
    xargs -r0 bash -c 'printf "%q\n" "$@"' sh
  ))"

Alternatively, with recent versions of bash, and assuming none of the array elements are empty, you could use associative arrays:

unset hash
typeset -A hash
for i in "${array[@]}"; do
  hash[$i]=
done
new_array=("${!hash[@]}")

With bash 4.4 and newer and with GNU sort:

readarray -td '' new_array < <(
  printf '%s\0' "${array[@]}" | LC_ALL=C sort -zu)

The order of the elements would not be the same in those different solutions.

With tcsh:

set -f new_array = ($array:q)

Would retain the first element (a b a => a b) like zsh's (u) expansion flag.

set -l new_array = ($array:q)

Would retain the last (a b a => b a). Those however remove empty elements from the array.

This solution worked for me.

ids=(1 2 3 2 1)
echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '

The above produces 1 2 3 as the output.

Shorter version as suggested by Costas could be,

printf "%s\n" "${ids[@]}" | sort -u | tr '\n' ' '

To store the end results to an array, you could do something like,

IFS=$' '
arr=($(printf "%s\n" "${ids[@]}" | sort -u | tr '\n' ' '))
unset IFS

Now, when I do an echo on arr, this is the output I get.

echo "${arr[@]}"
1 2 3

References

http://stackoverflow.com/a/13648438/1742825 http://stackoverflow.com/a/9449633/1742825

To do it entirely in the shell and put the result in an array,

declare -A seen
for word in one two three two one
do
        if [ ! "${seen[$word]}" ]
        then
                result+=("$word")
                seen[$word]=1
        fi
done
echo "${result[@]}"

In words: if we haven’t seen a given word yet, add it to the result array and flag it as having been seen.  Once a word has been seen, ignore subsequent appearances of it.