current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

Because of array depth issues in PHP, receiving this array from Python becomes truncated with an ellipsis ("..."). I'd like to process the array in Python before returning back to php.

Clarification: I need to maintain the inner sets [135, 121, 81]. These are R, G, B values and I'm tying to group sets that occur more than once. Values in sets need to maintain [1, 2, 3] sequence, NOT [1,2,3,4,5,6,7,8] as some answers have suggested below.

How would you simplify this 3D numpy.ndarray to a collection of unique RGB triples?

Here is how the array is printed by Python:

[[[135 121  81]
  [135 121  81]
  [135 121  81]
  ..., 
  [135 121  81]
  [135 121  81]
  [135 121  81]]

 [[135 121  81]
  [135 121  81]
  [135 121  81]
  ..., 
  [135 121  81]
  [135 121  81]
  [135 121  81]]

 [[ 67  68  29]
  [135 121  81]
  [ 67  68  29]
  ..., 
  [135 121  81]
  [135 121  81]
  [135 121  81]]

 ..., 
 [[200 170  19]
  [200 170  19]
  [200 170  19]
  ..., 
  [ 67  68  29]
  [ 67  68  29]
  [ 67  68  29]]

 [[200 170  19]
  [200 170  19]
  [200 170  19]
  ..., 
  [116 146  15]
  [116 146  15]
  [116 146  15]]

 [[200 170  19]
  [200 170  19]
  [200 170  19]
  ..., 
  [116 146  15]
  [116 146  15]
  [116 146  15]]]

Here is the code that I have attempted:

def uniquify(arr)
    keys = []

    for c in arr:
        if not c in keys:
            keys[c] = 1
        else:
            keys[c] += 1

    return keys

result = uniquify(items)
share|improve this question
    
That doesn't look like python code ... Can you show what the python "array" looks like? –  mgilson Jan 30 '13 at 2:07
    
I've updated the code to show the python array. –  stwhite Jan 30 '13 at 2:10
    
2D Array. Question was not fully updated at the same time. –  stwhite Jan 30 '13 at 2:30
    
@stwhite: OK, I'll change my answer to put the 2D answer first, instead of as an afterthought. But what's there should already work. –  abarnert Jan 30 '13 at 2:31
1  
Meanwhile, it would be simpler if you posted the expected output, instead of trying to describe it. –  abarnert Jan 30 '13 at 2:33
show 2 more comments

4 Answers

up vote 2 down vote accepted

Based on the representation of your "array", it looks like you're working with a numpy.ndarray. This becomes quite a simple problem if that is the case -- You can transform to a 1-D iterable simple by using the .flat attribute. To make it unique, you can just use a set:

set(array.flat)

This will give you a set, but you could easily get a list from it:

list(set(array.flat))

Here's how it works:

>>> array = np.zeros((10,12,42,53))
>>> list(set(array.flat))
[0.0]

As a side note, there's also np.unique which will give you the unique elements of your array as well.

>>> array = np.zeros((10,12),dtype=int)
>>> print array
[[0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0 0 0]]
>>> np.unique(array)
array([0])
>>> array[0,5] = 1
>>> array[4,10] = 42
>>> np.unique(array)
array([ 0,  1, 42])

I think I finally got this one figured out:

from itertools import product

items = set(tuple(a[itr+(slice(None),)]) for itr in product(*[range(x) for x in a.shape[:-1]]))
print items

Seems to work. Phew!

How this works -- the pieces that you want to keep as triplets are accessed as:

array[X,Y,:]

So, we just need to loop over all of the combinations of X and Y. That is exactly what itertools.product is good for. We can get the valid X and Y in an arbitrary number of dimensions:

[range(x) for x in array.shape[:-1]]

So we pass that to product:

indices_generator = product(*[range(x) for x in array.shape[:-1]])

Now we have something that will generate the first to indices -- We just need to construct a tuple to pass to __getitem__ that numpy will interpret as (X,Y,:) -- That's easy, we're already getting (X,Y) from indices_generator -- We just need to tack on an emtpy slice:

all_items = ( array[idx+(slice(None),)] for idx in indices_generator )

Now we can loop over all_items looking for the unique ones with a set:

unique_items = set(tuple(item) for item in all_items)

Now turn this back into a list, or a numpy array or whatever you want for the purposes of passing it back to PHP.

share|improve this answer
    
thanks for your suggestion. It's close, but I forgot to mention that I need to maintain the inner sets [135, 121, 81]. These are R, G, B values and I'm tying to group sets that occur more than once. –  stwhite Jan 30 '13 at 2:19
1  
@stwhite -- See update. –  mgilson Jan 30 '13 at 2:32
    
@stwhite: Are you actually using a 3D numpy array, as mgilson suspected? Or a list of lists of lists, as it sounds like from your question? Or something in between, like a list of 2D np.arrays or something? –  abarnert Jan 30 '13 at 2:51
    
@abarnert I am using numpy. I would assume it's only 3D, but I am measuring the same data right now... I wouldn't think numpy would use more dimensions dependent on data, but this is the first time I've used it. –  stwhite Jan 30 '13 at 2:52
    
@abarnert -- I suspected it was a numpy.ndarray because that's how they print (__str__). They leave gaps with ... and they don't put the commas in there. It's a bit annoying actually, but I think they do a bit better if you specifcally look at their __repr__. –  mgilson Jan 30 '13 at 3:00
show 5 more comments
up vote 1 down vote

Look at the recipes in the itertools documentation. There are flatten and unique_everseen functions that do exactly what you want.

So, you can copy and paste them. Or you can just pip install more-itertools so you just import them. Now, you can flatten the 3D array to 2D, and uniquify the 2D array with unique_everseen…

Except for one problem. The elements of your 2D array are lists, which are not hashable, so you have to convert them to something hashable. But that's easy:

def uniquify(arr3d):
    return unique_everseen(flatten(arr3d), tuple)

That's it.

And if you look at the implementations of those functions while you're pasting them, they're pretty simple. The only real trick here is using a set to hold the values seen so far: sets only hold one copy of each unique element (and can determine whether an element is already in the set very quickly).

In fact, if you don't need to preserve the ordering, it's even simpler:

def uniquify(arr3d):
    return set(tuple(x) for x in flatten(arr3d))

As a test, I copied your string and turned it into an actual Python list display, then did this:

inarray = [[[135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81]],
           [[135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81]],
           [[67, 68, 29],
            [135, 121, 81],
            [67, 68, 29],
            [135, 121, 81],
            [135, 121, 81],
            [135, 121, 81]],
           [[200, 170, 19],
            [200, 170, 19],
            [200, 170, 19],
            [67, 68, 29],
            [67, 68, 29],
            [67, 68, 29]],
           [[200, 170, 19],
            [200, 170, 19],
            [200, 170, 19],
            [116, 146, 15],
            [116, 146, 15],
            [116, 146, 15]],
           [[200, 170, 19],
            [200, 170, 19],
            [200, 170, 19],
            [116, 146, 15],
            [116, 146, 15],
            [116, 146, 15]]]
for val in uniquify(inarray):
    print(val)

The output was:

[135, 121, 81]
[67, 68, 29]
[200, 170, 19]
[116, 146, 15]

Is that what you wanted?

If you want it as a list of lists, that's just:

array2d = list(uniquify(array3d))

If you're used a simple set instead of unique_everseen, these will be tuples instead of lists, so if you need a list of lists:

array2d = [list(val) for val in uniquify(array3d)]
share|improve this answer
    
I'm getting a NameError: global name 'flatten' is not defined error on this. Tried intertools.flatten(arr3d) but it throws AttributeError: 'module' object has no attribute 'flatten' –  stwhite Jan 30 '13 at 2:35
1  
As the answer directly says, you have to either copy and paste them from the documentation, or import them from more_itertools after installing it. –  abarnert Jan 30 '13 at 2:37
add comment
up vote 0 down vote

Assuming the python list looks like [[[1,2,3], [4,5,6]], [[7,8,9]]] (that is, a list of lists of integers

mylist = [[[1,2,3], [4,5,6]], [[7,8,9]]]
items = set()
for sublist in mylist:
    for subsublist in sublist:
        for item in subsublist:
            items.add(item)

If you then specifically need a list, you can just cast it as so: items = list(items)

A set is a datatype that is similar to a list, but doesn't contain duplicates. A side-effect of the set datatype is the insertion order isn't preserved - if this is important to you you'll need something like:

mylist = [[[1,2,3], [4,5,6]], [[7,8,9]]]
items = []
for sublist in mylist:
    for subsublist in sublist:
        for item in subsublist:
            if not item in items:
                items.add(item)

Edit: based on your edit, you probably want this:

mylist = [[[1,2,3], [4,5,6]], [[7,8,9], [1,2,3]]]
items = []
for sublist in mylist:
    for item in sublist:
        if not item in items:
            items.append(item)
# items = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
share|improve this answer
add comment
up vote 0 down vote

itertools is your friend here:

>>> import itertools
>>> array = [1,1,1,2,2,2,3,3,3,4,5,6,6,6]
>>> [x[0] for x in itertools.groupby(array)]
[1, 2, 3, 4, 5, 6]

For example:

array = [[[135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81]],
         [[135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81],
          [135,121,81]],
         [[67,68,29],
          [135,121,81],
          [67,68,29],
          [135,121,81],
          [135,121,81],
          [135,121,81]]]

import itertools

new_array = list()
for inner in array:
    new_inner = [x[0] for x in itertools.groupby(inner)]
    new_array.append(new_inner)

Produces:

[ [ [135, 121, 81] ], 
  [ [135, 121, 81] ],
  [ [67, 68, 29],
    [135, 121, 81],
    [67, 68, 29],
    [135, 121, 81] ] ]

Not quite unique, but you can sort inner to get only unique.

share|improve this answer
    
This doesn't do what the OP actually wants. For one thing, he has a 3D array, not a 1D one. For another, it's not sorted, and groupby will only eliminate contiguous runs of a value, not any previous-seen value. –  abarnert Jan 30 '13 at 2:26
    
quite possible, I'm not very clear on what the output should be. –  isedev Jan 30 '13 at 2:35
    
Sorting inner doesn't get only unique values; you still end up with 3 copies of [138, 121, 81]. Also, you're returning a 3D array, but the question specifically asks for a 2D array. –  abarnert Jan 30 '13 at 2:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.