PHP adding to array in while loop

I'm having some trouble adding to an array during a while loop and was wondering if any of you could help me. Firstly, some background. I am looping through some sql results and trying to gather the results while grouping by various ids to make it easier to deal with later. It just seems to be one line of code that isn't working. There is some code below

while($row=mysql_fetch_assoc($res)){
if(!array_key_exists($row['foreign_key_value'],$contacts)){
    $contacts[$row['foreign_key_value']]=array();
}
if(!array_key_exists($row['uid'],$contacts['foreign_key_value'])){
    $contacts[$row['foreign_key_value']][$row['uid']]=array();
}
$contacts[$row['foreign_key_value']][$row['uid']][$row['rating_id']]=$row['rating_value'];  

}

It is the last line I am having trouble with, where I am adding rating_id and rating_value. The data I am looping through is 4 fields - foreign_key_value, uid, rating_id and rating_value. The structure I want to end up with looks like

array(1) {
  [73]=>
  array(2) {
    [9]=>
    array(1) {
      [4]=>
      string(1) "3"
    }
  [1762]=>
    array(1) {
      [1]=>
      string(1) "5"
    }
  }

I just cannot get the rating_id and rating_value to create more than one key value pair in the last array, where I am expecting 5 pairs. The only thing I am getting is the last pair selected. I really have no idea why I'm not getting the data I need, can anyone help?

abc667 - you're spot on. Thank you very much. I've been staring at it so long I missed that and I'm starting to feel like an idiot now