Scratch, 150 – 125 = 25

I'm here for the fun, not the golf.

Edit: The scoring is based on this proposal at the moment. If another scoring system is generally adopted here, feel free to notify me or to edit this answer! For a break-down of this count, please see the comments.

Main program:

Individual sprites:

This is the sprite with number 0. The other sprites have the same script, except the number.

Play with it online.

Screenshot:

Note: Please do not press two buttons within 0.3 seconds.

Bash 318 297 281 273-125=148

This is primarily a response to "Text probably won't work"; the following text-based bash script runs fine in Konsole, gnome-terminal etc. on my Ubuntu 14.04 machine. To create the regions of colour it sets the text background color. In fact, adding text makes the game more accessible to color-blind players. To make the game yet more accessible it reads the characters that the player needs to press (it assumes that espeak is installed). It also assumes that the only file matching /d*/ur*/ is /dev/urandom. For the regions of color to be of non-trivial size you probably want to set the text size to be quite large.

To play press y, r, g or b as appropriate.

set -e;d(){
(tput reset;echo 'x1r09mRx2g10mG
x3y11mYx4b14mBx0m'$s|sed s/.$1"//
s/[rgyb]..//g
s/x/`echo -e '\E'`[48;5;/g")
};x(){
d $c;espeak $c;d j
};l(){
for c in $o;do eval $1;x;done
};f(){
o=$o\ `tr -dc yrgb</d*/ur*|head -c1`
l;l 'read -n1 i;[ $c = $i ];s=$[s+1]';f
};f

See also the original ungolfed version.

The screenshots below are when the yellow light is on and the player's score is 7. The screenshot on the right has been desaturated to simulate colour-blindness.

Mathematica, 409 - 125 = 284

k = 2;
p = Tuples[{0, 1}, 2];
f[c_, p_] := 
 EventHandler[{c, Rectangle[p]}, 
  "MouseClicked" :> (AppendTo[x, p]; Beep[]; g)]
h[R_] := (i = 1; 
  RunScheduledTask[
   H = If[OddQ@i, 
     Beep[]; {EdgeForm[{Thickness[0.02], Black}], FaceForm[], 
      Rectangle@R[[Ceiling[i/2]]]}, {}]; i++, {.3, 2 Length@R}])
s := (m = 0; x = {}; h[R = RandomChoice[p, k]];)
g := (m++; If[Take[R, m] != x, k = 2; s, If[m == k, k++; s]])
Dynamic@Graphics[{MapThread[f, {{Yellow, Red, Blue, Green}, p}], H}, 
  PlotLabel -> k]
s