How to wire a button for input on an Attiny85 using Arduino code?

I am trying to "shrink" my arduino project to an Attiny85. But for some reason I cannot get an input button to work on the Attiny85? To simply the circuit and to make sure I didnt make a mistake somewhere I took everything off except for a button and an LED. First I tried it on the Arduino and used the example from http://arduino.cc/en/tutorial/button to try a simple example. I changed the ledPin to 10 which I plugged into my breadboard, through a resistor, to the LED and to ground. I plugged the button into pin 2 on the arduino and connected the ground wire of the button to a 10k resistor, then to ground. All of that works.

So then before I upload it to the tiny I changed the ledPin to 0 or 1 (it works on either one - I uploaded a simple blink sketch without the button code to test the LED) and then left the button pin to 2 as is shown by the diagram here http://www.pighixxx.com/pgdev/Temp/attiny.pdf or http://hlt.media.mit.edu/?p=1695 but nothing happens when the button is pressed. If I reverse the logic so the LED is on unless the button is pressed, then the LED is always on.

I cannot figure out what I am doing wrong. The same wiring works with the arduino, but I cannot get the attiny to register the button press. The final goal with the button would be to use the attachInterrupt(0, increment, CHANGE); for the button, but I cannot get even a simple sketch to work.

Any help is much appreciated! Thanks in advance!

Edit1:

// constants won't change. They're used here to 
// set pin numbers:
const int buttonPin = 2;     // the number of the pushbutton pin
const int ledPin =  1;      // the number of the LED pin

// variables will change:
int buttonState = 0;         // variable for reading the pushbutton status

void setup() {
  // initialize the LED pin as an output:
  pinMode(ledPin, OUTPUT);      
  // initialize the pushbutton pin as an input:
  pinMode(buttonPin, INPUT);  
}

void loop(){
  // read the state of the pushbutton value:
  buttonState = digitalRead(buttonPin);

  // check if the pushbutton is pressed.
  // if it is, the buttonState is HIGH:
  if (buttonState == HIGH) {     
    // turn LED on:    
    digitalWrite(ledPin, HIGH);  
  } 
  else {
    // turn LED off:
    digitalWrite(ledPin, LOW); 
  }
}