How do I write a debugger for Arduino?

Assuming you have enough time and money to spend on this project. ($50 is a bargain for a debugger, believe me. About a decade ago I wrote an assembler/debugger suite for a very small niche market microcontroller, the Maxim Integrated MAX1464, using a combination of SciTe and Perl/Tk. Writing your own debugger is a great educational project, if you have the time and money to complete it.)

Tools

Assuming you already have all the standard Arduino debug tools, and just want to learn the process of making a debug tool.

Assuming you've used the standard Arduino debug tools before, and written enough programs (sorry, you Arduino guys call them "sketches" for some reason) so that you have some code you feel comfortable debugging on. Setting a breakpoint in a simple "blinking LED" program isn't very satisfying, better to set breakpoints in a test program that actually does something. You'll also want to have some branching logic pathways to test breakpoints, and some variables (counters perhaps) so you can test reading values through the debug interface.

Assuming you have a couple of spare Arduino boards, because very likely during development some bad debug commands may get sent. Ideally you should have a device programmer capable of restoring the Arduino bootloader image that normally resides in the Atmel AVR processor. Sending arbitrary commands to the debug interface could possibly erase or mess up the bootloader.

Assuming you have access to customary EE lab tools: DMM, oscilloscope, soldering iron, etc.

Research

Assuming you already know how to make an Eclipse IDE plugin... (if you don't, that would be a question to research on Google or StackOverflow.) I'm treating this the same as if you were writing a plain command-line debugger or a gdb driver.

I'm not really sure which debug protocol you need to implement. Sadly, JTAG doesn't seem to be supported by Arduino. The ICSP in-circuit system programming header might give debug access, though it would require attaching some hardware to the board that you're debugging. If I understand correctly, the Arduino IDE supports debugging through the USB port, which is a very attractive setup.

Assuming you plan to use Atmel's proprietary DebugWIRE protocol, which someone else has kindly attempted to reverse-engineer here: http://www.ruemohr.org/docs/debugwire.html Ideally you'd get the real documentation from Atmel. The hazard of a reverse-engineered protocol like this (aside from possible IP infringement) is that the protocol could change without notice. Not having direct access to the team that designed the microcontroller, means you have to rely on whatever documentation you can find and whatever else you can discover experimentally.

Assuming you can use the USB bridge chip's DTR# to drive (and possibly sense) the RESET pin, to access DebugWire. If that USB bridge chip is an FTDIchip.com device, you'll want to use the bit-banging mode to drive and sense DTR#. (In normal "RS232 mode" this pin is not bidirectional.) Go to the FTDIchip.com website to get the D2XX programmer's guide and example code.

Since DebugWire is a protocol that uses one wire for all signalling, both directions, timing is absolutely critical. You must use an oscilloscope to observe the DebugWire signals on the RESET pin, otherwise you're "flying blind".

That should be enough to get you started. Initially, you'll have to figure out how to discover an attached Arduino, enter debug mode, set a breakpoint, and reset the part back into free-running mode. The "blinking LED" test program is good enough to see some results for this early stage.

Cheers