There are two parts of the semantics you will need to describe:
Static semantics usually take the form of type systems. I'd recommend looking at the books by Benjamin C Pierce and Robert Harper. Alternatively, you could write the rules of well-formed programs as a collection of (formal or informal) statements describing the validity conditions of your program. The formal approach taken by type systems provides a well-known framework for such specifications.
There are many formalisms for expressing the dynamic semantics to consider as your basis:
- denotational semantics --- gives meaning in terms of semantic domains
- operational semantics --- gives meaning in terms changes to an abstract machine states
- axiomatic semantics --- gives meaning in terms of assertions about the state of the machine
- algebraic semantics --- semantics is defined in terms of algebraic laws.
There should be plenty of books out there on these topics. A good broad introductory one is
Nielson and Nielson's Semantics with Applications. A downloadable version is available from the authors' site.
There is also a newer edition, but I haven't seen it.
Within each of these, there are choices to make. For example, if you decide to use operational semantics, there are large-step semantics, small step semantics, semantics based on labelled transition systems, semantics based on rewriting logic.
What kind of approach you should take will depend on what kind of language your DSL is, what kind of experience you have with designing semantics, and your personal preference.
Without any concrete details, I can help no further.
