Encoding of ASCII string in UTF8 XML document in Byte array

Never mind how the XML document is encoded for transmission. The right way to do what you want to do—encode certain non-ASCII characters so they survive the trip unscathed—is to use XML character references to represent the characters that need to be so preserved. For instance, your

ÐØë

is represented using XML character references as

ÐØë

The receiving [conformant] XML processor will/should/must convert those numeric character references back to the characters they represent. Here's some code that will do the trick:

public static string ConvertToXmlCharacterReference( this string xml )
{
  StringBuilder sb  = new StringBuilder( s.Length ) ;
  const char    SP  = '\u0020' ; // anything lower than SP is a control character
  const char    DEL = '\u007F' ; // anything above DEL isn't ASCII, per se.

  foreach( char ch in xml )
  {
    bool isPrintableAscii = ch >= SP && ch <= DEL ;

    if ( isPrintableAscii ) { sb.Append(ch)                             ; }
    else                    { sb.AppendFormat( "&#x{0:X4}" , (int) ch ) ; }

  }

  string instance = sb.ToString() ;
  return instance ;
}

You could also use a regular expression to make the replacement or write an XSLT that would do the same thing. But the task is so trivial, it doesn't really warrant that sort of approach. The above code is probably faster and less memory intensive and...it's easier to understand.

You should note though that since you want to preserve two different encodings in the same document, your conversion routine will need to differentiate between the conversion from "extended ASCII" to an XML character reference and the conversion from "ISO Latin 1" to an XML character reference.

In both cases, the character reference specifies a codepoint in the ISO/IEC 10646 character set — essentially unicode. You'll want to map the characters to the appropriate code point. Since string in the CLR world are UTF-16 encoded, that shouldn't be much of an issue. The above code should work fine, I believe, unless you've get something really oddball that doesn't play very nicely with UTF-16.

So.. System.Text.Encoding.ASCII.GetBytes(string) will probably do what you want.. convert a string into an ascii-encoded byte array.

You simply can't have 208, 216, 235 byte sequence in UTF-8 encoded string/byte array.

I hope you can save XML as ISO 8859-1 with or without mentioning it in XML <?xml version="1.0" encoding="XXXXXXXXXX"?> processing instruction (maybe even specifying invalid UTF-8 encoding in XML header).

Otherwise if your requirements are as you stated - just ask for exact expected byte array for given input and craft your own custom serialization (or maybe custom encoding, also not sure if it is possible).

The document must be encoded in UTF-8. The Lastname field only allows ASCII. City only allows ISOLatin1. The message must be put on the (IBM Websphere) MessageQueue as a IBytesMessage.

If that is the precise specification, then I think you might be misunderstanding it. Your task is not one of encoding, but one of validation/fallback. The entire document – including the Lastname and City fields – must be encoded as UTF-8. Quite simply, the XML document would be invalid if it declares its encoding as UTF-8 and then contains byte values that are not valid under that encoding.

Conveniently, ASCII overlaps with the first 128 codepoints of Unicode; Latin1 overlaps with the first 256.

To check whether Lastname can be represented as ASCII, then you could check that all its characters have codepoints within the 0–127 range.

bool isLastnameAscii = foo.Lastname.All(c => (int)c < 128);

To conform with your specification, you would have to force invalid characters to fall back to the replacement character (typically ?) by encoding the string as ASCII, and then decoding it back:

foo.Lastname = Encoding.ASCII.GetString(Encoding.ASCII.GetBytes(foo.Lastname));

Similarly for City:

bool isCityLatin1 = foo.City.All(c => (int)c < 256);

Encoding latin1 = Encoding.GetEncoding("iso-8859-1");
foo.City = latin1.GetString(latin1.GetBytes(foo.City));

Subsequently, you should just save everything as UTF-8.

My assumption is that your third-party software can correctly decode the XML document using UTF-8; however, it must then extract the Lastname and City fields, and use them somewhere where only ASCII and Latin1 are allowed. It imposes the restrictions on you in order to ensure that it would not be forced to incur data loss (because of the presence of disallowed characters).

Edit: This is the workaround that you’re proposing. I’m using Latin1 in the place of “Extended ASCII” because the latter term is ambiguous.

var x = new foo() { lastname = "John ÐØë", city = "John ÐØë", other = "—" };

using (var stream = new MemoryStream())
using (var utf8writer = new StreamWriter(stream, Encoding.UTF8))            
using (var latin1writer = new StreamWriter(stream, Encoding.GetEncoding("iso-8859-1")))
{
    utf8writer.WriteLine("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
    utf8writer.WriteLine("<foo>");
    utf8writer.Flush();

    latin1writer.WriteLine("  <lastname>" + SecurityElement.Escape(x.lastname) + "</lastname>");
    latin1writer.WriteLine("  <city>" + SecurityElement.Escape(x.city) + "</city>");
    latin1writer.Flush();

    utf8writer.WriteLine("  <other>" + SecurityElement.Escape(x.other) + "</other>");
    utf8writer.WriteLine("/<foo>");
    utf8writer.Flush();

    byte[] bytes = stream.ToArray();
}

SecurityElement.Escape replaces invalid XML characters in a string with their valid XML equivalent (e.g. < to &lt and & to &).