current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other.

Join them; it only takes a minute:

Sign up
Join the Stack Overflow community to:
  1. Ask programming questions
  2. Answer and help your peers
  3. Get recognized for your expertise
up vote 0 down vote favorite

I am trying to "spoof" a Firefox HTTP POST request in Java using java.net.HttpURLConnection. I use Wireshark to check the HTTP headers being sent, so I have (hopefully) reliable source of information, why the Java result doesn't match the ideal situation (using Firefox).

I have set all header fields exactly to the values that Firefox sends via HTTP and noticed, that the sequence of the header fields is not the same. The output for Firefox is like:

POST ...
**Host**
User-Agent
Accept
Accept-Language
Accept-Encoding
Referer
Connection
Content-Type
Content-Length

When I let wireshark tap off my implementation in Java, it gives me a slightly different sequence of fields:

POST...
**User-Agent**
Accept
Accept-Language
Accept-Encoding
Referer
Content-Type
Host
Connection
Content-Length

So basically, I have all the fields, just in a different order. I have also noticed that the Host field is sent with a different value: www.thewebsite.com (Firefox) <---> thewebsite.com (Java HttpURLConnection), although I pass on the String to httpUrlConnection.setRequestProperty with the "www."

I have not yet analyzed the byte output of Wireshark, but I know that the server is not returning the same Location in the header fields of my response.

My questions are:

(1) Is is possible to control the sequence the header fields in the request, and if yes is it possible to do using HttpURLConnection? If not, is it possible to directly control the bytes in the HTTP header using Java? [I don't own the server, so my only hope to get the POST method working is through my application pretending to be Firefox, the server is not really verbose, my only info are: Apache with PHP]

(2) Is there a way to fix the setRequestProperty() problem ("www") as described above?

(3) What else could matter? (Do I need to concern the underlying layers, TCP....?)

Thanks for any comments.

PS. I am trying to model a situation without cookies being sent, so that I can ignore the effect.

share|improve this question
2  
The order of headers has no importance. – Sotirios Delimanolis Dec 26 '13 at 22:47
    
Thanks. So what else could mess this up? TCP? – Kovács Imre Dec 26 '13 at 22:49
up vote 2 down vote

First, the order of the headers is irrelevant.

Second, in order to manually override the host header you need to set sun.net.http.allowRestrictedHeaders=true either in code

System.setProperty("sun.net.http.allowRestrictedHeaders", "true")

or at JVM start

-Dsun.net.http.allowRestrictedHeaders=true

This is a security precaution introduced by Oracle a while ago. That's because according to RFC

The Host request-header field specifies the Internet host and port number of the resource being requested, as obtained from the original URI given by the user or referring resource (generally an HTTP URL).

share|improve this answer
    
Thanks, everyone for commenting on this. If the order of the header fields doesn't matter, well in that case the solution is not at HTTP level. I give Apache's Http Components a try, otherwise I will really need to mess around with the underlying protocol layers. – Kovács Imre Dec 27 '13 at 11:54
up vote 2 down vote

the headers order is not important. the headers got by server are also out-of-order. And you can not control httpUrlConnection header order. But if you write your own TCP client, you can control your header order. like:

clientSocket = new Socket(serverHost, serverPort);

OutputStream os = clientSocket.getOutputStream();

String send = "GET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\nGET /?id=y2y HTTP/1.1\r\nConnection: keep-alive\r\nKeep-Alive: timeout=15, max=200\r\nHost: chillyc.info\r\n\r\n";

os.write(send.getBytes());

The Second question is answered by Marcel Stör in the first answer. a

share|improve this answer
up vote 0 down vote accepted

I got lucky with Apache Http Components, my guess is that the "Host" header's missing "www." made the difference, which can be set exactly as intended using Apache's HttpPost:

httpPost.setHeader("Host", "www.thewebsite.com");

The Wireshark output confirmed my suspicion. Also this time the TCP communication prior to my HTTP post looks different (client ---> server, server ---> client, client ---> server) instead of (client ---> server, server ---> client, client ---> server, client---> server).

Now I get the desired Location header value and the server is also setting the cookies. :)

For the most part, this question is resolved.

Actually I wanted to use the lightweihgt HttpUrlConnection because that's what the Android Developers blog suggesting. The System.setProperty("sun.net.http.allowRestrictedHeaders", "true") might work as well, if it allows to "www." in the Host value.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.