Data integrity with TCP

[Wolfgang Orth]

Hello Bruce and all.....

I just had to learn that sending data (XML) back and forth between Client and Server can be erroneous. At least when sending those data across the Internet.

The XML I expect has to have an even number of records returned. In some cases I get an un-even number back, which in turn means the XML is incomplete.

In addition I find errors (wrong characters) in the returned data.

This server is just a proof-of-concept, so no one is harmed yet! But its good to know that I need to integrate some integrity testing, either a CRC or MD5. And error-logging.

... and I had hoped TCP would handle that for me......

Seems as if I am way too lazy! Spoiled by Clarion.

Bye
Wolfgang

[Bruce]

Hi Wolfgang,

>> I just had to learn that sending data (XML) back and forth between Client and Server can be erroneous. At least when sending those data across the Internet.

Sorry, not true. TCP/IP is an "error correcting" protocol, which means that if you get a packet the data in it is guaranteed correct. Packets arrive unaltered, and in order.

Are you taking into account the fact that your stuff may arrive split across multiple packets? If you're using a simple server, and simple client (as distinct from a netWebClient) then you need to handle the possibility of multiple incoming packets.

Cheers
Bruce

[Wolfgang Orth]

> Sorry, not true.
> TCP/IP is an "error correcting" protocol, ...
> Packets arrive unaltered...

Hello Bruce,

I tend to believe you!

However.... as you can see below, the returned packets for the identical request are different.

The first request was sent to the server directly. It does not matter whether the request is sent to either the IP-Address or the fully qualified domain name, as long as the port is included.
In this case, the server listens to port 8999.

This comes back then: (the line with ### END ### is by me, to make clear the added empty line in the secend packet)

HTTP/1.0 200 OK
Date: Mon, 29 Aug 2011 08:16:52 GMT
Server: NetTalk-WebServer/4.56
Expires: Sun, 29 Aug 2010 08:16:52 GMT
Content-Type: text/xml
Cache-Control: no-store, no-cache, must-revalidate, private,post-check=0, pre-check=0, max-age=0
Set-Cookie: SESSIONID=431273635
Connection: close
 
<?xml version="1.0" encoding="ISO-8859-1"?>
<kix>
</kix>
 
###################### END ############################

Because in some networks the firewall settings are quite restrictive and prohibit anything but the common port, I made this server also accessible on port 80.
Because in my little local server I already run some other webserver, I use the Apache as a reverse proxy, according to Bejamin Krajmalniks artucle on ClaionMagazine.
http://www.clarionmag.com/cmag/v9/v9n02proxy.html
This is the same as what you did with that Multi-Site.DLL in Nettalk 5.

The second request was now sent to the same server, but to that different port, in this case port 80. The Apache puicks the request and forwards it to the server, actualy running on port 8999.

And this is what gets back - note the extra empty lines:
 
HTTP/1.1 200 OK
Date: Mon, 29 Aug 2011 08:24:40 GMT
Server: NetTalk-WebServer/4.56
Expires: Sun, 29 Aug 2010 08:24:40 GMT
Content-Type: text/xml
Cache-Control: no-store, no-cache, must-revalidate, private,post-check=0, pre-check=0, max-age=0
Set-Cookie: SESSIONID=1681448488
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
 
2d
<?xml version="1.0" encoding="ISO-8859-1"?>
 
19
<kix>
</kix>
 
0
 
 
###################### END ############################

I tried several requests, in anyone the line displaying "2d" and "0" were the same, I have no idea what 2d and 0 might mean. That line holding the "19" here changes the value, apparently depending on the length of the returned data. Big responses of several records can have a 377 there, or bigger. Oviously something depending on the size.

I also had to learn that inside the returned strings some strange characters were sprinkled into.

I agree that my construction, using the Apache here, is quite unusual and I do not insist on that (however I like this idea to overcome restrictive FW-rules).

My point here is to advert to that it _is_ possible to get quirky data back, although I would not have expected that.

The good news at all is, that addressing the server directly works well, the returned data are intact.

Thanks for reading,
Wolfgang

[Bruce]

Hi Wolfgang,

whew. Fortunately your very clear answer means that TCP/IP is indeed not broken.

the key is in your two requests - they're different. The first uses HTTP 1.0, the second is using HTTP 1.1.
Consequently in the first case the response is not using Transfer Encoding, whereas the in the second case it is using Transfer Encoding = Chunked. The rest of the packet is formed the way it is because of this encoding.

Presumably it's your Apache setup which is converting it from an HTTP 1.0 request to an HTTP 1.1 request.

cheers
Bruce

[Wolfgang Orth]

> is converting it from an HTTP 1.0 request to an HTTP 1.1 request

Oh My Gosh! Thats a difference of just some shabby 10%!

What a picky piece of software......