NetWebService Question

[de la Rosa]

Hi Bruce,

1. I have a NetWebClient sending a 512-byte structure to a NetWebserver NetWebPage. On the WebServer side, I de-randomized incoming data to avoid choking the SQL backend  by buffering to a memory table and reading off the buffer at intervals. - I get memory error with this setup.

2. To limit memory usage, I limited the no. records to buffer anyway the clients re-sends after a few minutes if it doesn't get any confirmation back. - No Memory errors.

3. But when about 150 or more clients simultaneously sends data, the NetWebPage stops receiving data. Threaded Pool didnt make a difference.

Question:

1. Will a NetWebService be able to handle simultaneous data better than a NetWebPage?
2. If yes, can I override what method to run on the NetWebService (if necessary) so I don't need to change the URL being posted to by the Clients?

Thanks,
Vic

[DonRidley]

Hello Vic,

I have been neck deep in NetWebServices for several months now.  Just yesterday I was testing a disconnected desktop / server setup and hammering the server just to see what it could handle.  The "worst" effect I observed was a log entry of "slow response."  "Slow Response" was seen very rarely and the JSON response was still sent - just "slow."  If you set the JSON return response from "Readability" to "Compactness", it goes from really fast to super really fast.

Oh, this is also being executed via a NetTalk Multi Site Host setup. 

I don't if this rambling helps you Vic but it may give you seomthing to think about.

Don

[de la Rosa]

Hi Don,

Thanks for that information, that is the reason I switched from XML encoded to plain structure (String over data), hoping to cut processing overhead and size of the payload.

Is the multi site implementation handling different applications, or designed to distribute load?

Thanks,
Vic

[Bruce]

Hi Vic,

let's cover a bunch of things at once;

>>On the WebServer side, I de-randomized incoming data

I don't know what "de-randomizing" is.

>> to avoid choking the SQL backend  by buffering to a memory table and reading off the buffer at intervals. - I get memory error with this setup.

I don't think this "buffer" is a good idea. Firstly, you've probably got bugs in this code (hence the errors) - secondly I would consider it unlikely that your incoming traffic would "choke" a SQL backend. Especially for what presumably are "simple writes". Personally I would dispense with this architecture. As your volume increases you can simply add grunt to your SQL server machine. Thirdly you would need to manage ram consumption by the memory table - and memory is your most limited resource in the server. So first thing to do is throw this away and just process each incoming request directly to the SQL server.

>> But when about 150 or more clients simultaneously sends data, the NetWebPage stops receiving data. Threaded Pool didn't make a difference.

"simultaneously" is an interesting concept here.  Because each request should only exist for a small fraction of time (probably no more than a few hundredths of a second". You can monitor server load on the performance tab - the interesting thing you'd want to watch is the "Maximum number of threads" item.

Perhaps you can tell us more about how often each client is posting? do they do multiple requests per second? Do they sustain that sort of traffic.

>> Will a NetWebService be able to handle simultaneous data better than a NetWebPage?

no. They are both just "procedures" in the app. The server doesn't care which one is receiving traffic.

>> If yes, can I override what method to run on the NetWebService (if necessary) so I don't need to change the URL being posted to by the Clients?

the URL is just the procedure name, so you wouldn't change the URL anyway.

>> I switched from XML encoded to plain structure (String over data), hoping to cut processing overhead and size of the payload.

A string over data will be faster and smaller than XML or JSON encoding. BUT in the long run you will encounter major issues with taking this service forward in terms of expandability and backwards/forwards compatibility. I think you will encounter much pain if you persist down this road.

Overall it sounds to me like you're optimizing the wrong thing. All optimization requires compromise, and I don't think you will like where your chose compromises are taking you...

It sounds to me like the root of your problem is a SQL server box with insufficient grunt. That's where I'd focus my attention if I were you.

Cheers
Bruce

[de la Rosa]

Hi Bruce,

I don't know what "de-randomizing" is.

-- since clients requests arrives at random, processing demand will fluctuate from light to
heavy. By buffering, processing is done at set timer intervals to what the backend can handle therefore "de-randomizing" processing load (",).
The constarints are:
1. I dont have control over the backend as it's customer's facilities.
2. when I write directly to the backend, it crashes on heavy load.
3. Have migrated from NetSimple a few years back to NetWebPage to handle threads as you suggested back then. With NetSimple problems starts above 15 clients, with NetWebPage problems started above 130 clients.
4. Have developed a replacement app based on distirbuted data and will eventually switch however it's not exactly cheap and simple to do the upgrade of all the sites in one go. Also, it must have the Grunts you mentioned by then.
5. Have to keep the clients as-is for the time being that's why I asked if I can overide what method to run on the NetWebService, for example now the client posts to url IP:port/upload_log and it's not that easy to change that to IP:port/webservicename/upload_log due to access restrictions.   
   


"simultaneously" is an interesting concept here.  Because each request should only exist for a small fraction of time (probably no more than a few hundredths of a second". You can monitor server load on the performance tab - the interesting thing you'd want to watch is the "Maximum number of threads" item.

Perhaps you can tell us more about how often each client is posting? do they do multiple requests per second? Do they sustain that sort of traffic.

-- at closing hour the sites will generate statistical reports and starts trying to send the records of the report to the consolidating server at the head office, this time is heavy as all sites will try to send at the same time (if they are lucky not to have been swicthed off before they are able to send), otherwise clients will restart trying to send in the morning when they are switched ON. All sites that have not been able to send will keep trying to post every minute.     


It sounds to me like the root of your problem is a SQL server box with insufficient grunt. That's where I'd focus my attention if I were you.

-- Absolutely, in fact I'm afraid the distributed data approach also might fail because of it. Not exactly sure I understand what you mean by Grunt or if I can add them without getting the db admin involved. The backend is MSSQL. So far my info on Grunt for MSSQl is not for production use. Can you elaborate on how it will improve backend processing?

Thanks!
Vic

[Bruce]

Hi Vic,

>> when I write directly to the backend, it crashes on heavy load.

This is a case where fixing the actual problem, rather than trying to work-around it, would be a better idea.

>>I don't have control over the backend as it's customer's facilities.

You don't have to control the backend, but you can speak to them about minimum requirements.

That said;

If you want to buffer, buffer to a local database - either a local SQL or local TPS database, not in-mem driver.
Presumably you have control over that, and the machine the server is running on.

Or, set the clients to send at a random time spread out over the night. If users turn the machines off then train them not to do that (who does that anyway?) or send from a machine at the client facility that is not turned off. To scale up, I would do this anyway. If the machine is switched off then don't resend immediately on restart, send sometime in say the next hour.
If the customer really wants to get the reports "by morning" then tell them to stop turning the machines off.

>> Not exactly sure I understand what you mean by Grunt

ram mostly in this case, but it's a slang term for "power" - so more CPU or more Ram or more bandwidth depending on what lack is causing it to crash.

>> ... or if I can add them without getting the db admin involved.

GET THE DB ADMIN INVOLVED!!! WTF would you want to pursue this sort of project and not have the db admin involved??? That's what they are there for.

Cheers
Bruce

[de la Rosa]

Hi Bruce,

This is a case where fixing the actual problem, rather than trying to work-around it, would be a better idea.

You don't have to control the backend, but you can speak to them about minimum requirements.

--Right, I just want to make sure to address or understand whatever is causing the reciever to stop accepting data before asking for more resources because at that point it has nothing to do with the backend yet.


If you want to buffer, buffer to a local database - either a local SQL or local TPS database, not in-mem driver. Presumably you have control over that, and the machine the server is running on.

-- I have limited the buffer to no more than 10 records to limit memory allocation but ok, will keep on looking for weak/bug points. Yes, I have control over the server app machine but not the db server machine.

Or, set the clients to send at a random time spread out over the night. If users turn the machines off then train them not to do that (who does that anyway?) or send from a machine at the client facility that is not turned off. To scale up, I would do this anyway. If the machine is switched off then don't resend immediately on restart, send sometime in say the next hour.If the customer really wants to get the reports "by morning" then tell them to stop turning the machines off.


GET THE DB ADMIN INVOLVED!!! WTF would you want to pursue this sort of project and not have the db admin involved??? That's what they are there for.

-- Haha, I know! The main objective of auto generating reports and sending to head office in this case is to get the reports because before this the sites didn't submit them regularly. So despite the hiccups, we're still able to offer the solution they need. Luckily, over the years you have been there to help us deliver.

Thanks,
Vic

[Matthew51]

It sounds like each client is sending 100 or more records one at a time, as I can't imagine 150 x 512 bytes would overwhelm any SQL server.  You should switch your clients over to sending in much larger batches.  You cold then send these batches to your SQL server in batches too.  I've had success using both Logout+Commit as well as PROP:SQL and massive insert statements.  Any time I've tried to access 100+ rows one at a time the best result is just slowdowns.

The other option is to tell the clients to wait a few seconds between each send.  If they don't need the reports till the next morning then you don't need to rush to get it sent and can take your time.  If they turn off the computer then it's their fault the data takes a long time to get their.