GPF when sending emails too quickly

[useless]

Got a situation where we need to send lots of emails and I'm getting the following error message

Runtime Error
Do you want to GPF?
Yes No

Basically I'm calling a process which loops through all emails in the "outbox" and in the TakeRecord embed starting a new procedure with the email id passed as a string parameter.

The newly started procedure is a hidden window which looks up the email record from the passed parameter and then attempts to send the email and either sends and updates the email record to say its sent or keeps retrying according to the users requirements until it eventually fails and logs the failure.

If the email fails to send, the window stays active but hidden waiting on a timer events until its time to resend the email again so this window could be open but hidden for hours potentially and could have hundreds or thousands of emails potentially waiting to send and thus hundreds or thousands of windows open although yet to test this because of this problem with the GPF windows.

Now obviously the runtime cant handle starting so many hidden windows which is why I get the "do you want gpf?" message so is there any way I can use the SendEmail class without a window to get around the problems with the Clarion runtime?

I have considered putting a sleep(x) before each start in the TakeRecords embed but this puts the whole app to sleep so emails just build up and get delayed until the customer stops using the system.

Anyway around this problem?

Edit: The reason for starting a new window on a new thread is because it can take a a number of seconds sending each email (some have large attachments) and talking to someone elses email server over a broadband connection (not an ISP's email server) can be slow so to wait for each email one after the other just creates a build of emails waiting to send.

[kevin plummer]

I think I would move the resend logic and number of attempts to the DB Table. I think most SMTP servers detect and block when they are sent massive amounts of email at a time not to mention domains also detecting this, blocking the smtp IP and possibly ending up on a blacklist which is a major PITA. I would however refer to the NT docs as I think there are some techniques discussed in sending bulk emails.

[Bruce]

Hi Richard,

you have a reasonably challenging problem, because there are a number of things stacked against you.

a) probably the biggest challenge is the fact that pretty much all email systems are specifically designed to prevent "sending lots of emails". Obviously the vast majority of email traffic (around 96%) is spam, so when you behavior resembles that of a spammer (regardless of how legit your situation might be) you can run into trouble. So this is something you'll need to work around quite a lot.

b) In the Clarion Runtime there are two primary limits with regards to lots of threads;

b1) - Ram. Each thread consumes a fair bit of ram (for starters each threaded Class, and threaded variable gets allocated Ram on startup) so the number of tables in your dict can have a big impact. Obviously windows has a 2 gig limit for 32 bit programs - but in practice if you consume Ram very quickly Windows may cut you off before that. The typical symptom here is that it "just vanishes". While this is likely not your current problem, you will need to have some sort of limit controller to ensure that "no more than x threads are running".

b2) the second issue has to do with how threads start, and where you start them from and so on. These threads all have a window, so ideally the thread should be allowed to startup, and get as far as the ACCEPT command before another window opens. A simple global variable set just before a Start, and unset at the end of the INIT method will probably be enough. (This is almost certainly the cause of the current GPF). I don't think it's the number of windows that matters here, as much as the timing of them starting up.

The GPF is helpful if you have something like GPF Reporter installed, because it'll tell you what specific code is causing the GPF.

Personally I might be inclined to change the architecture somewhat.
Put all the emails waiting to go in a table. Then have n threads running where the thread reads a record from the table, sends the mail, and deletes the record, or if it fails, set's a time field in the record, and writes it back.

This has a number of advantages;
a) the program can stop, and start, and it'll pick up where it left off.

b) you have control over the volume of outgoing mail (ie by governing the number of active threads).
Ultimately there will be a limit at which adding more threads does not actually send the mail faster (probably because of band-width), so this allows you to find that limit.

c) emails that are undeliverable can be marked in the table, and after x retries abandoned.

d) it scales across multiple processes and multiple machines. ie if you have a machine with lots of Ram, you can overcome the 2 gig limit by running multiple "senders". And this architechture would also scale up to multiple machines if necessary.

Cheers
Bruce

[useless]

I was having a think of a way around this problem playing with the code last night, firstly the GPF is caused by the clarion runtime not being able to handle too many windows being opened at the same time, like you say a timing issue.

I had been experimenting with the sleep function http://msdn.microsoft.com/en-us/library/windows/desktop/ms686298%28v=vs.85%29.aspx and although it states the thread will be paused, this is putting the entire app to sleep so C6 threads are not acting in a true win32 threaded style although this is possibly caused by the drivers accessing the db concurrently.

The C6 help is a bit contradictory as well, it states towards the top "Instead, use the Windows Sleep API function, which releases the CPU and lets other apps in the system effectively multitask." indicating or suggesting the whole app goes to sleep but further down the page it states " A value of zero causes the thread to relinquish the remainder of its time slice to any other thread of equal priority that is ready to run." indicating it should be affecting the thread only, but in my tests it was putting the whole app to sleep which I suspect is down to the drivers because of the way they work with MS SQL server.

So with this in mind and the starting of too many windows too quickly causing the GPF, what I decided I needed to do was to create x threads which cycles through the "outbox" and where an email id ends with 1, that would be processed by the 1st email thread, email ids ending in 2 would be processed by the 2nd email processing thread and so on. This way I can control the starting of each thread and window to avoid the window GPF's. If emails still build up the user can increase the number of threads that should run, lets say 20 threads or 100 threads and process the relevant email based on the emails id's last number or two, ie thread 30 would process every 30th email id and thread 20 would process each 20th email id if there were 30 threads running. The users can change this in their settings but I've limited it to a max of 250 as like you say the ram requirements for each thread could kill the box. There is a setting where they state max amount of ram to use, then when I start a new thread lookup the processes ram usage when the thread sends its first email, if its ok ramwise it then starts the next thread and repeats until the maximum number of threads that are allowed to run have been started.

I've rehashed the process window so it loops through each email id it needs to and just remains open but hidden processing them and rehashed it to make a note of emails which need retrying and then retries them when the time is right from a timer event. The process window also has the sendemail class moved onto it as well for obvious reasons.

This also puts a limit of 10 threads (or whatever the user sets in their email sending settings) in place so I dont end up with hundreds or thousands of threads starting and waiting to do retrys as well which could cause problems if net access suddenly went down and the system wasnt able to send emails.

So thats how I've got around it, regarding spam restrictions affecting the sending, the email servers we are sending to are pretty varied so its rare for the same email server to receive multiple emails except for the usual short burst of communication between people because I do a DNS/MX lookup and use the SMTP servers in the DNS/MX lookup to send direct to the server that is receiving emails for the email address but it depends on the activity and communication going between my customer and who they are communicating with. It might be just one email sent or if there are questions there might be 3 or 4 emails sent to their server with possibly attachments which could be a few MB in size.

For backup the last SMTP server that is used to send through is the ISP's SMTP and any commercial smtp servers also setup in the system.

I've also got the system setup to handle SMTP server restrictions which might be applied by some ISP's/commercial smtp servers like only being able to send x emails every x mins, hours, days, weeks or month &/or x T/G/M/Kb sent per min,hour,day,week or month which seems common in the US but not as much in the UK and parts of Europe, BT for example has no limits on sending emails through their SMTP server but has a 10Mb size limit but you do have to make sure the domain names used as the senders email address is registered with them otherwise they block the email being sent.

So thats how I've handled it, I think all bases are covered so its just a case of letting it go live now.