Problem with Mandrill Webhooks and large POSTs

[bshields]

Hi All (Probably Bruce),

I've got a tricky problem.

We use Mandrill to transport large volumes of emails roughly 200k emails per month.

Mandrill has a webhook where it POSTs back into our system some events in json (but URL encoded).

Nothing weird so far.

If the post data is large, say > 1Mb of json data (URL Encoded) the web server maxes out a CPU (and will continue to do so if more POSTs arrive). This does not happen if the POSTs are small (kilobytes, not megabytes).

I think if we wait long enough (hours) the thread may complete what its doing, but Mandrill has given up waiting for a response.

Important observations are:

1. Happens in NT10.40 & Clarion 10
2. Happens in NT11.30 & Clarion 11
3. Doesn't get to the NetWebPage code that is waiting to process the data (does get there if the payload is small).
4. If Ii simulate the POST via Postman it fails in the same manner as a POST direct from Mandrill.
5. If I decode the URL encoding and simulate the POST via Mandrill everything works as expected.

I suspect the built-in URL decoding system in NetTalk is not happy with the obscene amount of POST data that is requiring decoding > 1Mb.

I've attached a cut down APP that will hopefully compile (C11/NT11.30 - you might have to cut out some missing icons etc). But its just an empty web server with 1 Proc called MandrillHook (which doesn't even execute if the payload is large).

More importantly I've included a postman collection with the original post as it arrives from Mandrill that fails and another post that does the same thing but as i've already URL decoded the payload (and replaced & with + just to make sure it behaves the same as the first post) it succeeds.

Any help would be appreciated!

Regards
Bill

[Bruce]

hi Bill,

thanks for the example. I was able to compile it, and I set it to listen on port 88, plain HTTP.
I then imported the postman collection into postman, and changed the URL for Mandril Hook 1 and 2 to
http://127.0.0.1:88/MandrillHook

I clicked on Send for both hooks, and it seemed to work fine.
the first hook comes through as URL-Encoded data; about 1.2 megs in size.

content-length: 1269145
Connection: keep-alive

mandrill_events=%5B%7B%22event%22%3A%22open%22%2C%22ts%22%3A1592530510%2C%22user_agent%22%3A%22Micr ....

the second comes through as Json data, and is a bit smaller (840K ish)

content-length: 842397
Connection: keep-alive

mandrill_events=[{"event":"open","ts":1592530510,"user_agent":"Microsoft+Office\/16.0+(Microso...

I set the NetShowSend=>1 so I can see the reply in debugview, and in both cases I'm seeing
HTTP/1.1 200 OK
...
<success>1</success>
 
So that's all good and at least indicated there's nothing fundamentally wrong here.

I notice though that these might fall into the "small" category as being < 1 meg JSON data?
I'm assuming on your side though they are proving big enough to cause a problem.

the one thing you don't mention is which version of StringTheory you are using?
but that said, I'm not sure it matters. It would do more work (URL decoding) for the one that works for you, and no work for the one that does not. That said, worth checking I guess. I'm not sure if it does a JSONDecode on the incoming JSON, but there was a change in that method not so long ago.

On the up side it is at least working here, so we should be able to track down the difference. If you don't have any joy let me know and I can TeamViewer in and have a look on your machine.

cheers
Bruce

[bshields]

Hi Bruce,

Thanks for taking a look.

I realised (after posting the example) I probably have access to your code to look for whats causing it.

Like all things once its reproducible its practically solved, and at least it is reproducible here!

I'll let you know how I go.

Regards
Bill

[bshields]

Hi Bruce,

I traced it through into the NetWebServerWorkerBase._UnEscape procedure.

Specifically the "Check for Hex" section.

Given in my example there are 213374 hex codes that need translation, its not unsurprising it takes some time.

I've tweaked the procedure to use StringTheory's Replace function and in my example its reduced the duration from 360 seconds to 50 seconds.

Frankly, I think that will get me out of trouble.

I also downloaded NT11.39 to ensure nothing has been changed in NT since I last looked, and that code is identical to NT10.

Regards
Bill

[Bruce]

Hi Bill,

try changing the code in _UnEscape to use StringTheory URLDecode instead of replace.

I've refactored the method to read;

str  StringTheory
  CODE
  str.SetValue(clip(p_string))
  if band(p_Flags,Net:NoHex) = 0
    if band(p_Flags,Net:NoPlus) = 0
      str.UrlDecode('%','+')
    else
      str.UrlDecode('%','')
    end
  elsif band(p_Flags,Net:NoPlus) = 0 
    str.Replace('+',' ')
  end 
  if band(p_Flags,Net:NoColon) = 0
    str.Replace('__',':')
  end 
  return str.GetValue()

I'd be interested in feedback regarding time for this code versus time for a straight replace.

Cheers
Bruce

[bshields]

Hi Bruce,

The URLDecode method in StringTheory is the same as the _Unescape in NetTalk, so no faster.

Fortunately, StringTheory's Replace is very fast. I got more speed out of it, by using Replace for the popular characters I had to decode, then letting URLDecode handle the rest.

Being JSON, it was lots of [,],{ and }. Hundreds of thousands of them.

Ultimately, it was still too slow. As Mandrill keeps throwing packets at me.

My ultimate solution is crap (but effective) and frankly I was desperate. I modified _AddSetting to not automatically unescape the p_setting string in the event the parameter was clearly from mandrill.

The process i use to parse and process the JSON is decoupled from the POST function. I save the incoming JSON to disk into a queue and have a daemon that processes them in an orderly fashion. I just added Unescaping to the daemon.


To answer your question, using Replace on [ ] { } , then URLDecode (for the scraps) took the time from 180 seconds to 50 seconds.

Regards
Bill

[Bruce]

Next step is to try with StringTheory 3.27
In my testing here, with your sample, it's under a second...

I'll be uploading shortly. [update - it's now uploaded]

cheers
Bruce

[bshields]

Awesome. I'll give it a try.