NT11 - jFiles 1.72 - Fastmem - InMemory Driver

[Sibuya]

Hi,

Using Clarion 8EE I've made a desktop application using NetSimple Client to read Json from a RFID reader that I should process using InMemory Driver because reader sends a lot of packets very fast.

I was using jFiles to get fields data but application crashed with GPF when using jFiles. Adding Fastmem get worse. Sometimes trace gave me messages saying "JSONClass.HandleChar: Unexpected literal ("E") at position 3 ...." from json {"E":"000000000000000000883703","R":-7090,"A":1} and same message literal ("v") from json {"version": "18.615", "app_uptime": "9950001"}. I don't have GPF Reporter but seeing at registers values I saw that was callings from jfiles lib. I don't know if this occurs when another json message arrived when jFiles is still processing the old message.

I have threads (windows lauched by START) that write to memory tables, other that delete or update records and others that reads all data from memory tables.

IMDD says that is thread safe but when accessing tables from threads I got GPF.

Solved by doing the following:

- Not using jFiles stopped trace messages "Unexpected literal" and got lesser GPF. Replaced by SUB and INSTRING.
- All IMDD accesses sync'd with mutex got lesser GPF.
- When closing app with child windows opened accessing memory tables (emulating user behavior) got GPF. I have to not to use Fastmem and using MDI Synchronization template.

After all of this I got an application reasonable stable.

To process these fast json messages I got message from NetSimple/Process and pass to another thread to deal with memory tables in another thread.

Depending on quantity of tags that enters in the reader field it could generate hundreds of messages per second that could reach a thousand. In my tests an average o one or two hundreds per second.

Is there another way to process these fast messages?

Thank you.

Best regards,

Marcos Sibuya

[oggy]

One concept: use stringtheory instead of sub and/or instring.
For example:
1. StRFID.SetValue(RFIDInput)
2. extract what you want like this, and store all the data in localqueue
   StartPoint = 0
   EndPoint   = 0
FieldInQueue =  StRFID.FindBetween('starpoint','endpoint',StarPoint,EndPoint)
! or, for json:
 json.LoadString(StRFID)
 FieldInQueue = json.GetValuebyName('startpoint')
add(LocalQueue).....
After let say 1000 records, write those records to tps ....

P.S I have that kind of process. In one second there is about 500 inputs...
For now, just couple of times, the process failed...

Regards, Oggy

[Bruce]

Hi Marcos,

>> using NetSimple Client

Bear in mind that "text" coming from a device can easily be split into multiple packets. You will be processing in your .Process method, but you will need to find the "end of packet" marker in order to then pass the complete json payload to jFiles.

>> I was using jFiles to get fields data but application crashed with GPF when using jFiles.

I'd obviously love to see any (partial) JSON which causes a GPF. It should not GPF.
That said, the fact it GPF's suggests that you are either not parsing "complete" messages from the device, or possibly your techniques for multi-threading have flaws.

<< Adding Fastmem get worse.

I would not recommend adding Fastmem to this sort of setup until everything else is stable.

>> Sometimes trace gave me messages saying "JSONClass.HandleChar: Unexpected literal ("E") at position 3 ...." from json {"E":"000000000000000000883703","R":-7090,"A":1}

The exact log would be helpful. But it suggests the json is "invalid" in some way. There are a bunch of errors you might see in the log when it gets invalid Json - but none of those should cause a GPF.

>> IMDD says that is thread safe but when accessing tables from threads I got GPF.

I don't think it's IMDD which is your problem here - I think there are other threading issues in play. (For example how are you passing the incoming json to each thread etc.) You'd need to post an example app for more specific comment, but it sounds to me like it's probably you have thread-unsafe code in there somewhere.

>> - Not using jFiles stopped trace messages "Unexpected literal" and got lesser GPF. Replaced by SUB and INSTRING.
>> - All IMDD accesses sync'd with mutex got lesser GPF.
>> - When closing app with child windows opened accessing memory tables (emulating user behavior) got GPF. I have to not to use Fastmem and using MDI Synchronization template.

All of these sound like mitigations, which reduce the incidences of your root problem, but clearly aren't the cause of your actual GPF. It sounds like you're probably just "slowing things down" enough to make it more stable. But if you're still getting GPF's then clearly none of these is the cause.

>> To process these fast json messages I got message from NetSimple/Process and pass to another thread to deal with memory tables in another thread.

This step concerns me - lots of scope for problems here if you're not careful.

If you post an example app I may be able to give you more helpful (specific) advice.

Cheers
Bruce

[oggy]

I saw that you have GPF when using jFiles.
Couple months ago, i also had that kind of problem, just to realize that in my application, which is multi-dll, in dll where problem occurs, I did not check the flag on global template: "This is the part of Multi-DLL program".
Compile goes without errors, as expected, but, when I call procedure where jSon is, entire application crash...I did not telling that is your problem, but worth of checking, if your application is multi-dll
Regards, Oggy.

[Bruce]

correct. The setting is correct in DLL apps - but needs to be explicitly set in the Data DLL and EXE apps (in multi-app systems).

cheers
Bruce

[Sibuya]

Hi Oggy and Bruce,

Attached is my application. Sorry but it's portuguese but if you mind I could translate to better understanding.

Application is a standalone exe application (lib) not using multi DLL.

In dictionary you'll see MSSQL table Portal and ASCII file Template that are used to record results after processing in Memory tables. MSSQL or ASCII are optional to the user and using ASCII for tests.

Application will open Window "Leituras" using Net Simple to get messages from RFID reader that sends RFID tags information in one structure saying from what antenna read the tag.

All messages are sending with CR/LF at the end of each one and does not come more than one structure in one message.

Processing consists on determine tag direction of travel getting in's (Entradas) and out's (Saidas) from a portal with 4 antennas two for input and two for output. I'm doing this in memory tables. I have to determine direction of travel and after some time (configured) I have to delete from memory tables using another window named "Vida Tag" or "Tag lifetime".

There are more two windows that the support and install team can "monitor" what happens with memory tables in timed fashion about aproximately one second loading memory tables to a queue to be shown on screen.

In Net Simple Parse routine you can see that I commented old routine using Json.

Oggy, I was thinking to use queue but I have to constantly check tag's life to delete and show in browsers all modifications on-line making multi thread access to memory tables. So I decided to use IMDD. I had another application that I have to read serial port at fast pace and notice that string theory and json are very heavy loosing some data so using subs and instrings was a little more faster. Thank you for advices.

Bruce, RFID reader manufacturer support here in Brazil told me that the reader is sending one message structure terminated with CR/LF for each tag. So I put a trace on each message received in Net Simple and confirmed that. My test base was to show to the reader 40 tags at once so it generates an avalanche of 40 tags and repeat each tag every second. This is a reader configuration that depending on the other configuration could send me a constant avalanche os tags that could reach hundreds per second. I prevent user not to use this mode.

Seeing GPF messages registers against map table I could identify that came from a Json code area.

In final version I removed Fastmem and Json.

Atttached is my application.

I wish that you all had a very Happy Christmas and have a very good New Year!

Thank you all for support.

Best regars,

Marcos Sibuya

[Bruce]

Hi Marcus,

>> Bruce, RFID reader manufacturer support here in Brazil told me that the reader is sending one message structure terminated with CR/LF for each tag.

yes, just do not be confused between the term "Message" and "Packet".
They are sending "messages" on at a time, but just because they send them like that does not mean you will _receive_ them like that. Networking equipment is allowed to (and certainly does) add multiple messages together into 1 "packet" and they are also allows to split a message in to multiple packets.

So you must take into account that 1 message <> 1 packet. That is critical.

>> So I put a trace on each message received in Net Simple and confirmed that.

You can test till the cows come home, and it will _usually_ be 1 packet = 1 message - especially in developer testing. In the real world you _must_ allow for messages <> packets though, or you will have problems once you roll it out.

Cheers
Bruce

[Sibuya]

Hi Bruce,

Thank you for advice. You are right I agree with you.

TCP stack assembles network packages and could send truncated large chunk of data up to interface metric that is in general 1500. Should I consider that NetSimple in Process routine and NET:SimplePartialDataPacket could get a Packet truncated or you wait up to a end delimiter or a timeout to close a Packet and make it available to the application to parse?

Considering that the Package could have many Json structures separeted byt CR/LF could Json crash due max size or any other invalid character in the Package? Or even invalid structures inside due corrupt Package? I'm delimiting Max string size with 499 bytes.

Thank you.

Best regards,

Marcos Sibuya

[Bruce]

Hi Marcos,

In NetSimple you get whatever comes in, in the .Process method. There's no "protocol" in play at that point so no way for NetSimple to detect an "end of message". Where there is a known protocol (like web client, which then calls .PageReceived)it does it for you - but in this case you;re using Netsimple (and you need to because there does not appear to be a known protocol in place.)

>> Considering that the Package could have many Json structures separeted byt CR/LF could Json crash due max size or any other invalid character in the Package? Or even invalid structures inside due corrupt Package? I'm delimiting Max string size with 499 bytes.

no, it should never crash. If you have an example of any text that makes it crash I would be most interested in that. It may be "invalid json" in which case it will most likely fail to parse it correctly, but it should never crash.

cheers
Bruce

[Sibuya]

Hi Bruce,

Unfortunally I don't have the reader to test because it was deployed. I will ask for another one when available to test and let you know.

Thank you.

Best regards,

Marcos Sibuya