Webservices : how to detect "null" values when import json/xml data into queues

[AtoB]

Hi all,

(maybe this is more xFiles/jFiles related question ...)

whenever I import xml or json into a queue I nicely end up with all records in the Queue, but is there a way for me to detect wether an element was omitted or "null" when it imported into the queue?

a party is sending me an array of "objects" (records) I have to update, but it makes quite a difference wether some object properties are omitted/null or not (those fields need not be updated). I now end up with zeroes (for numeric fields) and empty strings, but I can't write these if the json didn't specify these fields ...

json is my top priority, but handling this via xml would be nice too

TIA,
Ton

[Bruce]

Hi Ton,

A clarion structure (group, list, variable etc) cannot tell the difference between a "blank" value and a null value. So if you test the variable itself you can't really tell the difference.

In order to know if the field existed in the original json or xml you would need to inspect the actual xml or json. That might get a bit tricky if the incoming structure is a list (since some records might have it, and some not) but I guess it's do'able.

If the field was omitted completely then I guess you could
a) load the structure from the json/xml (just to get the primary key values)
b) load the record off the disk and then
c) load the structure _again_ but without CLEARing the record. (I'd need to dig into the specifics for xFiles / jFiles if you were using those.)
d) put the record back

That way you'd have the "starting record" and add all the changes to that.

There's no "null" in XML, so if the field was omitted then it would simply not be updated in the record. If it was in Json, but set to null, then there may need to be a tweak to make that mean "don't update the field".

Cheers
Bruce

[AtoB]

Hi Bruce,

today I've studied the jfiles objects a little and decided that I could do what I want in the ".Fillstructure" procedure/method, so I tried to override the ".FillStructure" method of the JSONCLASS, by creating my own json class (inheriting from JSONCLASS).

This is not easily done I noticed: the class uses JSONCLASS and &JSONCLASS in lots of places (that all need to point to "my" class now) so I ended up overriding lots of methods (GetByName, GetObject, Get ...). This seems a little impractical

Is there an easy way to override this single method (.FillStructure) somehow, so I could modify it for my own needs? I try to keep away from modifying the base classes itself ..

(I'm thinking of providing the class with a reference to a Queue, in which I store alle ordinal Queue fields of the target Queue (that's used for loading the json) that are omitted (nIndex = 0 in .FillStructure) for each queue entry. Then adding a procedure .IsNull(QueuePosition, OrdinalFieldPosition) to the class and presto !)

regards,
Ton

[Bruce]

Hi Ton,

>> This is not easily done

no, it's not easily done at all. The JsonClass is "recursive" in the sense that it's a tree, where each node is a JsonClass, so overriding becomes very tricky.

If you check out the class you'll see a number of methods start with a check of the parent object. (ie they "cascade" themselves up the chain so the code in the topmost node runs. See .FormatValue for an example of this.

!---------------------------------------------------------------
JSONClass.FormatValue          Procedure(String pName, String pValue, *LONG pLiteralType)
  CODE
  if not self.parentObject &= Null
    ! cascades up so that embed code in the root object will execute.
    Return self.parentObject.FormatValue(pName,pValue,pLiteralType)
  else
    Return pValue
  end

This allows embed code in the uppermost FormatValue to run even though the actual object is much lower down.

If you needed to alter FillStructure then it would need a similar preamble, so you could embed in the procedure itself.

But this begs the question - what are you trying to do there? and why? Tell me more about the overrides you are making and maybe there's an alternative way.

cheers
Bruce

[AtoB]

Hi Bruce,

I'm developing an api layer where several parties can make use of.

I've got "tables" with rather large record structures that are exposed in this method for update action. The clients are supposed to supply only fields that need updating (for several reasons: saving bandwidth, not overwriting "old" data, less programming effort on the client side and different clients "own" different part of the record structure (different access rights))

I the "Primeparameters" section the Queue (which is my "target" structure) nicely gets filled from the json (array) that clients supply and I'm processing it afterwards manually in the buildresults section/routine. But after the queue is filled there is no way for me to detect omitted/null values.

So I dived into jFiles and saw a great opportunity in the .FillStructure procedure (where nIndex = 0 for a field if it is omitted) where I could it most efficiently (IMHO) without reprocessing the original json later on.

I probably would tie an external queue to the jsonclass and add entries for each record/field tuple that is omitted so I could later easily test for presence of these omitted entries ...

but maybe things can be done much simpler/faster ...

[Bruce]

Hi Ton,

I might be tempted to argue that your API design needs work.

In other words, if User Type A needs a method, then give them a method. If user type B needs a method (on the same table, but with "different clients own different part of the record structure" then perhaps that is a separate method.
Just trying to _document_ what goes where sounds like a nightmare to me.

That said;
I've thought a bit about your suggestion, but it gets very bogged down with lists. Nothing especially elegant is popping out at me. I've come up with a few suggestions in my head, but ultimately they all get very messy. And ultimately it comes back to the API design.

My biggest concern is not the omitted or null fields - that's probably solvable with priming of the fields with known "default" values. Rather my big concern would be security - ie what to do if the user sends you fields which they _shouldn't_ be sending. That's the key problem to solve first.

And that's where your choice of structure gets important. Using the table as the root structure, and trying to manage "who can write what fields there" is going to get ugly. So I'm back to where I started - perhaps it's best to create multiple API Methods, and in each use a Queue for the incoming list. Then manually move that queue into the table, taking security into account. This will likely solve the omitted issue as well because you could prime the queue records with "dud" values first - for example the name field might be primed to AAAAAAA so you can then see that primed field, and ignore it, when updating the table.

Cheers
Bruce

[AtoB]

Hi Bruce,

>I might be tempted to argue that your API design needs work.

you could always become a diplomat if programming doesn't work out any more :-) And you're probably right too: there is lots of room for improvement in my design.and creating multiple methods partly solves the access rights things too. But (being stubborn as always):

- I really would like to keep all things in one place (easy maintainance)
- the access rights are already taken care of (and result in errors from the backend), (docs is a thingy though ...)
- when the clients only specify modified fields there are no "dirty values" coming in (suppose both client one and two read field1=0 and field2=0 and afterwards client one sets/returns field1=1 and client two sets field2=1 (and both leave out the other field) there are no side effects to both update actions . If client1 returns field1=1, field2=0 and client2 returns field1=0, field2=1 then it depends on who came first, what the final result will be ...
- saving bandwidth is really important too

I've thought about the "dud" values too, but I first thougth of walking through all fields of the queue/group structure and setting things to "null" explicitly somewhere afther the clear(gr) in the .load method, but your "AAAAAA" brought me to something better perhaps, maybe you could incorporate into jFiles (and xFiles too?):

introduce two new equates in jFiles.inc:

jF:ClearNormal EQUATE(0)
jF:ClearHigh EQUATE(1)

then add an extra property to the JSON object:

ClearMode    BYTE

and then modify the following in the .load method:

!---------------------------------------------------------------
! Load JSON Object to Queue
!---------------------------------------------------------------
JSONClass.load  Procedure(QUEUE pQueue)
json         &JSONClass
result       long
x            long
gr           &group
match        long
  CODE
  self.Action = jf:Load
  self.Using = jf:Queue
  self.Q &= pQueue
  self.CascadeUp()  ! copy current properties up the tree
  if self.FreeQueueBeforeLoad
    free(pQueue)
  end
  gr &= pQueue

  loop x = 1 to self.records()
    json &= self.Get(x)
    case self.ClearMode
    of jF:ClearNormal
      clear(gr)
    of jF:ClearHigh
      clear(gr, 1)
    end
    ....

I think this would get me going. I now only have to set the .ClearMode property and change all my Queue elements to at least 4 character strings and then I can hanlde omitted values and real "null" values too!  It's fast and doesn't break any existing code (as far as I can tell ....)

Maybe the ".CascadeDown" method also need the following line (so you only have to set this property on the topmost instantiation of the recursive objects ...):

SELF.ClearMode         = pParent.ClearMode

What do you think of this?

(if you don't want to incorporate it, I'll do it myself anyway :-) )

regards,
Ton