endless loop done ftp demo nt 7.07

[Larry Sand]

In trying to replicate a problem using the ftp client we found one thing with the demo that may be related. The problem we see occurs when we put a large file 30+MB and when we get the "done" from the client we send a rename command and it triggers the error trap being called 3-60 times.  Sometimes when it's done the file is actually renamed. 

So In the demo connect to a server and put a file and leave the "Remote File" blank. 

I've attached a debugview capture.

Still trying to recreate the problem we see.

Cheers,
Larry Sand

[attachment deleted by admin]

[Bruce]

Hi Larry,

just FYI -

when big files are being sent or received NOOP commands are sent on the Control channel to keep the channel alive.

Different FTP servers deal with NOOP commands quite differently. So some servers _need_ NOOP, others ignore NOOP while files are transferring (and give you a bunch of errors at the end), others return a different code for NOOP - it's all over the map.

Your best bet might be to wait and see if the errorTraps come _before_ the rename, and then swallow all of them until they're done, then do the Rename.

cheers
Bruce

[Larry Sand]

Thanks for that Bruce,

It turns out that it is the NOOP and their processing that's causing the problem.

On a server that ignores them while processing they all come through after .done() is called for the putfile.  The problem is that the class and the example code rely on the ._command to know what command is executing. 

Since there are an unknown number of these queue up, it seems that the only way to know that replies are really replies to the ignored NOOP is to parse the reply.  Is it "standard" for ftp servers to reply with "200 Command NOOP okay"?  I'm guessing not.

>>"and then swallow all of them until they're done"

So how do you know they're done?  There's no record of how many NetTalk sent.  Picking some random time delay typically is a recipe for failure.  Maybe you could keep a counter of how many you sent, received, or ignored? Any other ideas?

Larry Sand

[Bruce]

Hi Larry,

>> Is it "standard" for ftp servers to reply with "200 Command NOOP okay"?  I'm guessing not.

Your guess would not be inaccurate. It's all over the map. Some respond with 200, some with 226, some immediately (ie the server is multi-threaded), some wait to the end, some spit an error, some do other wierd stuff. It's a real barrel of laughs...

which specific server are you using? maybe I can set it up here and duplicate the issue?

cheers
Bruce

[Larry Sand]

Hi Bruce,

>>which specific server are you using? maybe I can set it up here and duplicate the issue?

Well that's the problem, we do not control the servers our customers use.  This server sends the noop responses as a multi-part response after the file is received.  We see various levels of the symptom on a number of them.  I can setup access on a specific server for you but I still need a generic solution.  I suppose we could interrogate the server by sending it a noop so we can see the response.

If they can return 226 then handlePutFileReplies() will just call done and with 200 reply code it attempts to ignore it. 

What do you think about the object counting the number of noop it sends and receives.  At least then we'll have a chance to know that there are a bunch of replies outstanding and can time out when we cannot know when it's safe to proceed with more actions on the connection.

Larry Sand

[Bruce]

Hi Larry,

>> We see various levels of the symptom on a number of them.  I can setup access on a specific server for you but I still need a generic solution.

The way to a generic solution is one server at a time. There was a famous quote from Netscape once that their ftp code stretched to 5000 lines which seemed ridiculous. But it's an indication of how messed up the ftp spec is. It's one of the reasons I love the FTP protocol so deeply.

There are lots of servers out there, and there are _lots_ of different behaviors in all kinds of situations. There's a ton of code in NetTalk FTP which detects, and caters for various behaviors. And the only way to do that is to add code for each server behavior as we encounter it.

cheers
Bruce

[Larry Sand]

Hi Bruce,

Ok, so here's one more server then.

I modified the demo program and attached it.  It'll rename the file one timer tic after the done method is called for the putfile.  Also the "Refresh List" button will look for a file that doesn't exist.

* Start the FTP Client Demo
* Connect to the server (i'll send server info)
* Select a large "Local File" 100MB or so should work.
* Set the "Remote File"
* Press the "Refresh List" button, this is modified to look for a file that will not be found
* Press "Put"

Also attached the end of the log when the client finishes sending the file.

Thanks,
Larry Sand

[attachment deleted by admin]

[Larry Sand]

Hi Bruce,

Do you have everything you need to reproduce this?

Thanks,
Larry Sand

[Bruce]

I won't know till I try, and I haven't had a chance to try yet Larry....

[Larry Sand]

We wrote a workaround and have it working at thousands of sites now.  Just cannot use event driven programming with the FTP object.   Let me know if you ever get a chance to look at it since I don't like the kludge.

[Larry Sand]

Bruce,
>>
I won't know till I try, and I haven't had a chance to try yet Larry....
<<

It's been a year, did you reproduce this yet or shall I toss this in the pile with all the reports to SV?

Larry Sand

[Bruce]

ouch!
but point noted!

[Larry Sand]

Sorry Bruce, that was a bit coarse.
I was just annoyed by folks asking for examples and spending a bunch of time making them to never have them reviewed.

[Graham]

Hi Larry

Don't know if this is gonna help you at all but Bruce and I went over something similar quite a while ago.  He suggested a few tweaks to the noop procedure which worked for us.  Code below is in NetSimp.clw in libsrc folder - my changes commented with ! Graham


!------------------------------------------------------------------------------
NetFTPClientControl.NoOp Procedure  ()
  code
  self.log('NetFTPClientControl.NoOp','self._Command=' & clip(self._Command) & ' self._ControlDone=' & self._ControlDone & ' self._DataDone=' & self._DataDone)
  case lower(clip(self._Command))
  of 'getremotefile'
  orof 'putfile'
  orof 'appendtofile'
  orof 'getdirlisting'
    ! allowed to send a noop when busy with data port commands, _command property not changed though.
    !
    if ~self.Busy             ! Graham
        if (self._ControlDone = 0 and self._DataDone = 0) or self.busy = 0 ! in case control declared finished before data finished.
          self._NoOp()
        else
          self.LastNoop = clock()
        end
      else                               ! Graham
        self.LastNoop = clock()          ! Graham
      end                                ! Graham
  else
      if self.Busy
        ! Ignore command - as another command was busy
        self.error = ERROR:AlreadyBusy
        self.WinSockError = 0
        self.SSLError = 0
        self.FTPError = 0
        self._CallErrorTrap ('Can''t do a NoOp command, as the FTP object is already busy', 'NetFTPClientControl.NoOp')
        self.LastNoop = clock()
        return
      else
        self._NoOp()               ! Graham
        self._Command = 'NoOp'     ! Graham
      end
  !    self._NoOp()               ! Graham
  !    self._Command = 'NoOp'     ! Graham
  end
  return


Hope it helps
Regards
Graham

[Larry Sand]

Thanks Graham,

I'll review your code and see if it makes a difference.  Without looking up the code, I don't remember what our workaround was.

Cheers,
Larry Sand