Problems Controlling RAM Usage

[Rob Mikkelsen]

I have a procedure that produces a .CSV file of data requested by the user.  There is no limitation on the amount of data that may be obtained, so I decided to dump the entire database (approximately 900,000 entries).  First these entries are read into a queue and sorted, then presented to the user.  So far so good.

I expected the queue to grow rather large, so I fired up Task Manager to see how much RAM was used.  The number climbed to 150M before the data was written to the disk (which I find to be a high, but acceptable number). However, after the data is written and the procedure is executed, the RAM does not appear to be released. 

I had assumed that the RAM would automatically be freed after the local source procedure ended (it IS a local queue), but apparently not. 

I typed the queue, set a reference variable, and tried using NEW and DISPOSE.  No difference. 

I logged out, thinking that maybe garbage collection would take care of that.  No good, either.

On a positive note, I ran the same report a second time and the memory usage only increasd by about 4M, not another 140M as I had expected. 

I do have a five hour timeout on the session, so I will test it with a 20 minute session timeout so it will have a chance to clear, but that should only apply to session variables, not queue memory usage.

I have had this problem since day one where it appears that my NetTalk program is using but not releasing RAM.  Even when sitting idle, the RAM usage (according to Task Manager) continues to increase.  After several days or weeks of this, the RAM usage becomes astronomical.  There are no strange data types in the queue that need to be handled individually, just SHORT, LONG, DECIMAL and STRING variables.

What am I doing wrong?  Is there an ABC-specific method to free up the RAM?  I am, after all, fairly new to ABC and may miss some of its idosyncracies. 

Or is this simply that Task Manager is confused?  Is there some other software that I can use that will accurately display how much RAM is being consumed by my NetTalk app?

Thanks!

Rob Mikkelsen

[kevin plummer]

I always just use Free(MyQueue) - see if that helps.

[Rob Mikkelsen]

Tried that, too.  No change.

In fact, I even tried deleting the individual queue items as they were being written to the export file - still no good.  Certainly something should work!  Sor far, I have:

1.  Relied on leaving the procedure scope to release the RAM,

2.  Typed the queue, set reference variable LasPQ &LastMessage:PRintQueueType, then used LasPQ &=NEW(LastMessage:PrintQType) and DISPOSE(LasPQ) - no good,

3.  Deleted individual queue entries as they are written to the files (extremely slow but also ineffective)

4.  FREE() the queue when done - no good either.

5.  Checked to see if releasing the session queue resulted in releasing the RAM.  Nope.

This is not confined to a single computer - I have this program running at 62 locations with the same results.  I THINK I have tried everything available with the same results. 

I ran the program locally overnight (no additional access) and checked it this morning.  It was still sitting at 153M ram used.  I ran a separate RAM monitor program I donwloaded last night and it also confirmed the reduction in RAM to the system.  As I mentioned in my first post, running the report a second time does not result in an additional 150M RAM usage, but the original amount is never released back to the system, but the memory usage does increase slowly as the program sits idle.

I am not so concerned that I did not release the product, it just causes the occasional slow down and some maintenance headaches since the web server fails periodically.  I really want to worry about the software only when releasing new versions.

Thanks!

Rob

[Bruce]

Hi Rob,

Obviously the possibility for memory leaks in clarion exist, just like any other language - but I think in this case you've got all the bases covered. The fact that it "grows" in memory use would concern me - and I think you should look further into that. Especially if it is "growing" while the program appears to be doing nothing. That suggests a timer somewhere (probably on the web server window) doing something.

Alas though - using Task Manager to determine Ram usage is misleading. Because Windows is smarter than you think.

What happens is basically this. You program uses the 150 Megs of Ram, finishes with it and "frees" it. Windows noticed this, but has nothing better to do with the Ram at the moment. No other program is wanting Ram, and it has plenty of Ram for anyone who does ask. Since your program used this ram recently, it "leaves" the ram assigned to your program for now. That's why the next request just reused that 150 megs.

You'll see this all the time with a normal clarion windows app. On trigger that Windows uses for "taking the ram back" is if the app is minimised. If you "minimised" the server you'd probably see the number change a lot.

If you right click in task manager there are many other memory things you can measure. (for example Virtual vs Physical memory). And in WinEvent there are some memory functions for returning more "accurate" memory numbers.

Cheers
Bruce

[Rob Mikkelsen]

Bruce,

I have detected no change in RAM usage when minimizing the program to the system tray, but I will play with that a little.  I do have a couple timers running on the WebServer window so I will examine them.  I also have a hidden window that I use to relay the status of the system to our status board so that may also be a place to look.

While I am hopeful that the RAM issue is because Windows does not need to recover it, I will dig a little deeper and look at some of the Winevent RAM monitoring.

I will let you know what I find.

Thanks!

Rob

[Rob Mikkelsen]

I displayed the web server, the ran the long report again, and noted that the Task Manager indicated 153M RAM used.  I clicked on the "X" (close) button to drop it back into the system tray - no change.  I then restored the server and MINIMIZED it to the task bar.  Poof!  The RAM usaged dropped from 153M to 2M - the lowest I have EVER seen it in this app.

I let the program sit for a few hours and the RAM usage crept back up to 10M.  I showed the screen again, and just minimized it again and the RAM usage dropped to 1.3M - another record low.  If I keep doing this, will it start ADDING RAM to my system? <g>

The RAM usage does increase while the program is sitting idle, but it is collected when the program is minimized.  Is it safe to assume that this is because Windows does not need the RAM so it does not look to reclaim it?  I am perfectly happy with the "there is nothing wrong with your program" response.

I will, however, look at the timers and status window to see if I can figure out why the RAM usage ticks upward when it is not responding to requests.

Thanks again, Bruce!

Rob Mikkelsen

[Bruce]

Hi Rob,

As you noticed, "I clicked on the "X" (close) button to drop it back into the system tray" is not minimizing. Actually it just hides the window.

>> The RAM usage dropped from 153M to 2M - the lowest I have EVER seen it in this app.

well no, it didn't. The "Working set" of ram dropped to 2M. In other words, what this is showing you (if nothing else) is that Task Manager "Ram" is not an especially useful measure for determining memory leaks or anything else. (There are more columns you can turn on though that are better at it.)

So it's behavior neither shows a problem, nor shows there-is-no-problem. As a measurement for this it is not actually relevant.

Cheers
Bruce