NetTalk Central

Author Topic: MS SQL Tables Locking  (Read 137390 times)

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
MS SQL Tables Locking
« on: April 29, 2014, 12:01:56 AM »
Hi Everyone,

I'm still having problems with MS SQL tables being locked by the WebServer which in turn causes my win32 desktop app to freeze. The strange thing is that you can continue to use the web app while it has locked the sql tables.

The driver string for both desktop and web apps is: /BUSYHANDLING=2 /BUSYRETRIES=5 /LOGONSCREEN=FALSE /VERIFYVIASELECT=TRUE

Has anybody else out there actually resolved these issues without having to constantly monitor the WebServer and restart it? I've added code to automatically restart the webserver after a certain amount of time and to save the session queue to an XML (thanks to Kevin for the code) but I've found this does not work for a webserver running as a service, the service never restarts successfully.

Any help will be greatly appreciated. I almost have no hair left...

Regards,
Trent

kevin plummer

  • Hero Member
  • *****
  • Posts: 1195
    • View Profile
    • Production Accounting and Software Payroll
Re: MS SQL Tables Locking
« Reply #1 on: April 29, 2014, 01:06:31 AM »
Hi Trent,

here is my driver sting

/MULTIPLERESULTSETS=TRUE /BUSYHANDLING=2

I'm surprised you app is running properly without /MULTIPLERESULTSETS=TRUE but obviously it is.

SQLTIMEOUT is what got my SQL apps running smoothly. The hard thing was finding all the hand code to wrap.

For me it was a long journey so along the way I moved stuff to stored procedures, moved all code to Prop:SQL before I adding the SQL Timeouts.

So this app used to lock at least 10 times a day and another app was almost useless due to locking and both run 24/7 now.

Saving the session Q to XML was what I used during the dark locking days but I just don't have that problem any more.

Kev




trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #2 on: April 29, 2014, 02:58:08 AM »
Hi Kevin,

I've tried using both mars and busyhandling in the driver string but that seemed to make matters even worse. Also the Clarion docs state that you shouldn't use both at once as mars cancels out busyhandling. Another thing I found is that mars is not thread safe: http://msdn.microsoft.com/en-us/library/h32h3abf%28v=vs.110%29.aspx - check the 'Thread Safety' header. Not sure if this means anything when used with the NT WebServer...

- All hand code sql in web app is wrapped in sql timeouts.
- Moving everything to stored procedures will take months (300+ tables, multi-dll apps, and just me doing all of this) so isn't really an option.
- I'm not using any of Clarion's client or server side Auto Inc, only the GWB template.

The locking only occurs when both desktop and web app is being used at the same time. If web app is not running then no locking occurs in the desktop app. The desktop app doesn't even show a 'busy' message. It will just stay frozen until the table being interregated is unlocked. Does this mean the 'busyreties' isn't doing it's job? Or should I add the 'busymessage' option to the driver string?

If only the web app is running then no 'visible' locking can be seen.

Regards,
Trent

osquiabro

  • Hero Member
  • *****
  • Posts: 687
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #3 on: April 29, 2014, 03:28:37 AM »
is very strange, i have 1420 user connect to web aplication and 50 to desktop simultaneous, with SQL Server.

Some months ago i have a big trouble with locks and sql, but never identity what happen, the last  change before disaster was a SV sincrononize, after restore old DCT the problem disapear, I could never identify the problem.

My suggestion is try to identify the tables locked, and create small app with this tables without using the synchronize, only import tables and test..

use this query to see the locks:

select cmd,* from sys.sysprocesses
where blocked > 0

/BUSYHANDLING=2 /JOINTYPE=MSSQL /LOGONSCREEN=FALSE /MULTIPLEACTIVERESULTSETS=TRUE /VERIFYVIASELECT=TRUE

debzidoodle

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #4 on: April 29, 2014, 04:41:35 AM »
I recently went through this problem as well with one of my databases.  I have multiple databases, one for each client, and it was only happening on one database.  And it seemed to start all of a sudden, so at first it was really perplexing.  But I have it resolved now, and it was nothing I did in clarion, the issues were 100% backend related.
Here are some steps I used to find the issues

Look for your "most expensive queries".  I used this sql script as I could filter it for a time frame to see if I had made improvements using the following steps.  Change DATEADD(HOUR, -8, getdate()) to be to your liking.   The -8 gives you the last 8 hours of expensive queries.
Code: [Select]
SELECT TOP 50 DB_NAME(st.dbid) DatabaseName,
    SUM(qs.total_worker_time) as total_cpu_time,
    SUM(qs.execution_count) as total_execution_count,
    COUNT(*) as  number_of_statements,
    SUM(qs.total_worker_time)/SUM(qs.execution_count)/1000 AS AvgExecutionTime_seconds,
    st.text as Query,
     CONVERT(varchar(max), pl.query_plan)queryPlan,
    GETDATE() AS TimeStamp,
    qs.sql_handle
FROM
    sys.dm_exec_query_stats qs
    CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
cross apply sys.dm_exec_plan_attributes(qs.plan_handle) pa
cross apply sys.dm_exec_query_plan(qs.plan_handle) pl

WHERE  pa.attribute = 'dbid' and
      last_execution_time > DATEADD(HOUR, -8, getdate())
GROUP BY qs.plan_handle, st.text, qs.sql_handle,DB_NAME(st.dbid), CONVERT(varchar(max), pl.query_plan)

Copy a value from the query column, and paste it into a new query window.  Press ctrl+L if you are in SQL Management Studio.  This will show you the query plan.  At the top section of the results, look for a It should also tell you how much of a performance impact you will have from adding the index.  Right clicking on it should let you "Show Missing Index Details".  If the query you pasted is an object like a stored procedure or trigger, run each statement separately by highlighting it and pressing ctrl+L.  If you have insert\update statements, run it with the insert\update and again with only the select statement that feeds the insert\update. As these can have different query plans that may or may not show the missing indexes unless run separately.

The above really helped, but I was still having blocking issues.  So another item I did was to turn of "Use page locks when accessing the index".  You can find this by right clicking on your index in Management Studio, select properties and go to the options tab.  For a little background on what this does... My schema uses GUIDs as my primary key which is also my clustered index (how its stored on disk). Think of this index as a Phone Book.  Each page of your phone book has a list of GUIDS in alphabetical order.  Rather than locking the row you are editing, the query engine will sometimes use a "Page Level" lock, making it so that no one can edit any rows on that page while the transaction is active... and well if your transaction is expensive and runs a long time, or your transaction does not close, you are stuck.

The other setting in the same tab of your index properties to pay attention to is your "Fill Factor".  If you have a high transaction table, your fill factor should not be the default (usually around 80).  I set mine to 60 if it is a high transaction database.  This can make querying take a bit longer, but the inserts and updates go much faster.  I also turn on "Pad Index".  Think of this as how full you want your Page in your Phone Book to be.  If you have a page that is getting filled up quickly because it is 80% fill factor, it has to create new pages and shuffle your index to make room for the new data coming in.  Giving it a little more breathing room by padding and 60% fill factor makes that happen less often.  This is less of an issue if your index is on a sequential number as it just adds a new page to the end of the index.

After these things were done blocking went away.  Of course I also had the options you discussed earlier enabled as well.

Hopefully this helps,

Debra

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #5 on: April 29, 2014, 03:30:32 PM »
Hi Debra,

I tried running that query, then selecting a query and pressing Ctrl+L but run into an error. It seems that everytime I run that long query I get the same results. I can't tell what they are to do with. Attached is the query results in an excel spread sheet. I'm no SQL Expert and to be honest, why does the average programmer have to be an SQL Expert to figure out these issues? Shouldn't MS SQL by default not lock tables? Isn't that why we use MS SQL so that our programs can be multi-user and multi-threaded? If I run a query on a locked table in Management Studio I get an error message after about 30 seconds. Why don't I get any error messages in my win32 app?

Is there any way of turning off page locking and enabling fill factor in the clarion dict? Doing this for each individual client on every one of their databases is mind boggling. Also which index do you do these things on? One of the tables in question has 30 indexes not including the primary key.

Hi Osquiabro,

I haven't synced the dictionary at all.

To all.

Here is some more information:

Client A:

- Has 1x master server (Server 2008 R2) running MS SQL 2005. The WebServer is running on this server as well. The SQL Server instance is running at about 7gb RAM. The server has a total of 8gb RAM.
- Has 10+ Citrix terminal servers (Server 2008 R2 I think) for the win32 app. About 150 users use the win32 app on these terminal servers. I'm not sure of the exact number of terminal servers or users. The users run the win32 app via Citrix Xenapp from their local desktops at 50+ remote sites. Each terminal server hosts other Citrix apps, that do not freeze, and has about 4gb RAM each.
- Has an unknown number of web app users. The 50+ sites are hospitals so all staff are using the web app. If there are at least 40 staff at each site then there are 2000 web app users.
- The driver string for web app and win32 app is:  /MULTIPLEACTIVERESULTSETS=TRUE /BUSYRETRIES=5 /LOGONSCREEN=FALSE  /VERIFYVIASELECT=TRUE
- SQL Client Drivers are installed on all servers.
- The freeze can happen at any time on the win32 app. Some users are saying that they are not doing anything i.e. the win32 app is open but the user is not using it. Other users are saying that they can add new records but cannot update existing records.

Client B:

- Has 1x File Server (Server 2008 R2) that the WebServer is running on.
- Has 1x SQL Server (Server 2008 R2) running SQL Server 2008 R2. Instance is running at 4.7gb RAM. Server has 16gb RAM.
- Has 1x Terminal Server (Server 2008 R2) which only one win32 user uses.
- Has 5x Client Workstations (Windows 7) that the win32 app run on.
- Has 300+ web app users.
- The driver string for web app and win32 app is:  /MULTIPLEACTIVERESULTSETS=TRUE /BUSYRETRIES=5 /LOGONSCREEN=FALSE  /VERIFYVIASELECT=TRUE
- SQL Client Drivers are installed on all servers and workstations.
- Since changing this client to use MARS 99% of the locking issues have stopped. This was the client that I was having issues with before trying to integrate two of our win32 app databases together. Fixed with MARS and turning integration sql scripts into Stored Procedures.

Client C:

- Has 1x File Server (Server 2003) that the WebServer is running on.
- Has 1x SQL Server (Server 2003 I think) running SQL Server 2008 R2. Not sure how much RAM as I do not have access to this server.
- Has 4x Client Workstations (Windows 8.1) that the win32 app runs on.
- Has around 200 web users.
- The driver string for web app and win32 app is:  /BUSYHANDLING=2 /BUSYRETRIES=5 /LOGONSCREEN=FALSE  /VERIFYVIASELECT=TRUE
- SQL Client Drivers are installed on all servers and workstations.
- Same issues as Client A.

There is one difference between these clients: Clients A and C use the same functionality of the win32 app. Client B uses different functionality which is probably why they are not experiencing the same table locking issues.

Luckily the driver string is a variable inside an ini file on each server and workstation so I can easily change this. Client A takes weeks to book in an upgrade but I can work with Client C very easily. I will try and add in the MARS driver option with busyhandling and see if it makes a difference. Will report back shortly.

Does it matter which version of SQL Client Drivers are installed on the non-sql servers and workstations? Do they need to match the version of SQL that is installed?

If you have any other ideas please let me know.

Regards,
Trent

[attachment deleted by admin]

kevin plummer

  • Hero Member
  • *****
  • Posts: 1195
    • View Profile
    • Production Accounting and Software Payroll
Re: MS SQL Tables Locking
« Reply #6 on: April 29, 2014, 04:09:20 PM »
Some comments...

Client A. It's probably not the best to run the web server on the SQL server. Sure you avoid network access but SQL is very CPU and memory hungry and that server doesn't have much memory. More resources would help but would not fix the problem, maybe reduce it. In my experience the freeze generally happens in the web app which affects the win32 app.

Do you use FM3? This is where I set my driver string. It needs to be set before any files are opened and FM3 opens the first file.

Do you have access to SQL Management studio when tables are locked? This is where I would look at what table was locked. I would then look at where the table was used and add debug information to try to pinpoint problem area's. SQLTimeout was the final piece in the puzzle for me. I run my own servers and we use goglobal instead of citrix for win32 apps so it has been easier to debug.

Are you running NT8? I recall Bruce added something to trace Threads that hang. This may help identify problem are's in your code. Of course once a thread hangs this will cause other threads to hang so it is important to track down the first Thread.


trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #7 on: April 29, 2014, 04:21:03 PM »
Hi Kevin,

We do use FM3 but FM3 does not open the tables first in our app. We have separated FM3 into it's own exe and run that exe only when doing database upgrades.

Each SQL table in the dict has a variable as the driver string which is set/got in an ini file during startup.

This is on NT8 and NT7 WebServers. Clients B and C are using NT8. Client A is using NT7.

I will have to check out how to trace thread hangs, is there a video on this?

Regards,
Trent

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #8 on: April 29, 2014, 05:28:33 PM »
Another question: If you have both mars and busyhandling in the driver string, if SQL Client Drivers are not installed will the mars option be ignored or will the whole string be ignored?

debzidoodle

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #9 on: April 29, 2014, 06:17:42 PM »
Hi Trent,

Try the steps outlined in the pictures.  This will get you to the same place, but with a UI. Not all expensive queries will tell you an index is missing.  So to start with, just look through the items till you find one that has the green text with the missing index.

Try to choose ones that have a high impact ratings, some low impact indexes may not be worth it.  So testing is suggested for all indexes to see if their impact is worth their cost.  

By the way, the zip file in your prev. post was empty.

[attachment deleted by admin]

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #10 on: April 29, 2014, 07:35:57 PM »
Holy cow that's pretty awesome! It suggests keys for you to make to try and increase performance!

Thanks Debra. Sorry about the zip file - NTC must not like xlsx files.

I've updated a Client C use the following sql driver string: /MULTIPLEACTIVERESULTSETS=TRUE /BUSYHANDLING=2 /BUSYRETRIES=5 /BUSYMESSAGE=The connection is busy, please try again later. /LOGONSCREEN=FALSE /VERIFYVIASELECT=TRUE

Just waiting to see if the database tables lock again.

Regards,
Trent

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #11 on: April 30, 2014, 04:48:04 PM »
Hi Debra,

How do you open the expensive queries window in SQL 2005? The Activity Monitor does not have the options as displayed in your images.

Also the query doesn't work either - just get an error message about converting from XML.

Regards,
Trent

debzidoodle

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #12 on: April 30, 2014, 05:10:02 PM »
I am not sure if that query will work in 2005, some of those dynamic management views may be new in 2008. 
This link gives an explanation of 2005 activity monitor  http://technet.microsoft.com/en-us/library/ms175518(v=sql.90).aspx

Do you have access to a 2008 server?  They added quite a few tools to manager performance in 2008.

trent

  • Full Member
  • ***
  • Posts: 204
    • View Profile
    • Email
Re: MS SQL Tables Locking
« Reply #13 on: April 30, 2014, 05:42:53 PM »
Thanks, I've checked that link before and it's not very informative. Unfortunality this is for Client A and they only have SQL 2005. Makes things harder to find.

Yesterday I upgraded Client A and Client C to have the new driver string: /MULTIPLEACTIVERESULTSETS=TRUE /BUSYHANDLING=2 /BUSYRETRIES=5 /BUSYMESSAGE=The connection is busy, please try again later. /LOGONSCREEN=FALSE /VERIFYVIASELECT=TRUE

Client C has not emailed today saying that their desktop app is freezing but Client A has every desktop app freezing at the moment. I have restarted the WebServer but the app is still freezing so now I'm thinking it's not limited to the WebServer anymore... Also the BUSYMESSAGE doesn't appear, has anyone gotten this switch to work?

Delving into the desktop app now...

Regards,
Trent