Showing posts with label Memory. Show all posts
Showing posts with label Memory. Show all posts

Tuesday, 7 February 2012

A Script A Day - Day 7 - Memory Queries

Today's script is a collection of a few memory queries I've collected from various sources over the years.  These queries are by no means comprehensive but are ones I have certainly used the most.


/*
      -----------------------------------------------------------------
      Memory Queries
      -----------------------------------------------------------------
     
      For more SQL resources, check out SQLServer365.blogspot.co.uk

      -----------------------------------------------------------------

      You may alter this code for your own purposes.
      You may republish altered code as long as you give due credit.
      You must obtain prior permission before blogging this code.
 
      THIS CODE AND INFORMATION ARE PROVIDED "AS IS"
     
      -----------------------------------------------------------------
*/

-- Interrogating memory configuration.
-- Determine if this is a 32- or 64-bit SQL Server edition
DECLARE @ServerAddressing AS TINYINT
SELECT  @serverAddressing = CASE WHEN CHARINDEX('64',
                                                CAST(SERVERPROPERTY('Edition')
                                                     AS VARCHAR(100))) > 0
                                 THEN 64
                                 ELSE 32
                            END ;

SELECT  cpu_count / hyperthread_ratio AS SocketCount ,
        physical_memory_in_bytes / 1024 / 1024 AS physical_memory_mb ,
        virtual_memory_in_bytes / 1024 / 1024 AS sql_max_virtual_memory_mb ,
           -- same with other bpool columns as they are page oriented.
           -- Multiplying by 8 takes it to 8K, then / 1024 to convert to mb
        bpool_committed * 8 / 1024 AS buffer_pool_committed_mb ,
           --64 bit OS does not have limitations with addressing as 32 did
        CASE WHEN @serverAddressing = 32
             THEN CASE WHEN virtual_memory_in_bytes / 1024 /
                                                     ( 2048 * 1024 ) < 1
                       THEN 'off'
                       ELSE 'on'
                  END
             ELSE 'N/A on 64 bit'
        END AS [/3GB switch]
FROM    sys.dm_os_sys_info

-- Is NUMA enabled
SELECT  CASE COUNT(DISTINCT parent_node_id)
          WHEN 1 THEN 'NUMA disabled'
          ELSE 'NUMA enabled'
        END
FROM    sys.dm_os_schedulers
WHERE   parent_node_id <> 32 ;

-- Show memory consumption details
DBCC MEMORYSTATUS

-- System memory usage.
SELECT  total_physical_memory_kb / 1024 AS total_physical_memory_mb,
        available_physical_memory_kb / 1024 AS available_physical_memory_mb,
        total_page_file_kb / 1024 AS total_page_file_mb,
        available_page_file_kb / 1024 AS available_page_file_mb,
        system_memory_state_desc
FROM    sys.dm_os_sys_memory

-- Memory usage by the SQL Server process.
SELECT  physical_memory_in_use_kb,
        virtual_address_space_committed_kb,
        virtual_address_space_available_kb,
        page_fault_count,
        process_physical_memory_low,
        process_virtual_memory_low
FROM    sys.dm_os_process_memory

-- Get total buffer usage by database
SELECT  DB_NAME(database_id) AS [Database Name],
        COUNT(*) * 8 / 1024.0 AS [Cached Size (MB)]
FROM    sys.dm_os_buffer_descriptors
WHERE   database_id > 4 -- exclude system databases
        AND database_id <> 32767 -- exclude ResourceDB
GROUP BY DB_NAME(database_id)
ORDER BY [Cached Size (MB)] DESC ;

-- Breaks down buffers by object (table, index) in the buffer pool
SELECT  OBJECT_NAME(p.[object_id]) AS [ObjectName],
        p.index_id,
        COUNT(*) / 128 AS [Buffer size(MB)],
        COUNT(*) AS [Buffer_count]
FROM    sys.allocation_units AS a
        INNER JOIN sys.dm_os_buffer_descriptors AS b ON a.allocation_unit_id = b.allocation_unit_id
        INNER JOIN sys.partitions AS p ON a.container_id = p.hobt_id
WHERE   b.database_id = DB_ID() -- Change to suit
        AND p.[object_id] > 100 -- exclude system objects
GROUP BY p.[object_id],
        p.index_id
ORDER BY buffer_count DESC ;

-- Buffer Pool Usage for instance
SELECT TOP ( 20 )
        [type],
        SUM(single_pages_kb) AS [SPA Mem, Kb]
FROM    sys.dm_os_memory_clerks
GROUP BY [type]
ORDER BY SUM(single_pages_kb) DESC ;

-- Which queries have requested, or have had to wait for, large memory grants?
-- Shows the memory required by both running (non-null grant_time)
-- and waiting queries (null grant_time)
-- SQL Server 2008 version
SELECT  DB_NAME(st.[dbid]) AS [DatabaseName],
        mg.requested_memory_kb,
        mg.ideal_memory_kb,
        mg.request_time,
        mg.grant_time,
        mg.query_cost,
        mg.dop,
        st.[text]
FROM    sys.dm_exec_query_memory_grants AS mg
        CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS st
WHERE   mg.request_time < COALESCE(grant_time, '99991231')
ORDER BY mg.requested_memory_kb DESC ;

-- Shows the memory required by both running (non-null grant_time)
-- and waiting queries (null grant_time)
-- SQL Server 2005 version
SELECT  DB_NAME(st.[dbid]) AS [DatabaseName],
        mg.requested_memory_kb,
        mg.request_time,
        mg.grant_time,
        mg.query_cost,
        mg.dop,
        st.[text]
FROM    sys.dm_exec_query_memory_grants AS mg
        CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS st
WHERE   mg.request_time < COALESCE(grant_time, '99991231')
ORDER BY mg.requested_memory_kb DESC ;

-- Return the cache counters.
SELECT  [type],
        name,
        single_pages_kb,
        multi_pages_kb,
        single_pages_in_use_kb,
        multi_pages_in_use_kb,
        entries_count,
        entries_in_use_count
FROM    sys.dm_os_memory_cache_counters
ORDER BY [type],
        name ;

-- Investigating the use of the plan cache.
SELECT  name,
        [type],
        entries_count,
        entries_in_use_count
FROM    sys.dm_os_memory_cache_counters
WHERE   [type] IN ( 'CACHESTORE_SQLCP', 'CACHESTORE_OBJCP' )
               --ad hoc plans and object plans
ORDER BY name,
        [type]

-- AWE allocated memory
SELECT  SUM(awe_allocated_kb) / 1024 AS [AWE allocated, Mb]
FROM    sys.dm_os_memory_clerks

--  Single + Multi Page Memory Allocation
SELECT  type,
        SUM(virtual_memory_reserved_kb) AS [VM Reserved],
        SUM(virtual_memory_committed_kb) AS [VM Committed],
        SUM(awe_allocated_kb) AS [AWE Allocated],
        SUM(shared_memory_reserved_kb) AS [SM Reserved],
        SUM(shared_memory_committed_kb) AS [SM Committed],
        SUM(multi_pages_kb) AS [MultiPage Allocator],
        SUM(single_pages_kb) AS [SinlgePage Allocator]
FROM    sys.dm_os_memory_clerks
GROUP BY type

-- amount of mem allocated though multipage allocator interface
SELECT  SUM(multi_pages_kb)
FROM    sys.dm_os_memory_clerks

-- amount of mem allocated though multipage allocator interface DETAILED
SELECT  type,
        SUM(multi_pages_kb) AS sizeinkb
FROM    sys.dm_os_memory_clerks
WHERE   multi_pages_kb != 0
GROUP BY type

-- Detailed memory info per componant
DECLARE @total_alloc BIGINT 
DECLARE @tab TABLE
    (
      type NVARCHAR(128) COLLATE database_default,
      allocated BIGINT,
      virtual_res BIGINT,
      virtual_com BIGINT,
      awe BIGINT,
      shared_res BIGINT,
      shared_com BIGINT,
      topFive NVARCHAR(128),
      grand_total BIGINT
    ) ;
-- note that this total excludes buffer pool committed memory as it represents the largest
-- consumer which is normal
SELECT  @total_alloc = SUM(single_pages_kb + multi_pages_kb
                           + ( CASE WHEN type <> 'MEMORYCLERK_SQLBUFFERPOOL'
                                    THEN virtual_memory_committed_kb
                                    ELSE 0
                               END ) + shared_memory_committed_kb)
FROM    sys.dm_os_memory_clerks 
PRINT 'Total allocated (including from Buffer Pool): '
    + CAST(@total_alloc AS VARCHAR(10)) + ' Kb'
INSERT  INTO @tab
        SELECT  type,
                SUM(single_pages_kb + multi_pages_kb) AS allocated,
                SUM(virtual_memory_reserved_kb) AS vertual_res,
                SUM(virtual_memory_committed_kb) AS virtual_com,
                SUM(awe_allocated_kb) AS awe,
                SUM(shared_memory_reserved_kb) AS shared_res,
                SUM(shared_memory_committed_kb) AS shared_com,
                CASE WHEN ( ( SUM(single_pages_kb + multi_pages_kb
                                  + ( CASE WHEN type <> 'MEMORYCLERK_SQLBUFFERPOOL'
                                           THEN virtual_memory_committed_kb
                                           ELSE 0
                                      END ) + shared_memory_committed_kb) )
                            / ( @total_alloc + 0.0 ) ) >= 0.05 THEN type
                     ELSE 'Other'
                END AS topFive,
                ( SUM(single_pages_kb + multi_pages_kb
                      + ( CASE WHEN type <> 'MEMORYCLERK_SQLBUFFERPOOL'
                               THEN virtual_memory_committed_kb
                               ELSE 0
                          END ) + shared_memory_committed_kb) ) AS grand_total
        FROM    sys.dm_os_memory_clerks
        GROUP BY type
        ORDER BY ( SUM(single_pages_kb + multi_pages_kb
                       + ( CASE WHEN type <> 'MEMORYCLERK_SQLBUFFERPOOL'
                                THEN virtual_memory_committed_kb
                                ELSE 0
                           END ) + shared_memory_committed_kb) ) DESC
SELECT  *
FROM    @tab



Enjoy!


Chris

Wednesday, 18 January 2012

Don’t Be a Balloon!

I had a problem just before the festive period where one of our production SQL Servers was randomly grinding to a halt, queries which usually returned in milliseconds where running for minutes.  It took 4 days of performance monitoring / troubleshooting and some help from the SQL Community (when I though all was lost) to find the cause of the problem and get it resolved!

To give a little bit of background information about this particular environment we run multiple SQL Server clusters in various geographical locations across the globe which we replicate to from a central SQL Server cluster.  Each of these ‘child’ SQL Servers also replicates certain transactional articles back to the central environment.  Each server runs SQL Server 2005 Standard Edition SP3 x64 on Windows Server 2003, they have different CPU / Memory allocations designed to support their respective workloads.

The server encountering the problem had 50% more memory than another server and was running half the load.  The problem first came to light when a couple of overnight jobs which normally take a few minutes were running for several hours before I had to stop them.  I started investigating the issue looking at the usual DMV’s with no joy.

Below are a few of the DMV’s I used during my investigations. 

sys.dm_os_wait_stats
sys.dm_os_waiting_tasks
sys.dm_os_performance_counters
sys.dm_os_memory_allocations
sys.dm_os_memory_allocations
sys.dm_io_virtual_file_stats
sys.dm_os_schedulers
sys.dm_os_sys_memory

I recently read a fantastic book – Performance Tuning with SQL Server Dynamic Management Views by Louis Davidson and Tim Ford which although did not lead me to the problem allowed me to dismiss several possible causes.

I then used SQL Sentry Performance Advisor to monitor the server without much joy.  One thing I did notice is that the buffer cache would intermittently drop from its normal size of 6GB to 0?  I used perfmon to monitor SQLServer:Buffer Manager / Buffer Cache Hit Ratio / Page Life Expectancy / Database pages and just as SQL Sentry highlighted these would intermittently drop to 0, then build back up.

I tried finding a correlation between any jobs / processes and the dramatic degradation in performance again with no joy.  I then found errors in the error log similar to the below;

Source                  spid1s

Message
A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 115227 seconds. Working set (KB): 1635000, committed (KB): 6453712, memory utilization: 25%.

Why was SQL Server releasing memory?  I checked the server and the locked pages in memory privilege was not granted to the SQL Server service account so I added this and restarted the service after repointing the applications to another server only to find that it didn’t make any difference.  Whilst trawling the super information b road (sorry running joke I had at a previous employer in the days when the ‘super information highway’ was reeeaaalllly slow) I found that on 64bit systems locked pages in memory is not supported on SQL Server 2005 Standard Edition unless you are at SP3 CU4, great (or so I thought) lets plan deploying CU4 and with the locked pages in memory privilege and we should be ok.

Me being me, and I think this is a general DBA thing (maybe not, it might just be my inner geek) I wasn’t happy at just applying the CU, something somewhere was causing SQL Server to release memory, it had never been a problem before, it had only recently started happening.

I stumbled across an excellent article from Jonathan Kehayias about locked pages in memory, it was the Considerations for Virtual environments section that gave me my big break.  Jonathan describes a scenario called memory overcommit as below;

“Memory overcommit is a scenario where the memory allocated to the virtual machines running on the host exceeds the total amount of physical RAM available in the server.

When memory overcommit occurs, one of the first ways that the hypervisor reacts is to make use of a special driver, known as a balloon driver, which is installed in the VM as part of the VM tools. In essence, the hypervisor sets the balloon driver the task of reducing memory consumption in the VM to a target level, and the balloon driver responds by acquiring memory in the VM.”

Now I will hold my hand up here, I haven’t been in my new job long (2nd month at the time of the issue) and am still learning the environments to this day but that is no excuse.  This server is one of several which run my companies services for the whole of the UK and at the time I didn’t know that it was a Virtual Cluster which I should have.  Another issue is that I didn’t and still don’t have access to the VCentre for this server so I couldn’t see if memory over commit and or memory ballooning was a problem (or so I thought). 

Resorting to Twitter under the #sqlhelp hashtag, Jonathan came to my rescue and eventually after a few questions for further info he told me about some perf counters.  If the VM Tools are installed on a VM then some VM Counters are exposed, you can see where I’m going with this one can’t you?!



Adding this counter I was shocked, all 6GB of the buffer cache had been ‘ballooned’ to disk on the ESX host!  This explained the dramatic degradation in performance, after a call to the team who manage the VMware environment we moved the cluster to other hosts which had not overcommitted memory and performance was restored to normal levels.  Now I have always been sceptical about virtual infrastructure and SAN storage being used for my database servers.  However with careful design to make sure there is no contention / adequate resource with room for growth so environments can scale I have never had a problem of this magnitude before.

There are a couple of things I have learned from this problem;

1 – Make sure you have access to all the hardware which all the servers you manage run on (SAN / VMware etc. etc.) this makes life so much easier when troubleshooting!

2 – Make sure you are involved in every aspect of design and are part of the core decision making team for new environments!  Or if you inherit environments when you move jobs or from acquisitions check and double check everything!

And last and most importantly of all!

3 – Trust no one!  I learned this from a very early stage in my SQL Server career (thank you Mr David C. Riley).  Everyone starts off equal, equally useless (to be polite) until they prove themselves otherwise!

I really hope this helps someone with the same problem or prevents someone encountering the same issue.

Chris