• Summary (i.e. memory report download info)
• References:
• Documents
• Tools - unsupported
• White Papers

Introduction

The purpose of this document is to describe how memory is used and the tools, both supported and unsupported, that are availble to examine/report memory usage.  See below for details.

[ TOP ]

The memory line in the output of swapinfo...

• The memory line is infamously misleading and does not refer to acual physical memory use!!!! Rather it is the size of pseudoswap, which happens to be calculated to be 75% of the size of RAM (a.k.a. memory.)  As the name ("pseudo") implies, it does NOT exist. Pseudoswap is enabled by default with the kernel parameter swapmem_on(5) set to 1.  Don't worry about this line, just look at the total line for total used and sometimes it's interesting to look at the device PCT USED as an indication of how much swapping has occurred since the box was last rebooted. The memory line is also described  in the swapinfo(1m) man page.
  
• To read more on pseudoswap, refer to:
  
• Pseudo-Swap Space section of the Memory Management white paper
• "What is swap and why does a system need it?"   ITRC doc id  KBRC00001262
• "SYSADM: Swap and Pseudoswap Clarified"  ITRC Doc id  KBRC00010136
  
  

swapinfo description and example...

• Use 'swapinfo -tm' to get a complete/total picutre of swap usage (see below for example and details.)  Pay particular attention to the total line as it indicates how much swap space has been actually reseved for swap. When this percentage gets near 100%, processes will not start up (unable to fork process) and new shared memory segments can not be created.
  
  
• swapinfo -tm  example and explanation:

• dev line(s):
  
• are the actual physical swap device(s)
• show if swapping has actually occurred. In other words, the  PCT USED column in the dev lines represents the value last attained during a previous period of swapping. This is analogous to the high-water mark that a flood leaves.
• to check to see if swapping is currently occuring, use 'vmstat -v 5 5' to see if the 'po' (page outs) is sustained above 0.
• reserve line(s)
  
• indicate how much of the swap device(s) has(have) been set aside for memory should it need to be swapped.
• memory line:
  
• indicative of how much of pseudo-swap has been reserved
• when present, indicates  pseudo-swap is enabled   (i.e.  swapmem_on kernel paraemter is set to 1, which  is the default.) The size of pseudoswap is calculated to be 75% of the size of RAM (a.k.a. memory.)  In other words, it does not refer to acual physical memory use!!!!  Pseudo-swap was designed specifically for large memory systems for which acutal swapping is never (or rarely) expected to occur, so there’s less need to use actual physical disk space for swap. For more information, see swapmem_on(5) , which reads:
  
  
         In previous versions of HP-UX, system configuration required sufficient physical swap space for the maximum possible
         number of processes on the system. This is because HP-UX reserves swap space for a process when it is created, to
         ensure that a running process never needs to be killed due to insufficient swap.
         This was difficult, however, for systems needing gigabytes of swap space with gigabytes of physical memory, and those
         with workloads where the entire load would always be in core. This tunable was created to allow system swap space to
         be less than core  memory. To accomplish this, a portion of physical memory is set aside as 'pseudo-swap' space.
         While actual swap space is still available, processes still reserve all the swap they will need at fork or execute time from
         the physical device or file system swap. Once this swap is completely used, new processes do not reserve swap, and
         each page which would have been swapped to the physical device or file system is instead locked in memory and
         counted as part of the pseudo-swap space.
• total line:
•  the PCT USED value shown in the total line indicates how much swap space has been actually reseved for swap. When this percentage gets near 100%, processes will not start up (unable to fork process) and new shared memory segments can not be created.
  

[ TOP ]

A. Review Buffer Cache size

• The Buffer Cache size is, by default, 50% of RAM (see kernel parameter dbc_max_pct(5)). 
  
  
• Some consider there to be a buffer cache sweet spot -  400 Mb or 20% of memory, whichever is smaller. But of course, this may vary depending on the applications/dbs used on the system. 
  
  
• To check the current size of the buffer cache, either "sysdef | grep bufpages” (and multiply by 4096 to approximate the current size of buffer cache) or use glance’s memory screen to see what size “BufCache” is.

•   Note: Although buffer cache can be dynamic in size, decrease only occurs under memory pressure and then only decreases very slowly. So, the buffer cache often grows farily quickly to dbc_max_pct and only decreases (and slowly) when memory presssure is high.
  

          A: Choose one of the following methods:

[ TOP ]

B. Monitoring Memory Usage

• There are 3 ways for memory to be allocated, all requiring an equivalent amount of swap.

1. plain memory as allocated with  malloc(3C) system call.
2. shared memory as allocated with shmget(2) system call.
3. memory mapped files as allocated with mmap(2) system call.  

[ TOP ]             

1. plain memory
   
   • as allocated with malloc(3C), can be examined with: ps, procsize, kmeminfo, and glance.  See below for details on each of these utilities/tools.
   • is used by process' data space, stack space, and text space, limited by kernel parameters  maxdsiz(5), maxssiz(5), and maxtsiz(5) respectively, for 32-bit apps; and maxdsiz_64bit(5), maxssiz_64bit(5), and maxtsiz_64bit(5) respectively, for 64-bit apps.
   
   

• For Example,
  
•   # UNIX95= ps -eo vsz,ruser,pid,args | sort -rn | more
  26332 ids       1685 ./idsagent -a
     5296 root      2178 /usr/sbin/stm/uut/bin/tools/monitor/fpl_em
     4760 root      2713 /opt/perf/bin/rep_server -t SCOPE /var/opt/perf/datafiles/loggl
     4068 root      1487 /opt/perf/bin/scopeux
     4052 root      1243 /opt/dce/sbin/rpcd
     3364 root      2715 /opt/perf/bin/alarmgen -svr 2714 -t alarmgen /var/opt/perf/data
     3180 root      1465 /opt/perf/bin/midaemon
     3148 root      1495 /usr/sbin/swagentd -r
  
•  And then look at the 1st column in the output to see the amount of memory used by this process for data/text and stack. This value is in pages, so multiply by 4096 to determine the size in bytes.  Anytime you see that the size (SZ) is a four-digit number, that's relatively large, so it's one to watch over time and to see if it continues to grow, and therefore may have a memory leak.
• If you see a process called  mib2agt using an excessive amount of memory, this binary, mib2agt, has a known memory leak fixed in an 11.X patch called  PHSS_27858  (ITRC ftp site download).  This patch DOES NOT require a reboot. Furthermore, mib2agt can be killed and then restarted with “kill mib2agt_PID” and “/usr/sbin/mib2agt”. But only restart it if you have need of supporting SNMP requests (e.g. OpenView). If not needed, can be configured to not start at bootup by modifying /etc/rc.config.d/SnmpMib2.

• Alternatively, to look at both virtual size as well as the actual size:

• procsize  is available for download here:
  
    System:     hprc.external.hp.com  (192.170.19.51)
    Login:      eh
    Password:   spear9
    ftp://eh:spear9@hprc.external.hp.com/procsize.README.txt
    ftp://eh:spear9@hprc.external.hp.com/procsize.64bit.sh
    ftp://eh:spear9@hprc.external.hp.com/procsize.32bit.sh
  
  
• For example:
• Look at breakdown of memory usage, per process:
  
  # ./procsize -fnc | more
  pid Comm        UAREA   TEXT   DATA  STACK  SHMEM     IO  MMAP    Total
  2916 getty    v    4      5     6      4     0      0      349      369
  2287 prm3d    v   68      6   671    513     0      0    37212    38471
  .
  .
  .
  
  
  
• NOTE: numbers in the output of  procsize are the number of 4K pages. So, mulitply by 4096 to get a byte count.
  
• TRICK:  Here's the command to use to sort  by the 11th column (total memory usage)
  
  

• kmeminfo is an unsupported utility that can be used to examine/report memory usage.   
  kmeminfo is available for download here:
  
    System:     hprc.external.hp.com  (192.170.19.51)
    Login:      eh
    Password:   spear9
    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.README.txt
    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.64bit.sh
    ftp://eh:spear9@hprc.external.hp.com/kmeminfo.32bit.sh

• For example:

• A trial version of Glance is available on the application CDs (usally on cd #2 or #3)
  

• Glance is not available for download.
  
  Glance  product #'s
• 11.x s700:   B3691AA  B3699AA
  
• Trial version: B3691AA_TRY   B3699AA _TRY
• 11.x s800:   B3693AA   B3701AA 
• Trial version: B3693AA_TRY B3701AA_TRY

[ TOP ]

• 2) shared memory as allocated with shmget(2) system call.

• a) Can look at shared memory usage with  ipcs(1).  For example:

• b) Can look at shared memory usage with an unsupported  tool called shminfo

• shminfo is available for download here:
    System:     hprc.external.hp.com  (192.170.19.51)
    Login:      eh
    Password:   spear9
  

• For example:

• TRICK: 
• If a particular shared memory segment is of interest, and if you want to know which processes are attached to that shared memory segment, you can use shminfo -s id (where id is the shared memory identifier.)
• For example:
  

• c) Can look at shared memory usage by process with an unsupported utility called procsize, which breaks down memory by  various types, specifically: UAREA, TEXT, DATA, STACK, Shared  Memory, Memory Mapped files.

• For example:
• Look at breakdown of memory usage, per process:
  
  # ./procsize -fnc | more
  pid Comm        UAREA   TEXT   DATA  STACK  SHMEM     IO  MMAP    Total
  2916 getty    v    4      5     6      4     0      0   349      369
  2287 prm3d    v   68      6   671    513     0      0  37212    38471
  .
  .
  .
  
  
  
• TRICK: Here's the command to use to sort the processes by total memory usage, most to least.
  
  

• TRICK: Here's the command to use to sort the processes by shared memory usage, most to least.
  
  

• 3) memory mapped files as allocated with mmap(2) system call. 

• a) The are no system commands that will report memory mapped file usage
  
  
• b) Can use an unsupported utility called shminfo to see use of memory mapped memory.
  
  
• In the output of shminfo, memory mapped  files are shown as “OTHER”.
  
  
• See above for examples and download site for shminfo.
  
  
• c) Can use procsize to see which processes use memory mapped files
  
  
• In the output of procsize , memory mapped  files are shown under the "MMAP” column.
  
  
• See above for examples and download site for procsize
  
  
• TRICK: Use the following to sort by MMAP column:
     # ./procsize -fcn |sort -rnk 10 | more

[ TOP ]

C.  OS Memory Leaks / Hogs

• 1. Check for known OS Memory Leaks with an unsupported utility called kmeminfo The Response Center can assist with using the output of kmeminfo.
  

• 2. Checking for known memory hogs

• The JFS inode cache, is sized by the vx_ninode kernel parameter.  The default value of vx_ninode is determined by size of RAM and is different for 11.00 -vs- 11.11. 

• For example:
  
  • an 11.11 system with 8GB of memory, vx_ninode is defaulted to 256,000
  • an 11.0 system 8GB of memory, vx_ninode is defaulted to 144,000

• For most situations, a smaller value for vx_ninode is reasonable, say 20,000 for example
  
  
• Lowering vx_ninode results in a large savings of memory.
  
  
• To see the size of the JFS (3.3 and above) inode cache:

• To see how many JFS (3.3 and above) inodes are currently cached:

• To gage* the size of a systems JFS inode cache, looking at the output of kmeminfo, use the following table to know which bucket/arena JFS inode cache uses.

                         * NOTE: JFS inode cache is one of the consumers of bucket/arena.

[ TOP ]

D.  Application Memory Leaks

1. Use a tool to capture a baseline of memory use per process,
2. Then gather subsequent reports to see if there is a steady increase in memory use.
   

• it is out of swap
• it hit the maxdsiz limit (a kernel tunable parameter)
  
• it is out of addressible data space, e.g. max for 32-bit apps is ~940 Mb or ~1.9 Gb (for more details, see here)
  

[ TOP ]

E. 32-bit memory limitation, General Information

• Running without memory windows, HP-UX has limitations for shared resources on 32-bit applications. All applications in the system share and are limited to a total of 1.75GB of shared memory, 2.75GB if compiled as SHMEM_MAGIC. In a system with 16GB of physical memory, only 1.75 can be used for shared resources!!!! To address this limitation, a functional change has been made (Memory Windows was introduced by patches at 11.0 and is included in 11.11) to allow 32-bit processes to create unique memory windows for shared objects like shared memory.
  
  
• For 32 bit applications, the maximum size of a single shared memory segment a process can attach is limited to 1 GB (shmmax <= 0x40000000). The first 1 Gb resides in quadrant 3, and quadrant 4 only has 0.75 Gb reserved for shared memory and memory mapped files. So the total memory addressable by the default executable type is 1.75 Gb. 
  
  An executable type SHMEM_MAGIC has been defined which adds the use of quadrant 2 for shared memory and memory mapped files.  This additional 1 Gb results in a system-wide maximum of 2.75 GB of shared memory ­and memory mapped files address space. See below for more details about SHMEM_MAGIC. (note: for more on 32-bit memory limitation, see: “Understanding Shared Memory on PA-RISC Systems”, ITRC Doc id RCMEMKBAN00000027.)
  

• If you see “out of memory” or “not enough space” when running an application, and there is pleny of free swap space then the application may be requesting shared memory or may be mapping files to memory (with shmget() and mmap() system calls respectively) and the problem may due to 32-bit memory limitation/contention and the options are...
  
  
• Q: Is application 32-bit or 64-bit?
  
  
• A: To determine if an application binary is  32-bit or 64-bit, use the file(1)
  
  
• 32-bit example:
  

• SHMEM_MAGIC executables can address 2.75 Gb
  • For more details than explained below (about SHARE_MAGIC, EXEC_MAGIC, AND SHMEM_MAGIC), see ITRC doc id rcfaxmemory001 ("Shared_magic Explained")
    
    
• SUMMARY
  • To get SHMEM_MAGIC, the executable needs to have been previously linked with EXEC_MAGIC ("ld -N") and then can be chatr'ed to get SHMEM_MAGIC (i.e. with the "chatr -M" option.)
    
    
  • Check with the vendor's application support to see if their application supports SHMEM_MAGIC.
    
    
  • For example, for Sybase support of SHMEM_MAGIC, see: http://my.sybase.com/detail?id=1010341
    
    
  • PATCHES - Unlike 11.0 which doesn't require patches for SHMEM_MAGIC, 10.20 needs patches. The 10.20 patches are:
    
    • PHKL_16750 (for s700) PHKL_16751 (for s800)
      • These are LITS (Line-In-The-Sand) patches and will never be superseded.
        
    • PHSS_21110 (linker/ld patch)
      • Note: PHSS_21110, may be superseded. Please check for the latest patches at the IT ResourceCenter (ITRC) at the following web site:       http://www.itrc.hp.com/
        

• Q: Is application set up with SHMEM_MAGIC?
  
  
• A: To determine if the 32-bit application is setup with SHMEM_MAGIC or is capable of SHMEM_MAGIC, use the chatr(1) command. For example:

[ TOP ]

• Normally, 32bit apps can, as allowed by maxdsiz, can get up to ~940 Mb of data space, unless they have been linked with EXEC_MAGIC, in which case, they can get upwards of 1.9 Gb of data space.
  
  
• Alternatively,  chatr(1) may be used to enable the third quadrant private for data space.  Thus enabling an addition 1 Gb quadrant to be used for data space.
  
• For example:     chatr +q3p enable executable_name
  
  
• To determine what the executable is capable of, run:
  
         # chatr executable_name
  
  
• if it shows as "shared executable"  then it can only get to about 940Mb,
  
• if it shows "EXEC_MAGIC", then it can get upwards of 1.9 Gb of data space.

• maxdsiz  - maximum size (in bytes) of the data segment/space for any user process.

• What is Memory Windows?  Memory Windows is a functionality that was provided for consolidating 32-bit applications on 64-bit servers. Memory windows will allow each application/database to have its own shared memory space. There is still the 1.75GB limit (or 2.75GB limit with SHMEM_MAGIC) for each window, but there can be multiple windows on a system. Processes can not access more than one window. Testing will be needed to confirm if your applications can exist in a memory window. See below for more details.
  

• Overview - Running without memory windows, HP-UX has limitations for shared resources on 32-bit applications. All applications in the system are limited to a total of 1.75GB of shared memory, 2.75GB if compiled as SHMEM_MAGIC. In a system with 16GB of physical memory, only 1.75 can be used for shared resources!!!! To address this limitation, a functional change has been made (Memory Windows was introduced by patches at 11.0) to allow 32-bit processes to create unique memory windows for shared objects like shared memory.  This allows cooperating applications to create 1GB of shared resources without exhausting the system-wide resource. Part of the virtual address space remains globally visible to all processes, so that shared libraries are accessible no matter what memory window they are in.  The following customer-visible changes have been made for memory windows:
  
  • New kernel tunable, max_mem_window(5), to configure the number of memory windows a system can have.
  • New set of commands and files and their assciated man pages:
    
    • getmemwindow(1M)
      
    • setmemwindow(1M)
    • services.window(4)
  
  
  
• There is a Memory Windows White Paper in pdf format and is also availabine in an ascii text version -  /usr/share/doc/mem_wndws.txt
  
  
• There is a Memory Windows summary in the 11i release notes.
  
  
• Memory Windows are NOT supported by 
• OmniBack
• see "/usr/lib/dld.sl: Call to mmap() failed - TEXT /opt/omni/gui/bin/libcore.sl" ITRC doc id OV-EN007127
• OpenView (see non-support statement for HP-UX 11 Memory Windows in the OVO/UNIX 7.1 Release Notes.)
  
  
• Memory Windows are supported (contact Vendor for details) by:
• Oracle 8.06, or later "Using memory windows with Oracle" ITRC Doc Id: KBRC00009528

• GOTCHA: The default (SHARE_MAGIC) executable’s maximum size memory window is 1 gigabyte. Any consumption beyond 1 gigabyte consumes space from the 4th quadrant which is shared across *ALL* processes in the system. This is important, any application within a memory window that uses more than 1 gigabyte of shared memory consumes quadrant 4 resources that are shared by all processes no matter what memory window they occupy.

• Yet More Details...

• Check with the vendor's application support to see if they support Memory Windows.
  
  
• Patches - Memory Windows was originally introduced with 11.0 Patches: 
       PHKL_13810 (Kernel)
             PHCO_13811 (Commands)
  
  
• The current* patches are PHKL_18543 &  PHCO_23705
• * check the ITRC http://itrc.hp.com for the latest versions of patches.

• 11.11 (11i) does not need patches for Memory Windows.
  
  
  

• Is system already configured for Memory Windows?
  
• Either, test with  setmemwindow(1M), for example:
      # setmemwindow date
  
  
• Memory Windows is not configured if you get nothing from the date(1) command (i.e. nothing comes back), or if you get an error like this:
      Error(12), unable to set memory window(-1)
  
  
• Or check the kernel parameter max_mem_window to see if it has been set, with:
      # grep max_mem_window /stand/system

• NOTE: If you do NOT see max_mem_window(5) as a kernel configurable parameter in SAM, then you can install the latest 11.0 SAM patch  (SAM was first made aware of max_mem_window with PHCO_21187) or you can add max_mem_window to system file manually and then generate a new kernel.

• memwin_stat: 
• is an unsupported utility that displays information about shared  memory segments, processes and the Memory Windows themselves.
• is available for download, here:
    System:     hprc.external.hp.com  (192.170.19.51)
    Login:      contrib
    Password:   9unsupp8
        ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/memwin/memwin_stats
• and for download, here: 
    System:     hprc.external.hp.com  (192.170.19.51)
    Login:      eh
    Password:   spear9
    URLs:       
      ftp://eh:spear9@hprc.external.hp.com/memwin_stats.README.txt
      ftp://eh:spear9@hprc.external.hp.com/memwin_stats1100.sh
      ftp://eh:spear9@hprc.external.hp.com/memwin_stats1111.sh
      ftp://eh:spear9@hprc.external.hp.com/memwin_stats.html
  
• Footnote about memwin_stats availability:
• The PDF version of the Memory Windows White Paper (http://docs.hp.com/hpux/onlinedocs/os/memwn1_4.pdf) points the reader to an ftp site: ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/memwin/memwin_stats
• The text version of the Memory Windows White Paper (/usr/share/doc/mem_wndws.txt) points to an ftp site:
     ftp://hpchs.cup.hp.com/tools/11.X/memwin_stats.shar
• Example:

[ TOP ]

• Alternative ps command - Alternatively, you can use the UNIX95 options to look at both Virtual Size as well as the actual Size.
  
• Run:
  # UNIX95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | more

• For example:
  # UNIX95=1 ps -efo vsz,sz,pid,args |grep -v grep | sort -rnk 1 | more
  VSZ     SZ   PID COMMAND
  12252  627  2745 /opt/OV/bin/ovdbrun -c /var/opt/OV/share/databases/analysis/
  9060  1214  2362 /opt/omni/lbin/rds -d
  8808  1892  2677 /opt/hpwebjet-5.5/hpwebjetd
  
  

I.  Memory Usage from "physmem", "swapinfo", "top", and "glance".

• How do I undertand/resolve the different result about the memory usuage from "physmem", "swapinfo", "top", and "glance".

• Physical Memory
  • Can use dmesg to report Physical memory (RAM) info. For example:
    
       # dmesg | grep Phys
       Physical: 212992 Kbytes, lockable: 152792 Kbytes, available: 178188 Kbytes
    
    
  • Note:
  • This system has 2GB physical memory.
  • Lockable memory is used for
    • Process images and overhead locked using the plock() system call (see HP-UX Reference entry plock(2)).
    • Shared memory segments locked with the SHM_LOC command of the shmctl() system call (see HP-UX Reference entry shmctl(2)).
    • Miscellaneous dynamic kernel data structures used by the shared memory system and some drivers
    
    
    
  • Can also report Physical memory (RAM) size with adb:
    

• SWAP
  

• The "memory" line in the output of swapinfo is NOT physical memory, rather it is pseudoswap which is calculated to be 75% the size of RAM.
• MORE...

[ TOP ]

• top

         ^          ^             ^         ^                  ^

         |          |             |         |                  |

• Memory is not all of physcial, memory, it is:

1. Total physical memory in the system DEDICATED to text, data or stack segments for all processes on the system.
2. Total physical memory for runnable processes, as opposed to sleeping processes.
3. Total memory dedicated to text, data or stack segments for all processes on the system. Some of this is paged out to disk (that is, not all of this is in current physical memory.)
4. Total memory for runnable processes, as opposed to sleeping or stopped processes.
5. Physical memory the system considers to be unused and available to new processes. When this value is low, swapping is likely to occur.

• For more about top, see top(1) man page.  For example:
  

• SAM
  
• For SAM's System Properties, see "Help" for description of values shown, which are similar to those descriptions given above for top(1).
  
• ============================================================================
  SAM Areas:Performance Monitors:System Properties-> Memory screen:
  ============================================================================
                             System Properties (opie1)                        
     /-----------------------------------------------------------------------\
     |                                                                       ^
     | [ Refresh ]                                                            
     |                                                                        
     | /-----------\/--------\/------------------\/---------\/---------\      
     | | Processor || Memory || Operating System || Network || Dynamic |      
     | /------------/        \----------------------------------------------\ 
     | |/------------------------------------------------------------------\| 
     | ||Physical Memory:         3010.7 MB                                || 
     | ||Real Memory:                                                      || 
     | ||  Active:                452098.5 KB                              || 
     | ||  Total:                 516787.5 KB                              || 
     | ||Virtual Memory:                                                   || 
     | ||  Active:                1450485.6 KB                             |||
     | ||  Total:                 1538524.9 KB                             |||
     | ||Free Memory Pages:       340696 at 4 KB/page                      |||
     | ||Swap Space:                                                       |||
     | ||  Avail:                 4096 MB                                  |||
     | ||  Used:                  674 MB                                   ||v
     \-----------------------------------------------------------------------/
     -------------------------------------------------------------------------
      [   OK   ]                                                   [  Help  ] 
  
  ============================================================================
  SAM Areas:Performance Monitors:System Properties-> Dynamic screen:
  ============================================================================
                            System Properties (opie1)                        
     /-----------------------------------------------------------------------\
     |                                                                       ^
     | [ ] Auto Refresh                                                       
     |                                                                        
     | /-----------\/--------\/------------------\/---------\/---------\      
     | | Processor || Memory || Operating System || Network || Dynamic |      
     | /-----------------------------------------------------/         \----\ 
     | |/------------------------------------------------------------------\| 
     | ||Processor:                                                        || 
     | ||  Active Processors:      1                                       || 
     | ||Memory:                                                           || 
     | ||  Real Active:            460138.1 KB                             || 
     | ||  Virtual Active:         1464187.7 KB                            || 
     | ||  Free Memory Pages:      339804 at 4 KB/page                     |||
     | ||  Swap Space:                                                     |||
     | ||    Used:                 675 MB                                  |||
     | ||    Free:                 3421 MB                                 |||
     | ||Operating System:                                                 |||
     | ||  Unique Users Logged In: 1                                       ||v
     \-----------------------------------------------------------------------/
     -------------------------------------------------------------------------
      [   OK   ]                                                   [  Help  ]

• Glance

• Total VM: The total private virtual memory (in KBs unless otherwise specified) at  the end of the interval.  This is the sum of the virtual allocation of private data and stack regions for all processes.   

• Active VM: The total virtual memory (in KBs unless otherwise specified) allocated for processes currently are on the run queue or processes that have executed recently.  This is the sum of the virtual memory sizes of the data and stack regions for these processes.

• Sys Mem: The amount of physical memory KBs unless otherwise specified) used  by the system (kernel) during the interval.  System memory does not include the buffer cache.
  • On HP-UX 10.20 and 11.0, this metric does not include some kinds of dynamically allocated kernel memory, which has always been reported in the GBL_MEM_USER* metrics.              
  • On HP-UX 11i and beyond, this metric does include some kinds of dynamically allocated kernel memory.

• Buf Cache: The amount of physical memory (in KBs unless otherwise specified) used by the buffer cache during the interval. The buffer cache is a memory pool used by the system to stage disk IO data for the driver. 

• User Mem: The amount of physical memory (in KBs unless otherwise specified) allocated to user code and data at the end of the interval.  User memory regions include code, heap, stack, and other data areas including shared memory.  This does not include memory for buffer cache. 

• On HP-UX 10.20 and 11.0, this metric does include some kinds ofdynamically allocated kernel memory.    
• On HP-UX 11i and beyond, this metric does not include some kinds ofdynamically allocated kernel memory,  which now is reported in the GBL_MEM_SYS* metrics.               
• Large fluctuations in this metric can be caused by programs whichallocate large amounts of memory and then either release the memory or terminate.  A slow continual increase in this metric may indicate a program with a memory leak.

• Free Mem: The amount of memory not allocated (in KBs unless otherwise specified).  As this value drops, the likelihood  increases that swapping or paging out  to disk may occur to satisfy new memory requests.

• Phys Mem:  The amount of physical memory in the system (in KBs unless otherwisespecified).  Banks with bad memory are not counted.

• Note that on some machines, the Processor Dependent Code (PDC) code usesthe upper 1MB of memory and thus reports less than the actual physical memory of the system. Thus, on a system with 256MB of physical memory, this metric and dmesg(1M) might only report 267,386,880 bytes (255MB).  This is all the physical memory that software on the machine can access.

[ TOP ]

J. Troubleshooting Examples:  "Not enough space"  , "out of memory",  "Not enough core" (a.k.a. HPUX errno 12, ENOMEM.)

• "call to mmap failed" when accompanied by "not enough space": 
• If there is no lack of available swap, then the cause is typically contention for and/or fragmention of  the 32-bit address space used by shared memory and memory mapped files, which can be viewed using an unsupported utility called shminfo (example download).
  
  
• "Not enough space" (examples), "out of memory", or  "Not enough core" (a.k.a. HPUX errno 12, ENOMEM)
  • 1.) If the app/db is requesting shared memory, and the amount requested is more than the value of the shmmax(5) kernel parameter, then shmmax needs to be increased.
  • If it is determined that shmmax is not causing the failure, then the requested amount of shared memory could not be obtained due to lack of requested amount of contiguous memory (i.e  memory is fragmented).
  • 2.) Whether or not app/db is using shared memory, the problem may be caused by not enough free swap space. So, check to see if there is enough swap. Use 'swapinfo -tm' and see how much total free space there is.
    
    
  • 3.) If not a problem with shmmax nor with swap, then, the cause is most likely either data / stack kernel parameters OR shared memory configuration or contention/fragmentation.
    
    
    • To see if the problem is due to data / stack kernel parameters, determine which one, you can use tusc to trace the system calls to see which system call is failing and to see the ERRNO.
      
      
    • a.) If  malloc() ... system call equivalent in tusc output would be called "brk"... then check data/stack kernel parameters...
      
      
    • data/stack kernel parameters  will halt processes when their stack or data grows near (or attempts to pass) the maximum defined by maxssiz and maxdsiz (respectively.)  
      
    • NOTE: Prior to HP-UX 11.23 (a.k.a. HP-UX 11i Version 1.6), these kernel  parameters (maxssiz and  maxdsiz) are static and so changes to them require a reboot.
    • Normally 32bit apps can only get to ~940 Mb of data space.   32-bit apps can get upwards of 1.9 Gb of data space if the executable is compiled with EXEC_MAGIC (ld -N) or if chatr is used to enable third quadrant private (chatr +q3p enable executable_name).  To determine what the executable is capable of, use 'chatr executable_name' and if it shows as 'shared excutable' then it can only get to about 940Mb.
    • This is harder to determine and may need trial and error of increasing maxssiz and/or  maxdsiz until error stops.
    • If the process takes long enough to fail, then you can monitor it's stack and data usage with procsize.
    • The default for maxssiz is  8Mb and the default for maxdsiz is 64 Mb, they may need to be doubled, tripled, or quadrupled to resolve (i.e. unless Vendor recommends/knows good values, use trial-and-error.)
      
      
    • b.) If mmap()  or  shmget() is failing, then check for shared memory configuration or contention / fragmentation...
      
      
      • shared memory configuration or contention / fragmentation
      • Typically this is seen where other applications/dbs are using up shared memory to the extent that there is not any more left.
        
      • The options are, either:
      • reboot will defragment this memory.
      • or memory use should be reduced (by lowering the amount requested by the apps/dbs).
      • or temporarily shutdown apps/dbs that are using 32bit address space.
      • or use Memory Windows (Memory Windows allows 32-bit processes to create private/unique memory windows for shared objects like shared memory.)
        
      • To view the existing memory usage, including fragmentation and largest FREE memory segment, use shminfo.
        
      • For example.
      • To download shminfo.
        
      
      

• "Not enough space" examples:
  • /usr/lib/dld.sl:Call to mmap() failed - ZEROES /usr/lib/libdce.1

• From attempting to run sam in TUI (Text) mode with swap 99% used.

• OBAM INTERNAL ERROR: Cannot fork: Not enough space

• From attempting to run sam in GUI (graphic) mode with swap 99% used.
  

• sam: FATAL ERROR: Unable to load library "/usr/obam/lib/libIDMawrt.1": Not enough space

• From attempting to run sam in TUI (Text) mode with swap 99% used.
  

• /usr/lib/dld.sl:Call to mmap() failed - BSS /usr/lib/libnsl.1

• Seen during login [as root user] when system had swap at 99% used.

1. Download  the "memory" tool from the following ftp site:
         System:     hprc.external.hp.com  (192.170.19.51)
         Login:      eh
         Password:   spear9
         ftp://eh:spear9@hprc.external.hp.com/
   
   
2. Use ftp://eh:spear9@hprc.external.hp.com/memory.README.txt to see what to download, how to extract the memory tool from the compressed tar file, and how and when to run the tool.
   
   
3. Look for where all of the memory is being used.
   Work to understand memory use or reduce the memory use by applications/os/databases.
   Compare the results of running these when the memory issue is evident *and* after a reboot (as a baseline.)
   

• Memory Report Examples (including output of swapinfo, ps, procsize, shminfo, and kmeminfo):
• 11.11 
  
• 11.00
  

• shminfo example: