Home | WinRunner Quick Reference | LoadRunner Quick Reference | SilkTest Quick Reference | C Quck Reference | TestDirector Quick Reference | SQL Quck Reference | Unix Commands Quick Reference | Software Testing | Networking Quick Reference | VBScript Quick Reference
Software Testing Automation
LoadRunner Quick Reference

tiger.jpg

What's the difference between load and stress testing ?

1.    Stress testing is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, interrupts, etc.) needed to process that load.  The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful.  The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data).  Bugs and failure modes discovered under stress testing may or  may not be repaired depending on the application, the failure mode, consequences, etc.  The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.

2.    Load testing is subjecting a system to a statistically representative (usually) load.  The two main reasons for using such loads is in support of software reliability testing and in performance testing.  The term "load testing" by itself is too vague and imprecise to warrant use.  For example, do you mean representative load," "overload," "high load," etc.  In performance testing, load is varied from a minimum (zero) to the maximum level the system can sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.

3.    A third use of the term is as a test whose objective is to determine the maximum sustainable load the system can handle.   In this usage, "load testing" is merely testing at the highest transaction arrival rate in performance testing.

 

"Load testing" - Verifies the acceptability of the target-of-test's performance behavior under varying operational conditions (such as number of users, number of transactions, etc.) while the configuration remains constant.

"Stress testing" - Verifies the acceptability of the target-of-test's performance behavior when abnormal or extreme conditions are encountered, such as diminished resources or extremely high number of users.

 "Performance testing" is a class of tests implemented and executed to characterize and evaluate the performance related characteristics of the target-of-test such as the timing profiles, execution flow, response times, and operational reliability and limits.

-------------------------------------------------------------------

Simple definition.

Performance tests - test performance in any combination of ways you can think of.

Load - tests "stuff" under realistic, accurately modeled loads.

Stress - tests "stuff" under un-realistic, i.e. "stressful" conditions.

 

 Test Approach

 

Stress Test

Stress testing defines the number of concurrent users that the system can handle. This test verifies that the system has the capacity to handle large number of transactions during peak period. The purpose of this test is to find out what system resource is the most critical to the system performance. This test includes increasing the load on the system until the stress point is reached to identify the bottleneck as the following.

-         As load increases before the system reaches its stress point, performance results are linear and proportional to load.

-         As load increases after the system reaches its stress point, performance results are exponentially proportional to load.

-         This stress point is referred to as knee point.

 

Volume Test

Volume test includes running tests for a period of four hours to determine memory leaks and system behavior. System behavior is determined by increasing load up to 50 percent of system capacity and running for four hours.

             

Spike Test

Spike testing  (also known as bounce testing) simulates system behavior similarity to real production usage where load is irregular. The system is loaded to 50 percent capacity then loaded to maximum and release & repeated. This determines the system ability to recover when placed under sporadically changing loads.

 

Load Testing – Server Response time

This test determines server response time at different loads for critical transactions as mentioned below. It determines the maximum number of connections that servers can handle with better response time.

 

Failover Test

This test determines behavior of secondary system components when the primary component becomes unavailable through either failure or scheduled down time. Run the servers with 50% of load up to 30 minutes. Owing this period turnoff one or more servers from load balancing. Ensure that load is evenly distributed among remaining servers. Return the turned off servers to original state to determine if load begins balancing again

 

LoadRunner Monitors
Web Server resources(Apache)
# of busy servers - The number of servers in the busy state
# of idle serveres - The number of servers in idle state
Apache CPU usage - The % of time the cpu utilized by Apache
Hits/sec                - The HTTP request rate
KBystes sent/sec    - The rate at which data bytes are sent from the web server
 
Application server - Weblogic
 
Serverstate : State of the WebLogic Server, as inferred by the SNMP agent. Up implies the agent can contact the server; down implies the agent cannot contact the server
 
ServerHeapUsedPct: Percentage of heap space currently in use in the server.

ServerQuelenngth:
Specifies the maximum number of simultaneous requests that the server can hold in the queue. The default of 65536 requests represents a very large number of requests; outstanding requests in the queue should rarely, if ever reach this maximum value. Always leave the Queue Length at the default value of 65536 entries.
 
ServerQueThroughput:Current throughput of execute queue, expressed as number of requests processed per second
 
Unix server resource monitoring
 
 
HTTP status codes
OK 200  -- The request was fulfilled
Bad request 400
Not found 404 - The server has not found any thing matching to the
                      URL given
Internal Error 500
 
Load Average
Load Average is defined as the average number of jobs in the run queue. So a load average of 14.4 indicates that 14.4 processes were on average waiting to be served by the processors in your system (being 4 in your case if it was on web/app server or 2 if load average was on your database server).
 
Memory leak

A memory leak is the gradual loss of available computer memory when a program (an application or part of the  operating system) repeatedly fails to return memory that it has obtained for temporary use. As a result, the available memory for that application or that part of the operating system becomes exhausted and the program can no longer function. For a program that is frequently opened or called or that runs continuously, even a very small memory leak can eventually cause the program or the system to terminate. A memory leak is the result of a program bug.

Some operating systems provide memory leak detection so that a problem can be detected before an application or the operating system crashes. Some program development tools also provide automatic "housekeeping" for the developer. It is always the best programming practice to return memory and any temporary file to the operating system after the program no longer needs it.

 

Loadrunner stuff

 

What is new in loadrunner 8.1

Loadrunner quickstart – documentation step by step

Monitors – new integration of sitescope monitors

New protocols

VUGen

            Workflow wizard, thumbnails and transation editor, debugging capabilities

Recording options:

HTML based script – generates a separate step for each HTML user action.( for browser applications)

URL based script – captures all http requests  sent to the server as a result of each user action ( for non browser application)

 

When you want to scale your environment for higher number of vuser. You want to record HTTP or analog mode. Context sensitive or HTML recording heavily rely on caching mechanism. This may help your script look easier to read, however they will not scale to greater vusers. Analog (HTTP) scripts do record each and every request independently. This will make your script look lot bigger and cumbersome. Since HTTP statements are indepentent of each other they can help you simulate larger load (as caching not required).
Complex system doesn't warrant for any http or html recording. It's your call based on hardware you have.
Hope this helps

In loadrunner 7.0, HTML scripts are nearly as scalable as HTTP scripts. HTML scripts are shorter, higher level, require less corrolation and are easier to maintain. They now also run on UNIX. So HTML should be used whenever possible.

However in some cases HTML will not work, for instance if your transactions do not come from HTML, eg. Axtive X, Java applets, or javascript. In these cases HTTP mode is a good fallback.

Bottom line: Try HTML first. If this does not work, you can always use HTTP.

 

How do I obtain the routing table for my PC?
From an MS-DOS prompt type: "netstat -r" OR "route print". Both of these commands will display the routing table similar to the one below.
Active Routes:

Try running the "webtrace" command from the command prompt of the FROM machine.

   C:\>webtrace < server >

See what the results are and verify that no errors are generated.

Example:
D:\>webtrace superman

1 ) retry: 0 err: 0 000.000000 172.16.1.1 ? [000.000000]
2 ) retry: 0 err: 0 000.000000 172.16.2.10 SUPERMAN [000.000000]

The results here show that the timing is smaller than 000.0000000 seconds -- very fast.

Also, using standard traceroute shows similar data (although not as fine grained):

D:\>tracert superman

Tracing route to superman [172.16.2.10]
over a maximum of 30 hops:

  1    10 ms   <10 ms   <10 ms  172.16.1.1
  2   <10 ms   <10 ms   <10 ms  SUPERMAN [172.16.2.10]

Trace complete.

In VuGen:
The lr_log_message function prints messages to the output log. This log file is in the output.txt file within the script directory.

In the Controller:
The lr_log_message function prints messages to the log folder in the results directory as specified in Results -> Results Settings in the Controller. Note that this will NOT send messages to the Controller window.

To use lr_log_message, you need to make sure that the following is set in the Run-Time Settings:
1. Under General:Log, select "Enable Logging."
2. For the Log Options, select "Always send messages."

Process/Multi threading

If you run each Vuser as a process, then the driver (The Controller uses a driver program (e.g., mdrv.exe, r3vuser.exe) to run your Vusers) program is launched (and loaded) into the memory again and again for every instance of the Vuser. Loading the same driver program into memory
uses up large amounts of RAM (random access memory) and other system resources. This limits the numbers of Vusers that can be run on any load generator.

When there is no such a huge load to distribute among three load injectors, it is using only two web servers.

Run Vuser as a thread.

 

Run time setting

1.Bandwidth Networking speed – select maximum bandwidth

2.

Multi threading – ability to run more vusers per load generator – only threadsafe protocols to be run as multi threading

            If we run vuser as thread, the contoller launches only one driver program for every 50 users . This program launces several vusers , each vuser as thread.

The contoller uses a driver program ie mdrv.exe or r3vuser.exe to run vusers.If we run each vuser as a process , it launches drivers program in to memory again and again for every instance of vuser.This uses large amount of RAM and other system resources.

How to find which protocol is threadsafe

Solution: Vuser type with "EnableThreads=1" is threadsafe

Note: Do not modify this setting.

For LoadRunner 7.6 or above:
1. Go to <LoadRunner>\dat\protocols
2. Open up the relevant .lrp file (e.g. for Web, open the QTWeb.lrp file )
3. Search for "EnableThreads". If it is 1, the Vuser is threadsafe

 

Vusers per load generator

            50 users per load generator for machine with 512 mb memory

The primary factors affecting the script execution are Memory and CPU Utilization

Others are Network Bandwidth and Hard Disk speed

Some rules of thumb for monitoring load would be

  • CPU usage should stay below ~80-90% continuous usage.
  • Memory usage should stay within the physical limits of the machines memory. This means that the "Commit Charge Total" should be less than the "Physical Memory Total" which will ensures that minimal paging is done to simulate physical memory.

Memory footprint for web users

Memory required for each web vuser

Operating system

Win2000 Advance Server

XP

Protocol

 

Process

Thread

Process

Thread

(MB)

(MB)

(MB)

(MB)

WEB

Infra

6.82

13.74

8.77

8.98

Next

5.02

0.84

4.61

0.83

 

 

Monitors info

Run time monitors – number and status of vusers

Transaction monitor – response time

Web resourse monitor – measures statistics on web servers  - hits per second,

throughput,http responses, pages downloaded per second,

System resources  - average load, cpu utilization,  to monitor we need to configure rstatd deamon on unox servers.

Network delays

Firewall monitors

Web server resource monitors  - Apache – http:// server name/serverstatus?auto

-         # of busy servers

-         # of idele servers

-         Apache cpu usage

-         Hits per sec

-         Kbsent per second

-          

Web application server resource

Database server resource – cpu used by this session

-         bytes recived via SQL * net from client

-         logon current

-         user calls

-         dead locks

-         opened cursors current

-         total files open

-          

 

 

 

 

 

Weblogic - Currently (LoadRunner 8.0), WebLogic 5.x - 8.x are supported

            Loadrunner provides two different ways to monitor weblogic server depending on weblogic version that is SNMP based moniotor and JMX monitor

WebLogic Monitor (SNMP-based monitor)

WebLogic monitor (SNMP- based) gathers measurement based on information returned by the WebLogic SNMP agent. 

·        Availability: LoadRunner 6.0

·        Supported Server Version: WebLogic 5.x

·        Supported Server Platform: Windows NT, Windows 2000

WebLogic Monitor (JMX monitor)

WebLogic monitor (JMX) gathers measurement based on Java API. It uses the JMX interface, (which is a Sun extension to Java.

·          Availability: LoadRunner7.02 or above

·          Supported Server Version: WebLogic 6.x –7.x. For WebLogic8.x, please refer to the additional setup information below .

·          Supported Server Platform: Windows NT, Windows 2000, Solaris 2.6, AIX 5.1, HP-UX 11

How to configure the SNMP-based monitor in LoadRunner

 

1.      Click the WebLogic (SNMP) graph in the graph tree, and drag it into the right pane of the Run view.

2.      Right-click the graph and choose Add Measurement(s), or choose Monitors à Add Online Measurement.

3.      In the Monitored Server Machines section of the WebLogic (SNMP) ( or WebLogic) dialog box, click Add to enter the server name or IP address of the machine you want to monitor. Select the platform on which the machine runs, and click OK.

4.      In the Resource Measurements section of the WebLogic (SNMP) dialog box, click Add to select the measurements that you want to monitor. The WebLogic SNMP Resources dialog box displays the available measurements.

5.      Browse the WebLogic SNMP Objects tree.

6.      To measure an object, select it, and click Add. The following tables describe the measurements and server properties that can be monitored

 

The SNMP-based monitor can only monitor up to 25 measurements.

 

Database monitors

Oracle monitor has the following tables to see   V$STATNAME
   V$INSTANCE
   V$SESSION
   V$SESSTAT
   V$SYSSTAT
   V$PROCESS

By default, only V$SESSSTAT and V$SYSSTAT are available in the Oracle monitor. Using the custom query feature, you can define your own query for V$STATNAME, V$INSTANCE or V$SESSION to the Oracle database and view the result of this query (a single numerical value) in the Oracle online monitor graph.

 

 

Correlation

Correlation is fundamental to replaying scripts against sites that use dynamic values such as Session IDs, Database Primary Keys, and almost all security done over HTTP

            Automtic correlation – we can set rules to correlatre

            Manual correlation : record two scripts and highlight difference with windiiff

Counters

Average load

 

 

Bottlenecks

 

Functions:

Lr_eval_string – replaces a paramenter with current value

Lr_start_transaction,lr_end_transaction

Lr_log_message – logs a message to vuser log file

 

Load average

 

Load Average is defined as the average number of jobs in the run queue. So a load average of 14.4 indicates that 14.4 processes were on average waiting to be served by the processors in your system (being 4 in your case if it was on web/app server or 2 if load average was on your database server).

For a process to be in the 'run queue', it must be waiting on CPU (ie not waiting on I/O or any other resource).

 

Memory leak

A memory leak is the gradual loss of available computer memory when a program (an application or part of the operating system) repeatedly fails to return memory that it has obtained for temporary use. As a result, the available memory for that application or that part of the operating system becomes exhausted and the program can no longer function. For a program that is frequently opened or called or that runs continuously, even a very small memory leak can eventually cause the program or the system to terminate. A memory leak is the result of a program bug.

Some operating systems provide memory leak detection so that a problem can be detected before an application or the operating system crashes. Some program development tools also provide automatic "housekeeping" for the developer. It is always the best programming practice to return memory and any temporary file to the operating system after the program no longer needs it.

How to determine db delays

  use another DB monitor such as Omegamon and I can get all kinds of useful stats such as DB wait time, IO time, connection time, etc.  I can drill down and find where time is spent on a transaction.  I would think LRs database monitor provides similar capability although I am not too familiar with using the tool.

  

 Performance counters

Unix box

-         average load

-         cpu usage

-         cpu system

-         cpu user

-         idle time

-         swap memory

-         memory usage

Web server(Apache, IIS)

CPU usage

KB sent

# of idle servers

# of busy servers

Hits/sec

Database

            Logson current

            Opened cursors

            User calls

            Cpu used by

            Global locks

            Physical reads

            Byses sent

Weblogic

Idle threads

Que length

MEmory

 

Function Name

Description

calloc

Allocates an array and initializes all elements to zero.

free

Frees a block of memory.

malloc

Allocates a block of memory.

realloc

Reallocates (adjusts the size of) a block of memory.

 

Lr_decrypt – to decrypt passwords

Lr_message – message to vuser log and controller out put window

Lr_log_message – logs messages in output.txt in vuser script directory

Lr_output_message – to only out put window

Lr_error_message – send error message to out put window

 

Lr_think_time – for think time

 

 

Parameterization with group number, iteration number, random number

Update method

            Each occurance, each iteration, once

Importing data from existing database – with database query wizard from parameter list

-         by creating a new query

-         by manually entering sql quer y and connection string  through ODBC source

User defined functions

            A user defined function replaces a parameter with a value returned from a function located in an external dll. Create an external library files (dll) with function

 

Load dll using lr_load_dll function

            lr_load_dll("C:\\MyProject\\MyDll.dll");

To register a DLL or OCX file

Method 1:
1. Open a DOS window.
2. Navigate to the DLL's or OCX's directory.
3. Enter regsvr32.exe <filename>.<dll/ocx>.

lr_load_dll("C:\\...full_path_to_the_DLL...\\dll_name");
//Call your own function from dll_name, it should work now

rc = ci_unload_dll(ci_this_context, "C:\\...full_path_to_the_DLL...\\dll_name");
lr_message("\"rc==%d", rc); // check the return code for ci_unload_dll
//Call your own function from dll_name, it should fail now

 

 

 

Enter supporting content here