•  

User Story

QA department has a task to measure PortaBilling performance via Diameter interface, in the way like it is currently measured for RADIUS interface.

Use Cases

Common preconditions: Host(s) with test tool up and running; Test system with PortaBilling instance(s) up and running; IP connection is configured between host(s) with Test tool and PortaBilling instance(s).

Use case #1: Session template control

Roles: QA engineer, Test tool, PortaBilling

Use scenario #1.1: Create Diameter session template

Scenario: 

  • QA engineer creates a JSON-like(native Erlang in fact) file

  • QA engineer instructs test tool to parse prepared JSON-like file

  • Test tool determines sequence of Diameter requests which should be generated and sent to PortaBilling.

  • Test tool generates Diameter requests based on AVP image described by QA engineer

  • Test tool reacts on PortaBilling responses based on behavior prescribed by QA engineer


Use case #2: Test session parameters control

Roles: QA engineer, Test tool, PortaBilling

Preconditions: JSON-like template created.

Use scenario #2.1: Pre-configure test session parameters

Scenario: 

  • QA engineer pre-sets next parameters:
    1. limit for concurrent active sessions ('ACCT_SESSIONS_MAX_COUNT')(optionally)

    2. speed of 'INITIATION' requests generation ('INIT_RPS')

    3. number of 'UPDATE' requests within particular Diameter session ('UPD_COUNT')
    4. time-interval between sequential 'UPDATE' requests ('UPD_INTERVAL') within particular Diameter session
    5. time-interval between last-1 and last 'UPDATE' requests ('UPD_INTERVAL_LAST') within particular Diameter session
    6. duration of the test session ('TST_DURATION')(optionally)
    7. max timeout for PortaBilling to response ('BE_RESP_TIME_MAX')
  • QA engineer starts new test session using appropriate control of the Test tool

  • Test tool starts send/receive requests to/from PortaBilling according to pre-set settings.

Use scenario #2.2: Change pre-configured test session parameters on the 'fly'

Scenario: 

  • Test tool has test session in progress
  • After checking monitoring graphs of PortaBilling instance under the test, QA engineer realizes that he should decrease value of 'INIT_RPS'

  • QA engineer decreases INIT_RPS value using appropriate control of the Test tool

  • Test tool reacts on the event of parameters changing, and starts adjusting actual 'INIT_RPS' rate without interrupting current test session

Use scenario #2.3:  Control load "profile"

Scenario: 

  • QA engineer is going to perform test session with different values of 'UPD_COUNT'/'UPD_INTERVAL' /'UPD_INTERVAL_LAST' per particular Diameter session
  • QA engineer pre-sets several sub-session types(withing particular test session), in form: "Weight(aka %) - UPD_COUNT/UPD_INTERVAL/UPD_INTERVAL_LAST"

  • QA engineer starts the new test session using appropriate control of the Test tool

  • Test tool calculates and starts to generate requests according to the pre-set sub-session types

Use case # 3: Start/Stop/Terminate test session

Roles: QA engineer, Test tool, PortaBilling

Preconditions:  All required templates and pre-configuration was created/completed

Use scenario #3.1: Start test session

Scenario: 

  • QA engineer starts new test session using appropriate control of  the Test tool 
  • Test tool checks if the JSON-like template and other parameters are specified and correct
  • Test tool starts send/receive requests to/from PortaBilling, displays test session details to QA engineer

Use scenario #3.2: Explicitly stop test session

Scenario:

  • QA engineer has started new test session, but realized that he has pre-set something wrong 
  • QA engineer stops test session using appropriate control of  the Test tool 
  • Test tool reacts on request to stop currently running session, sends all required by RFC "terminate" requests to PortaBilling for all active Diameter sessions
  • Test tool reports appropriate status to QA engineer, as soon as all active Diameter sessions are terminated

Use scenario #3.2: Terminate test session by time-out

Scenario:

  • QA engineer is going to conduct 6 hours long test session
  • QA engineer sets optional parameter 'TST_DURATION'
  • QA engineer starts new test session using appropriate control of  the Test tool
  • Test tool starts send/receive requests to/from PortaBilling, displays test session details to QA engineer
  • Test tool reacts on 'TST_DURATION' interval exhaustion , sends all required by RFC "terminate" requests to PortaBilling for all active Diameter sessions
  • Test tool reports appropriate status to QA engineer, as soon as all active Diameter sessions are terminated

Use scenario #3.3: Terminate CC session by response time-out or incorrect response

Scenario:

  • QA engineer configures test scenario and runs test
  • PortaBilling (due to high load on some other factors) starts responding with delays (BE_RESP_TIME > BE_RESP_TIME_MAX) or some incorrect responses (received response does not match the template)
  • Test tool starts terminating such session (if any request except for CCA-T fails - session will be terminated with CCA-T. If CCA-T fails - no re-transmissions will be made)

Use case # 4: Test session parameters presentation

Roles: QA engineer, Test tool, PortaBilling

Preconditions: Test tool aggregates average values of mentioned below parameter(aka 'BE_RESP_TIME') withing some configurable basic interval(e.g. 30/60/120/etc. sec.). Resulted data is a bunch of such 'avg' values collected withing the time test session was active.

Use scenario #4.1: Display values of parameters while test session in progress

Scenario: 

  • QA engineer has to know what is the actual values of 'ACT_SESSIONS',  total speed of requests generation ('TOTAL_RPS'), PortaBilling response time ('BE_RESP_TIME')
  • QA engineer has to know what is the actual number of lost responses BE_RESP_NUM_LOST (if BE_RESP_TIME > BE_RESP_TIME_MAX)
  • QA engineer has to know what is the percentage of 'INITIATE/UPDATE/TERMINATE' requests is generated by Test tool in particular moment of time
  • QA engineer has to know what values of 'ACT_SESSIONS', 'INIT_RPS', 'TST_DURATION' he has explicitly set
  • QA engineer has to know how much time of 'TST_DURATION' has already passed
  • QA engineer checks appropriate functional of the Test tool to get required information

  • Test tool displays actual rates of mentioned parameters along with the pre-set values(if this value was really pre-set before test session has started).

Use scenario #4.2: Collect values change history once test session is completed

  • QA engineer has to know how values of 'ACT_SESSIONS', 'TOTAL_RPS', 'BE_RESP_TIME', 'BE_RESP_NUM_LOST' were changing while test session was in progress
  • QA engineer has to know what is the percentage of 'INITIATE/UPDATE/TERMINATE' requests generated by Test tool in particular moment of time
  • QA engineer checks appropriate control of the Test tool to generate results in CSV/XML/JSON
  • QA engineer uses external WI to parse and organize resulted data

Use case # 5: Test tool scaling

Roles: QA engineer, Test tool, PortaBilling

Preconditions: Performance of the host with running Test tool is not enough to fully utilize capacity of the PortaBilling instances under the test.

Use scenario #5.1: Utilize computing resources of 1+ hosts with running Test tool

  • QA engineer makes Test tool on some particular host aware about it's "neighbors", via appropriate configuration capabilities of the Test tool
  • QA engineer configures and starts new test session as 'usual' on some particular host with running Test tool
  • Test tool automatically distributes requests generation/responses processing among the available "neighbors"

Other requirements / constraints

  • Test tool should be capable to cope with traffic: 'ACT_SESSIONS'=100k 'INIT_RPS'=2k