Automation-friendly framework for Continuous Testing
Automation-friendly framework for Continuous Testing
JMeter ExecutorThis executor type is used by default, it uses Apache JMeter as underlying tool. JMeter Location & Auto-InstallationIf there is no JMeter installed at the configured path, Taurus will attempt to install latest JMeter and Plugins into this location, by default ~/.bzt/jmeter-taurus/{version}/bin/jmeter. You can change this setting to your preferred JMeter location (consider putting it into ~/.bzt-rc file). All module settings that relates to JMeter path and auto-installing are listed below: modules: jmeter: path: ~/.bzt/jmeter-taurus/bin/jmeter download-link: https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-{version}.zip version: 3.0 # minimal supported version of JMeter is 2.9 force-ctg: true # true by default detect-plugins: true plugins: - jpgc-json=2.2 - jmeter-ftp - jpgc-casutg force-ctg allows you to switch off the usage of ConcurrentThreadGroup for jmx script modifications purpose. This group provide steps execution parameter but requires Custom Thread Groups plugin (installed by default) With version parameter you can ask for specific tool version or use autodetect with auto value. In that case taurus will analyze content of jmx file and try to guess appropriate the JMeter version. plugins option lets you describe list of JMeter plugins you want to use. If plugins option isn't found only following plugins will be installed: jpgc-casutg, jpgc-dummy, jpgc-ffw, jpgc-fifo, jpgc-functions, jpgc-json, jpgc-perfmon, jpgc-prmctl, jpgc-tst. Keep in mind: you can change plugins list only for clean installation. If you already have JMeter placed at path you need to remove it for plugins installation purpose. JMeter Plugins Manager allows you to install necessary plugins for your jmx file automatically and this feature doesn't require clean installation. You can turn it off with detect-plugins option. If you use your own installation of JMeter (with path option) make sure it includes jmeter-plugins-manager 0.16 or newer. Run Existing JMX Fileexecution: - scenario: simple scenarios: simple: script: tests/jmx/dummy.jmx or simply bzt tests/jmx/dummy.jmx TODO: explain how multi-thread group will accept concurrency with maintained proportion JMeter Properties and VariablesThere are two places to specify JMeter properties: global at module-level and local at scenario-level. Scenario properties are merged into global properties and resulting set comes as input for JMeter, see corresponding .properties file in artifacts. You may also specify system properties for JMeter in system-properties section. They comes as system.properties file in artifacts. Global properties are set like this: modules: jmeter: properties: my-hostname: www.pre-test.com log_level.jmeter: WARN log_level.jmeter.threads: DEBUG system-properties: sun.net.http.allowRestrictedHeaders: "true" Scenario-level properties are set like this: scenarios: prop_example: properties: my-hostname: www.prod.com log_level.jmeter: DEBUG You can use properties in different parts of JMeter parameters to tune your script behaviour: execution: - concurrency: ${__P(my_conc,3)} # use `my_conc` prop or default=3 if property isn't found ramp-up: 30 hold-for: ${__P(my_hold,10)} scenario: with_prop modules: jmeter: properties: my_conc: 10 my_hold: 20 scenarios: with_prop: requests: - http://blazedemo.com/${__P(sub_dir)} - https://blazemeter.com/${__P(sub_dir)} properties: my_hold: 15 # scenario-level property has priority sub_dir: contacts Usage of variables are similar but they can be used on scenario level only: scenarios: sc_with_vars: variables: subdir: contacts ref: http://gettaurus.org requests: - url: http://blazedemo.com/${subdir} headers: Referer: ${ref} - url: https://blazemeter.com/${subdir} headers: Referer: ${ref} Open JMeter GUIWhen you want to verify or debug the JMX file that were generated from your requests scenario, you don't need to search for the file on disk, just enable GUI mode for JMeter module: modules: jmeter: gui: false # set it to true to open JMeter GUI instead of running non-GUI test For the command-line, use alias -gui or option -o modules.jmeter.gui=true, without the need to edit configuration file. Run JMeter in Distributed ModeDistributed mode for JMeter is enabled with simple option distributed under execution settings, listing JMeter servers under it: execution: - distributed: - host1.mynet.com - host2.mynet.com - host3.mynet.com scenario: some_scenario scenarios: some_scenario: script: my-test.jmx For accurate load calculation don't forget to choose different hostname values for slave hosts. If you have any properties specified in settings, they will be sent to remote nodes. Shutdown DelayBy default, Taurus tries to call graceful JMeter shutdown by using its UDP shutdown port (this works only for non-GUI). There is option to wait for JMeter to exit before killing it forcefully, called shutdown-wait. By default, its value is 5 seconds. Shutdown port number is searched automatically, starting from shutdown-port option value, by looking for unused ports. Modifications for Existing ScriptsJMeter executor allows you to apply some modifications to the JMX file before running JMeter (this affects both existing JMXes and generated from requests): scenarios: modification_example: script: tests/jmx/dummy.jmx modifications: disable: # Names of the tree elements to disable - Thread Group 1 enable: # Names of the tree elements to enable - Thread Group 2 set-prop: # Set element properties, selected as [Element Name]>[property name] "HTTP Sampler>HTTPSampler.connect_timeout": "0" "HTTP Sampler>HTTPSampler.protocol": "https" If selector for set-prop isn't found, taurus tries to create stringProp jmx element with last element of selector as name and sets given value for it. So you can create simple properties in jmx if it's necessary. Building Test Plan from ConfigScenario that has requests element makes Taurus to generate the script for underlying tools automatically. For now, this is available for JMeter and partially available for some other tools. The requests element must contain a list of requests, each with its settings and child elements (assertions, extractors). Also there are additional configuration elements for requests-based scenario, described below. Scenario is the sequence of steps and some settings that will be used by underlying tools (JMeter, Grinder, Gatling) on execution stage. Scenarios are listed in top-level scenarios element and referred from executions by their alias: scenarios: get-requests: # the alias for scenario requests: - http://localhost/1 - http://localhost/2 execution: - scenario: get-requests # alias from above is used Global SettingsScenario has some global settings: scenarios: get-requests: store-cache: true # browser cache simulation, enabled by default store-cookie: true # browser cookies simulation, enabled by default headers: # global headers header-name: header-value think-time: 1s500ms # global delay between each request timeout: 500ms # timeout for connecting, receiving results default-address: "https://www.blazedemo.com:8080" # http request defaults scheme, domain, port keepalive: true # true by default, applied on all requests in scenario retrieve-resources: true # true by default, retrieves all embedded resources from HTML pages retrieve-resources-regex: ^((?!google|facebook).)*$ # regular expression used to match any resource # URLs found in HTML document against. Unset by default concurrent-pool-size: 4 # concurrent pool size for resources download, 4 by default use-dns-cache-mgr: true # use DNS Cache Manager to test resources # behind dns load balancers. True by default. force-parent-sample: false # generate only parent sample for transaction controllers. # False by default content-encoding: utf-8 # global content encoding, applied to all requests. # Unset by default follow-redirects: true # follow redirects for all HTTP requests random-source-ip: false # use one of host IPs to send requests, chosen randomly. # False by default data-sources: # list of external data sources - path/to/my.csv # this is a shorthand form Note that timeout also sets duration assertion that will mark response failed if response time was more than timeout. If you want to use JMeter properties in default-address, you'll have to specify mandatory scheme and separate address/port. Like this: default-address: https://${__P(hostname)}:${__P(port)}. It's possible to use follow specific values for choosing of think-time:
Data sourcesTaurus allows you to use external CSV files for testing purposes. Here is a full list of options for this: scenarios: quick-test: requests: - http://blazedemo.com/${id}-${name} data-sources: - path: path/to/another.csv # this is full form; required option delimiter: ';' # CSV delimiter, auto-detected by default quoted: false # allow quoted data; ignored when random-order is true loop: true # loop over in case of end-of-file reached if true, stop executing if false variable-names: id,name # delimiter-separated list of variable names random-order: true # enables randomizing plugin; false by default When random-order is false, data extraction will proceed in direct manner. Data lines, which contain delimeters, will be read from the top down to the bottom, just the way they were written. Otherwise, the data will be extracted in a random way. Also variable-names can be omitted. In such case the first line of CSV file will be used as variable names. RequestsRequest objects can be of two kinds:
HTTP RequestsThe base element for requests scenario is HTTP Request. In its simplest form it contains just the URL as string: scenarios: get-requests: requests: - http://localhost/1 - http://localhost/2 The full form for request is dictionary, all fields except url are optional: scenarios: my-req: requests: - url: http://blazedemo.com/ # url to hit method: GET # request method (GET, POST, PUT, DELETE) label: homepage # sampler label body: 'request-body-string' # if present, will be used as body body: # generate query string based on parameters and request type param1: value1 param2: value2 body-file: path/to/file.txt # this file contents will be used as post body upload-files: # attach files to form (and enable multipart/form-data) - param: summaryReport # form parameter name path: report.pdf # path to file mime-type: application/pdf # optional, Taurus will attempt to guess it automatically headers: # local headers that override global Authentication: Token 1234567890 Referer: http://taurus.blazemeter/docs think-time: 1s # local think-time, overrides global timeout: 1s # local timeout, overrides global content-encoding: utf-8 # content encoding (at JMeter's level), unset by default follow-redirects: true # follow HTTP redirects random-source-ip: false # use one of host IPs to send the request (chosen randomly). # False by default extract-regexp: {} # explained below extract-jsonpath: {} # explained below assert: [] # explained below jsr223: [] # explained below Notes for upload-files:
ExtractorsExtractors are the objects that attached to request to take a piece of the response and use it in following requests. The concept is based on JMeter's extractors. The following types of extractors are supported:
To specify extractors in shorthand form, use following configuration: scenarios: my-req: requests: - url: http://blazedemo.com/ extract-regexp: # dictionary under it has form <var name>: <regular expression> page_title: <title>(\w+)</title> # must have at least one capture group extract-jsonpath: # dictionary under it has form <var name>: <JSONPath expression> varname: $.jsonpath[0].expression - url: http://blazedemo.com/${varname_1}/${page_title} # that's how we use those variables extract-css-jquery: # dictionary under it has form <var name>: <CSS/JQuery selector> extractor1: input[name~=my_input] - url: http://blazedemo.com/${varname}/${extractor1}.xml extract-xpath: title: /html/head/title Note that boundary extractor has no shorthand form. It can only be defined with full form. The full form for extractors is: scenarios: my-req: requests: - url: http://blazedemo.com/ extract-regexp: page_title: regexp: <title>(\w+)</title> # regular expression default: NOT_FOUND # default value to use when regexp not found match-no: 1 # if multiple values has matched, which match use (0=random) template: 1 # which capture group to take, integer or template string subject: body # subject for search scope: all # check main and sub-samples extract-jsonpath: varname: jsonpath: $.jsonpath[0] # jsonpath expression default: NOT_FOUND # default value to use when jsonpath not found from-variable: JM_VAR # JMeter variable for search concat: false # \ scope: variable # - see below match-no: 4 # / - url: http://blazedemo.com/${varname}/${page_title} extract-css-jquery: extractor2: expression: input[name=JMeter] attribute: value match-no: 1 default: NOT_FOUND scope: children # check sub-samples - url: http://blazedemo.com/${varname}/${extractor2}.xml extract-xpath: destination: xpath: /order/client/address default: NOT_FOUND validate-xml: false ignore-whitespace: true match-no: -1 use-namespaces: false use-tolerant-parser: false - url: http://blazedemo.com/${varname}.xml extract-boundary: pagetitle: subject: body # extractor scope. values are: body, body-unescaped, body-as-document, response-headers, request-headers, url, code, message left: <title> # left boundary to look for right: </title> # right boundary to look for match-no: 1 # match number. 0 for random default: DEFVAL # default value, if nothing is matched You can choose scope for applying expressions. Possible value for targets are:
Default value of scope is empty, it means search in main sample only. match-no allows to choose the specific result from several ones. Default value is 0 (random). To get all values you can use -1 - generation of variables varname_1, varname_2, etc. It means if you ask for some_var_name JMeter won't generate variable with exactly that name by default. Possible subjects for regexp are:
Parameters of jsonpath exractor concat, scope and match-num work only on JMeter >= 3.0 If several results are found they will be concatenated with ',' if concat. AssertionsAssertions are attached to request elements and used to set fail status on the response. Fail status for the response is not the same as response code for JMeter. Currently three types of response assertions are available. First one checks http response fields, its short form looks like this: scenarios: my-req: requests: - url: http://blazedemo.com/ assert: # contains list of regular expressions to check - .+App.+ The full form has following format: scenarios: my-req: requests: - url: http://blazedemo.com/ assert: - contains: # list of strings to check - .+App.+ subject: body # subject for search regexp: true # treat string as regular expression not: false # invert condition - fail if found assume-success: false # mark sample successful before asserting it Possible subjects are:
The second assertion type is used to perform validation of JSON response against JSONPath expression. scenarios: my-req: requests: - url: http://blazedemo.com/ assert-jsonpath: # contains list of options - "$." # if this JSONPath not found, assert will fail - "$.result[0]" # there can be multiple JSONPaths provided Full form: scenarios: my-req: requests: - url: http://blazedemo.com/ assert-jsonpath: - jsonpath: "$." # path to value, validation fails if path not exists validate: true # validate against expected value expected-value: "value" # the value we are expecting to validate, default: false regexp: true # if the value is regular expression, default: true expect-null: false # expected value is null invert: false # invert condition And the third assertion type uses XPath query to validate XML response. scenarios: assertion-demo: requests: - url: http://blazedemo.com/ assert-xpath: # contains list of xpath queries - "/bookstore/book" # if this XPath won't be matched, assert will fail - "/html/head/title" # you can provide multiple XPath queries Full form: scenarios: my-req: requests: - url: http://blazedemo.com/ assert-xpath: - xpath: "/html/head/title/text()='My title'" # query that compares XPath query result with some value use-tolerant-parser: false # use error-tolerant XML parser ignore-whitespace: true # ignore whitespaces in XML (has no effect when `use-tolerant-parser` is true) validate: false # validate XML against its schema (has no effect when `use-tolerant-parser` is true) invert: false # invert condition If sample is broken (RC isn't 200) and cause of assertion the same time, error message will be overwritten with assertion message. Sometimes both of them are important and should be saved into results file. For this case you can use following jmeter option: modules: jmeter: error-message-separator: ';' # joins error and assert message if presented JSR223 BlocksSometimes you may want to use a JSR223 Pre/Post Processor to execute a code block before or after each request. Taurus allows that with jsr223 block. Minimal example that will generate one JSR223 Post Processor. scenarios: jsr-example: requests: - url: http://blazedemo.com/ jsr223: 'vars.put("varname", "somevalue")' # inline script to execute, unless script-file is specified The example above uses defaults and inline script. If you want to use language different from groovy or use separate script file, please use extended form of jsr223 with key-value options. Each jsr223 element can define the following fields:
If execute field is set to after - Taurus will generate a JSR223 PostProcessor, if set to before - a PreProcessor. By default it's set to after. Long form: scenarios: jsr-example: requests: - url: http://blazedemo.com/ jsr223: - language: javascript script-file: preproc.js parameters: foo bar execute: before compile-cache: false - language: beanshell script-file: postproc.bsh execute: after Logic BlocksTaurus allows to control execution flow with the following constructs:
If Blocksif blocks allow conditional execution of HTTP requests. Each if block should contain a mandatory then field, and an optional else field. Both then and else fields should contain lists of requests. Here's a simple example: scenarios: if_example: variables: searchEngine: google requests: - if: '"${searchEngine}" == "google"' then: - https://google.com/ else: - https://bing.com/ Note that Taurus compiles if blocks to JMeter's If Controllers, so <condition> must be in JMeter's format. Logic blocks can also be nested: scenarios: nested_example: requests: - if: <condition1> then: - if: <condition2> then: - https://google.com/ else: - https://yahoo.com/ else: - https://bing.com/ And here's the real-world example of using if blocks: scenarios: complex: requests: # first request is a plain HTTP request that sets `status_code` # and `username` variables - url: https://api.example.com/v1/media/search extract-jsonpath: status_code: $.meta.code username: $.data.[0].user.username # branch on `status_code` value - if: '"${status_code}" == "200"' then: - https://example.com/${username} Once blocksonce blocks is executed only once (per thread). scenarios: loop_example: requests: - once: - http://blazedemo.com/ They're correspond to JMeter's Once Only Controllers. Loop Blocksloop blocks allow repeated execution of requests. Nested requests are to be specified with do field. scenarios: loop_example: requests: - loop: 10 do: - http://blazedemo.com/ If you want to loop requests forever, you can specify string forever as loop value. scenarios: forever_example: requests: - loop: forever do: - http://blazedemo.com/ Note that loop blocks correspond to JMeter's Loop Controllers. While Blockswhile block is similar to while loops in many programming languages. It allows conditional repeated execution of requests. while blocks are compiled to JMeter's While Controllers. scenarios: while_example: requests: - while: ${JMeterThread.last_sample_ok} do: - http://blazedemo.com/ Foreach Blocksforeach blocks allow you to iterate over a collection of values. They are compiled to JMeter ForEach Controllers. Syntax: requests: - foreach: <elementName> in <collection> do: - http://${elementName}/ Concrete example: scenarios: complex_foreach: requests: - url: https://api.example.com/v1/media/search extract-jsonpath: usernames: $.data.[:100].user.username # grab first 100 usernames - foreach: name in usernames do: - https://example.com/user/${name} Transaction Blockstransaction blocks allow wrapping http requests in a transaction. transaction blocks correspond to JMeter's Transaction Controllers. Example: scenarios: transaction_example: requests: - transaction: Customer Session force-parent-sample: false # False by default include-timers: true # add timers and pre-/post-processors execution time to samples do: - http://example.com/shop - http://example.com/shop/items/1 - http://example.com/shop/items/2 - http://example.com/card - http://example.com/checkout Take note: you can specify force-parent-sample on both levels - scenario and transaction. If both are found local (transaction) value has priority. Include Scenario Blocksinclude-scenario block allows you to include scenario in another one. You can use it to split your test plan into a few of independent scenarios that can be reused. Example: scenarios: login: data-sources: - logins.csv requests: - url: http://example.com/login method: POST body: user: ${username} password: ${password} logout: requests: - url: http://example.com/logout method: POST shop-session: requests: - include-scenario: login - http://example.com/shop/items/1 - http://example.com/checkout - include-scenario: logout Taurus translates each include-scenario block to a JMeter's Simple Controller and puts all scenario-level settings and requests there. Keep in mind: the following scenario-level parameters of including scenario have no effect for included ones:
Action Blocksaction block allows you to specify a thread-specific action that will be performed. You can use it to pause or stop the current thread, or force it to go to the next loop iteration. The following actions are available:
Actions can be applied to the following targets:
Examples: scenarios: action_example: requests: - action: pause target: current-thread pause-duration: 1s500ms - action: stop-now target: all-threads Set Variables Blocksset-variables block allows you to set JMeter variables from other variables. Example: scenarios: set_vars_example: variables: foo: BAR requests: - http://blazedemo.com/?foo=${foo} - set-variables: foo: BAZ This example will set initial value of ${foo} to be "BAR", but after first iteration it will be changed to "BAZ". HTTP AuthorizationSee RFC2617 for http authorization details You can use three follow forms for such purposes: scenarios: simply: authorization: url: auth_server_addr name: my_username password: my_pass It's the shortest form for quick setup. You can use several authorizations: scenarios: multi_auth: authorization: - url: auth_server_addr1 name: username1 password: pass1 - url: auth_server_addr2 name: username2 password: pass2 If you want to reset authorization for each test iteration you have to use clear flag and full form: scenarios: full_auth: authorization: clear: true # false by default list: - url: auth_server_addr1 name: username1 password: pass1 - url: auth_server_addr2 name: username2 password: pass2 Possible authorization params and their value are:
Required of them are username & password and one of url & domain. For implementation of authorization Taurus uses JMeter HTTP Authorization Manager. Client Certificate Based AuthorizationTaurus allows the use of client certificate based authorization using JMeter executor. There are generally two scenarios for client certificate based authentication. If you require only one certificate for your whole test:
modules: jmeter: system-properties: javax.net.ssl.keyStore: ${BASE_DIR}/test-data/my-client-certificates.p12 javax.net.ssl.keyStorePassword: MyClientCertificatePassword
INFO o.a.j.u.SSLManager: Total of 1 aliases loaded OK from keystore If you require multiple client certificates for your test (e.g. one per user session):
|
On this page:
Quick Links: |