The RESO Commander is the basis for automated Data Dictionary, Payloads, and Web API Certification.
To run the Commander as an automated testing tool, a Java 64-bit JDK must be installed.
The Commander has been tested with JDK 1.8, 10, and 11 at this point. JDK 11 is preferred.
Those using JDK 11+, please report issues if they arise.
To see whether you have the JDK installed, type the following using your local command line environment:
$ java -version
If you have a Java JDK installed, your output will look something like:
$ java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.10-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)
If you don’t see something like this, you need to install the JDK:
If you’re using macOS you can install the JDK using Homebrew. There are instructions for Windows and Linux here.
Note: there are known issues with Java/JDK 14+ and Groovy.
The Commander may be run in automated testing mode using a terminal.
Automated testing assumes that you have a Java 1.8+ JDK installed, as mentioned elsewhere in this README.
First, change into the directory you want to work in and clone the Commander repository.
You will need to have Git installed. Chances are you already do.
To check, open a command line and type git. If it’s present, it will print some info about the app.
If not, there are installation instructions here.
$ git clone https://github.com/RESOStandards/web-api-commander.git
C:\> git clone https://github.com/RESOStandards/web-api-commander.git
This will clone the repository into a directory called web-api-commander relative to whatever directory you’re currently in, which also means you’ll have a fresh copy of the latest code to execute.
To refresh the code after you have downloaded it, issue the command $ git pull in the root of the directory that was just created.
Cucumber is being used to describe acceptance criteria in a higher-level DSL rather than encapsulating all the test logic code. Cucumber’s DSL is called Gherkin and essentially allows backing test code to be organized in a logical manner that makes sense to analysts as well as programmers.
Under the hood, Gradle is being used for automation. It works across multiple platforms and is friendly with both Docker and Cucumber so that tests may be automated on CI/CD platforms such as Jenkins, Circle CI, Travis, or similar, and emit standard system codes during regression testing.
The Gradle wrapper provides a convenient way to automatically download Gradle when running tests.
After you have cloned the repository, as shown in a previous step, change into the directory containing the source code from GitHub. Convenience methods have been provided for the various certification tasks.
Once the Gradle Wrapper is set up, you should be able to run the ./gradlew tasks command in from the root of the
Commander source directory in a terminal window and see the list of available tasks.
$ ./gradlew tasks
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
...
There are both built-in tasks and RESO tasks.
The following section is what’s of interest here:
RESO Certification tasks
------------------------
testDataDictionary - RESO Data Dictionary Acceptance Tests
[Report location: build/certification/reports]
Examples:
RESOScript Example (Certification):
./gradlew testDataDictionary -Dversion=1.7 -DpathToRESOScript=/path/to/dd.resoscript -Dstrict=true
Metadata File Example (Local Testing):
./gradlew testDataDictionary -DpathToMetadata=/path/to/RESODataDictionaryMetadata.xml
Supported Parameters:
• version: Default is 1.7. Valid versions are [1.7, 2.0]
• pathToRESOScript: Path to config file in Data Dictionary format
• pathToMetadata: Local path to metadata file to validate. Not used for Certification
• strict: Default is false. Certification applicants MUST pass with -Dstrict=true
testWebApiCore - Web API Core Acceptance Tests
[Report location: build/certification/reports]
Example:
./gradlew testWebApiCore -Dversion=2.0.0 -DpathToRESOScript=/path/to/web-api-core.resoscript -DshowResponses=true
Supported Parameters:
• version: Default is 2.0.0. Valid versions are [2.0.0]
• useStringEnums: Default false. Use -DuseStringEnums=true if using string enumerations and the Lookup Resource
• useCollections: Default true. Use -DuseCollections=false if using OData IsFlags
• showResponses: Default false. Use -DshowResponses=true to see output of Web API Core requests
To use the automated RESO testing tools, you must have a JDK installed.
To use the Commander for automated Web API Core testing, you need a RESOScript.
For Web API 2.0.0 Server Core Certification, use this resoscript as a template.
For more information regarding Parameters and Client Settings, see the Web API Walkthrough.
The Cucumber BDD acceptance tests for Web API 2.0.0 Core certification are here. If you have any questions, please email us.
While you may use tags to filter tests as you choose, explained in the next section, it’s convenient to be able to run a predefined set of tests Web API Core Certification.
These tasks will also produce reports in the local /build/certification directory, relative to the project root, named according to which test you ran.
This will run the Core tests against the Web API 2.0.0 Server provided as WebAPIURI in your web-api-server.core.2.0.0.resoscript file.
Note: by default, the Commander assumes Edm.EnumType for single- and Collection(Edm.EnumType) for multiple-enumeration testing.
Pass -DuseStringEnums=true if you are using string enumerations.
Pass -DuseCollections=false if you are using IsFlags="true" instead.
$ ./gradlew testWebApiCore_2_0_0 -DpathToRESOScript=/path/to/your.web-api-server.core.2.0.0.resoscript -DshowResponses=true
C:\path\to\web-api-commander> gradlew testWebApiCore_2_0_0 -DpathToRESOScript=C:\path\to\your.web-api-server.core.2.0.0.resoscript -DshowResponses=true
*Note: the first time you run these tasks, they will take some time as the environment must be configured and code is being compiled from the contents of the source directory downloaded in the previous step.
A sample of the runtime terminal output follows:
> Task :testWebApiCore_2_0_0
@metadata-request @2.4.1
Scenario: REQ-WA103-END3 - Request and Validate Server Metadata
Using RESOScript: ./web-api-server.core.2.0.0.resoscript
Given a RESOScript file was provided
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file
Bearer token loaded... first 4 characters: test
Service root is: https://api.yourserver.com/OData
And a test container was successfully created from the given RESOScript
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication
Requesting XML Metadata from service root at: https://api.yourserver.com/OData
When XML Metadata are requested from the service root in "ClientSettings_WebAPIURI"
Asserted Response Code: 200, Server Response Code: 200
Then the server responds with a status code of 200
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01"
Validating XML Metadata response to ensure it's valid XML and matches OASIS OData XSDs...
See: https://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/schemas/
XMLMetadata string is valid XML!
And the XML Metadata response is valid XML
Validating XML Metadata response to ensure it's valid XML and matches OASIS OData XSDs...
See: https://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/schemas/
XMLMetadata string is valid XML!
XML Metadata is valid!
Edm Metadata is valid!
And the XML Metadata returned by the server are valid
And the XML Metadata returned by the server contains Edm metadata
And the Edm metadata returned by the server are valid
Found Default Entity Container: 'Default'
And the metadata contains a valid service document
Resource Name: Property
Allowed Resources: Property, Member, Office, Contacts, ContactListings, HistoryTransactional, InternetTracking, Media, OpenHouse, OUID, Prospecting, Queue, Rules, SavedSearch, Showing, Teams
And the given "Parameter_EndpointResource" resource exists within "Parameter_DD17_WellKnownResourceList"
Found EntityContainer for the given resource: 'Property'
And the metadata contains the "Parameter_EndpointResource" resource
Searching the default entity container for one of the following Standard Resources: Property, Member, Office, Media
Standard Resource Names requirement met!
And the metadata contains at least one resource from "Parameter_WebAPI102_RequiredResourceList"
1 Scenarios (1 passed)
15 Steps (15 passed)
0m4.093s
Detailed information will be added to a local commander.log file at runtime.
The Commander provides automated Data Dictionary acceptance testing for RESO Certification. The Data Dictionary testing specification is available here.
To use the RESO Commander for Data Dictionary testing, you must have the JDK installed and a local copy of the Commander repository. See RESO Certification before proceeding.
To use the Commander for automated Data Dictionary testing, you need a RESOScript.
For Data Dictionary Certification, use this resoscript as a template.
RESO Data Dictionary Certification is driven off of the official Data Dictionary spreadsheet for each version of the dictionary, currently DD 1.7.
Cucumber BDD acceptance tests are automatically generated from the local copy of the approved spreadsheet.
The generated Data Dictionary Cucumber BDD tests are located in this directory.
See the property.feature file, for example, for the RESO Property Resource acceptance tests.
If you have any questions, please email us.
There are predefined tasks for automated RESO Data Dictionary Certification using the Commander. These can be displayed using Gradle Tasks as well. See: test Data Dictionary
Note: the first time you run these tasks, they will take some time as the environment must be configured and code is being compiled from the contents of the source directory downloaded in the previous step.
This task tests for Data Dictionary compliance and generates a raw report in a timestamped local directory.
There are two ways to run automated testing to check for RESO compliant Web API metadata:
While RESOScript files and the use of strict mode are required for RESO Certification. In both cases, metadata are validated and then processed for RESO compliance.
The Commander allows for a local metadata file to be specified. Not only is this used for internal acceptance testing, but is useful for developers to troubleshoot metadata locally while working on compliance.
The Gradle task to validate local metadata can be run using the following command:
$ ./gradlew testDataDictionary -Dversion=1.7 -DpathToMetadata=/path/to/RESODataDictionary.xml
You may also pass a -Dstrict=true flag to see whether the given metadata file would pass Certification.
A raw report will be generated in a timestamped directory, and a commander.log will be generated during runtime.
During Certification, metadata are retrieved directly from an applicant’s Web API server using either OAuth2 Bearer Tokens or Client Credentials. Either authentication option is currently available for RESO Certification, depending on configuration, and the applicant will provide working RESOScripts when they apply for certification.
An example Data Dictionary RESOScript template can be found here.
Once a RESOScript file has been created, it may be used with the following command:
$ ./gradlew testDataDictionary -Dversion=1.7 -DpathToRESOScript=/path/to/dd.resoscript -DshowResponses=true
You may also pass a -Dstrict=true flag to see whether the given metadata file would pass Certification.
A raw report will be generated in a timestamped directory, and a commander.log will be generated during runtime.
To see examples of Data Dictionary testing output, you may use the ./gradlew testDataDictionaryReferenceMetadata
command to run the Data Dictionary acceptance tests on the RESO reference metadata.
There is additional documentation about how Data Dictionary testing works, including sample output, in the RESO Data Dictionary Specification.
You may filter by tags in any of the Web API or Data Dictionary tests. These are the items in the Cucumber .feature
files prefixed by an @ symbol. Expressions may also be used with tags.
This README doesn’t cover how to use tags, but the Commander supports them. For more information, see the Cucumber Documentation.
Run Web API Core Metadata Tests Only
$ gradle testWebApiCore_2_0_0 -DpathToRESOScript=/path/to/your.web-api-server.core.2.0.0.resoscript -Dcucumber.filter.tags="@metadata"
Run Data Dictionary Tests on IDX Fields Only
$ ./gradlew testDataDictionary_1_7 -DpathToRESOScript=/path/to/your/dd.resoscript -DshowResponses=true -Dcucumber.filter.tags="@IDX"