The Microsoft Regression Suite Automation Tool is something that I have been working with since its earliest release and something that I really like. Its not perfect, its not the solution to the worlds problems around testing D365 systems but it has its place in your toolbox and should not be underestimated, in my opinion.
Lets walk through the key steps that I do when I implement this for my customers:
Step 1 – I encourage the customer to setup a specific Lifecycle Services Business Process Modeler library (BPM) called RSAT or similar – within which they will manually create a list of all the tests that they are expecting can be carried out automatically. Typically this is at least 1 test per key business process – such as ‘Create a Transfer Order to move stock between Warehouses’ for example.
You can see below the specific RSAT library I created as an example
Step 2 – I then review this test list with them to see if what they expect, matches what is possible/practical. Remember RSAT is not great at ‘importing’ or ‘uploading’ data per se, and also I haven’t found a way to get it to ‘wait for a response’ other than by using a PowerAutomate flow to auto generate a ‘response’ and setting the time between actions to be a long’ish period to allow for it – but that’s another blog post for another day!
Step 3 – Once we have a list of processes to be tested the next step is to open D365 and Task Record those processes.
I encourage customers to setup specific, recognisable test data during the recordings, such as create customers called ‘RSAT-01’, products called ‘RSAT-PROD01’ and suppliers called ‘365Operations.co.uk‘ 🙂 The reason I do this is that I am sure that none of those values will exist in the customers production data-set, so when I execute these tests later on fresh copies of the production data I know I shouldn’t see any issues with a test failing because the ‘customer’ already exists, for example
During the task recording process I teach the teams to use the ‘right-click > Task recorder > copy‘ function to capture any field data they they believe will be used by any later test that needs to interact with this same order, journal or picking route for example. Essentially what this does it to add ‘power’ to the process later on when we come to look at ‘linking’ test cases sequentially to form a test of an end-to-end process.
Create a New product > ‘copy’ the item number for later use
Assign a cost price to a released product – this test will need the ‘item number created in test 1’, so because we copied it during the first task recording, then we can ‘paste’ it into this later test case using the RSAT Client. I will show you how later in this post.
Step 4 – Now we go back to the BPM and make sure that we have enabled the RSAT library to automatically sync to DevOps AND sync test cases. This syncing process will take some time (150 tests took approx 5mins to sync) but it should start immediately you select it.
I then check in DevOps that all the test cases have been created automatically, i do this by using a query that includes RSAT (or the name of the BPM library if different) in the title of the work item, as below:
If you open one of the test cases that have been created automatically by the sync, you should see that it has been populated with the test steps recorded AND there should be an attached file called ‘recording.xml’
Step 5 – Once I am sure all the test cases have been created, then I manually create a new Test Plan in DevOps called RSAT.
I add all the test cases to that test plan, this is a manual task but I tend add them all in a single step. Once they have been added you can then change the ‘order’ of them. The RSAT client will execute the tests in the order you have defined here inside the DevOps Test Plan
TIP: whilst you can build a folder structure in a DevOps Test Plan, I find that having a single tier means its much easier later in the RSAT Client to trigger a full test, as in the RSAT Client you can only ‘run’ 1 folder at a time.
Setting the test case order is a critical step especially if you have setup the test data during the task recording stage to process the data created by an earlier test, such as posting a delivery note against a sales order. If you haven’t created the sales order first, then that ‘create delivery note’ test will fail. So set the sequence of the tests correctly.
Note: Once I have the test plan created and sequenced I used ‘clone’ it using Microsoft Test Manager (now defunct but still works on my laptop). This allowed me to see the history of executions, BUT I soon realised that in DevOps you can see the results for each run in (Test Plans > Runs)
Step 6 – Its at this point I ask the customer for an RSAT specific user account to be created in the domain and also in D365. This account is given key security roles inside D365. We are typically testing functionality here, not security!
Step 7 – Now I install the RSAT application on a VM inside the customers tenant, and I will use the new D365 account the customer provided to set this up and to later execute the tests.
- You can download the latest RSAT tool from here: https://www.microsoft.com/en-us/download/details.aspx?id=57357
- You might need an Updated Chrome Driver which you can get from here: https://chromedriver.chromium.org/downloads
I connect the RSAT client to ALL of the AOS servers of the environment that I will be executing the tests against. This is actually not difficult, but sometimes I found it doesn’t work first time. There are multiple blog posts explaining how you do this – a good one is here from Microsoft: https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/perf-test/rsat/rsat-install-configure
Remember you will need to link the RSAT client machine to ALL the AOS servers inside the environment you are connecting to…not just one or you will see errors…..
I double-check we have the RSAT application connected to both:
- DevOps – this connects to this client to the specific Test Plan we created for RSAT
- D365 – This connects to the specified D365 environment that you will test against.
Step 8 – In the RSAT client I then select ‘load’ which connects to DevOps, and connects to the defined Test Plan and ‘pulls’ into the RSAT Client, the tests themselves. As you can see from the image below these get pulled in but the ‘Parameters’ column remains empty.
If you choose ‘Generate Test Execution and Parameters files’ this then uses the DevOps connection and for each test case selected, it extracts from DevOps the ‘test steps’ that are stored against a test case in the ‘recording.xml’ file. This file also provides the ‘data values’ that were used during the original recording process such as ‘item number’ etc. and includes any data you ‘copied’ as per point 4 earlier in this blog post.
You can see the files it generated if you look in the ‘working directory’ that you specificed when you earlier setup the RSAT Client after installing it:
You will then see that the ‘parameters’ column of the client has been filled in:
I can then open the MS Excel ‘parameters file’ for each test using the new Excel icon that has been added to the latest RSAT client version and here I can see the parameters and data values that this test will use when it is run. I can also edit them, which is really key…
Step 9 – Imagine we want to test an end-to-end Transfer Order between Warehouses. We know that we will want to carry out many actions against the same Transfer Order Number.
- Create a Transfer Order
- Pick a Transfer Order
- Ship a Transfer Order
- Registers an Arrival for a Transfer Order
- Receive a Transfer Order
During the Task Recording of the ‘Create a Transfer Order’ process we made sure to ‘Copy’ the Transfer Order Number field’s value (and a couple of other useful values). Now we can see in the Excel parameters file for the ‘Create a Transfer Order’ that the task recording created an entry in the variables section of the excel file. Its this ‘variable’ that we will copy/paste into the other ‘excel’ parameter files to make the link.
Now if we open the parameters file of the Pick a Transfer Order test – we can ‘paste’ that variable into the location where this task recording is needing to provide a Transfer Order ID
Essentially we have now linked the 2 test cases together. In this example when test 4464 runs first and creates a Transfer Order Number, then test 4465 will run and use the transfer order number created by test 4464 as the value of the transfer order’ that this test wants to update the pick list for.
- You created a BPM Library called RSAT
- You added business processes to that library
- You task recorded those process in D365 and used the ‘copy’ function
- You sync’d the BPM to DevOps
- You checked the Test cases had been created automatically by the sync
- You created a Test Plan in DevOps
- You added and sequenced the test cases to that test plan
- You installed and configured the RSAT client on a machine
- You loaded the tests into the RSAT client
- You created Parameter files for each test
- You edited the parameters files to link tests together where needed
Step 10 – Run Automated Testing
Now you should be looking at an RSAT client that has a long lists of tests to execute, like in the example below:
Make sure that you have selected all the cases, then Press RUN – and your automated tests will begin to run, one after the other.
As the tests execute you will see a browser session will be opened for each test as it executes, and you will see the login and all the mouse and data entries happen (if you want to sit and watch it!!). Note that the browser knows that its being controlled by test software.
Once the tests have finished running you will see in the RSAT client that test result. This is one of the 3 results:
- Not Executed
As well as updating the test results into the RSAT client, the client has also updated the test results back into DevOps, actually in 2 places:
- TEST PLAN > Each test in your plan will have had its ‘outcome’ value updated to reflect the results mentioned above.
- TEST RUNS > Each time you press RUN in the client a Test Run is created in DevOps
and also as a Test run
and if you click on each failed test you can see the detail of what failed, which step it failed on and why.
END. Hope this helps in your RSAT Testing.