RTW - API/Integration Testing
Integration testing is the functional and load/stress testing of web services and applications leveraging the communication interfaces (e.g. SOAP, MQ, webMethods, webSphere).
Recognize this?

"We are testing in our Production Acceptance Test environment while that's the first and only environment where everything works at best!"

Recognize that statement? Keep on reading...

Why Integration Testing

Due to manual execution, test execution was started when a user interface was available. So, testing was deemed to be at the end of a cycle. We speeded the test execution by automating the manual interaction (with tools like RFT).
RIT gives the ability to test without a GUI, directly sending messages/transactions to the application (e.g. webMethods, MQ, SOAP etc). That results in 2 time related benefits:

  1. Directly testing the application we want to test, which benefits also defect finding.
  2. No dependancy of availabiltiy of other applications (like a GUI-application), which enables test-early.

This results in a faster feedback on the behavior of the application under test.

Why Service Virtualization

The same technique used for Integration Testing, supplied by IBM Rational, can be leveraged for service virtualization. Especially in complex end to end testing environments this technique comes into play. Certain applications can only be tested when related applications are available. Virtual Services, created with Rational, behaves like the related applications. Service virtualization results that the application under test 'thinks' that the related applications are there. So these applications can be tested (incl interfaces) without having a full end to end environment available.

  1. No need to have full stack of running applications available.
    Think of installation cost of applications.
  2. Easier to predict behavior of virtualized related applications.
  3. Easier to manipulate test data.
  4. We can test error-situations (related application (virtualized) does not behave as expected)
  5. Less demand on the Production Acceptance Test environment!

This results in a faster feedback on the behavior of the application under test.

As an example: Suppose we have application A which needs B, which needs on his turn C. So for testing A, we need to install B and C, including the hardware and middleware. So a project starts with a actions to organize all this. When 'stubbing'/virtualizing B we might need a simple receive request and send answer to test A. Availablitiy of B and C is no concern for testing A.
At the end of you might want to have a full end to end environment executing complete end to end testing. But than we are confident that behavior is up to standards before we execute those costly tests!
Implementation Guide

RIT General

The very first steps of RIT-tesing is creation of functional tests. These can be used to define performance tests and or test created stubs.
Many techniques and tips described here for functional testing are applicable for performance testing and virtual services.

General RIT GUI

Technical View and Business View

You can switch between Technical View and Business View. Each view has specific features.

Figure: Technical View and Business View Icons

In the Business View one can rename the Actions to a more human friendly format.

Environments: Binding Logical and Physical

The testware created by RIT is defined at a logical level (e.g. 'a transaction server called Main'). By defining an environment you link that logical name to a specific implementation. Tests are so very easy to reuse.

Logical Element Test Environment 1
UAT1
Test Environment 2
UAT2
ACTIVE
Test Environment 3
PAT
MQ Manager1 n.a. 192.168.2.12 192.168.3.12
MQ Manager2 192.168.1.22 192.168.2.22 192.168.3.22

The logical definitions (first column) is done at the Architecture School - Logical View.
The physical definitions (other columns) is done at the Architecture School - Physical View.

Figure: The various Environments and the physical binding of the PAT environments. (GAT is the Dutch term for UAT).

Floating Windows

Useing a floating window of the test (contextual menu on the Tab) gives easy access to the test is self and the Test Lab concurrently.

Figure: Floating window of a test.

Architecture School

Logical View

Organize your Operations in a hierarchy.

Make note that a 'grouping' by UATx is not that evident, all tests can be shared across all UAT environments by switching environments.

Physical View

Email Server

After failing test you could send out a email to the responsible people to notify.

Figure: Physical setup of the email-server.

Schema Library

Select the schema and the rebuild becomes active. By rebuilding the references remains in tact.

Figure: Rebuilding (reloading) the schema.

Environmental Tasks

Tasks are defined in Architecture School and leveraged in the Scenario's. Steps can be:

  1. Command prompts
  2. IBM Optim runs
  3. HP Shunra Network Virtualization

Figure: Task to leverage Optim power.

Figure: Activating a task before execution of scenario.

Requirements Library

The Requirements Library stores 'specification messages'. Based on these requirements you can define messages in your tests.

Recording Studio

Test Factory

Here you can define and maintain your tests.

Main elements:

Functional Testing - Actions and Examples

Figure: All RIT actions

The actions in a list;

  1. Tag Data Store
  2. Test Data
    Use Directory Data Source to iterate across files in a certain directory.
  3. Message Switch
  4. Publish
  5. Receive Reply
  6. Receive Request
  7. Send Reply
  8. Send Request
  9. Subscribe

    Make most sense to use Append and New Line to append messages to a file or recreate a file with only the last message by deselecting Append and deselecting New Line.

    Figure: Use new line when adding a message to a log-file.

  10. UnSubscribe
  11. Close Case
  12. Modify Task
  13. Retrieve Case
  14. Retrieve Tasks
  15. Start Case
  16. Trigger Event
  17. Action Group
  18. Assert
  19. Descision - used in Check for File example
  20. Fail
  21. Fetch Test Data
  22. Iterate - used in Check for File example
  23. Iterate Test Data
  24. Iterate While - used in Check for File example
  25. Log
  26. Lookup Test Data
  27. Pass
  28. Run Test - used in Run Parallel Tests example
  29. Sleep
  30. Comment
  31. Compare Files
  32. Create Session
  33. Function
  34. GUI Interaction
  35. Map Action
  36. Run Command - used in Check for File example
  37. SQL Command
  38. SQL Query
  39. Stored Procedure
  40. User Interaction
  41. Start Timed Section
  42. Stop Timed Section
  43. Log Measurement

TAGs

Once I had a variable SrcRef (a list) and SrcRef_Current. The following statement

The following did not work:

indexValue = ( tags["TEST/ITERATION/NUMBER"] - 1 ).toString();
SrcRef_Current = SrcRef.get(indexValue);
Followed by a log of SrcRef_Current

The following did work:

indexValue = ( tags["TEST/ITERATION/NUMBER"] - 1 ).toString();
Dummy = SrcRef.get(indexValue);
Followed by a log of Dummy. This is probably due to the usage of the underscore in the name (not sure).

TAGs - A naming convention

A proposed naming convention:

  1. Use CAPITALS
  2. Use tag hierarchy HOTEL/ROOM/DOORCODE

Alternative could be using CaMeLcAse: HotelRoomDoorCode ...

Warning! A tag with a minus needs to be referenced with a tags[] notation.
NAME-FAMILY="Johnson"
Will result in error! The minus character will be treated as a minus-operation. Use:
tags["NAME-FAMILY"]="Johnson"

Passing TAGs

You can transfer variables between tests by using the same tag(/variable) in the tests.

Figure: RIT - Definition of Global Tag (deselecting "Confine scipe to this test").

Publish

Use publish to:

Subscribe

Don't wait unlimited, set timeout (ms) at something different that 0ms.

Action Group

You can define action groups to 'group' various actions. But be aware:

WARNING: Action Groups are ment to group messages and process them in parallel!!
Do NOT ASSUME actions are executed in sequence in a Action Group!!

Great Technote! RIT Action Group actions for parallel message handling explained

Once, I had Action Group of assignments and a message:

- Calculation1
- Calculation2
- Assignment
- Message

The message was spawned before the calculations and assignments of the tags were executed! An assumption is that this behavior is limited to communicaiton messages (not to messages creating files). This is currently under investigation by IBM Development.

Documentation on Action Groups

Log-file

Tip: Use consistent directories:

Figure: Good practice - write to a variable log-file.

An example can be found Check for File.

SQL and DB2

I got an error SQLCODE =-551 when using RIT to query the DB2 SAMPLES Database.

		GRANT SELECT ON DEPARTMENT TO USER DB2ADMIN;
		

To get a specific row of a DB

		SELECT DEPTNO,DEPTNAME FROM  COMPANY.DEPARTMENT WHERE DEPTNO='%%SelectDEPT%%'

Figure: Test containing the query.

A local DB2 database is used with the following physical definition:

Figure: Physical Definition of the database.

Check for File

To check for a predefined number of times (pace can be set), use the following:

Figure: Scenario to check for a file for a number of iterations

The CMD provides a exitCode in a tag.

Figure: Scenario to check for a file indefinitely.

It's advisable to "Limit the length of the entire iterate action", if the file is still not there a failure will occur.

Run Parallel Tests

This is by far not a performance test (no measurements done).

Figure: Running multiple tests in parallel.

The Run Test definition has the Run process in parallel and Parent waits for child to complete selected.

Figure: Run test definition.

Retcode

Figure: Clearing a queue leveraging RETCODE - make note the MQ has a timeout.

Search for Hotels

Leveraging the HotelFinder.wsdl

Assume the following table/data source:

CityHotel
AmersfoortGaaper
AmersfoortCentral City
AmsterdamHilton

On the input-tab of the stub you must tag the city.

Figure: The stub searches for hotels based on requested input, the city.

Test Lab

The area to execute tests interactively to test what's build. Use F5 to re-run last test.

The "Run ..." option gives the ability to schedule a test for a later time. This option used a RTVS PVU/SS license.

Results Gallery

Check out the results of a test. This (functional)information can also be viewed via the web-client (RTCP).

Integration Testing - Functional

Figure: RIT - A test fails while backend process is not available. Stub is started. Re-run test with success.

This set of capabilities is often referred as Rational Integration Tester (RIT). This is only available as part of RTW.

Integration Testing is only available as part of RTW.

Functional Testing - Using Tips

All using tips are described above in line with the GUI of RIT.

Integration Testing - Performance

The RIT performance testing is 'functional testing' and on top of that the definition of the performance test (including load etc) and references to the functional tests.

There are some 'additions' to make this work:

"We wanted to performance testing the application..."

Some years ago we had a single product for load and stress testing, Rational Performance Tester (RPT). This tool can do performance testing on web-based applications. It simulates various users working on the web-frontend of an system. By pushing the load one will get a better understanding of the behavior of the systems under these load-profiles. The supported communication protocol is mainly http(s).

The GreenHat company, with the GreenHat Tester product has also a performance test tool in it. This tool is used to push transactions to a service. With that one can validate if a service is performing well under certain load conditions. The supported communication is a broad variety including http(s), SOAP and MQ.

Now these two products are both available under the Rational Test Workbench product. This results in that the RPT (simulating users) as well as the GH-Test-Performance tool is available in there! So when installing the RTW product you might see the RIT product and RPT product, but not the GH-Test-Performance. This is part of the RIT product.

When going for larger loads than you need capacity. In the case of RPT we have two options for this. One option is to utilize various individual computers leveraging simulating the users (needs a Virtual Users License). The other option is to leverage a server on which the RPTS or RTVS software is installed. The RIT product, pushing transactions, can leverage the capacity of a server with RPTS or RTVS software installed.

In short:
RPT + Agents - Simulates users
RPT + Cloud-Agents - Simulates users
RPT + RTPS/RTVS - Simulates users
RIT + RTPS/RTVS - Product transactions

Make note that if you buy RIT, you buy RTW, which includes RPT.

Especially handy when running performance tests

Figure: The Word-wrap icon so you can see the complete message without scrolling.

Performance Testing - Using Tips

Statistics collection interval

You can limit the amount measurement data by increasing the Statistics collection interval. The Windows Performance Monitors and Test Engine Sequences stats are always every second.

Figure: A graph showing a statistics collection interval of 10seconds and the (blue) windows measurement of CPU.

Sequence Started Rate

Started "Sequence started rate".

Figure: 4 phases where initial target iterations 10/s, increment per phase 10/s, Even through period.

The phases are following each other without a pause period (e.g. without transactions), here we go from 10/s to, 20/s,30/s and finally 40/s.

Tip! You can view the results building up when tests are running.

Performance Testing: Add time-out on receive msg

This way connections are released after certain moment in time, especially handy when service/proxy are not functioning correctly.

Figure: A Timeout is set to 10 sec.

Performance Testing: Firewall issues

See technote Network port numbers used by IBM Rational Test Workbench Green Hat components for details of firewall issues that might come up when running performance tests.

Section: That piece of a test which is enclosed with a Timer.
Phase: Timed interval in performance test

Performance Test Sections:

      <!-- <instance name="engine1"/>
           <instance name="engine2"/> -->

Performance Testing: Threads

Out of threads error when running a performance test

Figure: A compare of two performance runs, by utilizing a 2nd window the scale can be totally different.

Note! Performance Test Results are only viewable in the RIT-GUI.

Unable to add Windows Performance Probe counters.


You needed to install the C++ libraries on RIT when building the test, as well as the machine to probe to run the probe.

Figure: Add counter does not work if MS C++ libs are not installed.

Tibco Hawk Probe counters.

Figure: Make sure you have selected the 'top' level to activate the REGEX

By setting an other value the the default for the Hawk.AMI.DisplayName, you can select which queue you want to monitor. Otherwise all are stack on top of ".Test"...
Images are blurred because they are from customer production environments.

Figure: Add a value for Hawk.AMI.DisplayName (note the number 1248)

Figure: Now we can select that specific probe

Message "CRRIT8529E / CRRIT8551E Probe definition does not have a Hosting Agent defined"


The test transport was using a real production service on the Internet. That service was not under our control, so we could not run an Agent on this server, resulting that we could not run the probe.
To probe a non-related machine (not the RIT, or service machine). Solution is to add a dummy HTTP or Database server to the test operation references to run the probe.

Figure: You must add a relation between the service and the dummy HTTP Server (xRTVS-server here).

Figure: A dummy HTTP server, bound to machine to probe.

Close and re-open the performance test to see an updated Probe tab with the new server in it and use this for the probe (activate).

Agent not started error with performance test


The hostname of RIT was sent to the Agent to talk back on. The Agent did not know the hostname/IP address. Solution is to add the RIT machine to the hosts file on the Agent.

Alternative you can put the IP address of the RIT machine in the library manager to return the calls.

Figure: The return address of the RIT machine from communication with the Agent.

Stub not picking up messages


We modified the transport to use the RIT-Platform Proxy to make requests, the stub then issues the redirect rule
Remember to reload by re-open the project or right-click and Disconnect the HTTP transport to pick up this change
Ensure that the hostname is of the Proxy host so that the remote performance test can access it

On the physical resource

Figure: A Proxy Host and Proxy Port is added.

Linux Librarysettings.xml

A sample Linux Library Settings ( should be located in $HOME/.rit8 )

librarysettings.xml

Adapt this file to your environment using basic editors like VI.

Virtual Service creation

Similar way as you created the functional test you can save the communication as a 'stub' or Virtual Service. The development of Virtual Services is done in RIT. This service can than be deployed in the RTVS. When started other applications can leverage the virtual service as if the replaced application is there.

More detail, see RTVS »

Wake Up Stub

When a stub needs to check if certain files are present at a regular interval (no initiation from sending a message), use the following approach.

Figure: Define the startup and time events in the behavior tab.

Figure: Add to the activity the scheduleTimerEvent.

The used code to trigger the timer once after 30 seconds is:

		timer.scheduleTimerEvent("timer", tags["SESSION/KEY/sessionID"], 30, java.util.concurrent.TimeUnit.SECONDS)

It's only once so you have to trigger timer it self again in the code. Make note that stubs are limited in RIT for 5 minutes.

Multi Event

This example shows how the addNumbers stub can be expanded into a definition where it can process the SOAPAction "Add" and "Substract".

  1. First create a basic addNumbers stub and test.
  2. In the Operation Definition you have to do the following:
    1. Open the MEP of the operation, Stub-Tab
    2. Disable any filtering on SOAPAction words, the rules in RTCP will be based on this definition.

      Figure: Disable any filtering on SOAPAction words.

  3. In the Stub Definition you have to do the following:
    1. Apply the field-filter on the SOAPAction "Add" in the Request

      Figure: Disable any filtering on SOAPAction words.

    2. Now you can add a a new Stub Event with a SOAPAction "Substract" in the same way as the "Add"

      Figure: Disable any filtering on SOAPAction words.

You can have hard coded values. If you want to calculate, adding the tags in the simple way will give a concatenation of strings, not a calculation. We assume you have quicktagged the needed message fields (create a tag). So you first have to convert them into integers. This can be done in the following way.

tags["return"]=parseInt(tags["arg0"])-parseInt(tags["arg1"]);

Installation Tips

Documentation Library Manager

Make sure that all components are of the same release. This minimizes the risk of introducing communication issues.

Installing 'RIT and RTVS'

Frequently we get the question to "install RIT and RTVS" or even simply "Install GreenHat". Often they mean the RIT and the RTVS server so a POC can start. Than you need the following components (please check documentation):

  1. Installation manager (should be part of SETUP disk or check IIM)
  2. A Database installation (if not already there) - bring your own database with you!
  3. Rational Integration Tester
    Use installation files:
    • RIT_8.7_MP_ML.zip
  4. Rational Integration Tester Platform Pack
    Use installation files:
    • RIT_8.7_MP_ML.zip
  5. Rational Test Control Panel
    Use installation files:
    • RIT_8.7_Control_Panel.zip
  6. RPTS/RTVS Agent
    Use installation files:
    • RIT_AGENT_8.7_MP_ML.zip
  7. Create DB
  8. Load schema

DB2-Express C responce file to install DB2 on Windows Server 2012: PROD_EXPC.rsp (check contents before installing).

Validation of simple/basic setup:

  1. Creation of Project
  2. Load WSDL file
  3. Create Stub based on MEP
  4. Create Test based on MEP
  5. Run Stub
  6. Run Test (will fail -> Overwrite expected message)
  7. Re-Run Test
  8. Publish Stub
  9. Start Stub in RTVS
  10. Create Suite add Test
  11. Run Suite
  12. Check Results in RTCP

RIT-RQM integration

The RQM/RIT combination is extremely powerful! Yes you can incorporate RIT tests in a build (using ANT). The result is that the build manager defines what is tested and reporting is nothing more than pass/fail of complete build (exaggerate). Advantage is that it gives (developers) a quick result, which is good.
The next level of testing is managed by testers, different data, propper planning and reporting, more management. A layer of RQM is essential. A successful build triggers a suite execution. What's in the suite is defined by testers. It's for testers very easy to adapt what to be tested in the setup (of test). Results are stored in the propper context of test plans. Reporting is generated, done automatically.

RQM - RIT Setup

Executing a Rational Integration Tester test remotely from Rational Quality Manager

Figure: Config of RIT-RQM integration.

With this defintion in place one can export RIT Tests and RIT Suites to RQM.

Figure: Listing 'Rational Integration Test' as test scripts in RQM.

Note! The RIT-agent is the RQM-Adapter!

An example of a RIT Agent setup. Change the Agent.config file

   <!--
     Multiple RQM projects can be supported by adding multiple <rqm> sections to the configuration
     however RIT projects on disk can only be used by one instance of RunTests at a time. So either
     a separate ghtprojectbaselocation should be specified for each RQM project or it should be
     arranged that a RIT project is only used from one RQM project.
     -->
   <rqm enabled="true">
       <username value="clmadmin"/>
       <password value="clmadmin"/>
       <url value="https://fibula5.vanlint5.nl:9443/qm"/>
       <rqmproject value="Sandbox (Quality Management)"/>
       <ghtprojectbaselocation value="C:\Users\IBM_ADMIN"/>
       <runtests value="D:\Program Files\IBM\RIT-Agent\RunTests.exe"/>
       <rqmAdapterId value="MyFixedGHTesterId"/>
   </rqm>

The project file is located at: C:\Users\IBM_ADMIN\MyProject.ghp, note that a complete hard-coded reference to RunTests is applied.

Figure: How the adapter pops-up in the RQM application.

Issue: Adapter not available when schedule RIT test case.
I sometimes have a issue with availability of appropriate RIT adapter when running a Test Case referencing a RIT-test. This might be caused by my environment where I switch a lot from machines, configurations etc. When creating a TCER (manually or on the fly), there is no appropriate adapter.

A 'work around' I found was:

  1. Re-start your browser session (logoff, logon)

If needed additional:

  1. Stop the adapter
  2. Delete the adapter definition in RQM
  3. Delete the adapter definition in RTCP
  4. Re-start the adapter
  5. Re-start your browser session.

Above is on an 'as-is' basis, I had some success with it.

I think if you run a RIT-suite before any RIT-test from RQM it will blow-up the Adapter....(100% not sure).

Issue: SSL Connection Error when Exporting

Figure: SSL Connection error.

Got this message RIT8.5.0.1 and RQM 5.0.1. Known issue when running on Tomcat. This is already from about 4.0.7+...

RQM - RIT Structure

This chapter describes some aspects of initiating RIT tests from an RQM environment. It assumes that an integration is in place, like described here above.

Note! For clarification

More CLM Artifacts

The goal is to start a 'normal' RQM-TC or RQM-TS and with that RIT-Tests. The detailed results must be in RTCP, but for test-initiation (CI) and reporting purposes the RQM-TCR/TSR must also be available. Both results must be related.

From the Enhancement 56441:

If a test is run directly in RIT then its results are *not* stored in a database so can't be displayed again later (MVL: Good for test-developement).
The reports are only stored in a database IF

1) The rest is run as part of a test suite (regardless of how that test suite is initiated - from RIT, from RQM, from HP QC)
2) If a Test Cycle is active for that project and the user has joined. In this case all tests will write their reports to a database

When a test suite is run from RQM then the information that is returned to RQM and displayed in the RQM UI contains a link to the RTCP page that will display the report.
Note that this only occurs if the RIT project has been configured with an RTCP server in the project settings.

If a test (not a suite) is run from RQM then no RTCP link is provided because the test report is not in the database.
If a test is run from RQM AND the project has an active test cycle then an RTCP link should be provided as there will be results in the database. (Confirmed)
		

Some experimenting I found the following best (order of creation is not relevant, adapters must be active):

This will result that within RQM the suite is executed (and can be executed based on a finalizing build). Results are available in RQM to report on (e.g. with Dashboards or RRDG/RPE). When needed the user can drill from RQM to RTCP to go into detailed results of each test (not applicable for RIT Performance while this is not implemented in RTCP).

A resulting hierarchy can be build:

ArtifactWhy?Example/Remark 
RQM Test Planteam Communication
Planning and Reporting
Traceability
Test Plan for the new customer data moduleWhat
RQM Test SuiteGrouping of test-goals
Ability to have a single point for starting execution
Example 1: Validate functionality of a new customer data module
Example 2: Check all credit cards payments
RQM Test CaseSingle Independent Test GoalExample 1: In new module being able to request customer-data
Example 2: Check a specific credit card payment
RQM Test ScriptImplementation of Test GoalGoal can be implemented in various ways (think of channels, webservice, webGUI)
Remark: Generated by exporting RIT Suites and RIT Tests to RQM
RIT SuiteImplementation of Test Goal in execution toolThe RIT suite has the ability to transfer data from one RIT Test to the next RIT Test (see RIT Tags). Remark: can be imported as RQM Test Script (*1)How
RIT TestVarious interactions with the applicationRemark: can be imported as RQM Test Script

Remark *1: It looks that a RIT Suite should be imported into a RQM Suite, it is not! The current implementation is good. If the RIT Suite would be imported into a RQM Suite, than it could not be nested in an other RQM Suite. With the current implementation it's possible to let the RIT-engineer, focussing on the how, create his own suite. The RQM-engineer, focussing on the what, can incorporate the RIT Suite, imported as a RQM Test Case into a RQM Test Suite.

Switching from one RIT Test to the next is faster than switching from one RQM Test Case to the next! This is due to the architecture, and structure.

RIT-RQM Integration

Check RIT - RTC Integration.

Starting Stopping Agents

Sample scripts (for linux) can be found at D:\Program Files\IBM\RationalIntegrationTester\examples\ExampleInitScripts

DFDL & HL7

  1. IBM DFDL Introduction
  2. DFDL schemas on github
  3. Nice document HL7 V2 compared V3

DB Schema

See RTCP Install

Troubleshooting and Debugging

Some URLs.

RIT-Start with Log-screen on

You can start RIT from the command prompt to view messages with:

GHTester.exe -consoleLog

Figure: RIT Console Log enabled

You can print messages to this console leveraging the Custom Function printConsole

			-Dcom.ghc.ghTester.gui.console.trim=nnnnn
		
where nnnn are the number of chars held in memory.

Switching on RIT-Logging

Check technote RIT-Agent log files

Replace Agent with RIT for GUI Logging...

Sample Test Applications

Some small test applications which can be used for demonstration, learning, evaluation or testing.

IBM RIT - Addnumbers

A small 'server' and client application which is supplied by RIT.

http://localhost:8088/addNumbers?wsdl

Examples on Github

RIT Examples

Gloabal Weather Service

I used this Weather service from http://www.weather.com/ for validating RIT functionality. I found it's not a very solid service in responce, so you might end up in various issues.
http://www.webservicex.com/globalweather.asmx?WSDL
GetWeatherResult.xsd

Find Hotels

Sample wdsl file to search for one or multiple hotels in a city.

Leveraging the HotelFinder.wsdl

Other Resources

Business Partner Contacts

Not sure where to put them, but you might want to get in contact with one of the following business partners. The list is not in any particular order.