Using ChatGPT for Automated Testing

It’s been all over the news and if you haven’t heard about it by now it’s time to check out ChatGPT from OpenAI. It’s an incredibly powerful tool that runs on Artificial Intelligence. The concept is that you can ask it almost anything or give it any task and it would, hypothetically, respond with the correct answer or complete the task for you. For example, people use it to write speeches, compose emails, research topics, or even write code. While the model is nowhere near perfect it’s still very impressive and contains lots of good information. Our very own Mark Harris recently demonstrated this capability to me with a simple script that controls an instrument for automated testing. I decided to go down this rabbit hole and experiment with ChatGPT and automated testing. In this article we will walk through how to get onto ChatGPT, learn how to ask it questions, and, most importantly, understand what the right questions are needed in order to achieve the results you’re looking for.

Getting Started with ChatGPT

Before diving in you’ll want to get familiar with ChatGPT and what it has to offer. First navigate to chat.openai.com and sign up or log in with an account. At the time of writing this article the tool was being offered for free so you should be able to jump in right away. Start with basic questions such as “what is 1 + 1?” or “what color is the sky?” After that you can start to give it tasks such as composing an essay, drafting an email, or writing a simple function in software. The interface is clean and simple, just type in the message box at the bottom like you would send any message and click on send (or hit enter).

Figure 1: Message box with ChatGPT

Writing Software Libraries

Once you’re comfortable with having a conversation with ChatGPT you can start experimenting with getting it to generate code for you. Basic requests like “write a function in C to add two numbers” is a trivial task that it can easily handle. It starts to get complicated once you add more details to the request. I like to have all my code buttoned up in neat little classes so I explicitly tell ChatGPT to create a class that does something for me. Here is an example:

Figure 2: Request for class generation

You’ll also notice I asked it to generate tests for me using Pytest. It handled this request as well but, unfortunately, the code had a bunch of bugs in it that were somewhat challenging to fix. It was still a great process to go through to see how it structured code and unit tests. I, certainly, learned a thing or two by watching it generate code in a way that I would not have done myself.

Generating Instructional Code in ChatGPT

One approach described above is to have AI hand you the tool (i.e. software library) so that you can go build the product (i.e. test script). Another approach is to give ChatGPT specific test specifications and have it generate the code for you. This is great for generating quick and dirty test scripts to get you going before needing to tweak your setup and test cases. Let’s take a look at an example.

I’ve provided the following, very simple, instruction set to ChatGPT to test a switching power converter:

Write me a script in Python using Pytest to perform the following:

1. Turn on a Rigol DP832 power supply and set it to 5V and 2A on channel 1

2. Turn on a Rigol DL3021 electronic load and set the load to 1.5A constant current for 3 minutes

3. Turn off the DP832 and DL3021

I haven’t even baked in any specific tests in this case. All I want to do is observe that my DC-DC converter can handle a 1.5A load for 3 minutes. The generated code was pretty decent but every now and then I would get bugs that made it a bit challenging to fix. You can also tell ChatGPT to bake in specific assertions or test for specific failures but I was only interested in testing the basics with the AI app. Additionally, you can even request it to write you a test class (with or without mocks) to validate the code. Again, not everything was lined up perfectly. I had to massage the code itself and the test class to get it working perfectly. 

Conclusion

All in all, the experimentation with ChatGPT was a good experience. It was nice to have an “aggregator” to put some concepts together in code without having to do the manual work or searching across the web for examples. While it’s not perfect it was still helpful and fun to experiment with. After reviewing this article you should have a decent understanding on how to:

1. Access ChatGPT
2. Ask it basic questions
3. Create software libraries based on a request
4. Create test execution code based on a request

I strongly encourage you to check it out and either run the examples that I have provided in this article or try some of your own out. Keep in mind that ChatGPT is still, technically, in beta so don’t set your expectations too high. Be patient and make sure to tell it when it gets things wrong so it can “learn” from its mistakes. Happy coding!