How to Build a Custom GPT Action to Talk to Your Hardware

In the early days of ChatGPT (which actually wasn’t very long ago), OpenAI evolved with different ways to reach the outside internet through the use of their LLM. With Code Interpreter, one could Curl a website and pull the contents of it. Custom add-ons also allowed you to hit external websites but required special permissions from OpenAI. OpenAI also added browsing capabilities with Microsoft Bing that could hit the internet as well. All of these options, however, did not prove to be very sustainable for the growth and demand that users and developers commanded on OpenAI’s systems. OpenAI has, thus, come up recently with a new platform called GPT Actions. GPT Actions allow you to create custom “GPTs” that can tie into different servers to provide AI enabled engines on top of an existing web service.

So what does all of this mean for a hardware engineer that pretends to understand what AI is? How does AI controlled hardware sound to you? Sounds ominous, potentially dangerous, and incredibly exciting all at the same time! In this article we’re going to dive into how to set up a webserver that will hook up to your GPT Action. We’ll even look at how to enable ChatGPT to control the IOs on your Raspberry Pi, opening up the capability to many new and exciting AI driven projects that the average engineer never had access to…until now.

How it Works

Before diving into the code it’s important to note that this is an evolving field and is rapidly changing. It is also a technology dependent on OpenAI and requires a subscription to the paid version of ChatGPT. It’s almost certain that competitors will be rolling out their versions of “GPT Actions” but, at the time of writing this article, GPT Actions (in my opinion) was the simplest way to hook up a fully functional Large Language Model (LLM) to your web service.

The concept behind a GPT Action service is pretty straightforward. We point our custom GPT to a webservice, tell it what the inputs and outputs are, and let it run. The implementation, however, proved to be much more complicated than I had imagined (which was the impetus for writing this article). Here is a diagram of how the service works:

Figure 1: Server Interactions

As you can see, we provide OpenAI with a webservice and an interface document (OpenAPI) that tells us how to interact with our hardware. We will start with a very trivial example that does not involve hardware and then graduate to a simple LED toggle example.

Secure Server Configuration

For hardware engineers who are, at best, familiar with embedded software, this section may not be as trivial to them as it would to someone working in IT or Cloud Infrastructure. As both an Electrical and Software engineer, setting up a fully SSL encrypted server was a little bit of a head scratcher to me - mostly because there are so many ways to do it. Unfortunately, this is a prerequisite from OpenAI so we’re going to have to muscle through it. If you already have a secure way of hosting a web server (e.g. TLS termination at your load balancer) you can skip this section. Like me, most folks reading this article will want to experiment with their home server. Keep in mind that you still need to open up ports 80 and 443 to the outside world through your router settings. Instructions on how to configure your router’s firewall and port forwarding is out of the scope of this article but there are plenty of resources online you can find to help you with it.

To get started we need to request an SSL certificate. This can be done using paid services (e.g. Cloudfare), free third-party services (e.g. ZeroSSL), or a direct mechanism using Let’s Encrypt. In this tutorial we’re going to use Let’s Encrypt via certbot. You’ll also need a domain name that points to your server’s IP address. You can either purchase one or get a dynamic DNS name using Cloudfare (paid) or No-IP (free). Keep in mind that forwarding your dynamic DNS name to your IP requires an “A” record type and not URL redirection.

To request a 90 day SSL certificate from Let’s Encrypt we’ll need to utilize Certbot. They have executables that can be run on any operating system. We’re going to run the simplest command to acquire the certificates. Here is the command to acquire the certificate:

sudo certbot certonly --standalone --agree-tos -v --preferred-challenges http -d your_domain.com

If you’re on Windows you won’t use the sudo part. Make sure to replace your_domain.com with the purchased domain name or the one you received from the dynamic DNS service (e.g. No-IP). A folder containing a series of certificate files will be generated. At this point it’s time to clone the repository template: https://gitlab.com/ai-examples/gpt-actions-webserver/. Move the certbot generated folder to the repository once you’ve cloned it.

In order to get SSL redirects working properly (i.e. ensuring all traffic is encrypted) we’ll want to set up Nginx. Create a folder within your repository called nginx and a new file called default.conf.  Using this template, replace your_domain.com with the domain name that was used when you requested your SSL certificate using certbot:

server {
    listen 80;
    server_name your_domain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name your_domain.com;
 
    ssl_certificate /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/privkey.pem;
 
    location / {
        proxy_pass http://web:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Running the Application

At this point you should have SSL encryption correctly set up and ready to go. Now we’re going to focus on the basic web application that can communicate with our custom GPT. You can find all the code in the repository so we’ll just highlight how to run it rather than the nitty gritty details of what goes into the application.

Given the nature of an SSL encrypted web service, I’ve abstracted away the web application from the actual SSL encryption. To do that we need Nginx, which redirects traffic and serves up the certificates that we received from Let’s Encrypt via Certbot. To fully encapsulate everything into a neat little package I’ve wrapped it into a Docker Compose. This means you’ll need Docker to run this example. It’s a bit cumbersome and tedious to get set up initially but this is where software is moving to these days, thus, a project like this is a great place to get acquainted with Docker. As a side note, all descriptions and instructions have been placed in the README.md file within the repository. As long as you followed the previous steps above you should be able to run docker compose up and all the components (i.e. Nginx and the web application) should build and start running. Remember that this application is just a simple “Hello World” type of example and can be run from anywhere.

After running the docker compose up command you should see an output like this:

Figure 2: Terminal Output

You should test your application by navigating to http://localhost:8000. You will see a test output that says {"status":"OK"}. You should also see the same result by navigating to http://your_domain.com where “your_domain.com” is the domain name you set up in the previous step. To really validate that your router and domain settings are correct you should test this outside of your network (e.g. on your cell phone with WiFi turned off).

Configuring the GPT Action

Now that we’ve validated that our basic web server is working we need to hook it up to our Custom GPT Action. OpenAI requires a document called an OpenAPI which tells their servers how to interact with yours. There is already an openapi.json that you can copy and paste directly into the GPT Actions settings. To get to your settings click on your name in the bottom left corner and then click on “My GPTs”:

Figure 3: ChatGPT Settings

Click on Create a GPT, then Create new Action at the bottom:

Figure 4: GPT Action Configuration Settings

After that you can copy and paste the contents of openapi.json in the Schema box:

Figure 5: GPT Action Schema Settings

You’ll need to replace “https://your_doman.com” with the address of your actual domain. At that point a few test buttons will pop up. Assuming your webserver is running you can click on a test button and see if OpenAI can communicate with your webserver. If that all checks out then you can save/publish the GPT (at the top right corner) and start playing with it. Make sure to publish this only for yourself since this is connecting to a private network (presumably at home).

Running the GPT

If you managed to get through all of the above sections and successfully tested your GPT action then congratulations! Now it’s time to really test it all out. I started a new ChatGPT conversation but this time using my Custom GPT which I call “Instrument Controller.” To make it simple I asked for the status of my server:

Figure 6: Server Status using GPT Actions

And my Nginx logs (from the command line where I ran Docker Compose) confirms the traffic as well:

"GET / HTTP/1.1" 200 15 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" "-"

We also have a random number generator as well:

Figure 7: Getting Random Numbers from the Server using GPT Actions

And the traffic from Nginx also confirms that we received 5 requests from our server.

Raspberry Pi LED Toggle

At this point we’ve demonstrated how to perform software actions using GPT but what about controlling hardware? In this article we’ve already tackled a lot so we’ll just take a look at a basic blinky LED example.

To get started, run the same process as above on a Raspberry Pi. Note, you can use the same SSL certificate and repository code that you used on your home computer (assuming all devices are in the same network). Once you have everything ready to go run the following command:

BUILD_TYPE=rpi docker compose up --build

This example now contains the ability to toggle an onboard LED on pin 26 (GPIO7). You can modify the pin number in app/RPiDemoApp.py. You will also need to update the custom GPT’s OpenAPI schema with the opeanapi-rpi.json schema provided in the repository as well (since we’ve added a new function).

At this point you should be able to toggle the LED using the chat interface with ChatGPT:

Figure 8: Toggling an LED on the Raspberry Pi using GPT Actions

Congratulations on making it this far! This was a complex tutorial and you will, undoubtedly, have questions and/or challenges. ChatGPT is making changes to this feature on a daily basis and finding simpler ways to walk beginners through the process. Look out for updated documentation and even AI assistant bots that can help you through the process.

Conclusion

In this article we took a look at custom GPT Actions, how to create them, and build all the scaffolding around it to serve up an SSL encrypted web service from our home computer. In addition to building a basic web service we were able to extend this code to control an LED from the Raspberry Pi. This opens up the possibility of controlling and communicating with much more complex hardware while utilizing “AI in the loop“ (play on “hardware in the loop”). In future articles we will explore such topics and concepts, leveraging this example as the foundation.