Automated Mirroring Between Concord Pro and Your Git Server

Altium Concord Pro™ as a standalone product and brand name has been discontinued and the capabilities are now available as part of our Altium enterprise solutions. Learn more here.

In Linking Managed Projects with your Git Server using Altium Concord Pro, we reviewed how to mirror Git repositories from your Altium Concord Pro™ server to your favorite Git server (i.e., Github, Bitbucket, Gitlab, etc.). In this article, we will review an approach that automates creation of a Git server mirror and includes comprehensive logging. What we'll setup here can run as a Windows service on your Altium Concord Pro server.

Overview

Altium Concord Pro uses an internal Git server to store Managed Projects. At the time of writing this article, Concord Pro has no way to store all Managed Projects on an external Git server natively. We will be walking through an example script and architecture that automates the process of mirroring all the Managed Projects on a Concord Pro server to another Git server. To get the code and create your own Git server mirror, you can download it from this Gitlab repository.

Why should we use a Git server mirror? There could be a myriad of reasons why one would want to store their designs on an external Git server versus Concord Pro. The primary reason is to maintain a comprehensive revision history backup. You may want a simple way to managed projects from your on-premises Concord Pro server. If a project is deleted from the Concord Pro server, it can be retained on the Git server mirror. This does more than just mirror project files, it also stores a history of commits, branches, etc.

Git Server Mirror Architecture

There are a few key components that make up this service:

1. The Watchdog Python Library: This monitors the Git repository folders on the Concord Pro server and triggers an event when any files change
2. Windows Server library utilizing pywin32: This turns a basic Python script into a Windows service so it can be run on your Windows Server without interruption.
3. The FirebirdSQL Python Library: This enables us to get into the Concord Pro’s Firebird database and extract information about the Managed Projects
4. Your favorite Git server’s API: In this example, the Atlassian (Bitbucket) Python Library is used to enumerate Bitbucket repositories and create new ones when needed.

The code is broken down into two pieces: the lookup table and the synchronizer. The lookup table (also known as the “repo dictionary” in the code) serves as a linker between each Concord Pro Managed Project repository and your target/remote repository. Since Concord Pro uses GUIDs, the repository name/description lies within the Firebird database. In this example, the lookup table, or dict, uses the GUID as the key and the remote repository (on your remote Git server) as the value. The synchronizer piece uses the Watchdog library to recursively monitor the Git folder in the Concord Pro installation folder for any file change events. When the master file for any folder (i.e. repository) changes, it is relatively safe to assume that the Git HEAD has been moved to a new hash. This, essentially, means that the repository has been updated. This triggers an event to be generated which tells the synchronizer to mirror the GUID’s repository from the Concord Pro server to the remote repository on the remote Git server (using the repo dictionary lookup table). Once the git mirroring occurs, the transaction completes, and the next event in the queue (should one exist) repeats the process.

The following is a block diagram depicting the two pieces as Python scripts:

1. Git-to-Git Cross Reference: This is the lookup table that is generated in one script
2. Watchdog Python Library: This is the synchronizer that runs in the main script

The Lookup Table

There are many ways to generate the lookup table that links the Concord Pro Managed Projects to the remote Git server’s repositories. One very trivial way would be to manually create the remote repository and then feed in a text file to the Synchronizer script. This script generates a dictionary object, which instructs the synchronizer on which Managed Project mirrors to which remote Git repository. In this example, the remote Git server is being managed by Bitbucket. Therefore, the Atlassian (Bitbucket) API has been used to validate existing repositories and create new ones when necessary.

As mentioned above, since the Managed Projects use GUID for their repository naming scheme, it is necessary to perform a lookup function in the Firebird database to get the “true” name of the Managed Project. After retrieving the name of the project, this can be used to create the repository on the remote server. For example, a repository GUID might be “E9B1952E-BAFE-4A79-801B-76C0F750A8D7” with the Managed Project’s name as “My Repo.” That repository will not exist on the remote server initially, and you may need to create a generic rule: all spaces get replaced with a dash, and all characters get converted to lowercase. This will result in your remote repository name becoming “my-repo.” You issue an API call to create the repository and then store it in your dictionary table/file. Once this lookup table initialization has completed (including creating repositories that do not exist yet on the remote server), you are ready to start the synchronizer.

The Synchronizer

As discussed above, synchronizer looks for file changes in the Concord Pro Managed Projects folder. It then mirrors it to the remote server using the lookup table provided by the repo dictionary script. There are auxiliary functions such as comprehensive logging and Windows service management that happen in the background, but they are not part of the core functionality of the script. While this script is not exhaustive and not foolproof, it is a good demonstration on how one can create an automated mirroring service between Concord Pro and a remote Git server mirror.

Conclusion

In this article, we reviewed and provided an example on how to automatically create a Git server mirror and populate it with the Managed Projects in your Altium Concord Pro instance. You can use a remote Git server such as Github, Gitlab, or Bitbucket as your Git server mirror to store your Altium Concord Pro projects. We looked at the architecture and the services needed to put together this solution and then explained how they all fit together.

This example does not cover every corner case. However, this type of mirroring is easy to implement as a service on a Windows server that hosts Altium Concord Pro. We've taken advantage of the API on an Atlassian (Bitbucket) server through their Python library, but you could also implement this on another Git server as long as you have API access. If you're working on your own custom server, you can follow these same steps with your own code to create a Git server mirror of your Altium Concord Pro projects. 

Talk to an Altium expert today to learn more.