Script Management

In Switchboard, everything is governed by configuration files. This includes:

• every ingestion
• any upload to the data warehouse
• any transformations

These are all governed by a Switchboard script recipe.

Recipes can be found under the “Scripts” section of the left hand navigation

This section contains four subsections:

Published

Contains scripts that are considered “published” and running in production, meaning the data will be collected as scheduled and added to your primary data store.

Drafts

Contains scripts that are not considered published. This a work area for creating new scripts that are incomplete and may contain errors or require testing.

Tests

Chronological listing of test runs of scripts from your drafts section.

History

Contains all the revisions of your published and draft scripts.

Updating a Script

To make an update to an existing script, such changing something about one of the ingestions, create a new draft.

Go to “Scripts > Drafts” using the left hand navigation. Click the “Create Draft” button which creates a sandbox that allows you to make changes, look for errors, and run tests.

The sandbox will automatically validate your script.

In example below we have removed the “ad unit” from a Google Ad X recipe as a column as well as from its schema.

When the script is saved, this results in a validation error where that column is later referenced in the recipe where it trying later on in the script to transform something that doesn’t exist. This error can be rectified by deleting the offending line of code.

Testing a Draft Script

Once all issues have been resolved with the new draft, you will want to test it.

Navigate to “Scripts > Tests” in the lefthand navigation. This is the test runs area. You can access the currently running test here.

The test run will distinguish between tests that are currently downloading new data and when it gets to the processing stage.

Everything from the test will be logged in the Events UI.

As part of running the test, you will add some test parameters. Typically for a big query export you will provide test_table and test_dataset, as shown in the example below. These parameters can left in the script and will not effect production data. You can only have one test location per export block.

Click test, and this will ask you what exports you want to test.

Note that you have the option of testing the export with existing downloaded data. Using this option will prevent the test execution from downloading new data, which will allow the test to execute faster. Testing with existing data is ideal in cases where only the transformation portion of the recipe has been changed.

The test will run. The system will pull down the data, ensure that the script runs successfully, and send the data to your test location in the data warehouse.

It is possible to set up multiple tests to run in your sandbox.

The “Tests” subheading under “Scripts” contains the history of tests the have been previously run.

Publishing a Script

Once you have tested your recipe and it has completed to your satisfaction, return to your draft recipe under “Scripts > Drafts” in the lefthand navigation. Select the draft you were working on.

Click the blue “Publish” button in the upper right hand side of the scren to make the current content of a script the actively running version.

The user interface will momentarily be covered with a modal semi-transparent overlap that disables the main window as it goes into a “Publishing Draft” state to prevent further changes to the script as the system is updated. The recipe will then now running in production.

Note that because Switchboard preserves the history of your recipes, it is always possible go back through the history of all of the different updates and understand what changes have happened over time or to revert to a previous version.

Script History

If you want to go through the history of all of the updates you’ve made, select “Scripts > History” from the lefthand navigation bar.

Note the difference between the contents of the drafts and history sections. Drafts haven’t been merged into production while history represents a previous known state of the system.

In the screenshot above you can sandboxes that have been created, but not put into production.

In the case where this history is particularly lengthy, it is possible archive entries so they no longer appear. This history can always be restored.

If you wish, you can revert to a previous version of a recipe. To revert, create a new draft from a previous version by hovering your mouse over the plus sign in the Options column, then publish that version.

Note that sometimes previous versions that were functional may contain errors because of subsequent changes to the required parameters of a script (e.g. a new parameter such as an “API version” has been added).