Script Management
In Switchboard, everything is governed by configuration files. This includes:
- every ingestion
- any upload to the data warehouse
- any transformations
These are all governed by a Switchboard script recipe.
Recipes can be found under the “Scripts” section of the left hand navigation
This section contains four subsections:
- Published
- Contains scripts that are considered “published” and running in production, meaning the data will be collected as scheduled and added to your primary data store.
- Drafts
- Contains scripts that are not considered published. This a work area for creating new scripts that are incomplete and may contain errors or require testing.
- Tests
- Chronological listing of test runs of scripts from your drafts section.
- History
- Contains all the revisions of your published and draft scripts.
Updating a Script
To make an update to an existing script, such changing something about one of the ingestions, you must create a new draft (otherwise you will be in “Read Only” mode).
Go to “Scripts > Drafts” using the left hand navigation.
Click the “Create Draft” button next to the name of your script. Doing so creates a sandbox that allows you to make changes, look for errors, and run tests.
You can rename the script by simply clicking on its name in the upper right corner next to the page icon and typing.
The sandbox will automatically validate your script. Validations are done when you hit the “Save” button in the top toolbar.
Problems, such as missing keys or unused columns, will be highlighted in the script and listed with an explanation in the Problems section.
In the example below, the key has been renamed such that it is invalid.
When the script is saved, this results in a validation error where that column is later referenced in the recipe where it trying later on in the script to transform something that doesn’t exist. This error can be rectified by deleting the offending line of code.
Sections of the Script Editor
The left hand navigation provides a number of subsections with different tools to help you better manage your scripts.
Files
The files section allows you to navigate all the scripts that you have created.
Imports
The import section shows you all the different sources you have defined in your scripts from which are you are pulling data.
Exports
The exports section shows you all the destinations you have defined to which you are pushing data.
Changes
The changes sections shows you a diff of the original version of your file and the draft you are currently working on, allowing you to easily see additions (in green) and deletions (in red) from your previous version.
Testing a Draft Script
Once all issues have been resolved with the new draft, you will want to test it.
Tests may be initiated by clicking on the “Test” button at the top of the script editor.
As part of running the test, you will add some test parameters. Typically for a big query export you will provide test_table and test_dataset, as shown in the example below. These parameters can left in the script and will not effect production data. You can only have one test location per export block.
Click test, and this will ask you what exports you want to test.
The test will be launched, to see it its progress, click the “Test Runs” button at the top of the editor or exit the editor (click the X in the upper right) and navigate to “Scripts > Tests” in the lefthand navigation.
This is the test runs area. You can access the currently running test here.
The test run will distinguish between tests that are currently downloading new data and when it gets to the processing stage.
Everything from the test will be logged in the Events UI.
Note that you have the option of testing the export with existing downloaded data. Using this option will prevent the test execution from downloading new data, which will allow the test to execute faster. Testing with existing data is ideal in cases where only the transformation portion of the recipe has been changed.
The test will run. The system will pull down the data, ensure that the script runs successfully, and send the data to your test location in the data warehouse.
It is possible to set up multiple tests to run in your sandbox.
The “Tests” subheading under “Scripts” contains the history of tests the have been previously run.
Publishing a Script
Once you have tested your recipe and it has completed to your satisfaction, return to your draft recipe under “Scripts > Drafts” in the lefthand navigation. Select the draft you were working on.
Click the “Publish” button in the top navigation of the screen to make the current content of a script the actively running version.
The user interface will momentarily be covered with a modal semi-transparent overlap that disables the main window as it goes into a “Publishing Draft” state to prevent further changes to the script as the system is updated. The recipe will then now running in production.
Note that because Switchboard preserves the history of your recipes, it is always possible go back through the history of all of the different updates and understand what changes have happened over time or to revert to a previous version.
Script History
If you want to go through the history of all of the updates you’ve made, select “Scripts > History” from the lefthand navigation bar.
Note the difference between the contents of the drafts and history sections. Drafts haven’t been merged into production while history represents a previous known state of the system.
In the screenshot above you can sandboxes that have been created, but not put into production.
In the case where this history is particularly lengthy, it is possible archive entries so they no longer appear. This history can always be restored.
If you wish, you can revert to a previous version of a recipe. To revert, create a new draft from a previous version by hovering your mouse over the plus sign in the Options column, then publish that version.
Note that sometimes previous versions that were functional may contain errors because of subsequent changes to the required parameters of a script (e.g. a new parameter such as an “API version” has been added).