Event Logging

Switchboard logs every operation that happens within the system.

Switchboard has three types of eventslogs.

• Logs that are specific for downloads found under “Downloads > Files” in the lefthand navigation.

• Logs that are specific to uploads, meaning data that’s going out of the system into a data warehouse or some file-based destination, are found under “Uploads > Files”

• Logs for all events, including more general events, are found under “Events.”

Just about everything goes into these logs and this logging can be broken down by all events, errors and warnings.

This part of the application is where you can find detailed information about processing, delays, API problems, etc.

These logs are split out because they have slightly different functionality.

Downloads

For the downloads shown under “Downloads > Files”, anything that comes into the system can be filtered by the various types of data operations.

Imports

Corresponds to the “import” or “download” labels declared in your SBS recipes.

Created By

Download data may be created by different sources.

Production

Data imported as part of day-to-day operations.

Backfill

Data that is manually added to the system from a previous date.

Test

Data comes from the testing sandbox, which is outside of production or backfill and can be filtered from view.

Status

Unfinished

Indicates a download that is in progress and but not complete

Completed

Indicates the data for the download has finished downloading (any row in the table with green check mark has been completed).

Failed

Indicates something happened during the download that prevented it from completing, such as an API breaking in the middle of an import or a file was moved from its location when download started.

Dirty Rows

Indicates switchboard that has some amount of rows that doesn’t match the schema defined in an SBS.

Skipped

Indicates that a download was abandoned because something changed. For instance, if you queued up several backfills and then immediately to make a change to the recipe. Since the downloads you previously initiated would be attached to a previous recipe, those downloads would be invalid because they would be attached to an old recipe that is no longer in production. The system abandons downloads as a way to maintain data quality.

Sorting by Date - Started/Finished or Report Date

You can organize data by when the download was initiated (Started/Finished) or by the date the report data was generated (Report Date).

In the above example you can see the downloads started on the 1st of April.

The toggle for “Report Date” that allows you to look at the report date of the file downloaded.

Report Date allows you see to all the files where there’s data inside the files for a given date (e.g. February 10th).

This is useful if you had backfills done for your reporting data from previous months that are intermingled with data pulled as part of the regular retrieval of data done for production. For instance you may have data pulled from a given day in February that was subsequently pulled in as part of a backfill with a date range that includes that date.

The Downloads page allows you see data that was accessed today, but it can also show you data about what’s in the data set.

Uploads

Switchboard has a dedicated log for the uploads it does of customer data to their chosen endpoint whether that is a data warehouse, flat file, or FTP site.

Different data warehouses have different processes for having data sent to them and Switchboard’s recipes ensure that each upload matches the schema for the data warehouse.

Upload Properties

The following are the properties of each upload action:

Ready

Timestamp when data was processed and ready to be sent to the endpoint.

Started

When the process of exporting to the endpoint started. The mechanism for transferring the data varies based on the endpoint.

Finished

When the export of the data to the data warehouse completed. Switchboard ensures the data is exported correctly.

Description

Shows the name of the export as defined in the recipe and its destination, such as a URL to BigQuery and the name of the table in our example below.

Filtering Uploads

There are various options for filtering the Upload events.

Created By

Allows you to narrow the Uploads shown by whether the Upload was created by Backfill, Production, or Test.

Status

Allows you narrow the Uploads shown by whether they are Completed, Failed, or Unfinished. Sometimes uploads fail because the data warehouse is broken or there was a network error. if something is in the process of being exported to data warehouse it is marked as “Unfinished.”

Date Based Filtering

Start / Finished

Allows you to pick an arbitrary date through calendar widget for when the export to the data warehouse started and finished.

Report Date

Pertains to the date of the export (if it has a date) — i.e. report data from a specific date. This is particularly useful in finding data that has been backfilled.

Upload Details

When you click on the description of a given upload you can see the source download data that went into a given export — this is useful given that Switchboard allows you to do joins of data from different sources.

Upload Charts

Charts visualize the volume of data being exported to the data warehouse. You can see the amount of data as the total number of rows that were exported. This may fluctuate based on the type of data source.

Events Log

The Events Log is a quick way to see what it going on with the system.

This log contains all events in the system, including uploads and downloads. The information pertaining to uploads and downloads is much more granular, logging points in time for any given operation such as the creation of the upland, when it was started, etc.

The log also has information about the start and stop of processing, which, in addition to any possible transformations, also includes validity checks (matches expected schema, that there were no empty rows, etc.)

Comments

The Event Log allows you to add comments about a given event to tag those that are significant. It is also possible to filter to only events that have comments.

Severity

If the system has issues processing information, such as the system is in the process of adding capacity to a large data source or there is an API error.

In general the system will attempt to retry an operation, but it is possible that an error is unrecoverable, such as an issue with the vendor’s API.