The Challenge

When you’re running an attraction for thousands of visitors, situations can change in minutes or seconds. Connect&GO’s clients needed reports and data feeds they can use to proactively direct resources and staff. Before finding Estuary, Connect&GO integrated its 12+ MySQL instances into Snowflake using a self-hosted ELT solution that often overloaded the small data engineering team and required them to use other teams' resources. More importantly, data was moved in batch every 45 minutes from MySQL to Snowflake, which was too big of a delay for their customers who wanted more real-time visibility.

The team knew they needed streaming data pipelines, but switching to a more complex self-hosted solution, like Kafka, would have been an even greater strain on tight resources. Meanwhile, the managed batch ELT services they considered — while an improvement, would add too much latency for Connect&GO’s customers.

All the other vendors out there are batch-based, so you'd have five-minute turnarounds for data refreshes. For us, finding something that was both pretty cost-effective [with latency] close to the second was very attractive. 

Alexandre Pelletier, Senior Data Engineer, Connect&GO

The Solution

The team was searching for a fully managed solution for capturing and streaming in change data from MySQL to Snowflake. They required that the solution wouldn’t impact their production databases, and so preferred a platform capturing change data from a write-ahead log. And given the tight budget constraints most data teams have come under, they wanted to find something cost-effective and sustainably priced as they scale. During their search, they discovered Estuary. Estuary was the only solution they found that combined the ease of use of a managed ELT tool with true data streaming.

Connect&GO connected their MySQL databases through the Estuary UI and quickly saw their large existing DBs backfill within a couple hours on top of Estuary’s distributed stream processing framework. Once connected, Connect&GO was able to quickly set up a Snowflake destination and lock in their desired write schema for the data. Data typically landed from their MySQL DB to Snowflake within 15 seconds... the vast majority of this latency deriving from limitations on Snowflake’s ability to ingest. Beyond just a typical point-to-point ELT/ETL setup, Connect&GO’s team felt ready to cost-efficiently scale knowing that data ingested using Estuary lands in their data lake, allowing them to spin up a new fully-backfilled pipeline in minutes.

Using Estuary’s low-latency connectors, Connect&GO empowers customers to access to their most highly requested datasets in seconds. Through real-time reports, Connect&GO enabled Parks are driving higher revenue and optimized guest experience with real-time insight on capacity availability and foot traffic by access zone.