Google BigQuery to Databricks migration
Leading provider of Crypto solutions for businesses
Problem we found
Leading provider of Crypto solutions for businesses wanted to centralize its data to provide new products & services to its customers.
Customer Vision
- Get all payments related data on one platform that can scale, process very high volumes of data in near real time and is cost efficient
- Easy addition of new products and services for their customer
- Reduce the cost of data processing
Technical Pain
- Data was siloed in different platforms, and different applications were unable to communicate with each other
- High cost of data processing, with slow data processing speed
- Need for simplified data architecture
Solution we implemented
- Migrated off BigQuery in 4 months, ahead of time of budget
- Reviewed current architecture and Databricks Jobs
- Integrated siloed applications including Heap analytics, Salesforce and Jira
- Helped the customer with ETL, DDL, and Unit Testing of each pipeline
- Highly collaborative communication allowing our teams to work transparently and successfully to develop use cases
$100k
Monthly reduction in data processing cost
Positive Outcomes
- Saved $100,000/ month in data processing costs
- Simplified data architecture allowing the customer to quickly create new use cases & services for clients, at low cost
- Significant improvement in data processing performance and Time to Insight
- Completed the project ahead of schedule and below budget
“Your team has performed outstandingly. Particularly I would like to call out Santosh Chouhan. Santosh has demonstrated expert knowledge over the BQ migration process and this is reflected in the respect and trust he has built with the customer. Santosh is always prepared, has amazing attention to detail, and instills a sense of calmness despite the complex nature of the project. It would be an honor to have additional opportunities to work with Santosh on future engagements”. Tyler Gainey, Sr. Project Manager, Services, Databricks