xpate serves e-commerce sellers who want an all-in-one service for most finance management needs.

Since 2018, xpate’s range of services had expanded to include 6 finance-processing tools. But the company’s transaction reconciliation remained a daily chore, requiring manual labor to:

• Collect transaction data from all core systems into a .json file
• Import it into Excel
• Run intensive processing, mapping, and matching in Excel

After a “productive collaboration” with The Software House’s engineering team in a previous project, xpate’s CTO Ruben Vosmeer was “happy to extend it” with a new challenge.

His business goal was to automate the platform’s accounting.

Ruben wanted to start by creating a data lake that could also be funded in part by AWS’ “Proof of Concept” program, since xpate had been a long-time AWS customer.

xpate used the same product team from us it has been working with since 2022 to create the data lake and secure the AWS grant. 

A Cloud Architect, 4 Developers, a DevOps Engineer, a QAA Engineer, and a Project Manager handled the project fully while reporting to xpate’s CTO.

The Software House’s Team and xpate’s Engineering Consultant implemented an AWS data lake in 11 months. Securing $18,000 in financing from AWS took another 6 weeks.

Technology choice

AWS Cloud, Lambda + Next.js, Serverless framework, AWS Glue + Scala, AWS Quicksight, Aurora Serverless

What xpate achieved

xpate sources information into their new data lake from three points — the payment gateway (which we upgraded months earlier), the ledger service, and the customer authorization service.

The project allowed Ruben’s team to monitor the success/failure rate metric for finance transactions, which was critical fo xpate’s goal of improving the reporting workflow and accounting accuracy.

The logic

1. Lambdas, set up in Node.js, receive data from external APIs and push them to S3 data lake ingestion buckets

2. Changes to S3 trigger AWS Glue, which is set up in Scala

3. AWS Glue collects data and maps it to an Aurora Serverless database that serves as a data warehouse

4. The Glue service also saves the data back to the S3 data lake in parquet format

5. xpate reviews the intel with AWS Quicksight, which allowed their team to set up a range of dashboards

AWS Proof of Concept Program

The “Proof of Concept” program rewards businesses in contract with AWS for embedding Amazon’s services into their digital product.

xpate qualified for the program under the condition they’d continue to use the AWS cloud for a given time.

The Software House applied for a grant from AWS for two parts of the projects.

Once xpate had the data lake implemented, AWS paid out $18,000 to the company.

Challenges

AWS Glue runs on Scala, which we didn’t work with before, yet the service was the best option for the job.

After our team adapted the language, we faced a roadblock while tweaking Aurora Serverless. It didn’t have a native integration with AWS Glue.

The engineering team had to create a custom query for AWS Glue so that it could push data to Aurora.

AWS Lambda

A compute service that runs code on Amazon’s infrastructure and provides automatic resource management to free engineers from server management. Used for backend operations and business logic.

AWS Glue

Serverless-based data processing service that can integrate information from multiple sources. Used for software development, analytics, or machine learning.

Amazon S3

Stored documents used by the product and data backups as a reliable and easy-to-manage and configure service.

Amazon Aurora Serverless

An on-demand service for autoscaling Amazon Aurora’s database capacity.

AWS Quicksight

A business intelligence service that transforms collected data into a readable format ready to be reported.