xpate build a data lake to automate transaction analysis and earned a $18K grant from AWS
The company introduced an operational process for transaction reconciliation in 11 months by implementing a data lake on AWS.
About xpate
The xpate group provides payments services in Europe and Asia through multiple products.
The latest initiative provides an all-in-one payment and banking solution to e-commerce merchants in the UK with a planned service expansion to mainland Europe in late 2023.
The platform provides a great variety of ways for merchants to receive payments (from traditional credit cards, payment links to alternative payments methods such as Google Pay and Apple Pay), and comes with pre-built plugins, check-out pages, and its API to support any type of merchant, from a starter to an expert.
Country
UK
Industry
Fintech
Timeline
09.2023 - ongoing
The xpate group provides payments services in Europe and Asia through multiple products.
The latest initiative provides an all-in-one payment and banking solution to e-commerce merchants in the UK with a planned service expansion to mainland Europe in late 2023.
The platform provides a great variety of ways for merchants to receive payments (from traditional credit cards, payment links to alternative payments methods such as Google Pay and Apple Pay), and comes with pre-built plugins, check-out pages, and its API to support any type of merchant, from a starter to an expert.
The challenge
xpate serves e-commerce sellers who want an all-in-one service for most finance management needs.
Since 2018, xpate’s range of services had expanded to include 6 finance-processing tools. But the company’s transaction reconciliation remained a daily chore, requiring manual labor to:
- Collect transaction data from all core systems into a .json file
- Import it into Excel
- Run intensive processing, mapping, and matching in Excel
After a “productive collaboration” with The Software House’s engineering team in a previous project, xpate’s CTO Ruben Vosmeer was “happy to extend it” with a new challenge.
His business goal was to automate the platform’s accounting.
Ruben wanted to start by creating a data lake that could also be funded in part by AWS’ “Proof of Concept” program, since xpate had been a long-time AWS customer.
Partnership goal To build the company's first data lake and secure $18,000 of AWS funding for xpate
Before
xpate’s financial analysts spend over an hour on manual work daily to collect, process, and analyze transaction data in search of incorrect transaction records
After
The platform sources first and third-party data into one dashboard automatically through the data lake, saving dozens of hours in financial reviews
The Software House's cloud engineering team led the development process for xpate
We worked on
Team formation
xpate used the same product team from us it has been working with since 2022 to create the data lake and secure the AWS grant.
A Cloud Architect, 4 Developers, a DevOps Engineer, a QAA Engineer, and a Project Manager handled the project fully while reporting to xpate’s CTO.
"The team is highly qualified and meets my expectations.
Additionally, I appreciate having a certain level of control during the process. The Software House is great at merging their team with ours and co-delivering.
Recently, The Software House has made their teammates more resilient to uncertainty."
Our partnership
The Software House’s Team and xpate’s Engineering Consultant implemented an AWS data lake in 11 months. Securing $18,000 in financing from AWS took another 6 weeks.
Technology choice
AWS Cloud, Lambda + Next.js, Serverless framework, AWS Glue + Scala, AWS Quicksight, Aurora Serverless
What xpate achieved
xpate sources information into their new data lake from three points — the payment gateway (which we upgraded months earlier), the ledger service, and the customer authorization service.
The project allowed Ruben’s team to monitor the success/failure rate metric for finance transactions, which was critical fo xpate’s goal of improving the reporting workflow and accounting accuracy.
The logic
1. Lambdas, set up in Node.js, receive data from external APIs and push them to S3 data lake ingestion buckets
2. Changes to S3 trigger AWS Glue, which is set up in Scala
3. AWS Glue collects data and maps it to an Aurora Serverless database that serves as a data warehouse
4. The Glue service also saves the data back to the S3 data lake in parquet format
5. xpate reviews the intel with AWS Quicksight, which allowed their team to set up a range of dashboards
AWS Proof of Concept Program
The “Proof of Concept” program rewards businesses in contract with AWS for embedding Amazon’s services into their digital product.
xpate qualified for the program under the condition they’d continue to use the AWS cloud for a given time.
The Software House applied for a grant from AWS for two parts of the projects.
Once xpate had the data lake implemented, AWS paid out $18,000 to the company.
Challenges
AWS Glue runs on Scala, which we didn’t work with before, yet the service was the best option for the job.
After our team adapted the language, we faced a roadblock while tweaking Aurora Serverless. It didn’t have a native integration with AWS Glue.
The engineering team had to create a custom query for AWS Glue so that it could push data to Aurora.
How xpate used AWS
AWS Lambda
A compute service that runs code on Amazon’s infrastructure and provides automatic resource management to free engineers from server management. Used for backend operations and business logic.
AWS Glue
Serverless-based data processing service that can integrate information from multiple sources. Used for software development, analytics, or machine learning.
Amazon S3
Stored documents used by the product and data backups as a reliable and easy-to-manage and configure service.
Amazon Aurora Serverless
An on-demand service for autoscaling Amazon Aurora’s database capacity.
AWS Quicksight
A business intelligence service that transforms collected data into a readable format ready to be reported.
Deliver the project with a product team recommended by 98% of CTOs
On average, tech leaders from 10+ industries rate us ★5 for quality and speed. Build you product with confidence by using our 11 years of experience in software development.
Work only with scalable technologies
Expect a software solution designed for stability, usability, and scalability thanks to next-gen technologies used at The Software House: microservices, serverless, and cloud computing
290+ professionals available
People with the skill set your project needs are within reach – developers, cloud engineers, DevOps, architects, and product designers
Rated 4.8/5 on Clutch.co
We follow a simple mantra that worked for +160 software projects we delivered with success — everything we build must be of great value to you and your clients
"There was no issue that they left unaddressed."
As said by Smartum Oy’s Product Manager in a review with a ★5 for Quality and Willingness to refer.
Smartum upgraded their transaction platform from Drupal to one that supports React.
