In this article, we walk you through our journey of migrating workloads from an on-premises data center to the AWS Cloud. The last server in our data center was decommissioned in December 2024
What is Motion Data?
Motion Data by T-Systems harnesses the power of geo-information analytics from Deutsche Telekom's mobile network to provide valuable insights into user movement and behavior. By analyzing anonymized mass movement data, combined with demographic details, businesses can make smarter, data-driven decisions across various industries like retail, tourism, and public transport. Motion Data enables companies to optimize strategies such as store location planning, marketing efforts, and transportation services by identifying high-traffic areas, understanding visitor patterns, and enhancing customer experiences. Fully GDPR-compliant, Motion Data ensures privacy while delivering actionable insights that drive efficiency and growth.
The Scope of the Migration
The migration targeted environments used for post-processing, QA, and development for our Motion Data product. Motion Data provides mass movement data aggregated from individual mobile customers' movements and is sold to companies in retail, tourism, and transport sectors. Since we operated in a colocation setup, we were responsible for maintaining all the hardware and software for these environments.
Why We Moved to AWS Cloud
When we received a notice of a 50% cost increase from the colocation data center, we began evaluating a cloud migration strategy and conducting cost estimations. Ultimately, AWS emerged as the best option, offering not only cost savings but also solutions to several existing challenges, such as:
- Outdated technology constraints: The latest versions of Python were unusable due to underlying Hadoop restrictions
- High maintenance effort: Significant resources were required for managing OS, Kubernetes, Postgres, Jupyter, and Hadoop
- Scattered file storage: Various storage solutions (local file system, NFS, SFTP, HDFS) made file-like persistence management inefficient
- Fragmented compute: Running multiple compute technologies (physical machines, LXC/KVM containers, Kubernetes, YARN) added to operational overhead
- Hardware maintenance burden: Maintaining hardware was resource-intensive and limited our ability to scale efficiently
Planning and migration strategy
During the planning phase, we identified and prioritized workloads for migration, assessed costs, and determined the most suitable migration strategy. AWS recommends six migration strategies, known as the "6 R’s": Rehost, Replatform, Repurchase, Refactor, Retire, and Retain.
We opted for Replatforming, allowing us to leverage the cloud’s benefits, including managed services, scalability, and faster innovation. Additionally, AWS provided an opportunity to replace our aging Hadoop stack. Since we were already using Python and Spark for distributed data processing, we transitioned from Hadoop’s Spark on YARN and HDFS stack to Spark on Kubernetes and S3 with minimal code changes.
Key improvements with AWS
By migrating to AWS, we were able to implement significant improvements:
- Extensive use of managed services
- Kubernetes (EKS)
- Spark (EMR on EKS)
- Jupyter notebooks (EMR Studio)
- PostgreSQL (RDS)
- Consolidation of compute workloads into managed Kubernetes (EKS)
- Unification of file-like persistence into a single S3 bucket
- Right-sizing of compute workloads via Karpenter in EKS
- Future-ready architecture with the option to go serverless
Results achieved
The migration delivered tangible benefits:
- 35% lower infrastructure costs compared to pre-cost-increase levels in the data center
- Access to up-to-date technology stacks:
- Python
- Spark
- Kubernetes
- Jupyter
- Postgresql
- No resource constraints for compute workloads and file-like persistence
Migrating to Cloud, we significantly improved our infrastructure efficiency, reduced operational overhead, and positioned ourselves for future scalability and innovation. This journey reinforced the value of cloud adoption, ensuring that our Motion Data product remains competitive and future-proof.
