So, as I mentioned in my last post, I started a new consulting gig recently at Mesh-AI. It’s been a whirlwind of activity, and I wanted to share some of the interesting things I’m working on.

Now, I don’t know if I can name the actual client but it’s a major UK energy provider that generates and supplies electricity.

The consultancy works in squads and ours has few data engineers, a platform engineer (me), a software engineer, along with a number of leaders to keep an eye on things. As the lead (and only) platform engineer, I feel deeply responsible for making sure the team has the tools and infrastructure they need.

I’m currently working on two projects:

  • Building a Video AI Engine Platform on AWS using Terraform. It’s a video analysis application that processes uploaded videos using AWS AI/ML services to identify customer vulnerabilities. It’s currently a human review process, that takes hours, and the goal is to automate it to provide near real-time insights.

    So this involves setting up:

    • S3 buckets for video and metadata storage with triggering events.
    • AWS Lambda Python functions for processing.
    • AWS Rekognition for video analysis.
    • AWS Transcribe for speech-to-text conversion.
    • A Cloudront distribution, with a WAF attached, with:
      • An Origin for a React frontend hosted in S3.
      • An Origin for API access through API Gateway to a Python Lambdas.
      • An Origin for video access via signed S3 URLS.
    • Cloudfront for content delivery.
    • Dynamodb to provide data to the UI
    • SNS for notifications.
    • Cognito for user authentication.
    • And a extensive Github Actions CI/CD pipeline for automated deployments.

    All the while ensuring security, scalability, and cost-efficiency - along with complying with the client’s extensive patterns and policies.

  • The second project is helping out with building an AI scenario framework for a weather-dependent energy simulation system for their Wholesale Market Optimisation team. It simulates weather regimes, wind speeds, and solar irradiance to support energy market forecasting and analysis.

    • This one quite simply involves building a very modular Lambda based framework in Python that sets up S3 input and output buckets to fetch climate data, and will eventually pul configuration from a RDS database - and then trigger simulations.
    • The client here is curiously using Serverless Framework for IaC, so I’m getting familiar with that tool as well.

Whist this is all meaty stuff and it’s exciting to be at the forefront of integrating AI/ML, I have to say that I’m really enjoying working with both the Mesh-AI team and the client’s team. Everyone is super smart, collaborative, and driven to make a real impact. And, despite me hating commuting, working in the heart of London a few days a week never gets old. I’ve found an excellent place to get get tacos near the office, can attend a lot of tech meetups again, and just generally feel energised by the city’s vibe.

As always, you can add my RSS feed to your reader of choice and if you made it this far thanks for reading!

Chris