Monthly-Update on AutoSpotting

Monthly Update August 2022 - Liftoff

Sat, 10 Sep 2022 20:36:12 +0100

Hello,

I’m happy to announce that I finally left my full-time job last week, so this is my first status update as self-employed. Over the last few weeks I’ve been vacationing a lot So far there’s not much to share

It’s been a while since my previous update in here, which was at the end of October.

I’m now trying to resume these monthly updates, so in this first post after this gap I’d like to give a few updates covering the last months, and also my plans going forward.

Monthly Update October 2021

Sun, 31 Oct 2021 20:36:12 +0100

Hello,

As usual, here’s my latest monthly report.

Going forward I’ll release these reports here on the CloudUtil blog and I’ll only share the link through my mailing list and our new Slack #announcements channel.

If you want to be notified when I write a new one, don’t forget to sign up to the CloudUtil newsletter shown at the bottom of the page, or join us on Slack.

I also published as blog posts most emails I had sent to the AutoSpotting newsletter over the last year, so you can see how it changed over this time. I had to edit them a bit to fix URLs and clean them up a bit, bear with me in case there were any issues.

Monthly Update September 2021

Mon, 04 Oct 2021 03:36:28 +0100

Hello,

As usual, here’s my latest monthly report.

AutoSpotting

Since the previous month there hasn’t been much going on in the mainline repo, at least not from my side. The only thing I did there was updating dependencies, including instance type information for the recently released M6i instance types.

From the community side, I’m very thankful to mello7tre who contributed a couple of significant changes:

allow us to reuse existing Spot instances instead of always launching new ones, saving some time when doing Spot replacements in certain situations.
added a global configuration flag (defaults to off) that allows us to disable the Instance Rebalancing Recommendation events, which sometimes get emitted without resulting in a Spot termination, creating additional instance churn. The caveat is that with this enabled you only get the 2-minute termination notification, and don’t get to run the instance termination lifecycle hooks, but it gives you a bit less instance churn if you don’t need those.

This month I also updated the Terraform code to match CloudFormation and documented it as officially supported installation method on the AWS Marketplace.

Monthly Update August 2021

Sun, 29 Aug 2021 10:13:42 +0100

Hello,

As usual, here’s my monthly progress report, this time covering August 2021.

AutoSpotting

As I mentioned in my previous report, I’ve been working on releasing the new AutoSpotting stable build on the AWS Marketplace. After many months of hard work, I’m happy to announce that the release finally landed on the Marketplace a few days ago, and it’s probably the first serverless paid product built with Lambda available on the Marketplace.

Monthly Update July 2021

Mon, 09 Aug 2021 07:24:54 +0100

Hello,

Another month, another community update. This one comes about a week late as I’ve been on a long vacation and some things had to give.

Progress report - July 2021

AutoSpotting - throughout my vacation I’ve been working a lot on a still unreleased branch that prepares AutoSpotting for being published on the AWS marketplace as a paid Docker product, charging a small percentage of the savings. This was way harder than I expected, as AutoSpotting is probably the only Lambda-based software being made available as paid Docker product so a lots of things had to be figured out and tweaked for making it work under the Marketplace constraints. There are many not-so-functional changes under the hood, such as calculating the generated savings, repackaging the Lambda to use Docker images in order to fit the Docker product requirements of the marketplace, charging a percentage of the savings amount through the Marketplace APIs from a new Fargate task(don’t ask! ;-) ), etc. I’ve learned a lot while building this(I’ll probably publish a technical blog post about it at some point) and it was a lot of fun and even though it basically works, I still have a number of things to figure out or update(especially docs). My goal is to have this published by the end of August, fingers crossed! I’ll do my best to not cause much impact to the OSS users, but some things might be rough on the OSS side for a while after this code lands.

Monthly Update June 2021

Sun, 04 Jul 2021 11:54:03 +0100

Hello,

Another month, another AutoSpotting community update.

Progress report - June 2021

EBS-Optimizer - To be honest this month I haven’t made as much progress on AutoSpotting as I had hoped, but I started a couple of new workstreams which kept me busy. The first is EBS-Optimizer, my new tool dedicated for EBS volume upgrades from GP2 to GP3, resulting in up to 20% better cost and often better performance than GP2 volumes.

Monthly Update May 2021

Wed, 02 Jun 2021 11:40:52 +0100

Hello,

As promised in my last monthly email, I’m now sharing monthly the progress achieved in the previous month and what’s coming soon.

I’ve also resumed the monthly AskMeAnything-style community calls, and below you can see the recording of our previous one.

(The recording of this call is unfortunately no longer available online.)

Progress report May 2021

I helped polish further the contribution by Alberto (mello7tre on Github), which simplified the event handling logic by introducing a FIFO SQS queue for serializing AutoScaling operations (such as temporarily extending the maximum capacity and suspending AutoScaling processes) instead of previously calling an auxiliary Lambda function with concurrency 1. The code and internal architecture is significantly cleaner and much more robust after our initial tests and I’m very happy with the way it works. I’d really appreciate if you could give it a try in a test environment and please let me know if you run into any issues.
this month I also merged the EBS storage upgrade logic that converts GP2 volumes to 20% cheaper and more performant GP3 volumes for EBS volumes smaller than 170GB. There’s also a change for similarly converting IO2 volumes to IO3, but that’s unlikely to make much of a difference for Spot use cases. This is also available in the current nightly build, and here’s how it looks like for GP2 volumes:

Monthly Update April 2021

Sat, 01 May 2021 21:48:07 +0100

Hello,

It’s been a while since the previous status update and also I really missed seeing some of you like I did in our monthly webinars.

I decided to give a status update for the last few months and considering it’s the beginning of the month going forward I’ll do my best to do this on a monthly basis, sharing the progress since the previous month and what’s coming soon.

Monthly Update October 2020

Fri, 16 Oct 2020 21:54:24 +0100

Hello,

I’m happy to announce that the massive code change that has been in development since more than a year ago has been merged and after a few further improvements it’s now ready for Alpha/Beta testing by a wider audience.

What’s new?

Event-based instance replacement

In a nutshell, when deployed using the latest CloudFormation infrastructure code, AutoSpotting is now reacting to all new instance launches, in addition to the legacy cron-based execution mode that’s still available as before. Under normal circumstances newly launched on-demand instances are now immediately replaced with spot clones, avoiding to run the spot clone in parallel to the AutoScaling group they were launched for which has always been a quirk of AutoSpotting when compared to its alternatives.