Using a battery for intraday trading as a hedging strategy
At the end of 2023, traders from Essent’s Energy Markets department came up with the idea to investigate the potential of using a battery for automated trading on the intraday market as a hedging strategy. The prior goal was to deploy an optimization framework that allows for fully automated battery asset steering on the intraday market with a minimum required timeframe of 15 minutes. This is because trades are done on the hourly and quarter hourly intraday market. As Dataworkz, we were asked to join the project team in the role of software engineer. Our task would be to build and deploy a robust and reliable cloud application that runs fully automated as frequent as possible.
First steps
Early 2024 I joined the project team. The other members already made a prototype of a linear optimization algorithm that could successfully create an optimal charging schedule with a quarter-hourly granularity for a battery based on historical intraday market prices. Also, we already had access to an external market API that could provide us with real-time market positions and prices, and to an API used to steer the battery.
Together, we outlined the desired end state of our application. We planned to bring all the pieces of prototype code together and create a production-ready application. The general idea was as follows. Every iteration:
- The current state of the battery, such as state of charge, is retrieved from the external API that steers the battery
- The real-time market position and prices are retrieved from the external market API with a quarter-hourly granularity
- A linear optimization algorithm runs to create a charging schedule with a quarter-hourly granularity that maximizes the profit
- Orders are sent to the external market API
- The schedule is sent to the API that steers the battery
- All relevant log data is uploaded to Azure Storage
Requirements
From the start we had several crucial requirements which affected the design and the choice of tooling.
- The application must run only during business hours, at least the first period until we were confident enough it could run without supervision.
- The application must be configurable without stopping. For example, the battery has certain limitations when it comes to charging power or capacity. When there is maintenance, this may change. We wanted to be able to update the settings without deploying a new version.
- Runs must be reproducible for auditing and analysis purposes. This means we need to log the data from the runs.
- The application must run as frequent as possible, but at least every 15 minutes.
- The application must avoid trading when an error occurs when communicating with the market or battery API.
Algorithm
To create an optimal charging schedule for the battery, we choose for a linear optimization approach. Our application is written in Python and for the linear optimization we use the Pyomo library. The objective of the linear optimization problem is to maximize the profit. You get a higher profit if you charge the battery when prices are low, and discharge when prices are high. Hence, the input for our linear optimization algorithm consists of the following:
- The current planned schedule for the battery
- The intraday market prices
The constraints are mainly related to the battery’s state and limitations, such as state of charge and maximum capacity and charge power. Obvious constraints are of course that the algorithm cannot produce a schedule where the battery charges more than its capacity, or at a higher rate than possible.
The output of this algorithm is an optimal schedule for the battery together with the amount of energy we need to buy or sell at the intraday market for every time interval. The intraday orders are then sent to the market API, and the corresponding battery schedule is sent to the battery steering API. These APIs are not maintained by our team.
Tooling
Essent’s Energy Markets department runs their tooling and applications on Azure. For version management we have an Azure DevOps repository. Every release, a Docker image is built in an Azure DevOps pipeline.
Because of the first requirement list above, about running the application during business hours, we decided to use a Cron schedule. In our repository we have a crontab file stating the schedule and the Python run command. Our Docker file ends by executing the crontab file.
The Docker image is built in the Azure DevOps pipeline and published to Azure Container Registry.
To run the algorithm, we choose for an Azure Container App application. The container app loads the image from the container registry and runs it according to the schedule. The main reason why we choose for an Azure Container App instead of, for instance, an Azure Function, is because we needed to build a Docker image where we can install a binary executable of a third-party solver for our Pyomo optimization algorithm. This solver is installed in our Docker image, and the path to the executable is set as environment variable which we use in the Pyomo model.
Logging and monitoring
In the list of requirements, we also mentioned reproducibility and auditability. This is important in the first place for us. We want log data to analyze the performance, and to compare newer, improved algorithms, or to investigate why certain decisions were made by the algorithm. On the other hand, we are trading on the intraday market, and we must be able to show and clarify upon request by an audit or risk department which trades we executed and why.
For these logging purposes, we upload all relevant data to Azure Storage at the end of a run. This includes among others:
- Metadata, such as start of the run, initial state of charge and solution status
- Project data, such as version and dependencies
- Market prices
- Initial battery schedule
- Final battery schedule
- Market trades
All these files are saved as JSON files in Azure Storage every run.
We have developed a very basic dashboard built with Plotly Dash to show the runs of the current day. This dashboard uses the logged JSON files from Azure Storage described above.
Results and next steps
We went live with our intraday battery dispatch optimization application in September 2024. Since that moment the application runs once every minute during business hours (9-17), which is much faster than our initial requirement of once per quarter. It runs with almost no downtime or failed runs, and without supervision. The occasional downtime is only caused by battery maintenance, and not by application failure. There are also some days where an error occurred in a few runs. This has often to do with an error response from one of the two external API’s. For example, the collection of intraday market prices fails for a reason beyond our control. Our application then aborts the run. This is to prevent from doing a trade without being able to execute the schedule in the battery.
The results are very positive, and currently the organization is seeing if we can increase the use of batteries for flexibility optimization. For our own project we are currently working on an implementation of using the battery for aFRR (automatic frequency restoration reserve).