Push or poll? How to handle large data sets with ease

Becoming the number one global aggregator of energy data is the vision we are working towards at Metry. Everyday we are working with millions of data points: collecting, updating and serving them. Obviously, a large data set brings its own challenges at different levels.

The challenge with API’s

Our responsibility at Metry is not only limited to collecting energy data – to make data useful it needs to be delivered to different 3rd party systems. I hear you saying “that’s great, why don’t you use an API to communicate?”. Good news is we already have a standard, neat looking, Restful API to make all the data available for our customers. Bad news is that it does not scale at this volume.

Our customers require the most up to date energy data to fulfill their purpose and bring value to our customers. However, in today’s metering technology, devices can not guarantee that a measurement is always accurate. There are various reasons for a wrong measurement; hardware failures and meter changes are the most common ones. But the most important factor for us to consider is that energy measurements can be changed over time.

One simple mistake…

Energy measurements being corrected is a part of reality when you’re working with energy data. But when they are corrected, and when that information is received, matters greatly. Examples would help us to make the case clearer:

3rd of May 03:30 – An electricity meter in an apartment building with 4 storeys reports 6342 kwh of energy usage for the entire month of April in 2021. Metry successfully store the measurement as 2021-04 : 6342 kwh

7th of May 00:22 – An error has been detected regarding the metering device and consumption of electricity for the month of April has been recalculated as 8572 kwh. Metry successfully updates its own database as 2021-04 : 8572 kwh

Everything looks ok from Metry’s perspective: we received the first measurement successfully and then, when it has been corrected, we have successfully updated our database. But a slight change of perspective makes us see things differently.

What is the measurement the facility manager sees in their 3rd party system if they have synced data with Metry only before the correction happened? The answer is simple: they would see 6342 kwh instead of 8572 kwh. This could cause the facility manager to make wrong decisions, do wrong reporting and so on. One simple mistake can have so many consequences, including financial ones.

Polling data drains resources

To mitigate this situation, our 3rd party system partners usually go the path of “polling” data from the Metry API. It is as simple as making the same requests periodically to see if the data has changed or not. If data is the same on both ends, no action is required. If data is different at Metry, the 3rd party updates their database with the latest information.

But polling data is a constant resource drain for both Metry and for the 3rd party that’s polling the data – and since the measurement change rate is quite small, more than 98% of the recurring http requests would most likely be in vain. And believe it or not, polling unnecessary data also has a negative impact on the environment.

– Considering that every http request has a carbon footprint, we are not only consuming computing resources, we are also emitting CO2 for no purpose. Metry is responding to millions of http requests every day, so having a goal to reduce this overhead was a no brainer for us! says Ugur Atar, Head of Software Development at Metry.

This is the reason why we built Metry Streams. We needed a solution where we can seamlessly communicate with our customers when a new measurement arrives or is updated. Instead of constantly polling data from Metry, pushing up to date data to an external system only when needed is much more efficient and environmentally friendly.

Simple and sustainable collection with Metry Streams and Exports

Metry Streams is a solution developed by Metry to overcome the challenge of notifying our customers when energy measurements are updated.

Simply, we keep track of your energy measurements for each metering point you have in the Metry registry. When new measurements arrive, or an existing one gets updated, we write it in a special buffer to be able to communicate with you from a separate channel than our Restful API.

This separate channel is called Exports. We can push new or updated measurements (consumption data) to your system periodically (every hour, day or month) – and we only push the data you have not received before. You simply get only the data you need, an incremental update every time. That way, both parties can reduce unnecessary bandwidth overhead, implementation complexity and all the tracing challenges.

Exports push data in CSV format which is a plain and simple format that makes data exchange and implementation lightweight. Any modern programming language has support for parsing CSV files. Read more about our generic format here.

For sending the CSV file mentioned above, we support the following target systems:

• FTP and SFTP servers
• HTTP server
• Email
• AWS S3 bucket
• Azure Blob Storage

Read more about how you can receive the delivery here.

Metry Streams is a hassle free solution for making seamless integrations between Metry and a 3rd party software. All you need to do is map the metering points from Metry to your own system and implement a CSV parser that can accept regularly incoming files. The rest is taken care of by Metry!

Are you ready to make your energy data collection even smoother and, in extent, reduce your carbon footprint? Talk to us and we’ll walk you through the process!