WP3 Smart Meters - First face-to-face project meeting
March 30 – April 1, 2016
Statistics Estonia, Tallinn
Organizer: Maiki Ilves
Participants: Alexander Kowarik (AT), Olav Grøndal (DK), Peter Stoltze (DK), Dan Wu (SE), Maiki Ilves (EE), Toomas Kirt (EE), Tauno Tamm (EE).
Minutes
1. Welcome
Maiki welcomed all participants and introduced the agenda of the meeting. No additions to the agenda were proposed by the participants.
2. Review of the current status and future perspectives regarding the availability of smart meters in other European countries. Presentation by Toomas.
A smart meter is an electronic device that records consumption of electric energy at intervals an hour or less and communicates that information at least daily back to the utility for monitoring and billing. Smart meters enable two-way communication between the meter and a Distribution System Operator (DSO) and the central system can determine the interval of recordings. The European Union's Third Energy Package set a goal to further open up the energy market and increase competition between energy providers. New smart grids are energy networks that can automatically monitor energy flows by smart meters and adjust to changes in energy supply and demand accordingly. The European Commission has proposed a deployment plan for smart electricity meters in the EU Member States on the basis of economic assessments of long-term costs and benefits and achieve almost 72% deployment rate by 2020. It is expected that 195 million smart meters will be deployed in EU by 2020. The participating counties of the WP3 have the plans as follows: Austria has set a mandatory roll out strategy and expects achieve penetration rate 95% (5.7 million smart meters) by 2019 and data refresh rate is 15 minutes; Denmark has set a mandatory roll out strategy and expects achieve penetration rate 100% (3.28 million) by 2020 and data refresh rate is 15 minutes; Estonia has set a mandatory roll out strategy and expects achieve penetration rate 100% (709 thousand) by 2017 and data refresh rate is 1 hour; Sweden has set a voluntary roll out strategy and has achieved penetration rate 100% (5.2 million) by 2009 and data refresh rate is 1 hour. All over the world there are several big smart meters deployment projects and according to the report generated by Telefónica Digital it is predicted there will be deployed almost 800 million smart meters by 2020.
3. Review of available data of Smart Energy Meters in DK. Presentation by Olav.
In Denmark there is one datahub for electricity consumption data owned by Energinet.dk. Statistics Denmark has experience of combining company data with smart meters data. One year smart meters data (aggregated on one year). Almost all companies have smart meters, not so with households. Known problems with the data source are that smart meters are connected in the database to ID of paying client, who is not necessarily the consumer. E.g., renters are not in smart meters database. Some meters do not have company information attached, at all. Misclassification problem of NACE.
Discussion: datahub owners very interested in collaboration, they do not have infrastructure to give data researchers. Statistics DK will give access for research purposes and Energinet.dk is very interested in that. Within 1-2 months will get access to new data, not yet decided whether half an hour data or slightly aggregated (depends on the IT readiness). Problems: own produced electricity (solar panels, wind) is not observed.
4. Review of available data of Smart Energy Meters in AT. Presentation by Alexander.
Overview of the situation in Austria: lot of grid operators (each state has at least one), goal by 2020 is that coverage of the smart meters is 95%. Smart meters have 15 minutes reporting interval. Each customer can opt out regarding getting the smart meter. It is expected that not more than 5% of population opt out. E-control (regulatory body is Austria) has permission to access aggregated data (higher level aggregation than day). One loophole: ministry can make special request that NSI can get access to electricity data but there is no infrastructure for it. The best case scenario for NSI: access to daily aggregated data, however ministry is not keen to make special provision for the NSI at the moment. IT department is just now developing big data skills, but the current infrastructure and philosophy does not support big data.
Discussion regarding possibility of linking data: companies have several IDs (tax office, social security), no one ID used in the country. Inside office one ID is used.
5. Review of available data of Smart Energy Meters in Sweden. Presentation by Dan.
Overview of the situation in Sweden: There is national action plan for the smart grids. There are 380 owners of grid operators in the country and for three last years hourly consumption reporting is in place. Creating one datahub is in the planning stage, SCB needs to describe the use cases and has opportunity to influence the content of the datahub. Current energy statistics source at SCB is survey on electricity consumption, heating and hot water consumption by multi-dwelling houses and small houses.
SCB lawyers put stop to gaining access to data in order to first study the privacy issues of the project (Nov 2016). In case access will be gained, then it will be from some individual electricity providers, and aggregated data. At the moment, monthly data seems to be feasible.
6. Review of available data of Smart Energy Meters in Estonia. Presentation by Toomas.
The Estonian electricity system connects the power stations in Estonia, the network operators and electricity consumers. The largest Estonian producer of electricity and heat energy are Eesti and Balti power stations running on oil shale, owned by Eesti Energia, which provide over 90% of the electricity produced in Estonia. From January 1st, 2013, Estonia’s electricity market is completely open and all customers are eligible consumers. The Estonian Data Hub system is maintained by Elering and it is a software/hardware solution that manages the exchange of electricity metering data between market participants, supports the process of changing electricity suppliers in the market, and archives the metering data of electricity consumption. It is expected that by the end of 2016 all the metering points are replaced by smart meters in Estonia. Statistics Estonia has got a copy of the data in the Elering’s data hub in 2015.
7. Review of use of Smart Meters in official statistics – case-studies in other countries outside the WP3.
Presentation by Olav and Alexander: UK Smart meter experiment to estimate in Ireland the occupancy rate for given day. Data used was from survey and smart meters for 6500 households, smart meter data was for half hour periods. 8 algorithms were developed for classifying a home as unoccupied for a given day.
Presentation by Dan: In the second year of UNECE Big Data TF smart meters pilots were carried out. Information found about three projects.
- UK innovation lab project
- Canada project: created hourly consumption synthetic data based in Ireland data
- Ireland project: machine learning exercise to categorize the households. For test group household budget survey households were used.
Discussion: It was decided not to spend too much time to find more use cases as it was believed that most relevant cases are covered. Dan will look into UNECE Big Data TF internal website and share relevant documents from there if there are any.
8. Discussion and agreement of the outline of the first report about data access and data handling and agreeing on responsibilities.
Group agreed on following outline and responsibilities regarding the Report 1.
Introduction – topics to be covered:
- The aim of the project (just referring what and why it is done) – very brief, in the beginning of every report, could be shared with all the WP reports. Responsibility: EE
- The aim of the WP 3 - to see whether the same data source and methods and techniques could be used for all countries, is it possible to replace the traditional data sources (all or partly). Responsibility: EE
- The aim of the report 1 - what we will cover in this report. Responsibility: SE
- Use of electricity data in official statistics
- Traditional statistics of electricity consumption produced by WP3 countries - How it is produced, Concepts (what are statistical units: enterprise, household), what is produced, who uses, why uses, quality of outputs, European policies and requirements. What are problems today and where we want to go and what are the challenges? Responsibility: ALL
- Expected results/business cases of the big data approach – improvements what we can do to existing statistics (e.g., increased frequency). Rather short. Responsibility: ALL
- Smart Electricity Meters
- Smart Meters in Europe. Responsibility: EE
- Smart meters in the rest of the world. Responsibility: EE
- Examples of studies with smart meters for official statistics
- UK 2 cases. Responsibility: DK, SE
- Canada case. Responsibility: AT
- Singapore case. Responsibility: DK
- Ireland cases. Responsibility: SE
Each case summarized to 1 paragraph + reference to the material
Data access – topics to be covered:
- Description of electricity market in the partner countries (Who produces, who distributes, who collects the data ...)
- Availability of smart meters in the partner countries
- Data holders/owners in the partner countries
1 paragraph general summary (Responsibility: SE) and then one page per country. Responsibility: EE, AT, DK, SE
- Describe data transmission practise and possible technical solutions. For example, is there already something developed for the admin data and could be used for the big data. Responsibility: EE, DK
- Questionnaire to all EU country representatives and analysis of that data. Responsibility: DK
List of possible questions to ask:
Does your NSI already have access to smart meters data? What kind of smart meter data (electricity, gas, water)?
If no, then has your NSI considered getting access to smart meter data?
Have you encountered any legal problems getting access to smart meters data?
Do you know whether your country has central data hub for smart meter data?
Are you aware of any projects outside Europe where smart meters data is used to produce statistics? If yes, please send us the materials/links.
- Sustainability of the source. Address it shortly. Responsibility: DK
Legal aspects regarding the access to the data and linking data to other data sources
- Description of the process getting access to the data
- legal environment in the country
Half a page per country EE, AT, DK, SE
- Collaboration model examples. Responsibility: DK, EE.
- Privacy. Won’t go into privacy topic, too complex to cover. We will just mention shortly that public opinion may be of concern. Reference to the article by ONS McKenna, E., Richardson, I., & Thomson, M. (2012). Smart meter data: Balancing consumer privacy concerns with legitimate applications. Energy Policy, 41, 807–814. doi:10.1016/j.enpol.2011.11.049 Responsibility: EE
Data handling
- Description of the available data
- structure, records (what are units?) , attributes and definitions Responsibility: EE, DK
- quality (incl. missing values), outliers in the dataset by box-plot. Possible indicators to be computed: uniqueness of identifiers, item nonresponse, misclassification cases, etc.
Peter (DK) will send full list of indicators developed in admin. data project and we will choose relevant indicators next WEBEX meeting. Responsibility: EE
- Assessment of the current coverage (households and businesses) and expected coverage in the near future. Responsibility: EE, DK
- Technological requirements to pre-process data (hardware and software). Describing our experience: what was used and what worked, what didn’t. Responsibility: EE, DK, AT
- Processing raw data and transforming it into usable cleaned datasets to be used in the next tasks. Responsibility: EE, DK
- describe as fully as possible what has happened to data before it arrived to NSI and what was done in NSI before statistics computation, including adding geo-ID, item imputation.
- common schemas for all countries, is it possible?
- Visualising first results to demonstrate the potential (or not) of the big data sources for official statistics. Possible outputs:
- Maps
- Data confrontation – benchmarking against the existing results.
- Infograph with data from EE and DK
- Interactive demo of proof of concept (based on synthetic data)
Objectives for generating synthetic data. Responsibility: AT
Conclusions. Responsibility: SE
9. Discussion and agreement on the timetable of the first report.
Group agreed on following timetable of Report 1:
29.04.2016 Introduction part: countries output ready
29.04.2016 Objectives for generating synthetic data ready
30.05.2016 Data access part: countries output ready
08.06.2016 Data handling part: EE part ready
20.06.2016 Final deadline for the output !!!
13-15.06.2016 Tallinn meeting, demo ready
28.06.2016 SE sends draft version to partners for commenting
28-29.06.2016 Commenting by partners
30.06.2016 Draft report to be sent to review board
20-28.07.2016 Commenting by partners and finalizing report
29.07.2016 Final version ready, sending to Eurostat
Collaboration: main way to collaborate in the project would be to share the software experiences and program code. It means we should aim to the common structure of data.
Ideas for analysis
Do basic classification (household vs. business, household type classification) all smart meters in the dataset.
Possible other data sources that could be linked with smart meters data. During discussion following sources were mentioned as possible sources for linking:
- business register (DK, EE)
- population register (DK, EE)
- building register (EE)
- address register (EE for geocoding)
Group discussed our vision of methodology, quality and technological frameworks for the study.
Methodology framework – to be further elaborated by AT.
Data editing – will be partly covered in Report 1
Linking (incl. record matching)
Creating new variables (classification)
Different classes of methods, criteria to find the best method
Quality framework – to be elaborated by EE, SE
Input quality: indicators will be agreed on the next WEBEX meetings and will be included in Report 1.
Output quality: indicators will depend on whether we will follow Eurostat’s quality dimensions (e.g. timeliness, coherence, comparability over time and between countries) or some other quality dimensions. For example, UNECE (2014) quality framework for big data.
Discussion on which are possible error sources for this data source: classification error, modelling error, unit error (delineation error), coverage error (if not whole population), linking error, measurement error (if meter not smart). Discussion point for the future: which of these can we quantify for our report? Another discussion point for the future: should burden aspect be covered under quality?
Technology framework – to be elaborated by DK
11. Brainstorming to come up with potential new statistical products based on smart meters data.
Ideas that were mentioned:
- consumption by building type (buildings efficiency)
- consumption by heating source of the building
- relation to some economic statistics (turnover, GDP)
- compare and relate to the weather information
- seasonal occupancy rate (inbound tourism)
- linking with mobile data to estimate the main residence of the individual, estimate the size of the households, to form the households (if no household register), labour status of the individual
- profile the households behaviour into classes by e.g. sleeping time, working time, cooking time
- predicting the consumption based on the temperature information
- consumption by electric cars (in stations)
12. Discussion and agreement of the other project management issues –ways of communication (Skype), channels of communication (Wiki?)
It was decided that we use WEBEX possibility to communicate. Frequency of the meetings was decided to be in every three weeks.
Planned WEBEX meetings are held 14-15.00 (CEST)
05.04 - Testing WEBEX 16-16.30
20.04 – frameworks, objective of synthetic data, Peter: input quality indicators
11.05 – introduction and data access commenting
01.06 – preparation for the 13-15 meeting, data handling part commenting
29.06 – report 1 draft version comments
27.07 – report 1 review board comments, finalize report
It was decided that next face-to-face meeting will take place in Vienna and tentative time of the meeting is 14-15 November 2016.
13. Discussion on expectations and requirements regarding a synthetic data set. Presentation by Alexander.
Three possible objectives for synthetic data:
- 1Technical testing in UNECE sandbox (find out whether it is possible to use Irish data in Sandbox) – fixed structure and volume of data are important. Goal is to test how much time/ resources are needed to process N sized datasets.
- Generate demo output i.e. prototypes for different outputs can be generated before results from real data are available. No specific country should be mentioned. Amount of data needed: random draw of 1000 smart meters (500 businesses and 500 households), hourly data, 1 year, additional information on aggregated level needed.
- Generating synthetic population for prototyping new kind of statistics and test the scaling and benchmark the algorithms (enterprises, households). Real consumption data and household and businesses unit information is needed. Consumption information daily level is needed of 5000 smart meters of households for 1 year.
14. Examining the Estonian data set and discussion on sharing Estonian data with Statistics Austria for creating synthetic data purposes.
Introduction of Estonian data by Toomas and Tauno.
In 2015 it was agreed that Elering provides data of smart meter recordings from period 2013- 2014 to Statistics Estonia. The provided database consists of all tables but the most important tables are metering data, metering points, agreements and consumer which contain information about by whom and where electricity was consumed. The dataset contains hourly recordings from 709 000 metering points and the data amounts to 1.5 TB. Recordings per year result in ca 365 * 24 * 709000 = 6 210 840 000 data rows. From the database sample time series for two subjects (one household, one enterprise) were selected and first visualizations were showed. Graphs indicated that there is clear difference in consumption pattern. Linking metering point data with geo information enabled to visualize consumption density map.
Discussion: Access to microdata is needed for AT. Denmark will find out whether and on what conditions access through their research centre is possible. Maiki helps Alexander with getting access to some Estonian data by sending relevant information to sign contract for use of confidential data for scientific purposes.