Meter study¶
The Meter Study is a research project at the University of Oxford’s Environmental Change Institute (ECI) collecting data on household demographics, activities and electricity use. The aim is to better understand the timing of electricity use in households and identify potential routes towards flexibility in electricity use, which could support the emergence of more sustainable energy systems.
Modules in this Repository¶
Meter¶
This module provides SQL functionality to connect, query and upload to the database.
// .. automodule:: meter // :members:
Meter interface¶
The Meter interface is a terminal based app to configure the research tools of the Meter study and support data handling with the MySQL database. The forms support Mutt-style navigation and key bindings, based on the npyScreen package.
Meter mailer¶
A utility to send emails to mailing lists and contacts in the Meter Database. Built on a similar framework as the Meter interface.
See full Documentation for more detail.
The Meter Database¶
This section explains
- the types of data collected as part of the Meter project;
- how these data are structured, stored and retrieved; and
- the tools used to manage the data.
This is a live document. For the latest version, please visit Meter/docs.
What data are collected?¶
Meter collects three types of data:
- Household and individual survey information
- Individual activity information
- Electricity readings at household level
Each of these data are explained in more detail here.
Database schema for Meter Data
Household and individual survey information¶
Survey information is grouped into three tables:
Contact: This is personal information with restricted access (all other data will become public). It is provided by participants when registering via energy-use.org/hhq.php.
Household: Socio-demographic information about household composition (number of people, ages, housetype, appliances, electricity bills income). This table also keeps the ‘date_choice’ - the preferred date for the study, and ‘status’ (see Section ‘Status progression’).
Individual: Collected on the study day via the booklet or app. Covers individual information, such as age, occupation, working hours, use of appliances...
Individual activity information¶
Recorded via booklet or app. Each entry contains:
Time (dt_activity): Freely chosen by participant (unlike in conventional Time-use research, where 10 minute windows are prescribed). Certain times of particular interest will be suggested in the app (e.g. 5:30pm, 6pm, 8am). ‘dt’ stands for DateTime in the format ‘2012-11-13 23:59:59’.
Time recorded (dt_recorded): The time when the entry was made. In case of the app, entries can be back-dated (what did you do 20 minutes ago?). Booklets may be coded up days later. Entries made via the web-interface later still. This time stamp helps to identify ‘how devoiced’ the entry is from the actual event.
Time use code (tuc): Numeric code based on an extended version of the Harmonised European Time-Use Survey codes (@eurostat14). See field ‘ID’ in activities.json.
Activity: Plain text description of the activity. See field ‘title’ in activities.json.
Location: Numeric code for location:
- home
- travelling
- work
- public place
- outdoors
- other
- not specified
Enjoyment: Numeric code for enjoyment:
- not at all
- not very much
- neutral
- somewhat
- very much
- not specified
Electricity readings¶
Collected via current clamp connected to an Android phone using DMon software. Recordings are taken every second and stored in a csv file with two columns: a DateTime stamp ‘2012-11-13 23:59:59’) and the reading in Watt. The file name refers to the Meta table ‘idMeta’ value followed by underscore and a sequential number for each time readings were started for this device with this ‘idMeta’. Often the xxxx_2.csv file contains the ‘real’ readings, because the phone was started up for testing before shipment. A xxxx_03.csv file is created when starting up the phone to process the data.
When uploading to the database this file is copied to the server first to speed up the database processing. Each reading is a row with an ID (idElectricity) the reference to the Meta entry (Meta_idMeta), the DateTime stamp and the reading value in Watt.
Database structure¶
The Meta table is central to the data structure as shown in Figure 1. The name is perhaps an unfortunate left over from early trials with data structure. It could equally be called the ‘devices’, ‘instruments’ or ‘study events’.
Each row in this table has a unique ID (idMeta) and represents either a booklet or a phone that has been sent out. The entry is created as part of the equipment set up. Each household gets one entry for the electricity recorder (eMeter) they receive and one for each booklet or activity app. DataType for an eMeter is ‘E’, whereas a booklet or device with the activity app is labelled ‘A’.
When devices are returned the CollectionDate, which is ‘NULL’ until now is updated.
Individual and Activity data share the same idMeta value, because they come from the same instrument and can thus be linked. Individual/Activity data and Electricity readings can be linked to their Household via the Household_idHousehold reference in their Meta entry and vice versa: To find all Activities for a given Household with the ID 1234 one could use the following SQL statement:
SELECT idMeta
FROM Meta
WHERE Household_idHousehold = 1234 AND DataType = 'A';
This will return one idMeta for each individual which has completed a study day. The activities can then be retrieved for each of these returned values. If 6789 was returned, the activity record can be accessed with:
SELECT dt_activity,activity
FROM Activities
WHERE Meta_idMeta = 6789;
This will produce a list of DateTime values and the description of the activity.
Further information¶
The data handing is principally conducted with the Meter Interface. This repository includes a dummy database file in /dbDummy which can be used for experimenting. To set this up you require a working MySQL version on your machine. Create a user and grant privileges to the data.
To gain access to the live Meter database on www.energy-use.org you will need to be granted access.
Configuring Devices¶
Prepare root/flash¶
Install packages (.apk)
- root
- Insecure
- Flashify
- AppRemover
- AppHider
- AppLock
- ButtonRemap
- Tasker
Copy
- recovery.img
Settings
- Lock settings
- Select Lock screen
- None
Open “Uninstall” (AppRemover)
- System app
- delete about 43 apps
- User app > remove:
- MyEE
- SuperBattery
Open “AppHider”
- hide all but the downloaded apps
- hide Settings
- Pin Code -> Never Show
Open “Tasker”
- Profile > [+] > Event > Display > Off
- [+] task
- App > Launch > Meter
Open “Button Remapper”
- enable service > OK
- [+] Short and long press
- BACK > do nothing
- Home > do nothing
Open “AppLock”
- lock all but Meter
Create Backup¶
adb reboot recovery
- BACKUP
- tick all
- > Options > enable compression
- swipe to backup (392s)
Copy backup folder to local machine
adb pull /sdcard/TWRP ./TWRP
eMeter setup¶
Requires: Pixi 3 - Android 4.2.2
Assume device is rooted, Insecured
From bash with USB connected:
adb push recovery.img /sdcard/recovery.img
adb install Flashify.apk
adb push ./TWRP /sdcard/TWRP
Open “Flashify”
- OK, OK, OK
- Recovery image
- select /sdcard/recovery.img
- Flashing (2 min)
adb reboot recovery
- RESTORE , select, swipe
adb shell
mkdir /sdcard/METER/
exit
adb push id.txt /sdcard/METER
aMeter setup¶
Requires: Pixi 4 - Android 6
Start device (Skip or ignore of offered options)
> Settings > About Phone
- tap Build Number 5 times to enable Developer options
> Settings > Developer Options
- enable OEM unlocking (allow bootloader unlock)
- enable USB debugging (allow to connect via USB)
- Connect device via USB
- Tick trust this device on device
- on computer, open terminal in the folder where recoverypixi4.img resides (this file can be found in this repository under flash_aMeter/)
NOTE you may need to reconnect the USB cable between each step
adb reboot bootloader
fastboot oem unlock
Confirm with <VOL UP>
fastboot flash recovery recoverypixi4.img
fastboot format userdata
- remove the battery (tough but quick way to switch off)
- insert the SD card with the backup (see flash_aMeter/TWRP/)
- insert battery
- start phone with Power + VOL UK (this is equivalent to fastboot reboot)
- select <Restore>
- select <Storage location>
- select <SD card>
- pick directory and swipe to restore
- when done, remove battery and SD card
- insert battery and start with Power button
Set Time¶
Use this command to set the time (also available in Meter Interface > Menu > Device > Set time)
'adb shell date -s `date "+%Y%m%d.%H%M%S"`'
- on device, go to > Settings > Time and Date
It may be necessary to set the time manually to the right time zone first.
To make the time ‘stick’ it is necessary to manually confirm it. For this open either time or date and tap OK.
MeterActivities (JSON)¶
A json structure for activities and their navigation
Screens¶
Lists keys to activities.json Each screen key has six such ‘activity’ keys
Activities¶
Each entry can be thought of as a button on the screen
- “caption” is displayed on the button
- “title” describes the activity on the home screen and is used as plain text description in visualisations
- “ID” is the time-use-code based on an extended version of HETUS
- “help” optionally displayed in the app to explain the meaning of a button
- “value” used for ranking questions or for numerical calculations, such as relative adjustment of time by, say, 15 minutes
- “icon” and image file in png format stored in /img
- “next” key in screen.json to display the next screen after pressing this button
LegendHousehold¶
- all columns in the Household table of the Meter database are listed as keys
- under each key possible values are assigned their ‘meaning’ as plain text
- “q” denotes the question asked to which these are the anwers
- not all possible answers are listed. For instance, counted values like “How many screens do you have”, can be left as numerical values
- Note all these values have been uploaded to Table Legend in the Meter schema using json2sql
LegendIndividual¶
- same as Legend Household, but for table Individual on Meter database
- Note all these values have been uploaded to Table Legend in the Meter schema using /utils/json2sql.py
Other Meter Repositories¶
Activity recorder¶
The app to record activities is kept in the `Meter App`__ repository (see MeterApp documentation).
