1Click飲みRomoCartTempescope色色[:iroiro]Other Projects


Myra, an intelligent robotic lighting system for your living room

Myra is a robotic spotlight that adjusts autonomously to optimize the lighting condition in your home.
By sensing what you're doing (e.g reading, sleeping, eating, etc) using depth sensors, it changes the orientations of the lights to ensure you always have the perfect lighting, no fumbling around with switches needed.
I started this project in my spare time in December 2013.  I intend to opensource everything when I find the time.

See Myra in action:

The Objective
The motivation behind this project is simple: to free people's lives from the constraints of the "fixed" lighting system.

Conventional room lighting is static (or only mildly flexible).
- For example, many rooms (especially in Europe) are designed with a dark ambient, with some spots of light. 
This is aesthetically fine, but what if you decide that you want to read a book in the middle of the room? You can't.
With conventional lights, actions of people (e.g reading) are constrained to particular locations (e.g next to reading lamps).
- Another example is if a single room holds multiple functions, like a living room and dining room (as is often the case in small countries like Japan.)
It would make sense to be able to change the lighting based on how you're currently using the room;
when you're eating, you want to light up the table and hide everything else,
but perhaps if you're watching TV, you just want some light on the walls.
Again, the conventional static lighting system wouldn't allow this (very easily), so when a room is multi-functional, people have to compromise and live with suboptimal, inaesthetic lighting.

Such problems could be solved if we had a lighting system that:
(a) can orient light and shadow in any configuration
(b) does (a) with minimal user input (preferably completely autonomously)

This is the idea behind building Myra, a robotic spotlight array that will configure itself automatically by recognizing the residents' activities.

The Prototype

The Myra system consists of three components:
(1) The Myra Light: this is a robotic arm with an LED.  An array of these are placed around the room, and are controlled by a central PC.
(2) An RGBD sensor (e.g Microsoft Kinect, or Asus Xtion): this allows the PC to "see" where the residents are, and what they're doing.
(3) A PC: the PC analyses the readings from the sensor to classify user activities, decides how to light the room, and sends commands to individual Myra lights.

Components of the Myra system; an array of Myra lights (left) and an RGBD sensor (right)

(1) Building the Myra Light
The hardware is very simple. (schematics etc will be uploaded somewhere at some point)
It is an Arduino controlled wireless robot arm with a high-powered LED.
The LED is fitted with a lens, which gives a fairly strong light beam of 15° arc.
A Myra Light on my curtain rail

BOM (for a single Myra light):
The entire setup (for a single light) should be less than $60.
I have attached all of my lights to the curtain rail, using an attachment that I made using plywood.
You also need a 315MHz transceiver to wirelessly control the light from your PC.

The software that goes on the Myra light is a very simple Arduino sketch;
receive orientation (θ,φ) and the intensity (k) from the 315MHz receiver with VirtualWire.h, and then set the orientations of the servo (and the intensity of the LED) accordingly.

(2) Setting up the RGBD sensor
Any off-the-shelf RGBD sensor will do.
I used an Xtion Pro Live, because it requires no external power and so is easier to handle.

(3) Developing the controller PC
All of the intelligent decisions take place in a central PC that reads input from the sensor, and sends commands to the Myra lights.

The current system is written in Java, using OpenNI and NiTE for processing the RGBD sensor feed.
Visualizations are done using JMonkeyEngine.

The figure below shows the control pipeline for the Myra system:
Control pipeline for Myra

I won't go into the specifics of each step (I'll do that in a future post), but here is what each step does.
1) You live your life. Myra requires no user input.

2) The RGBD sensor collects a point cloud of your room at 60Hz.

3) Myra uses NiTE to extract people from the point cloud. NiTE tags people, and their features (e.g head, shoulders, hands, etc) automatically.  

4) This is where much of the hard work happens.  Myra first classifies the state of each person into 5 different activities: reading, watching TV, standing, walking, sleeping.
Then, "lighting targets" are set according to the person't activity, and their postures.
For example, if a person is "reading", then a target is set on the person's arm closest to their head.
 -> this allows Myra to continually adjust the light so that your book is never in shadow. 
If the person is "walking", multiple targets are set: one at the predicted destination, and others at various features in the path up to the destination.
 -> this is useful in cases like, when you are watching a film and you need to go to the toilet, or you get up in the middle of the night to get a glass of water.  There's no need to find the light switch, because Myra automatically shows the way to the door, without disturbing anyone else.

5) After lighting targets have been found, Myra assigns a robot light to point at one (or more) of the lighting targets.  The video above shows a scene where Myra picks the best light to shine on a book I am holding.

6) Finally, the orientations of each of the lights are sent wirelessly to each robot light.

Steps 1~6 are repeated about 3 times a second. 

(4) Calibration
When the system is first setup, Myra doesn't know where each of the lights are with respect to the sensor.
This is essential information, because it needs to be able to know which values of (θ,φ) would shine light on a particular position in the sensor's coordinate system.

Myra calibrating itself
I'm not going to go into details here, but I have developed a fully automatic calibration system (as shown in the video); I will explain how this works in a future post.
Essentially, the system points the light in random directions, and sees where the center of the lights lye.
It then uses simulated annealing to find the best-fit estimation of the light orientation.

So what can it do?
Here are some of the things that I have got Myra to do so far:

- automatically generate "skins" for different functions of your living room
The same room looks entirely different just from the lighting.

- point of interest tracking: ensure that whatever posture you're in, your book will always be lit
Myra follows you, however absurd your favorite posture may be.
- create a "path of light" to wherever you're going, so you can always find your way anywhere safely and without stress
You're never in the dark with Myra.
- automatically turn off when you fall asleep

- a "targeted alarm clock" that wakes a person up by shining targeted light at the target's face, without disturbing anyone else

Myra is a project still very much in development.
I don't think the use cases I've described above fully exploit the potential of the very new concept of an autonomous robotic light array.
When I find the time, I will release all schematics and code in open source.
I will appreciate any comments or suggestions.
Thanks for reading.


If you want to do well in Topcoder, live in Japan, South Korea, or Cape Verde

Topcoder Single Round Matches are held at incredibly obscure times if you live in certain parts of the world (e.g Japan).
Most of the SRMs in the past 3 years were held at 5pm(GMT)=1am(Japan Standard Time), which means that East Asians may have been competing in SRMs half asleep.
The purpose of this quick analysis is to examine the effects of time on Topcoder performance.

Summary of Results
- Competing at night may have detrimental effects on performance.
- There's no evidence to suggest that a particular time from noon~evening is better than another.
- The current Topcoder schedule favors East Asians, and disfavors Americans.

The Topcoder Data Feed provides data such as users' countries and past competition results.
I used data for all rated Div1 SRM results from Jan 2012 to Jan 2014 (=102 events)
A timezone was assigned to each user based on their country.
For countries with multiple timezones, I assigned the median (e.g CST for USA).

As an example, here's a visualization of the average absolute score of all Div1 results for each timezone.
Fig1. Average Div1 score for each timezone (Blue: 120 ~ Red:212)
You can see that there is great inter-timezone difference in the average score.  Parts of South America have average of 120 (UTC-5), whereas East Asia has an average score of 212 (UTC+9).
The cause of difference may include factors like the number of competitors, education, etc.
In order to see just the effect of time on the score, I need to be able to normalize-out the inter-timezone differences.

I therefore calculated the "Normalized Score" metric to represent a user's relative performance in an SRM event.
For a particular user, I have a list of the user's score in each SRM he has competed in.
For each of these scores, I normalize the score by the average and standard deviation of all scores of events within 70 days.
This allows us to quantify how much better the user performed in an event compared to "usual", independent of other factors.

Local Time vs Normalized Score
For each SRM event, I calculated the local time for each user, and the average "Normalized Score" for each local time.
(The local time is the time at the timezone where you live. If you live in Japan, and the competition is at 5pm(GMT), your local time is 1am, and your score will be added to the normalized score average for 1am.)
The following graph shows the results:
Fig 2. Average Normalized Scores for each localized time

The maximum average Normal Score is at 23pm (Normalized Score 0.09), and the minimum is (surprisingly) at 10am (Normalized Score -0.11)
From between 10am to 23pm, there is no clear relationship between time and the Normalized Score, suggesting that people may perform equally well at any of these times.
At night, there is a clear decreasing trend from 3am to 8am.
It is surprising that the average Normalized Score at 1am and 2am are both positive, meaning that people are performing better at these times than usual.
Note that the standard deviation at each of these data points were around 0.9, which means that none of these data points have significant difference.  Therefore, none of these trends are evidential.

Where's the best place to live?
The graph below shows the chance of an SRM event starting at a particular time.
Fig3. Ratio of SRM start times per starting time (GMT)

For each timezone, by using this frequency distribution added with the normalized score per local time (Fig.1), we can calculate the expected "Normalized Score" of a particular timezone:
for GMT time [t], let the chance of an SRM event happening at t be: p(t) (as in Fig.2)
for local time [u], let the expected normalized score of a participant competing at local time u be: s(u)
then, for a particular timezone with offset [h] from GMT, 
the expected normalized score is: Σ[0<=t<24]. p(t).s(t+h)
This results are shown below (Fig4, Fig5):
Fig4. Expected Normalized Score of a timezone
Fig5. Expected Normalized Score of a timezone (as a heap map)

Interestingly, the further east you go, the better chance you have of doing well.
Note that these scores are based on the Normalized Score, which means that it is purely a representation of the effect of living at a particular timezone, when SRMs are scheduled according to the frequency distribution as in Fig.2. (and is therefore independent of other factors like education, etc)
The best places to live in are:
UTC+9 (Japan, South Korea, Indonesia)
UTC-1 (Parts of Greenland, Cape Verde)

I showed that there is a (non-significant) trend suggesting that competing after 2am has a negative effect on SRM performance.
However, people perform relatively well on competitions that start before 3am.
During the day, there is nothing to show that a particular time is better than another.

I also showed that the best places to live for Topcoder are East Asia, and the middle of the Atlantic, and the worst place to live is USA(!)
This is contrary to my initial presumption that the current schedule is unfavorable for East Asians (and I owe Topcoder an appology).

Some future work:
- how about other factors? weather? investment in education?
- what's the best event schedule to ensure fairness to all timezones?