Patrick Winston Explains Deep Learning

Patrick Winston is one of the greatest teachers at M.I.T., and for 27 years was Director of the Artificial Intelligence Laboratory (which later became part of CSAIL).

Patrick teaches 6.034, the undergraduate introduction to AI at M.I.T. and a recent set of his lectures is available as videos.

I want to point people to lectures 12a and 12b (linked individually below). In these two lectures he goes from zero to a full explanation of deep learning, how it works, how nets are trained, what are the interesting problems, what are the limitations, and what were the key breakthrough ideas that took 25 years of hard thinking by the inventors of deep learning to discover.

The only prerequisite is understanding differential calculus. These lectures are fantastic. They really get at the key technical ideas in a very understandable way. The biggest network analyzed in lecture 12a only has two neurons, and the biggest one drawn only has four neurons. But don’t be disturbed. He is laying the groundwork for 12b, where he explains how deep learning works, shows simulations, and shows results.

This is teaching at its best. Listen to every sentence. They all build the understanding.

I just wish all the people not in AI who talk at length about AI and the future in the press had this level of technical understanding of what they are talking about. Spend two hours on these lectures and you will have that understanding.

At YouTube, 12a Neural Nets, and 12b Deep Neural Nets.

Robot Is A Hijacked Word

The word “robot” has been hijacked. Twice. (Or thrice, if we want to be pedantic, but I won’t be.)


The word “robot” was introduced into the English language by the play R.U.R., written in Czech by Karel Capek, and first performed in Prague on January 25, 1921. R.U.R. stands for Rossumovi Univerzálni Roboti, though even in the first edition of the play, according to the play’s Wikipedia page, published by Aventium in Prague in 1920, the cover designed by Karel’s brother Josef Capek, had the English version as the title, Rossum’s Universal Robots, even though the play within was in Czech.

According to Science Friday, the word robot comes from an old Church Slavonic word, robota, meaning “servitude”, “forced labor”, or “drudgery”. And the more you look the more references indicate that it is not known whether Karel or Josef suggested the word.

But, in any case, in the play the robots were not electro-mechanical devices, in the way I have used the word robot all my life, in agreement with encyclopedias and Wikipedia. Instead they were “living flesh and blood creatures”, made from an artificial protoplasm. They “may be mistaken for humans and can think for themselves”. Both quotations here are from the Wikipedia page about the play, linked to above. According to Science Friday they “lack nothing but a soul”.

This is the common story, more or less, about where the word robot came from.


But, but, maybe not.  According to this report the word “robot” first appeared in English in 1839. 1839! It says that robot at that time referred not to an individual, neither machine, nor protoplasm, nor electro-mechanical , but rather to a system, a “central European system of serfdom, by which a tenant’s rent was paid in forced labour or service”. Ultimately that word came from the same Slavonic root.

So perhaps in English the word “robot” changed in meaning between 1839 and 1920. Though realistically perhaps no one who picked it up from the Capek brothers in 1920 had ever heard of it from the old 1839 meaning. And in any case it seems such a different use that I don’t think it really is a hijacking. Just as “field” was not “hijacked” in going from a field of wheat to a field of study.

I am not going to count this as a “robot” hijacking.


In 1920 “robot” referred to humans without souls, manufactured from protoplasm. But that meaning changed quickly.


By the time I was deciding what really interested me in life, the word “robot” had turned into meaning a machine, as given by the online English Oxford Living Dictionaries, where it defines the word as:

A machine capable of carrying out a complex series of actions automatically, especially one programmable by a computer.

Here is the cover of the January 1939 edition of Amazing Stories.

There is a robot, a machine, right on the cover, illustrating a story titled I, Robot. But for me the big news is that the author of that story is Eando Binder (a nom de plume for Earl and Otto Binder), rather than Isaac Asimov–we’ll get back to Dr. Asimov in just a minute. I have found one slightly earlier reference to mechanical robots, but it is only a passing reference. In this Wikipedia list of fictional robots there is a story titled Robots Return, by Robot Moore Williams, dated 1938. Unfortunately I do not have the text of either of the 1938 or 1939 stories, so can’t tell whether the authors assume that their readers implicitly understand to what the word “robot” refers.

However, in 1940, only twenty years after the R.U.R. play was first published with the English version of robot on its cover, Isaac Asimov published his story Strange Playfellow in Super Science Stories, an American pulp science fiction magazine (of the Binder story he said “It certainly caught my attention” and that he started work on his story two months later). Later, retitled as Robbie, the story was the first one that appeared in Asimov’s collection of stories published as the book I, Robot, on December 2, 1950.  I have a 1975 reprint of a republished version of that book from 1968. In that reprint, at the bottom of the third page, after describing a little girl, Gloria, playing with a mechanical humanoid Robbie, Asimov uses the word “robot” to refer to Robbie (note the alliteration) in a very casual way, as though the reader should know the word robot.  The word “robot” got hijacked in just 20 years, in the popular culture, or at least in the science fiction popular culture, from meaning a humanoid made of protoplasm in a Czech language play in Prague, to meaning a machine that could walk, play with, and communicate with humans. Note that 1940 was before programmable computers existed, so there was some more evolution to get to the definition involving computers as quoted above.

In the 1920’s such mechanical humanoids seem to have been referred to as “automatons”.

I have no idea what contributed to that transformation of the word “robot”, but I am eager to see any citations that might be offered in the comments section.

So now we have the first real hijacking of “robot”. But there has been a more recent one. I may not care deeply about the earlier hijacking, but I sure do care about this one. I am an old school roboticist in the sense of the definition in italics four paragraphs back. And my meaning has been hijacked!!


In the more recent hijacking, “robot” has come to mean some sort of mindless software program, that does things that are relentless, or sometimes even cruel, though sometimes amusing and helpful. This new use is getting so bad that it is often hard to tell from a headline with the word “robot” to which form of robot the story refers. And I think it is giving electromechanical robots, my life’s work, a bad name.

I think it starts with this secondary definition which also appears with the English Oxford Living Dictionaries definition from above.

Used to refer to a person who behaves in a mechanical or unemotional manner.

And then it includes an example of usage: ‘public servants are not expected to be mindless robots’.

Now Asimov’s robots have not been mindless, and none of mine have ever been mindless (well, perhaps my insect-based robots were a little mindless, certainly not conscious in any way). But the first industrial robots introduced into a GM automobile plant in 1961 certainly were mindless. They did the same thing over and over, without sensing the world, and did not care whether or not the parts or sheet metal they were operating on was even there. And woe be a person who got in their way.  They had no idea there was someone there, and even had no idea that someone, or anyone for that matter, existed, or even could exist. They did not have computers controlling them.

I may be wrong, but I trace the hijacking of the word “robot” to two things that happened in 1994. I am guessing that there is some earlier history, but that I am just not aware of it. Ultimately it is all the fault of the World Wide Web…

Tim Berners-Lee invented the World Wide Web and put up the first Web page in 1991. Explosive growth in the Web started soon after, and by 1994 there were multiple attempts to automatically index the whole Web. Today we use Google or Bing, but 1994 was well before either of those existed. The way today’s search engines, and those of 1994, know what is out on the Web is that they work in the background building a constantly updated index. Whenever someone searches for something it is compared to the current state of the index, and that is what is actually searched right then and there–not all the Web pages spread all over the world. But the program that does search all those pages in the background is known as a Web Crawler.

By 1994 some people wanted to stop Web Crawlers from indexing their site, and a convention came about to put a file named robots.txt in the root directory of a web site. That file would never be noticed by Web browsers, but a politely written Web Crawler would read it and see if it was forbidden from indexing the site, or if it was to stay away from particular parts of the site, or how often the owner of the site thought it reasonable to be crawled. The contents of such a file follow the robots exclusion standard of which an early version was established in February 1994.

You can see a list of 302 known Web Crawlers (listed as a “Robots Database”!) currently active (I was surprised that there are so many!).  Of the 302 Web Crawlers, 29 include “robot” in their name (including “Robbie the Robot”), 18 include “bot” (including “Googlebot”), three have “robo”, and one has “robi”.  Web Crawlers, which are not physical robots, have certainly taken on “robot” as part of their identity. Web Crawlers became “robots”, I guess, for their mindless search of the Web, indexing it all, and following every link without understanding what was there.

That same year there was another innovation. There had been programs, all the way back to the sixties, that could engage in forms of back and forth typed language with humans. In 1994 they got a new name, chatterbots, or chat bots, and some of them had a more or less permanent existence on particular Web pages. Probably they attracted the suffix “bot”, as they could seem rather mindless and repetitive, again harking back to the dictionary example above of mindlessness of robots.

Now we had both Web Crawlers and programs that could converse (mostly badly) in English that had taken parts of their class names from the word “robot”. Neither were independent machines. They were just software. Robots had gone from protoplasm, to electromechanical, to purely software.  While no one is really building machines from protoplasm, some of us are building electromechanical devices that are quite useful in the world. Our robots are not just programs.

Since 1994 the situation has only gotten worse! “Robo”, “bot” and “robot” have been used for more and more sorts of programs.  In a 2011 article, Erin McKean pointed out that there were “robo” prefixes, and “bot” suffixes, and that at that time, in general, robo has a slightly more sinister meaning than bot. There was “Robocop”, definitely sinister, and there were annoying “robocalls” to our phones, “robo-trading” in stocks caused the 2010 “Flash Crash” of the markets, and “robo-signers” were people signing foreclosure documents in the mortgage crisis. Chat bots, twitter bots, etc., could be annoying, but were not sinister.

Now both sides are bad. We see malicious chat bots filling chat rooms intended for humans, we see “botnets” of zombie computers, taken over by hackers to launch massive denials of service on people or companies or governments all over the world.

Here is a list of varieties of bots, all software entities. Some good, some bad.

At some web sites, such as, it seems to be all about “bots”, but it is hard to tell which ones are software or if any have a hardware component at all. Now “bots” seems to have become a generalized word for all aspects of A.I., deep learning, big data, and IoT. It is sucking all up before it.

The word “robot” and its components have been taken to new meanings in the last twenty or so years.


Here’s the bottom line. My version of the word “robot” has been hijacked. And since that is how I define myself, a guy who builds robots (according to my definition) this is of great concern. I don’t think we can ever reclaim the word robot (no more so than reclaim the word “hacker”, which used to be only pristine goodness forty years ago, and I was honored when anyone ever referred to me as a hacker, even more so as a “robot hacker”). I think the only thing to do is to replace it.

How about a new word? What should we call good old fashioned robots? GOFR perhaps?

One that comes to mind is “droid”, a shortened version of “android”, which before it was a phone software system meant a robot with a human appearance. Droid distinguishes itself from the hijacked version of android which refers to phone software, and is generally understood by people to mean an electromechanical entity, an old style robot. Star Wars is largely responsible for that general perception. But that is also the problem in trying to use the word more generally. The Star Wars franchise has the word “droid” completely bottled up and copyrighted. I know of three robot start up companies that wanted to use the word “droid”, and all three gave up in the face of legal problems in trying to do that.

No droids. Unfortunately.

So…what should the new word be? Put your suggestions in the comment section^{\big 1}, and let’s see what we can come up with!

^{\big 1}I manually filter all comments, as the majority of comments posted are actually advertisements for male erectile dysfunction drugs…

Megatrend: The Demographic Inversion

Megatrends are inexorable, at least for some decades, and drive major changes in our world. They may change our planet permanently, they change the flow of money within our society, they drive people to move where they live, they kill some people and cause others to be born, they change the fortunes of individuals, and they change the shape of our technology. When a megatrend is in full swing it is quite visible for all to see, and with a little thought we can make predictions about many of the things it will cause to change in our world. Having accurate predictions in hand empowers individuals to make canny decisions.

A few of today’s megatrends are global warming, urbanization, increased use and abuse of antibiotics, the spread of invasive species, increased plastic in the seas, rapid species extinctions, and changing demographics of the world’s human population. We see all these happening, but we can feel powerless to change them, and certainly they are impossible to change in the short term.

In this post I will talk about the demographic inversion that is happening in many parts of the world, and how that will drive much of our technology, and in particular our robotics technology, for the next thirty to fifty years.

We know how many 25 to 29 year olds there are in China today, and we know an upper bound on how many there will be in 20 years–less than two thirds of the number today. No political directives, government coercion, or technological break throughs are going to change that. This population trend is real and now unavoidable, and it is man and woman made. We can’t change this fact about the world twenty years hence. Given the proportion of the world’s population that is in China, and that more than half the other countries have similar trends, the aging of human society is a megatrend.

Magnitude of the inversion

Here is the data on which I based my comments above, in a diagram I got from the CIA World Factbook, yes, that CIA.

This is a pretty standard format for showing population distributions. Male and female are separated to the left and right, and then their age at the date of the data is histogrammed into five year intervals. This is a snapshot of the age distribution of the Chinese population in 2016. One can see the impact on the birth rate of the hardships of the cultural revolution, followed by a population boom a dozen years later. And then we see the impact of the one child policy as it was enforced more and more strongly following its introduction in 1979, with an uptick in the echo of the earlier population boom. The one child policy was phased out in 2015, but there is some evidence that the culture has been changed enough that one child couples might continue at a higher rate than in many other countries with equally strong economies. We also see here the impact of the cultural desire for male children over female children when there is a restriction to only one child. Not all the extra female children were necessarily subject to abortion or infanticide however, as it is strongly believed that there is a large ghost population of female children in China, girls who existence is hidden from authorities.

Here is the same data for Japan, and now we see a truly scary trend.

There are just less and less younger people in Japan in an unbroken trend lasting for forty years. Given the fertility age profile for women, forty years of decrease is really hard to turn around to an upward trend without immigration which Japan very much eschews.

The real impact is on age distribution. Again, from the CIA World Factbook, currently 27.28% of the Japanese population is 65 or older, compared with only 15.25% of the US population, or 6.93% of the Mexican population. In Japan in 2016 there were for every 1,000 people only 7.8 births while there were 9.6 deaths. Looked at another way, the average number of births per woman in Japan is now only 1.41, compared to the obvious number of 2.0 needed for population replacement, but more like 2.1 to cover early deaths. While the population of Japan is shrinking, the ratio of older people to younger people is going to get larger and larger.  There is detailed coverage of the aging of Japan in Wikipedia, and here is a critical graph (under Creative Commons License) from that article:

By 2050 predictions are that over 35% of the population of Japan will be 65 or older. All those people are already in existence. Only a truly stupendously radical change in birth rate can possible change the percentage. There are no indications that such a change is coming. Here is another way of looking at how the population is changing, this one from 2005 from the Statistics Bureau of the Japan Ministry of Health, Labor, and Welfare.

This is the typical, if somewhat more extreme in this case, change of shape of the population of developed countries in the world. Population is changing from being bottom heavy in age to top heavy. This is true of Europe, North America, Japan, Korea, China, and much of South America.

The current population of the world is about 7.5 billion people.  Predicting how that number will change over the next century still has a lot of uncertainty, largely driven by uncertainty over whether the fertility rate (how many children each woman has on average) will drop as quickly in sub-saharan Africa as it has in the rest of the world whenever the standard of living increases. But the growth in the world population does seem to be slowing down. On average, then, the pattern of an increasing ratio of older to younger people that we have seen above for China and Japan will be the overall pattern for the world over the next thirty to forty years. India is lagging Japan in this regard by about 50 years, so it will soon start happening there too, even as India becomes the most populous country in the world, surpassing China.

Consequences of the inversion

Thirty years ago in much of the developed world we saw schools built to handle the post war baby boomers close after that bubble of children worked its way through the system. Today we are starting to see the consequences of the demographic inversion. It is showing up in two ways: (1) less young people filling certain job categories, creating a pull on automation in those arenas, and (2) uncertainty about how care can be provided for the huge overhang of elderly and very old people we are about to see.

(1) Less workers. Some jobs that seemed attractive in earlier times no longer seem quite as attractive as younger people have more educational opportunities, and aspirations for more interesting jobs. Two categories for which this is particularly true are farming and factory work. Farming has the additional negative aspect of often requiring people to live away from particular urban areas that they might otherwise choose with more geographically mobile professions. These jobs both require physical labor, often in unpleasant conditions. Neither is a job that many people would take up at a later stage in life.

Food supply

The average age of a Japanese farmer is now 67, and in all developed nations the average age is 60.  Agriculture ministers from the G7 last year were worried about how this high age could lead to issues over food security. And as the world population is still increasing, the need for food also increases.

The Japanese government is increasing its support for more robots to be developed to help with farming. Japanese farms tend to be small and intensely farmed–rice paddies, often on terraced slopes, and greenhouses for vegetables. They are looking at very small robotic tractors to mechanize formerly manual processes in rice paddies and wearable devices, exoskeletons of sorts, to help elderly people, now that their strength is waning, continue to do the same lifting tasks with fruits and vegetables that they have done for a lifetime.

In the US farms tend to be larger, and for things like wheat farming a lot of large farm equipment is already roboticized.  Production versions of many large pieces of farm equipment, such as those made by John Deere^{\big 1} (see this story from the Washington Post for an example) have been capable of level 3 autonomous driving (see my blog post for a definition) for many years, and can even be used at level 4 with no one in the cab (see this 2013 YouTube video for an example).

There is now robotics research around the world for robots to help with fruits and vegetables. At robotics conferences one can see prototype machines for weeding, for precision application of herbicides and insecticides, and for picking fruits and vegetables. All these parts of farming currently require lots of labor. In the US and Europe only immigrants are willing to do this labor, and with backlashes against immigration it leaves the land owners with no choice but to look for robotic workers, despite the political rhetoric that immigrants are taking jobs that citizens want–it is just not true.

Tied into this is are completely new ways to do food production. We are starting to see more and more computer controlled indoor farming systems both in research labs in Universities and in companies, and as turn key solutions from small suppliers such as Indoor Farms of America and Cubic Farms, to name just two. The key idea is to put computation in the loop, carefully monitoring and controlling temperature, humidity, lighting, water delivery, and nutrient delivery. These solutions use tiny amounts of water compared to conventional outdoor farming. More advanced research solutions use computer vision to monitor crop growth and put that information into the controlling algorithms. So far we have not seen plays in this space from large established companies, but I have seen research experiments in the labs of major IT suppliers in both Taiwan and mainland China. We now have enough computation in the cloud to monitor every single plant that will eventually be consumed by humans. Farming still requires clouds, jut entirely different ones than historically. Indoor farms promise much more reliable sources of food than those that rely on outside weather cooperating.

Once food is grown it requires processing, and that too is labor intensive, especially for meat or fish of any sort. We are still a few years away from bionically grown meat that is practical, so in the meantime, again driven by lack of immigrants and a shortage of young workers, food processing is turning more and more to automation and robots. This includes both red meat cutting and poultry processing. These jobs are hard and unpleasant, and lead to many repetitive stress injuries. There are now many industrial robots in both the US and Australia being used to do some of these tasks. Reliance on robots will continue to grow as the population ages.


Manufacturing is an area where there has been great fear of job loss due to automation.  But my own experience in manufacturing in both China and the United States over the last 20 years is that there is a shortage of manufacturing labor and that the labor force is aging.

I have been involved in manufacturing in mainland China since 1997. In the early years it was with iRobot, and the companies I worked with were all based in Taipei or Hong Kong, but had plants in Guangdong Province. Around 2004 I started to notice that we were losing a lot of workers over Golden Week, the biggest holiday period in China where most people travel in unbelievably crowded trains over incredibly long periods to go “home” to visit parents and other family. We started seeing that our production lines were suffering after Golden Week as so many workers would not return, and it might take a couple of months to make up for those losses. This sometimes had real impact on our business with a drop in deliverable product.

By around 2005 in my role as Director of CSAIL (Computer Science and Artificial Intelligence Laboratory) at M.I.T., I was working with a number of very big manufacturing companies based in Taipei, with plants spread over a much wider area of mainland China. These companies did not have their own brands then, but many now do. They built many of the IT products that we in North America use on a daily basis. They were working with M.I.T. in order to move up the value chain from being OEMs (Original Equipment Manufacturers) for US brands to having the technology and internal R&D to develop their own unique and branded products. The message I got over and over again, often from the original founders who had started the companies in the 1970’s, but were now enabling a new generation of management to take the companies forward, was that it was getting harder and harder to get sufficient labor in China. I remember one particular discussion (and I most likely don’t have the exact wording correctly here, but this is how I remember it): “in the old days we would put up a single sign, 3 inches by 5 inches, and the next morning we would have a line of prospective workers around the block–now we employ advertising agencies in Shenzhen and run ads on TV, and still we can’t get enough workers”. They told me that their two biggest problems in China were worker recruiting and worker retention. That was in 2005.

Today as I talk to manufacturers in China I find that very well run companies, with lots of retention strategies in place will have a labor turn over rate of 15% per month (per month!!). Less well run companies will have up to 30% per month. Imagine trying to run a business with that level of turnover.

The reasons for the drop in eager manufacturing labor in China are complex. The demographic charts above tell a big part of the story. But another part is that the general standard of living has risen in China, people have more access to education, and they have higher aspirations. They don’t want to work in a factory at repetitive jobs–they want more meaningful work. All humans want to do meaningful things once they are beyond desperately trying to survive.

At the same time as this, I was working as an advisor to John Deere, visiting many of their factories in the US, and seeing how they were suffering from an aging workforce with no prospects for younger replacement labor to come along, in towns in the mid-west where the young would leave for bigger cities as soon as they had a chance. It wasn’t that there were not jobs in those plants and smaller cities, but that the youth was heading for bigger cities.

Those trends that I was seeing back then have been borne out in the decade since.

In this 2013 story at Bloomberg it is reported that the median age of a highly skilled US manufacturing worker was 56. And this 2013 report from the Manufacturing Institute compares how the median age of all manufacturing workers, skilled or unskilled, is rising compared to the age of other workers.

We can see that the median age of a manufacturing worker is going up by a year every two or three years, and that it is going up faster than for other non-farm jobs. In the 12 years shown here, manufacturing workers have gone from being 1.1 years older than other workers, to 2.7 years older.

My observations of a decade ago alerted me to the fact that worldwide we would be having a shortage of labor for manufacturing. This has indeed come to pass. Naturally I thought about whether robots could make up the shortfall. But it seemed to me that industrial robots of the time were not up to the task.

While there were a lot of robots working around the world in automobile factories they existed in a complete apartheid from humans. In the body and paint shops it was all robots and no humans, and in the assembly lines themselves it was all humans and no robots. The fundamental reason for this was that industrial robots were not safe to be around. They had no sensors to detect humans and therefore no way to avoid hitting them, with tremendous force, should humans stray into their workspace.

For a set of tasks which robots could do by themselves, repeatedly and reliably, humans were banished from that part of the factory, both for the safety of the humans, and so that humans wouldn’t mess up the totally controlled order needed for robots with hardly any sensors to see changes in the world. Where there  were tasks that humans but not robots could do, the solution was to have humans do all the tasks, and to banish robots.

There were many consequences of this dichotomy. First, it meant that the installation of robots required a well thought out restructuring of a factory floor, and turned into a process that could easily take a year, and involve much peripheral equipment, such as precision feeders, to support the robots. Second, because the robots and people were segregated there was no thought put into ease of use of robots, and use of modern user interfaces–there were no human users! Third, the robots had to be 100% successful at everything they were tasked with doing, or otherwise a human would have to enter the robot domain to patch things up, and that meant that all the robots, perhaps hundreds of them, would have to be stopped from operating. Fourth, the human factory workers had no direct experience of robots, no understanding of them as tools (as they did of electric drills, say), and so the alienness of them contributed to the “us” vs. “them” narrative that the press loves to propagate.

So in 2008 I founded Rethink Robotics (née Heartland Robotics) developing smart collaborative robots to address these shortcomings of industrial robots. The idea, and reality, is that the robots are safe to work side by side with humans so there is no longer any segregation. The robots come with force sensing and built in cameras so that out of the box they are useful and require much less peripheral equipment to control the environment for them. The robots have modern user interfaces, which, like the user interface of a smart phone, teach a user how to use the system as the user explores, so that ordinary factory workers and technicians can use the robots as tools. The robots get regular software upgrades, just like our phones and computers, so more value is added to them during their lifetime. And lastly, the robots are easily able to control other equipment around them, again reducing the set up and integration time.

These sorts of robots are growing in popularity, and are leading a revolution in automation of the 90% of factories in the world that are not automobile factories. They are beginning to answer the severe labor shortage, world wide, that the demographic inversion is causing.


Much of technology over the last 50 years has been used as a way of shifting service labor to the end user, and thus reduce the number of workers needed in order to provide a service. Examples of this are bank ATMs, self service gas stations, supermarkets rather than separate butcher shops, fruit and vegetable stores, and dry goods stores with all the goods behind a counter, vending machines, automated checkout lines, check-in kiosks at airports, on line travel reservation web sites, word processors, cheap printers, and electronic calendars replacing administrative assistants, etc.

One technological development that is different in this regard is fulfillment services for online shopping. No longer do we need to travel to a physical location for a particular sort of purchase, walk into the store, take physical possession of the object, take it out to our car, transport it to our house, and then unpack it. Now we choose an object, or a wide variety of objects that would previously be located at geographically separated locations, and they arrive at our homes with no more personal effort from us.

This has shifted and restructured labor for getting goods to our houses. And it is a mixture of it being a challenge to get enough workers in the chain of steps that run from online order to the goods being in the home, and the fact that this convenience will let the elderly stay in their homes longer, beyond when they are up for all the active shopping that yesteryear would have required.

The chain of events in fulfilling an order is roughly as follows:

  1. Go to many different locations in a warehouse where each of the ordered objects are stored.
  2. Pick up each object.
  3. Pack them in in a box.
  4. Move the box from the warehouse to some sort of long distance transportation node.
  5. Transport the box a long distance to a terminal node.
  6. Take the box from the terminal node and deliver it to the customer’s house

This set of steps is stereotyped and may not be exactly what happens for every order. For instance sometimes there is only one item in the order. Sometimes steps four and five might involve a combination of steps from a shipping depot for a particular carrier (e.g., FedEx or UPS), to their airport operations, then two flights to a final airport, then a truck to their distribution node. The final delivery may be via multiple steps, first to the post office and then the regular postal service, or a special postal service, or perhaps a direct truck delivery from the carrier’s shipping node.

Fulfillment companies have to hire lots of temporary labor at different times of the year to meet demand, and even at the best of times it can be hard to get labor willing to do some of these jobs. So those companies are trying to automate many of the steps, and that often involves robots.

Let’s look at Amazon, for instance.

In steps 1 and 2, above, people have to move around all over enormous warehouses in order to get the items together to fulfill an order. This step is called picking, as the person needs to pick up all the items for the order. One of the first improvements in efficiency was to have a single person be picking for multiple orders at the same time. A program would group together a set of orders so that the person could carry all the items in their cart, but chose the orders so that if a person was picking up items A and B for one order, it would find a second order where perhaps there was an item C that the person would pass by going from item A to item B. When the person got back to the packing station for step 3, it might be that they packed all the items for their orders, or it could be that a different person there would do all the packing, so that pickers and packers were doing specialized jobs.

Most of what a picker did then was to move around from place to place. That is something that robots can be made to do rather easily these days. But the actual picking up of one of maybe hundreds of thousands of different sorts of objects is something that our robots are not at all good at. See, for instance, my quick take on research needed on robot hands.

A start up company in Boston, Kiva Systems, tackled this in a brilliant way. Kiva asked its customers to store all their fulfillment items on standardized shelving modules. The pickers all had fixed workstations to one side of the warehouse. Then small flat-ish robots would go out, drive under a shelving unit, left it up, and bring it to a picker, arriving just as their hands were free ready to pick a new item. A screen would tell the person what to pick, an LED on the relevant shelf would light up, and miraculously the item would be in easy reach at just the right moment. By the time the picker had scanned the item with a bar code reader to confirm they had the right object the next shelving unit would be right there ready for their next pick.

Humans are still way better at picking than robots, but it is easy to automate moving around. The brilliance here was to change what was moving. In the old way the human picker moved from shelf to shelf. In the new way, the shelves moved around to the picker.

This approach turned out to be such an increase in efficiency that Amazon bought the company, and has since expanded it greatly into Amazon Robotics based in Boston. All new Amazon distribution centers are using this technology.

As a side effect, that left other fulfillment companies without the option of using Kiva Systems, so now there are at least five start up robotics companies in Boston attacking the same problem–it will be interesting to see when each of them comes out of stealth mode how they manage to work around the intellectual property covered by Kiva/Amazon patents.

Amazon now runs an annual “pick challenge” at a major robotics conference that moves from continent to continent on a yearly basis. They are encouraging academic researchers to compete on how well their robots (it is really their algorithms–over half the competitors use robots from my company, Rethink Robotics) are able to do the picking task.

Amazon regularly announces large hiring goals for workers in fulfillment centers but they just can not get enough. Only through automation will they be able to meet the challenges.

Amazon, and others, are also looking at step 6 from above. In Amazon’s case they are talking about using drones to deliver products to the home. Just three weeks ago I was present in Palm Springs for what turned out to be Amazon’s first public delivery by drone in North America. The delivery was to Jeff Bezos himself and the box contained many bottles of sunscreen. If only I had realized it was the first such delivery I would have taken a sunscreen bottle when Jeff offered to hand one to me from the box he had just opened!

Other companies are looking at other solutions for that last leg of delivery, and there are regularly press stories about small robots meant to autonomously drive on sidewalks to get things to houses inside lock boxes that only the recipient will be able to open. Other solutions offered involve using the trunk space of on-demand car services, and have the drivers drop the goods at people’s houses between taking paying customers on journeys.

All these step 6 solutions are in their early days, and it remains to be seen whether any of the current proposals really work out, or whether others will be needed.

But in any case there is a clear demand for these current steps, and if researchers can make progress there is plenty of room for robots to work in the actual picking and the packing, besides where current systems are being developed.

(2) Assistance for the elderly. When we look at the ratio of working age people to those sixty five and older we see remarkable changes over half a century or so. In Japan the ratio is going from about nine to one to two to one. In the US it is not quite as extreme, but still extreme. This means that there can not possibly be as large a pool of workers to provide care services for the elderly as there was in the past, and even worse, from that point of view, modern medicine is extending the lives of the elderly so that they are able to survive when they are much older and much frailer.

Well, you say, we should go back to the old way, where the families looked after the elderly. But think what that means for China with the effect of its one child policy. After only two generations of it, a youngish couple will have one child of their own, but four parents and eight grand parents to look after, with no help from siblings or cousins. A modern Chinese couple is raising one child and is solely responsible for 12 older people. Yikes!

In both China and the US, and probably other places there is the additional problem that people move vast distances for work, and so they are not geographically coupled to where their parents are.

I think this means that ultimately most people will have to end up in managed care facilities, but that there will be much smaller pools of human workers to provide services.

Care services for the elderly

Most people resist moving into a managed care facility for as long as possible. This resistance is going to be part of the solution. If care services can come to people’s homes, in whatever form, then people will be able to stay in their own homes much longer.

I think that the elderly want independence and dignity. Those are two very important words, independence and dignity.

At the same time as people get older and more frail they face many challenges. There is danger that they might fall and not be able to get up. They may forget to take their medicines. They may have trouble getting into and out of bed. They may have trouble carrying packages from someplace outside their house that a delivery system delivers it to, into their house. They may have trouble reaching high shelves that they have used all their lives. They may have trouble keeping their house clean, putting things into and out of dish washing machines and clothes washing machines. They may have trouble folding their laundry and putting it away. Ultimately they may have difficulty dressing and undressing themselves. They may have trouble getting enough exercise without the risk of breaking fragile bones. They make have trouble using the bath or shower. They may have trouble using the toilet.

The longer they can get assistance in doing all these things with independence and dignity, the longer they can stay in their own homes.

I think that this is where a lot of robotics technology is going to be applied over the next thirty years, as the baby boomers slide into older and older age.

It is a little too early for actual robots for most of these challenges at this point. We need lots more research in robotics labs. There are a few labs in the US that are looking at these problems, but already in Japan it is a priority and one sees many demonstration robotic systems at big robot conferences in Japan, where research institutes and Universities show off their early ideas on what robots might do to help the elderly with some of these challenges.

I want to stress that this is not research into robot companions for the elderly. Rather it is research into machines that the elderly will be able to use and control, machines that will give them both independence and dignity. I think we all want to stave off the day when we will need a person to wipe our bum, and various sorts of machines can preserve that dignity for longer.

It is early days yet in research on these topics. It is only just now starting to appear on people’s radars that these sorts of robots will be necessary, and that there will be an incredibly large market for them. Today’s research prototypes are too early for commercialization. But the demographic megatrend is going to put a tremendous pull on them. Before too long VCs are going to see a long line of people want to give pitches for funding for various companies to develop such robots. And large established companies in adjacent non-robotic markets are going to be faced with how to transform themselves in home elder care robotics companies.

Driver assist

In my January essay on self driving cars I referenced the levels of autonomy that are usually used to describe just how much self driving is done by the car. Levels 4 and 5 involve no human input at all, the car does indeed drive itself. I am becoming more skeptical by the day that we will have level 4 or level 5 autonomy in our cars any time soon, except in very restricted and special geographies. Level 3 which lets the driver take their hands off the wheel and their attention off of driving is also going to be harder than many people think as the switch back to the human taking over in tricky circumstances is going to be hard to pull off quickly enough when it is needed. Levels 1 and 2 however, where the person is observing what happens and takes over when they need to, will, I think, be fairly commonplace in just a few years.

Levels 1 and 2 for autonomy in cars are going to make them so much more safer, even for bad drivers. And the elderly usually get progressively worse at driving. These levels will take over parking, lane keeping, and eventually almost all of braking, accelerating, and steering, but with the driver’s hands still on the steering wheel.

The technology for all of levels 1 through 5 is really robotics technologies. As we go up through the autonomy levels the cars are progressively becoming robots more and more.

This, for now, is my final arena where the aging population is going to be a pull on new robotic technology. The longer that an elderly person can drive, the longer they can have their independence, the longer they will be able to stay in their own homes, and the longer they can get by without relying on individual services provided to them by someone younger.

Car companies already recognize the need to cater more to an elderly driving population, and many driver assist features that they are introducing will indeed extend the time that many drivers will be able to drive.


The demographic inversion in our population, a megatrend that is happening whether we like it or not, is becoming a significant pull on the need for new automation, IT, and robotics. It is pulling us to more robotic solution in all stages of food production, in manufacturing, in fulfillment and delivery to the home, soon in care services for the elderly, and even now in driver assist features in cars.

^{\big 1}Full disclosure.  For many years I was a member of John Deere’s Global Innovation and Technology Advisory Council.