5 minute REST API with Docker

Docker is a wonderful technology to manage environment consistency. It delivers on the promise of Vagrant without having to use Vagrant. All the things that you thought you’d be able to do with virtual machines are easy with Docker.

In this post I wanted to see how long it would take me to start a REST API using a Dockerfile. It actually took more than 5 minutes, but if you knew what you were doing it would take even less.

Firstly, a warning: this won’t work on Windows. Docker on Windows is just not worth trying.
Continue reading

Kanye West: Businessman

As of March 28 it still isn’t for sale. Kanye’s newest album, The Life Of Pablo, was “released” on February 14, but not for sale – it could only be heard through the streaming platform Tidal.

You can download an illegal copy, of course, but you won’t get the most recent version. That’s because it is a “living breathing changing creative expression“.

Take away the music and admire the innovative marketing effort undertaken here. This is an updated business model in which you pay for access to the latest version of an artist’s album. Apparently the album will never be released on CD, but remains to be seen if it will come out on rival platforms for purchase.

For other artists, Kanye is demonstrating a platform which encourages creative turmoil and evolution; compare to other platforms with their onerous approval processes. This creative control is something that artists have never been exposed to – new opportunities for new ideas.

This takes many of the things users love about YouTube – the interaction, the iteration, bite sized updates – and integrates them with the celebrity and self-promotion of mainstream pop.


Debugging Spark logs with YARN

Debugging Spark can be quite painful. The error that you see in your application, or even in your job, might not reflect the underlying problem. Here’s how I dig into the executor logs to find what (might) be really happening.

From the YARN job list, find your job. If it’s no longer running you’ll need to go to the History page.


From here you can navigate to the driver log. There are two here, because the job was retried.


The driver log records the Spark startup, and will log some of the information that the executors provide. Unfortunately if they crash or become unresponsive they often won’t report their status – they’ll just stop.


The driver log will tell you which executor exited, which is a nice way of saying that it failed. You will have to trawl the driver log a little to find these; there will probably be multiple per job. You now know the name of the container that failed.


Now search the driver log for the launch details for the problem container (just do a CTRL+C, CTRL+F in Chrome). Each executor has a “YARN executor launch context” block like the below.


If you browse to the SPARK_LOG_URL_STDERR path you will see the detailed log for that container / executor, wherein you will (hopefully) see a more detailed description of what went wrong. At least you should be able to identify the root ‘Caused by’ exception, and Google from there.

American health care – the “Medical-Industrial Complex”

When I first moved to the USA I had the strong impression that the health care system was designed to deliver quality care – the outcomes were good. It’s well acknowledged the the US Healthcare system is the most expensive in the world with annual costs of almost $10k per person, but delivers below OECD average life expectancy.


My assumption was that this is mostly due to economic inequality. I’m still not clear on that, but I have some insight into the quality and cost of care.

The *attentiveness* of medical workers is pretty good. You can easily get an appointment, in short time, provided you have insurance. You’ll meet some medical aide who takes blood pressure and pulse updating your records before you are able to speak to a ‘doctor’. Everything is wonderfully clean and sterile, the chairs are comfortable, the gowns & outfits are very medically appropriate.

My doctor has given me what I would consider to be some extreme advice. The most recent was a mild fever or allergy – I went to the doctor with complaints of headache, body ache (back, shoulders), lethargy and weakness. I requested some blood work to ensure nothing ‘serious’ was happening, and for general advice – rest or medicate? After all, this is ‘free’ due to my insurance.

“Any other symptoms?”

Well, my stomach was cramping and aching this morning, so I skipped breakfast to allow it to rest too.

Cue some stomach prodding, my discomfort, and a declaration that I need to get a CT Scan.

Wait, what? A CT scan? Yes, I might have appendicitis. Loss of appetite and stomach pain. The stomach pain isn’t in the right place, or severe enough, and I consciously chose to miss breakfast rather than lose my appetite, and there are a bunch of other signs that something else is going on in my body, but the best course of action was to get a CT scan.

The median cost of a abdominal CT scan in Chicago is $1000.

I suggested that instead I monitor my pain levels, and simply proceed with the scan if the pain increases. The doctor begrudgingly agreed. The ran blood tests, and I’m sure they made a tidy sum from the insurance company.

So in this system of detached health insurance, who is monitoring costs? The doctor doesn’t care about spending frivolously. The insurance company gathers their health premium from the subscriber; do they question the excessive expenses that appear in their system? And of course the customer aka ‘patient’ blindly follows their doctors’ sage advice, thinking that they need such procedures.

Health care costs are something like 18% of the US economy.



Retraining after automation

Andrew Ng, professor at Stanford and founder of Coursera (one the first and most popular online schools, where Andrew’s Machine Learning course is extremely popular), recently discussed the difficulty of retraining people as their positions are transformed or made redundant through technology.

Historically technology has created challenges for labor.

The U.S. took 200 years to get from 98% to 2% farming employment. Over that span of 200 years we could retrain the descendants of farmers.

With this technology today, that transformation might happen much faster.

What our educational system has never done is train many people who are alive today. Things like Coursera are our best shot, but I don’t think they’re sufficient. People in the government and academia should have serious discussions about this.

Even if we could retrain people, where are the ‘new jobs’ going to come from?



International competition after automation

Today we figure that production of goods is a highly competitive business, especially in cost of production. Most people are familiar with the offshoring / outsourcing trend over the past 40+ years. Many people are also aware of the progression of low cost providers: the initial outsource provider was Japan; after that Hong Kong, Taiwan, Korea, China. As these countries themselves develop stronger economies, their wages increase and they in turn outsource to neighboring (& cheaper) providers.

Note that the biggest impact in cost of production in these countries is labor, both directly and indirectly. But what happens under an automated and robotic workforce, where labor is a one-off cost rather than an ongoing expense?

One possibility is hyper-localization. Local factories (really just resource storage facilities) where robots can process or create a large number of products. Given unlimited expertise at a fixed, one-off cost – why can’t that happen in your city, or even in your neighborhood?

The increasing automation of work

It wasn’t that long ago – within my lifetime – that most work at factories and on assembly lines was done by humans. I’ve seen video from the 80s of plastics manufacturers and car factories; hundreds of people working with their hands. Compare with a modern cutting edge factory, where large robots swing arms through the air, working all at once as items pass through conveyors belts. Machines & robots turned out to be excellent workers, especially at repetitive, high-throughput tasks.

With today’s accelerating technology, especially given the huge investments in AI due to the internet and smartphone revolution, we’re on the cusp of many more jobs being replaced by robots. The most high profile of these is driving.

According to NPR’s “Planet Money”- The Most Common Job in Every State, truck driving is the most common job in more than half of the United States. This includes freight, delivery, couriers, forklift, and other drivers carrying goods. This covers millions of jobs.

Driving a truck is much more complex than tightening a screw, but as computers advance so too will the robots on the assembly line. A self driving truck is not a regular truck with a robot at the steering wheel, but is more like a computerized truck packed with sensors, GPS, and a powerful computer which can safely manoeuver traffic and road conditions.

Mercedes’ recently accounced their vision for a self driving truck which should be on the roads in 2015. Even more urgent is likely to be the self driving car, which is most likely to be developed by Google or Uber as a taxi-like service.

Driving is just one example of jobs that are likely to become irrelevant due to the progress of technology. What else? Customer service: using a kiosk, talking to a computer, or using an online service instead of interacting with a person. Brokers & agents for insurance, travel, real estate, cars, tax: replaced by websites allowing you to easily search for something cheaper or better value. I’m sure there are many more.

What can you do better than a robot?

RobotWorkerI often talk to people about the future, about automation and the capabilities of robots in today’s and tomorrow’s world. Predictions about the future can overestimate (flying cars) or underestimate (video phones) the eventual reality. I don’t doubt that we *could* have flying cars if it was something we wanted enough – more research, profit incentive, and consumer demand would undoubtedly lead to some flying car of some sort. But technology is driven by what people are prepared to pay for – mostly new smartphones, tablets, and voice controlled doohickeys.

I’m going to assume that intelligent robots are going to happen. Robots have been mainstays in factories for many years, with increasing automation. We see some very famous robots in action – the Honda robot, ASIMO, is one example. And those robots that play soccer. YouTube is full of videos of research labs with robots that can run, catch, play pingpong.

In the future, if you want to be employable, make sure you choose a career in which you’ll be more valuable than a robot.