2015-12-17

Poka Yoke

or Building Boxes for Others to Think Inside Of...

 Alternatively: ending our subheading with a preposition.

Here's the scenario:  You have an important task that you need to get done quickly.  Maybe it's a paper to write, or an analysis of performance data in a spreadsheet to perform, and you find yourself fighting with the tools more than working with them.

User interfaces are hard, and they don't get enough attention.   Examples: the ribbon in Microsoft Office, which is clearly the result of designing a word processor, by committee, for illiterate people (but I digress).

About 8 years ago now, Evil Conglomerate Customer (TM), which had not yet jumped on the Agile bandwagon, and was still paying lip service to 6 sigma, decided to overhaul the interface that its engineers and scientists used to interact with computing resources.

One of the design principles of this interface is Poka Yoke.   Poka Yoka, in a nutshell, is "mistake proofing."  In a manufacturing context, it is designing tools and processes that minimize the opportunities for defects to occur.  (Incidentally, manufacturing quality metrics include the idea of DPMO, or Defects Per Million Opportunities, the lower this number, the higher the quality of the manufacturing process.  One way to lower this number, is to lower the number of opportunities for defects to be introduced in the manufacturing process, which is the ultimate goal of Poka Yoke).

For me the canonical example of Poka Yoke in a manufacturing process is the use of a special lug nut wrench in an automobile assembly line with 5 heads, so all 5 nuts are tightened at once.   This would reduce occurrences of cars leaving the station with only 4 nuts on a wheel.

What the wikipedia article linked above doesn't cover is that Poka Yoke is equally applicable and useful in the design process.

Supercomputers, like the ones I work on (an aside: I don't work on Blue Waters, but Evil Conglomerate Customer (TM) has their own, they have authorized press releases stating that we're a Cray shop, but I am not at liberty to divulge any more than that), are expensive.   They represent a major investment by the controlling organization, and management expects, and is entitled to, a return on that investment.

And the ROI is HUGE. (Now take a second to get the Fuccillo guy out of your head).

Part of maximizing that ROI is reducing the DPMO of the actual processing that the supercomputer performs.   In our case, this is typically metric assload (technical term) of Navier-Stokes equations.   The engineers and scientists understand this stuff, because that' the job that they were hired to do:

Scene: A suburban housewife is preparing bag lunches for her children to bring to school.   After cutting the crusts off of the Wonderbread (tm) and Bologna sandwich, and packing it on top of a frozen juice box, to keep the whole thing cold until lunch she calls up:

"Billy, would you like the intake to the combustion chamber to inject a laminar fuel stream, or should it attempt to premix with...."

And I'm losing my fucking mind.

Anywho, the designers understand the engineering problem that needs solving, and assemble the necessary data, input parameters, geometry, etc to submit to a solver of the N-S equations, and eventually, converge on a "solution."

Supercomputers are complicated!  This is the "getting started" guide for new users of Blue Waters, which since it's public domain, I can use as a relevant example (the software we use is slightly different, and even more complicated than that used on Blue Waters).

So now we have to maximize or ROI, not only on our supercomputing investments, but also our investments in manpower and brain power.  We want our engineers assembling the metaphorical school lunches (why did I even go there?) , not flailing away uselessly with arcane batch submission scripts while our big fancy computer goes under-utilized.

No science gets done, and that make GlaDOS sad.


Now back to Poka Yoke, DPMO and maybe some other buzzwords I forgot before.   Situations where complicated systems interact with experts in fields other than said complicated systems, are rife with opportunities for defects to occur.  In a supercomputing context, these defects manifest themselves primarily in two ways.  First wasted time, and second, underutilization.

Wasted time is a killer, and it happens like this:   I'm a pointy-headed-chrome-dome-egghead-scientist who happens to be an expert in analyzing CAT scan images, and little else.   Like Einstein, I can't comb my hair.  I also leave my bike helmet on all day, because I'll just have to put it back on when I leave anyway (Paramount made a movie about me!).  The point of all of this is that while I'm brilliant in my field, I'm not so good outside of it.

When our users like the above are confronted with the low level complexities of the batch system, or hardware, they inevitably make "rookie mistakes."  Things like attempting to run a job with more processors than were requested (because the number of processors has to appear in 2 places, and it is very easy to only update one).  The end result is a job that spends 8 hours waiting in a queue for execution, dies immediately, wasting 8 hours of time.

A second way this can happen is if the parameters supplied by the user for the job don't match the input data.   The end result is the same.

The way we use Poka Yoke in the design process to reduce these opportunities for defects is abstract away most of the complexity, and place it behind a simple web interface (or CLI, for our hardcore users).  Our users can focus on the data & equations, so we realize a better ROI on their time.

On the web interface we have also defined relationships between parameters, so that we reduce cases of input data not matching it.   For certain solvers, our web form with actually dynamically change based on what parameters are selected, bringing forward important parameters that are implied by previous selections, and hiding ones that are incompatible.

The end result is fewer defects in the jobs submitted, better ROI on the engineers time, and better ROI on the supercomputing investments. That is why (and how) Poka Yoke applies to the design process, and why it is worth considering as in integrate part thereof.

Humblebrag: I wrote the software that does this hardware abstraction and dynamic interface alteration.  I work on "Computers.  Big Ones."  but I'm not the bearded sysadmin from hell who belittles you for not understanding how the systems work.  My job is to enable your job.

And I love it.

k

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.