Tuesday 29 November 2011

Exploded and collapsed layers

Now a range of regions can be "exploded" into new layers in the stack, and stacked layers can be collapsed again into sets of regions. First random test image of this behaviour behaved correctly, but is somewhat disturbing.

Wittgenstein vindicated - types are not a hierarchy

In the days when object-orientation was new to most programmers, we were occasionally surprised by apparently profound failings of metaphysical conception among our colleagues. In one case that was notorious among members of our team for a long time, an early draft of a programming tutorial described inheritance in terms of the parts of a car - the class "car" should inherit the class "wheel", the tutorial explained.

Wittgenstein might have observed a problem in the metaphysics of object-oriented programming, rather than an ontological disconnect in the life of that author (and of every person who had read that draft before I saw it). Is it really the case that we always know what category is a kind of what other category?

Well, I spent most of today resolving a similar conceptual reversal. Until today, I had assumed that "content" was a kind of "layer". Now I've decided that layer is a kind of content. Of course the real problem is that neither is fully a kind of the other. Most development projects involve epistemological compromises, and expert users of the software we create (if the category terms are revealed) manage to subtly change their previous understanding of those words, in that context, to anticipate the actual system behaviour. Unfortunately, when programming in Java, reversing the position of two classes in the inheritance hierarchy is far from a trivial exercise.

Saturday 26 November 2011

Standardising geometry

I resisted the temptation, when I started coding, to create my own basic geometry classes - the Java libraries are full of dependencies on basic classes like Point and Rectangle. But they seem to require far more mundane repetition than you might hope from an object oriented language (C++ operator overloading and auto-type casting, where are you when I need you).

I didn't mind at first, because it was kind of meditative, doing all those repeated operations of x + width, y + height and so on. But I finally cracked, and created some new Point and Rectangle classes (well, Location and BoundingBox) that do everyday stuff more conveniently. Of course, now that I have 10,000 lines of code, much of it doing geometry, I wished I'd started a lot earlier. And quite a few things didn't work afterward.

Never mind. Almost back to where I started, and my ink has bounding boxes again.

Friday 25 November 2011

Defining selection regions

I wanted to be able to define a subset of regions within a layer as an interactively-edited selection. This turned out to provide a nice example of the tension between formalised algorithms and "intuitive" user intention.

The basic idea was that the user should be able to draw a line around the items of interest, or that a previously defined image layer could act as a region selection mask. In either case, an enclosed region is clearly intent to select something.

The enclosure could be defined computationally using either a flood fill, or a convex hull. However, neither of those corresponds very well to informal graphical conventions. A partially closed boundary still indicates containment, but couldn't be flood-filled, while a concavity should certainly be respected, if specifically drawn to exclude something.

As a result, I spent nearly a day inventing a more informal alternative to the well-known formal algorithms. It seems to work as expected in most cases, and while less efficient than a good flood fill, is faster than a naive convex hull. I doubt there is much future to the field of informal algorithm design, but hopefully this will be good enough to suffice for the rest of my project.

Wednesday 23 November 2011

Adding a play button

I'm very impressed by the new Lua-based end-user game programming app for the iPad, Codify. Their approach to the need to distinguish between code manipulation and game interaction is to use a simple play button. That's what we expect with macro recorders, of course, though many environments complicate that metaphor. In Scratch, for example, you can still interact with the code while the program is executing, and it can control its own execution state through use of the green flag.

So after a final Cambridge drink with Sam Aaron and David Coyle last night, where Sam pressed me on exactly what kind of execution this language does, I thought I should add my own play button today. Fortunately, it worked as expected pretty much straight away, and created a whole bunch of new layers on the fly.

Tuesday 22 November 2011

Some usable parameterised layers

So it's taken a few days, but now I have operations with value parameters, and a usable subdevice that can be used for viewing and modifying the order of the layer stack.

The result can be used to create basic scripted interactions, with some interesting behaviours resulting from layer dependencies (this image).

From here, the next step is a choice between two optons: a) adding a greater variety of parameter and operation types (e.g. parameterising the image thresholding, or adding rotate and scale operations to the current vector translate), or b) extending the computation model (e.g. processing layer regions as sets - they can already include references to other layers, or interactive commands, so this would extend the power of the language pretty substantially).

Wednesday 16 November 2011

Pain and sub-devices (drag and drop)

I've never seen an API for drag-and-drop that I liked. The Java 2 version is truly horrendous. Unfortunately, if you create notational sub-devices and want the window manager to help with them (as per a recent blog entry - it was nice to have the layer stack sub-device in a separate pane, and perhaps make it scrollable in future), then you find yourself in the horrible world of wanting to coordinate graphical behaviour between different window system components.

Hence a morning spent (wasted?) trying to get to grips with the Java DnD API. I really don't recommend it. After giving up on that, I spent an afternoon trying to roll my own. Guess what - if you press the mouse in one component, then drag into another, the new component does not receive any events (not drag events, not motion events). You can tell it has entered, but then no more information. The really crude response (as implemented by yours truly) is to collect the events in the drag source, and pipe them over to the drop target. The astute reader will have noticed that this means recreating the complexity of the DnD architecture that I was trying to avoid in the first place (well not quite, because I have avoided data transfer between applications). Even if I had got Java DnD working, apparently it doesn't work under Windows anyway.

So as of the end of the day, I can finally drag items off the layer stack and onto another layer. Here's a picture of an interim work product, sometime during today (drop target rendering not quite sorted!).

A reflective note - one of the rants that I give my HCI lecture class is that most GUI applications do not really implement direct manipulation. This is a classic example of why they don't - it's just so difficult to for application developers to do it properly. It's in the interests of the API vendors to make every GUI function as much like a disguised command line as possible - because that is easier for them to implement.

Tuesday 15 November 2011

Making mundane sub-devices

As promised in the last entry, I had to make a sub-device to improve the usability of layer manipulation in many significant ways. It's interesting that this threw me back into the world of the conventional GUI, as I had to start making things like split-panes, and then adding buttons to initiate some of those useful functions that had been missing. Unfortunately, early attempts are both boring and annoying (as Luke tells me is true of all programming).

(yes, a boring sub-device)

And this was followed by some tasks that were at least a bit more layer-like, but with more of those tricky rendering decisions ... where a visual cue is needed to indicate the metaphor, but none of the options are quite as attractive as one might have hoped. For example, choosing between these two options of slightly tilted cards to make the ordering of the layer stack clear:

Sunday 13 November 2011

Reaching a usability nadir

With some parameterised operations, a few literals, and layers that return values, it was finally time to make something that resembles a program. The result was unusable to a remarkable degree. Friday ended in 20 minutes struggle with an environment offering absurdly high premature commitment, viscosity (layers can't yet be deleted or re-ordered), and poor visibility. The last is perhaps the only interesting one. All layers are visible, but it's not always easy to tell what order they are in. As Thomas observed early on (in comparison to Photoshop) I need a sub-device.

Thursday 10 November 2011

Collaging parameters

Some nice juxtapositions today. After a couple of days trying to get literal values working properly, I finally sorted out my rusty statistics and trigonometry enough to derive magnitudes and orientations from images. That meant that I could build a first function that is parameterised by those. More to come tomorrow.

But meanwhile, a spurious picture generated at some point during this morning's experimentation with those vectors (they may not have been right at this point - mistakenly using a figure in degrees where radians are expected takes a lot longer to diagnose than the opposite).

Tuesday 8 November 2011

Don't forget the secondary notation

We're always having to remind visual language designers that they need some secondary notation. (Otherwise known as "comments", for any non-CDs readers stumbling across this). You might say that, up until this point, almost everything has been secondary notation because the execution semantics of my representation has been vague-to-non-existent.

However, in a remarkable display of self-discipline, I have created a secondary notation mechanism before starting on the execution semantics. Here's what it looks like at this stage:

Monday 7 November 2011

Getting thresholding right

Eric Saund told me ages ago that good thresholding is the key to working with captured sketch images. So today has been spent putting together a (hopefully) moderately competent adaptive thresholding algorithm for background removal. The basic approach is inspired by the technique that Pierre Wellner created for the Digital Desk, and that has been used for years in the Rainbow Group and elsewhere since then. However, I've added some enhancements based on the histogram method that Alistair Stead used in his adaptive blob detector last year.

Here's a lovely image, created from a bunch of images that I happened to have lying around my desktop as I was testing auto-detection and alpha blending of white and black backgrounds.

On the purely mechanical side, a couple of hours wasted trying to get access to the built-in MacBook camera from Java so that I could wave arbitrary bits of paper in front of the screen. Unfortunately, this seems to get you implicated in some kind of religious war between Quicktime and Java Media Framework, such that nobody wants to tell you how to do it.

Friday 4 November 2011

Working with Beryl - is it sketching?

I've spent several days in a fog of nasty bugs, as it becomes increasingly clear that my model for managing selection (as discussed in week one) was wrong. I've implemented an alternative, but it doesn't work yet.

On to more cheerful matters - Beryl has proposed a mini-workshop for my first week in Auckland, since Gem Stapleton is visiting that week too. Beryl summarised the relevant research interests of those involved, and said that I was working on "sketching for visual programming" (with a query whether that was correct).

So that was a nice point for reflection. It's certainly true that I've asked to work with Beryl because the work her group does on sketching will be a valuable input to the layer language. As to whether my stuff is sketching, I don't think anything I've done yet looks like any previous piece of work in the sketching community. But on the other hand, it definitely builds on the ideas of mixed-formality representations described in the paper that Luke, Beryl and I wrote with Dave Gray.

Tuesday 1 November 2011

Premature optimisation ...

... is the root of all evil, according to either Dijkstra, Knuth or Hoare. Whoever it was, I figure that this is good company to be in. (Assuming that the original statement was a product of personal experience).

Unfortunately, premature optimisation is not an easy habit to give up. You might as well say that unnecessarily general class hierarchies are the root of all evil (another of my recent problems).

In a combination of both problems, I spent most of yesterday reversing the rendering order of my basic layer abstraction. The first implementation had rendered each layer in turn, followed by those above it. There is no point in recalculating a layer that hasn't changed, so unchanged regions from layers above are preserved in the local cache. Of course (?), this is completely the wrong way around. It is the layer on top that should be the main priority for rendering, because this is where the user is currently looking/interacting. It is the layers underneath that should be cached, because they are unlikely to change.

So failure to take a user-centric view in the original architecture meant that I had to completely reverse the caching order (not yet finished, in fact). If I hadn't spent so much time implementing the caching in the first place, reversing the rendering stack would have been a one hour job instead of a two day one. A clear case of premature something or other.