Or should that be octopodes? Anyway, this is so much more interesting than so-called psychic octopuses: an octopus whose mimicry can make it more conspicuous, not less.
The 'mimic' octopus (Thaumoctopus mimicus - now, there's a name that Terry Pratchett would appreciate) is arguably the best colour-changer on the block, & it combines its colour-trickery with changes in apparent shape that allow it to be beautifully camouflaged. This sort of cryptic behaviour (aka 'crypsis') is well-known in cephalopods, which have very flexible bodies & so can take on a whole range of different postures (aka polyphenism) that may result in them looking nothing like an octopus or squid, at all. Huffard et al. (2010) comment that this sort of shape-changing ability "may impair their predators' formation and use of a search image." In other words, the poor old predator can never be quite sure what its potential dinner is going to look like.
What makes T.mimicus unusual is its ability to increase its visibility, which you would expect to be rather counter-productive if done in the presence of predators. It's not entirely alone in this: the dreaded blue-ringed octopus flashes those irisdescent blue rings as a warning, although you can apparently also see them when the little beast is sitting still. Huffard & her colleagues also list Octopus cyanea, which can copy both the shape & the colour of a parrotfish as it swims above the reef. However, they feel that the best example of what they call 'conspicuous defence' is shown by T.mimicus: it's something called 'flatfish swimming', where the animal looks like a flatfish, swims like a flatfish, & is coloured like a flatfish. In other words, not only does it copy flatfish morphology, but their behaviour as well. Even its eyes are placed as you'd see them in a flounder! While the authors identify other ocotopodes that show this behaviour, the mimic octopus is the only one that also uses a conspicuous colouration. The question Huffard et al. set out to answer, is 'why'? How did this counter-intuitive suite of behavioural traits evolve?
The first thing you'd need to demonstrate is that the behaviour you're studying is heritable, & not simply the result of learning. It turns out that other studies have recorded flatfish swimming in a naive, lab-reared octopus that had never had the chance to see a flatfish. What's more, other species of octopus, found in the same habitats as flatfish, don't show this behaviour. So it's fair to assume "that the ability to express this behaviour is a genetically determined presence/absence trait" (Huffard et al. 2010), although, given the intelligence of cephalopods, it's a fair guess that within 'flatfish swimmers' some degree of learning may also come into play.
Another question is - are we looking at an exaptation or an adaptation? 'Exaptation' is a term (first coined, I think, by Stephen Jay Gould) used to describe traits that orignally evolved in other circumstances & with a different function, but have since been co-opted for another role. Feathers, for example, probably evolved via natural selection working on animals with slightly better thermoregulatory ability (they're excellent insulators), but were subsequently co-opted for flight. With adaptations, you'd predict that physical changes & any related behaviour patterns would evolve together.
In the octopus context, Huffard & her colleagues considered several distinct traits related to 'flatfish swimming' in T.mimicus: a change from nocturnal to diurnal behaviour; 'dorsoventrally compressed' swimming - the bit that mimics the shape of a flounder or other flatfish; high-contrast (& hence high visibility) dark-brown & light coloured body patterns, visible both while swimming & at rest; & mimicing the actual behaviour of flatfish (ie the way they undulate just above the seabed. They predicted that the diurnal behaviour came first, ahead of the swim-like-a-flounder behaviour, & examined a number of related octopode species for the presence/absence of these traits. At the same time, genetic analyses allowed them to build up a phylogenetic tree for Thaumoctopus mimicus & its relatives, which they could then map the various behaviour patterns onto.
From the DNA analyses, the team concluded that the mimic octopus's closest relative is Wunderpus photogenicus. (No, I'm not making that up! Isn't it a wonderful name?) W. photogenicus, however, is crepuscular - it's most active in the dim light of dawn & dusk. Along with several other species, the researchers placed these 2 species in a group they called the 'Long-Armed Sand Octopus' clade (or LASO, for short). Mapping various traits onto the family tree that includes LASO, they concluded that both dorsoventrally-compressed swimming and 'flatfish swimming' probably evolved in the most recent common ancestor of the LASO group, along with another hallmark of these species: their rather long arms (for which the group is named).
But what would be the selective advantage of simply looking like a non-toxic flatfish? After all, flounders are generally quite palatable, even if you do have to watch out for the bones. One possibility put forward by Huffard et al. is that some 'gape predators' (fish that effectively 'inhale' their prey when they open their mouths rapidly) can't get their mouths around an adult flounder. They also suggest that swimming with a dorsoventrally flattened body may be a more energetically-efficient manner of moving around if you're an octopus with disproportionately long, skinny arms, although there's no data provided on this one.
On the issue of visually-conspicuous colour patterns, in the LASO clade these seem to be restricted to the mimic octopus and its wonderful photogenic relative (sorry, I mean W.photogenicus; I just couldn't resist!). While other species use some startling colour patterns in defense (basically bluffing that they're nastier or more toxic than is really the case), only mimicus & photogenicus have the same colour pattern when they're resting. The team suggest a couple of possibilities here: the colours may act as disruptive colouration, a form of camouflage that makes it harder to see the actual body outline of the animal. Or they may have been selected for because wavy stripey arms somewhat resemble highly toxic sea snakes.
While these may read like 'just-so stories', the team do offer ways of testing them. They suggest examining the highly visible pale/white component of the animal's colouration to see how well it matches the seabed of their normal habitat, which is black sand with white shell fragments on it. It should also be possible to test mimicus for toxic or unpalatable chemicals - it really could be nasty to eat, in which case the colour patterns take on a whole new role, that of warning predators to avoid it altogether.
Now, none of this mimicry is perfect - after all, to a human observer the 'lionfish' copycat isn't all that close, and even the 'swim-like-a-fish' appearance and behaviour is flawed. But to expect perfection would be to fall into the trap of thinking that a perfect result is the normal outcome of natural selection. It isn't. The existing behaviours might be just enough to give a hungry predator pause as it eyes up its dinner, and '[these] decisions may cause enough confusion to allow 'mimic' octopuses to escape predation' (Huffard et al. 2010) & live to copy flounders another day.
CL Huffard, N Saarman, H Hamilton & WB Simison (2010). The evolution of conspicuous facultative mimicry in octopuses: an example of secondary adaptation? Biological Journal of the Linnaean Society, 101, 68-77