I've just finished reading Your inner fish (Shubin, 2008) - honestly, I can't recommend it highly enough. But for anyone who hasn't bought the book yet, let's look at what another part of our anatomy - our ears - has to tell us about our evolutionary past.
Mammals are unusual in the ear department: we have a fleshy external ear, for a start, & that's something you won't see in any other animal group. But we differ internally as well. Where we have those three tiny middle-ear bones (hammer, anvil & stirrup, or - to give them their scientific names, malleus, incus, & stapes), reptiles, birds & amphibians have only one. So where do the mammalian ear bones come from?
Shubin points out that this question was originally answered way back in 1837, when a German anatomist (Karl Reichert) was studying embryos of mammals & reptiles. In both groups, the early embryos have structures called gill arches, & Reichert was looking at where these structures ended up later in development. Much to his surprise, he found that the bits that became bones in the reptile jaw - became 2 of the ear bones in mammals. (The third bone, the stapes, is found in both groups.) This was borne out by later studies - & implied that it should be possible to follow this change in mammals.
Well, in some fossil sites there are an abundance of fossils that have a mix of characteristics: in some ways they're reptile-like, but other features, particularly their teeth, look mammalian. And there's a good sequence of these 'mammal-like reptiles' - some are very reptile-like indeed while others are hard to tell from 'true' mammals. The most reptilian have just the one middle-ear bone, & their lower jaw is made up of quite a collection of separate bones. As you move onto looking at more- & more-mammal-like fossils, two of the bones originally associated with the jaw joint become smaller & smaller & their position alters, until they end up in the middle ear as the malleus & incus. At the same time the number of bones in the lower jaw decreases, until you end up with just one, the dentary.
The third bone - the stapes - has an even older origin: it began life as the hyomandibula bone in fish. This is a bone that links the lower jaw of a fish to its skull. Again, it's possible to trace this transition - and what's more, Shubin notes that the relationship is confirmed by the nerves that supply the hyomandibula in a snapper or shark, and the stapes in mammals (the facial nerve in both cases).
So you really do have an inner fish.
N. Shubin (2008) Your inner fish: a journey into the 3.5-billion-year history of the human body. Pantheon.