So touch is a crucial part of how we relate to the world around us. It affected the progression of drawing through history and it visible in the way people first learn to draw. Color, too, is categorized by people in a similar way as shapes are, but they can't be lumped together. You can't really feel color, you can only see it, and even though Harold Speed lumps it into the same discussion on touch (Chapter 3, Oil Techniques and Materials), I don't see how it applies. It has more to do with how the brain works and the differences between the two hemispheres.
So when an inexperienced person is asked to paint a red apple they'll reach for the red without really looking at the colors in front of them even though the actual color of objects fluctuates enormously depending on the lighting. They'll paint the shadows as a darker red, when in fact there's tons of other color in them. I mean sure, the apple's still red and going to be painted with red but there are many other colors involved, especially in the shadows.
It's due to the left brain categorizing things as they appear in "standard lighting" (kind of vague, you know what I mean). So we know that such and such object is this color and we remember it that way, along with that shape and texture we learned from our sense of touch. It's pretty much a system to put a name to all the information hitting our senses. It's completely necessary for language, which is centered in the left hemisphere.
The right hemisphere processes and understands visuals in a more pattern-based way and doesn't categorize like the left hemisphere. Learning to see (read: draw/paint) is a lot about learning to use that side of the brain. Sure, you won't have a name for that color, but you'll know what it is.