You can argue that grains are not whole foods, because they can't be eaten raw.
You can argue that grains are whole foods, but only if the bran layer has not been removed.
However, if taking the "skin" off something can make it a non-whole food, then it likewise can be argued that meat (the way Americans eat it) is not a whole food - because we're not eating the whole animal, far too many of us us are only eating the muscle tissue.
You can therefore argue that "meat" is not a whole food unless you're also eating the skin, blood, fat, internal organs, connective tissue, tendon, fascia, bone marrow and at least some of the bone and cartilage.
It's actually not a bad argument. There was a problem (I believe in Colorado) when during a very rough winter, well-meaning people were leaving grocery store meat in the woods for the cougars and other predators, and wild life conservation experts were seeing animals get sick because they weren't getting the nutrients from the skin, bones, internal organs, and connective tissues.
The Inuit (native Eskimo) diet is often held up as proof that humans can live quite well and healthfully on a diet of almost exclusively animal diet (although they do eat berries and other plant foods when they're available), but the traditional diet isn't wasteful. All parts of the animal are used. So the skin, fat, eyeballs, brains.... all gets eaten.
I think when trying to do a "natural" diet, you do have to at least consider what the natural proportions are. We're so removed from a natural world, it's not always easy to determine that.
Still, until only a couple generations ago, people used to say about farm animals such as pigs "everything but the oink" and for beef "everything but the moo."
We don't eat that way any more.
I think sometimes whole foods is sometimes used to argue that you can eat anything "natural" in any portion or proportion. I think we have to remember what portions and proportions would be in a "nautral" environment (it's not always easy).
Honey would be a rare treat, because it wasn't always available, and rarely in large quantities.
Sweet fruit would be rare and seasonal - and there would be a lot of competition for it, so no one would get unlimited quantities of it.
Meat would be lean during most of the year, and you'd have to eat every scrap. You'ld also have to work really hard to get it.
It's really hard to duplicate a natural diets, because we're 15,000 years removed from it. We've bred sugar into our fruits and our vegetables, and the fiber out of them. So there aren't many foods that resemble their wild counterparts.
What we feed our meat animals even makes just the muscle tissue very different from their wild counterparts.
It's no wonder that we struggle with the definitions of "whole" and "natural" (and even "paleo" - because the modern equivalents of paleo foods aren't always that equivalent).

Let that milk sit for a while and you get cream. No processing .. just natural. Honey ... about as unprocessed a sweetener as you can get. Whether or not it's fructose is utterly and completely irrelevant to the fact that you can eat honey straight from the comb w/out any processing. It's a whole food.
I remember eating in a vegetarian restaurant and being criticized for eating faux shrimp because they resembled 'shrimp'. lol Now I would not eat them because they are so processed and full of crap. 