12:20 p.m. on October 8, 2008 (EDT)
The short answer is that everyone's feet are different.
Slightly longer - insoles are intended to more closely customize the fit of the footwear and control the motion of the foot inside the footwear to avoid problems like blisters, bunions, and other problems developed by improper fit and wrong socks.
Still longer, with the background information -
Shoes and boots are made to a set of standardized foot shapes, called "lasts". The set of lasts used by each manufacturer are different from one another, which is why the first set of steps in buying shoes and boots is to find a brand, model, size, width, etc that fits as well as possible.
Insoles are a method of refining the fit and personal comfort of the footwear. Some people have high arches, some have flat feet, and some fall somewhere in between. Some people have narrow heels compared to the rest of their foot, some have wide heels. Some people find soft insoles, others prefer firmer support. Some people when walking/running/hiking pronate (most), some suppinate.
Manufacturers of quality boots, running shoes, and other specialty shoes sell their footwear with only a "placeholder" insole, with the expectation that people using such footwear will get specialized insoles, maybe even custom insoles. If you look inside most quality boots, you will see that the inside is "unfinished", with the intention that you will put a customized insole in. And insoles wear out at about the same rate as the soles, so you will be replacing the insoles at the same time you re-sole the boot.
So the second major part of properly fitting boots or running shoes (or other special-activity footwear) is getting insoles that more closely customize the fit to your feet.
Another major part is a proper set of socks for the activity. For most of the uses of people who come to Trailspace, this means a thin wicking liner sock and a thick insulating outer sock. But it might include VBL socks, socks with a specific set of padding (thicker sections of the sock along the top, under the ball, under the heel, etc). Or it might be a double layer sock (like Wright makes). Or it might be a single very thin sock.
You could, of course, get custom made boots and shoes to fit your feet. Or you could go barefoot. Most people do not do so. Even custom boots, made by quality cobblers, are often intended to have a removeable custom insole (maybe even a specialized orthotic).
It used to be that it was necessary to break boots and specialty shoes in. However, many boots and specialty shoes are made with synthetic materials that do not change shape to conform to your foot with wearing. So you must be sure to get as good a fit as possible to start with, then fine tune the fit with insoles and other inserts (insoles are not the only adjustment item).
Yes, you can get away with Crocs, sandals, barefoot, or even hike in worn-out dress shoes. Dress shoes, by the way, are not used for long walks or hikes by most people, so people generally do not notice the difference (until they develop foot problems).
If you take the insoles out after hiking, they will dry out much more quickly. And you really should rotate your everyday shoes to give them a chance to dry and rebound.
Your feet are complex and unique. Getting a proper fit for that through-hike of the AT or PCT is essential if you do not want tired feet at the end of each day or blisters or other foot problems.