The Kaigos restrict themselves to a very certain part of this cave system for the very reason that it is so vast and large. Wanderers from the surface would find the ever-present paranoia to inevitably drive people to madness. Sure enough there are dangers in the dark chasms, but the worst danger is oneself. Many nomads have been driven insane and ended up killing companions out of fear, or ran off to hide in the environment, only to be eaten or never seen again.
Even the Kaigos are unsure of the influence the caves have on people's minds, but they report no such incident happening with their own people. It may be due to rampant spirits lurking about, or a corrupted flow of energy... perhaps spores released into the air from the cave fungus. No one knows. The only sure thing is that if you are to venture into these caves for whatever preposterous reason... keep a strong hold of your mind, and keep a weapon at hand.