In the germ world, fungi usually lack the flair of viruses or bacteria. To people with normal, healthy immune systems, a fungus will rarely show itself — even though you carry around a microscopic film of fungus on your hair and skin, and take in invisible clouds of fungal spores with each breath. While many other microbes prefer to make a living through disease and death, a fungus is often content to wait for its host to die of something else.This article by Laura Beil nicely explains where moulds come from and why we don't get infected by them as a general rule.
In fact, throughout the history of civilization fungi have mostly been humans’ friends, providing the bounty of bread and beer, recycling trash and enabling plants to extract nutrients from the soil. Scientists estimate that roughly 1.5 million species of fungus inhabit the Earth, but only a handful are capable of causing human disease.
Problem is, when they do, fungi can be remarkably lethal
It goes on to suggest why some people do get infected and what genetic differences they may have compared with people who don't get infected, and finally discusses why they are so difficult to treat!
It is well worth a read.