There are two main aspergillus infections that directly involve allergy. One is ABPA and the other is allergic rhinitis. In both cases the patient has an allergic reaction against the infective material - this is completely different from an inflammation of infected tissue, which is the more usual case. The fungus does not invade the tissue but simply triggers the allergic response which can become chronic.

Breathing in spores from the air can cause more problems for these patients as they are already primed to react to the fungus so they should avoid situation where they breathe in too many spores e.g. damp houses, gardening, composting etc.

The medical charity Allergy UK explain what an allergy is very well so I will quote directly from their website:

What is Allergy?

The term allergy is used to describe a response, within the body, to a substance, which is not necessarily harmful in itself, but results in an immune response and a reaction that causes symptoms and disease in a predisposed person, which in turn can cause inconvenience, or a great deal of misery.

An allergy is everything from a runny nose, itchy eyes and palate to skin rash. It aggravates the sense of smell, sight, tastes and touch causing irritation, extreme disability and sometimes fatality. It occurs when the body's immune system overreacts to normally harmless substances.

Allergy is widespread and affects approximately one in four of the population in the UK at some time in their lives. Each year the numbers are increasing by 5% with as many as half of all those affected being children.

What causes an Allergy ?

Allergic reactions are caused by substances in the environment known as allergens. Almost anything can be an allergen for someone. Allergens contain protein, which is often regarded as a constituent of the food we we eat.
In fact it is an organic compound, containing hydrogen, oxygen and nitrogen, which form an important part of living organisms.

The most common allergens are:
pollen from trees and grasses, house dust mite, moulds, pets such as cats and dogs, insects like wasps and bees, industrial and household chemicals, medicines, and foods such as milk and eggs.

Less common allergens include nuts, fruit and latex.

There are some non-protein allergens which include drugs such as penicillin. For these to cause an allergic response they need to be bound to a protein once they are in the body.

An allergic person's immune system believes allergens to be damaging and so produces a special type of antibody (IgE) to attack the invading material. This leads other blood cells to release further chemicals (including histamine) which together cause the symptoms of an allergic reaction.

The most common symptoms are:
sneezing , runny nose, itchy eyes and ears, severe wheezing, coughing, shortness of breath, sinus problems, a sore palate and nettle-like rash.

It should be understood that all the symptoms mentioned can be caused by factors other than allergy. Indeed some of the conditions are diseases in themselves.

When asthma, eczema, headaches, lethargy, loss of concentration and sensitivity to everyday foods such as cheese, fish and fruit are taken into account the full scale of allergy be appreciated.

The Allergy UK website goes on to further explain what an intolerance is, what a multiple chemical sensitivity (MCS) is and how these are all diagnosed and treated.