Basically:
Epoxy when used with the same fibreglass as polyester. Is stronger, less prone to cracking and does not shrink over time.
Epoxy generally takes longer to cure, has a set mix ratio and can cure in a vacuum. Poly cure time is dependant on amount of catalyst used and usually comes mixed with styrene which evaporates during curing.
Polyester is smelly and the fumes are bad. Also the catalyst is nasty if not treated with care.
Epoxy is not so smelly and fumes are not a big issue. Repeated exposure to skin can cause an allergic reaction.
Epoxy is expensive, Polyester is half the price.
Poly is fine epoxy is usually better. Depends of the application.
There is also vinyl ester which is a mixture of both.