From spongy moth caterpillars to torrential rains, this garden season has been plagued by one challenge after another.