What did you use to glue the fabric down with? You need to do this in stages, not all at once. Basically, you need to start at the large and small speaker openings. Attach the fabric to the pod 1" completely around the perimeter of the large opening and up to the small speaker opening with High Heat Resistant contact adhesive. Don't cut out the openings until you are done. Once that is solid and the glue is dry, you can pull the rest of the fabric out and away from that point. The low spots you will have to glue to hold the fabric in place, but the rest doesn't have to be glued on the front, just on the back. If you can staple into the back side, that would be easier than gluing.
No one lives forever, the trick is creating something that will.