There are some things that can't be fixed by any amount of adjustment. No amount of pulling the buttons tighter will fix this. After looking at this closer, I think the problem is that you are gluing the sew foam and vinyl panel flat. I have had problems like this before with motorcycle seats. You need to use your seat as a fixture, lightly glue your sew foam to the seat foam so you can remove it easily later, and then glue the vinyl to the concave sew foam. The vinyl needs to be glued firmly to the sew foam by gluing both the sew foam and the vinyl. Remove the panel, sew around the perimeter, and then sew the seat cover. This should leave you with a concave panel. Then when you sew it all together attach it to the seat frame and put the buttons in it won't go crazy on you.
No one lives forever, the trick is creating something that will.