Adjust the rear brakes until they drag slightly and see if that changes the pedal. Depending on the vehicle, the parking brake adjustment will effect the pedal feel. Most modern cars w/drum brakes are self adjusting. If they are functioning as they should, repeated stops made while BACKING UP will adjust the rear brake shoes correctly. Otherwise if the adjusters are not working correctly, you'll need to use a brake spoon to physically adjust the shoes to the drums.
If there's excessive wear, the drums may need turned or replaced along w/new shoes.