Increased environment sound while speed decreases

When you are cruising at 340kts, and you set the speed to, say 250kts, there is a rush of sound until the speed reaches 250kts.  I think this is backwards.