When changing from A to D, the first finger remains in position on the 2nd fret of the 3rd string, and acts as a pivot around which the other fingers move. This is an important technique, making many chord changes easier and smoother.
Timing is of the utmost importance, so keep a slow, steady count. Smooth chord changes are far more desirable than speed at this stage of your development. Speed will come with time and practice (be patient).