In buffer when hydrogen ion is added, it will neutralized by the base in buffer.  Hydroxide ion will be neutralized by the acid.  On the overall pH of the buffer solution, these neutralization reactions will not show much effect.
While when you select an acid as a buffer solution, try to use an acid acid that has a pH closed to your desired pH.  This will help your buffer to achieve nearly equivalent amount of acid and conjugate base, so that it will enable to neutralize as much as H+ and OH -.   _{Your Comment}