When you create or modify an album, during the photo selection process, click on the small pencil to the right of the image and modify your caption.
You can then replace or change the caption. The photo must be selected (thanks to the orange frame) for the button to appear. If the photo is not selected, you will not be able to change the caption.