According to the datasheet the AD620 is not a rail-to-rail amplifier:

I think the above means that the output can go about 1-1.5 V near either Vs- or Vs+ and no closer.

What will I have to do to amplify a sine wave and get a nice 5 Vpp output? Can I just use a higher Vs? I will try using my ~8.2 V Lilion batteries.

In the above circuit the grey is the virtual ground between -4.1 V and +4.1 v (two halves of the 8.2 V Lilon). When I test using my waveform generator its ground will be connected to the virtual ground in the circuit. Also my oscilloscope's ground will be connected to it, too.

Experiment success! I was able to amplify a 1 Vpp signal to 5 Vpp.

Side notes:

• The potentiometer on the battery was useless. I thought that maybe I could use it to easily shift the level of the output but it had zero effect.
• The potentiometer did allow me to shift the maximum and minimum of the output level, or in other words to shift the rail level of the inamp. Shifting it too far in one direction caused flats in the peaks and valleys of the waveform.
• The potentiometer as gain resistor worked perfectly. It let me change the gain on the fly.