This is pretty straight forward, how big are the batteries? Lead acid batteries don't get much smaller than C-cell batteries. Coin cells don't get much larger than a quarter. There are also standard sizes, such as AA and 9V which may be desirable.
Weight and power density
This is a performance issue: higher quality (and more expensive) batteries will have a higher power density. If weight is an important part of your project, you will want to go with a lighter, high-density battery. Often this is expressed in Watts-hours per Kilogram.
Price is pretty much proportional to power-density (you pay more for higher density) and proportional to power capacity (you pay more for more capacity). The more power you want in a smaller, lighter package the more you will have to pay.
The voltage of a battery cell is determined by the chemistry used inside. For example, all Alkaline cells are 1.5V, all lead-acid's are 2V, and lithiums are 3V. Batteries can be made of multiple cells, so for example, you'll rarely see a 2V lead-acid battery. Usually they are connected together inside to make a 6V, 12V or 24V battery. Likewise, most electronics use multiple alkalines to generate the voltage they need to run.
Don't forget that voltage is a 'nominal' measurement, a "1.5V" AA battery actually starts out at 1.6V and then quickly drops down to 1.5 and then slowly drifts down to 1.0V at which point the battery is considered 'dead'.
Some batteries are rechargable, usually they can be recharged 100's of times.
There are a few different ways to 'measure' batteries, here are the ones I will be comparing