There's more on this in The Catholic Bunker section, but:
Start off buying a bit of extra food you already eat with each visit to the store, to not drop a pretty penny all at once. Buy stuff you already use, just a bit extra on each visit, but make sure it's the less perishable stuff (think cans, and peanut butter, then tuna, spam, canned beef, canned green beans, [canned spinach is awful IMO], different beans [beans are a nearly perfect food]). Store amounts based on calories per day/per person-- if you don't actually look at calories and do the multiplication, you'll prepare meals that won't satiate and you will go through your food storage faster than expected.
Buy a *cheap* multivitamin from Wal-mart, your diet will be repetitive and this is better than nothing.
Do not screw around with water. Spend the money. This means, if you don't have well or spring, buy a 50gallon PETE plastic drum (est. 2-3 gallons per family member per day). For water, buy them new (you're not supposed to be able to buy reconditioned drums for potable water, but people will lie what theyre using it for to save 20 bucks. They should be blue. If your municipal water is chlorinated you will not need to add a few tsp of (unscented) bleach (*not* the 'splashless') when you fill the drum, but it won't hurt except to make it taste chlorinated. Don't screw around with a a bunch of water bottles, milk juge, etc. They're not suitable for long term storage, breakdown easily, don't save much money or time versus doing it right, and aren't opaque (usually clear plastic!) so allow light to grow stuff inside the water, and allow things to leach through the plastic, and stack poorly if at all.
Here in everything-overpriced California, such a drum is 40-50 bucks, you will see them cheaper. You'll need a hand siphon pump/bung wrench. Just spend the 10 bucks instead of making things harder on yourself.
Store bleach. It cleans 'stuff', even dishes, of course floors/surfaces and even wounds (ouch), sterilizes things, and can make water potable. Stores well, and is dirt cheap. People don't think about sanitation. Store other cleaning items of course, but bleach is very versatile (and cheap).
Once you got cans of meat and greens (that you will eat) from your regular grocery store, buy canned wheat and other items from the mormons (Deseret store). The price is hard to beat, the quality is great, long shelf life, wheat is versatile.
Often forgotten: get some seasonings and spices, pick up some hot sauce (hot sauce has quite a shelf life!). It will help vary food flavor (did I mention your diet will be repetitive?). Get some Tang for Vit C and to vary your water flavor sometimes.
If you have a cool dry cellar or large pantry, preserve garden grown vegatables, stews, meats, even pasta dishes!: a thousand times more tasty than cans.
Get Spigarelli's book (Crisis Preparedness Handbook) once you've done some basics.
Weapons. Ammo. Lots of light (candles, lamps, survival candles and wicklamps), dynamo/crank LED lights, some NOAA weather and other band radio that cranks, etc. First aid items are a post in themselves. A way to cook besides fuel fires (there are entire sites with solar oven plans you can read for free and you can cook with cardboard and foil or mirrors if you have sun). If you have ample fuel sources fine, but solar ovens are small/light and portable and cleaner (and make no attention-getting smoke!) and easily improvised thus you can make one if you find yourself in unfamiliar or urban surroundings. It's knowledge worth having.
On this last note: I always says the *most* important thing is not 'stuff' but skills, yet people stock up first and defer learning indefinitely. Do not do this. *Learn*
Learn first aid. Learn first aid, but beyond that: military training manuals teach essentially field surgeries. How to deal with infections, set bones, severe burns (you'll need to make field IVs!), growing food, hunting, defense, orienteering/navigation/hiking, sewing and on and on.
Skills are much more important than stuff: Weapons without a marine are useless, a marine without a rifle is still lethal.