I just recently jumped into vintage computers myself, and in the last couple of months have picked up a few IBM PCs. You've already received lots of great advice, but since I just did this, maybe I can help as well.
I bought my first PC 5150 on eBay for about $140 (closer to $200 with shipping). It was an early 16KB-64KB unit, and it came with a CGA card, an all in one card with serial and parallel ports, a 5153 CGA monitor (unfortunately DOA due to shipping damage), and appears to have an issue with one of the memory chips that keeps it from loading some DOS and programs. It's a good-looking machine, but it was filthy inside and the drives took a lot of cleaning before they recognized my floppies.
On the other hand, I was watching Craigslist for PC parts and accessories, and noticed somebody giving away a PC. It turned out to be a fantastic machine and my primary now - 640KB of RAM, spotless inside, CGA card, all-in-one card, works great. The only issue that the b: drive isn't reading anything and isn't responding to cleaning, so I may need to give it some additional attention.
But my point is this - how much you pay is directly related to how patient you are and how much work you want to put into it. It's easy and quick to go onto eBay and buy a system, but you'll pay for it. If you keep your eyes open and are willing to wait, you might be able to get a less-traveled machine for less (or no money).
Parts haven't been too hard to find. I have a computer recycler near me that had a few CGA monitors for sale for about $30, so I picked one up to replace the broken 5153. I also picked up a cheap parallel port Zip drive to serve as an inexpensive hard drive substitute that didn't require me to make the PC itself non-original. The trickiest part is bootstrapping the machine if software doesn't come with it - most ways to get software onto your machine require you to boot into some sort of DOS. If you're interested, I'd be happy to send a 5.25" floppy out to you with basic DOS tools on it, along with a trial version of Klaus' Zip drive driver. That way you could pick up a cheap Zip drive and use that to move data back and forth. Or you could install a 5.25" drive on your older machine, as long as you make sure it's 360KB and not 1.2MB - I tried the 1.2MB on my bridge computer and it didn't work, but the 360KB works great.
Hope that helps!