Several other ways to do this, none of which require duplicating data:
1) Create a chart from a named formula that refers to the PT data. The advantage of this approach is that if the PT shrinks/grows, the chart auto-adjusts.
2) Create an empty chart and then specify the source as the PT data. The key here is to not include any of the PT headers.
2a) Select the empty chart, then Chart | Add Data... and specify the PT cells of interest
2b) Select the empty chart, then Chart | Source Data... | Series tab | specify the X and Y values as required
2c) Select the PT data range, copy, select the chart, and Edit | Paste Special...
2d) Select the PT data range, move the cursor over the edge so that you can drage the range, and drag it on to the chart.
Of all the ways listed in item 2 above, 2b probably gives you the most flexibility -- especially if you have multiple row fields.
I ended up referencing the pivot table in a regular range using GETPIVOTDATA function, and regular old ="some cell". Then I created a chart from that range. Now everything works normally. Hid the new range under the chart for asthetics. Will probably forget I did that one day, and drive my self bonkers trying to figure out where that range is. Thanks for the opinion mrhartley.