Hi Hawley,
I'm surprised that you are running into memory problems with only on the order of 5000 rows. Of course, the amount of memory requires depends on a lot of factors, like how complex the formulas are, how many columns are involved, and even whether cell formatting, validation, conditional formatting, etc., are involved.
One way to considerably reduce the memory associated with complex formulas is to convert them to VBA UDFs. Usually, even a very complex formula can be converted to a UDF that has just a few arguments and makes the cell formula much simpler.
The formula memory can be reduced even further by eliminating the cell formulas via the use of a VBA macro that does the calculations and just writes the results into the cells. This way, the cells contain only values, but no formulas. Macros can also replace conditional formatting, validation, etc., if these are contributing to the memory problem.
If you would like an example of converting cell formulas to UDFs and macros, let me know and I will put together a simple one.