Reaching Excel's Limits

ExcelAtEverything

Active Member
Joined
Jan 30, 2021
Messages
351
Office Version
  1. 2019
Platform
  1. Windows
Hello friends,

I am working on an Excel doc in a book with more data than I've ever worked with before. I have sheets in this WB that by year's end will be clipping near the 1,048,576 max for a worksheet. I also have other sheets in the same WB which will all be doing the same. The reason is that this Excel doc that I am working on will be pulling in new data each day to be analyzed, and it's a lot of data. My sheet is already starting to hang and "think" often when I hit the enter key.

My question is... what will happen once it reaches the limit? Also, any expert advice to try to alleviate the potential issue?

Thanks!
 
And Tupe77 again, thanks. My response sounded angry at some points, but I'm not. I'm just frustrated.
 
Upvote 0

Excel Facts

How to total the visible cells?
From the first blank cell below a filtered data set, press Alt+=. Instead of SUM, you will get SUBTOTAL(9,)
Tupe77, yeah I appreciate the help. But while I understood your clarification, things like "misusing the forum" and "things I you can easily look up on the internet" come off like it's on purpose or out of laziness. What you said later on is correct, but you don't understood the details of how this project ended up in my lap and how it came to be that I built it up incorrectly from the start. And I do look things up on the internet all day every day. But when I need additional help in understanding something because it's either not working correctly for me or because I'm just having trouble grasping it, I turn to this forum of all-stars. I've been guilty of asking a question here & there that's an easy one to many people and easily found on the internet. But that doesn't mean that I just do that with everything. Hindsight is 20/20, and you don't know what you don't know... you know? And I didn't ask how to frame it up correctly early on, because It wasn't an issue.... until it became an issue. But then I solved that issue. But then another... but then that too was solved. But then they started getting worse and more plentiful, and then grew into major problems... and here I am. There were parts of the data that I didn't even have access to for a while, so I started building the shell of the project out based on what I was told, which turned out to be not even close to the truth. So yes, now I am in deep on a project that I would never have agreed to do had I known (A) the extent of it (B) the poor quality of their currect reporting, and (C) that it required me to do things that I did not know how to do at all. Like PQ for instance. Never used it before, never saw it... it was foreign to me. But backed into a corner, I stayed up for 2 dyas straight learning as much as I could about it, then I rearranged my project for that. Now however, I'm seeing that it needs to be arranged in a different way. But had I known that before... etc. But again, you don't know what you don't know. And also you learn the what to do's & what not to do's as you grow. I'm obviously not at you r level of expertise with Exce, but I'm sure that if you & I were in a recording studio then things might be reversed. You wouldn't know the best way to do it until you became experienced at doing it. I used to be all into Excel. But I'm a music producer now. A producer who has not been able to work on music for 2 weeks now because of this project that I thought would certainly be done by now. So I'm just on here very humbly asking for help from people who are way smarter than me with Excel. If anyone doesn't want to help me, then I respect that decision. I'm in the situation that I'm in for a myriad of reasons. I'm just trying to find a way to get back in front of it.
I'm sorry that you understood me completely wrong. I do not criticize you because you are asking here, but sorry because you're wasting your own time by asking the wrong questions.

I really don’t understand anything about how the studio works, but I know I won’t learn it from the equipment manuals. I’m also not a professional with Excel but just an enthusiast, so I know very well that I can’t know what I don’t know.

However, the biggest difference between us and top professionals like RoryA and Fluff is not the knowledge of formulas or VBA code, but their experience and understanding of the project as a whole.
(I know the difference is huge in coding too, so I'm not trying to downplay it)
You can easily learn to write code, but learning why you write it in code requires years or decades of experience.

I don’t know how much the professionals here are willing to spend their time helping, but I would recommend asking them about your project as a whole. You can get help from others us for formulas and other smaller things. I've been following your posts in the forum and I understand your situation as far what you've been told. Personally, I would need help in the implementation of the project you described and especially in the planning of the implementation.

I suggested you split your data into months or weeks, but RoryA immediately saw that you might not need that data at all because you could get the data directly from the connection. That's why it's worth asking them, because they would be able to tell you how to achieve your goal, and not how to do something...

I wish you luck with the project, but I hope you get help from some professional in planning, otherwise you will solve a huge number of problems that should not have been dealt with at all.

English is not my native lanquage, so if I write something very strange, it's probably because of it.
 
Upvote 0
My thoughts on how to deal with the problem is that to calculate all your year to date , month to date, totals and averages you do not need to store the entire year worth of values. All you need is yesterdays values and a count of the number updates you have got for this year or month. So each time you get an update you add the todays values to yesterdays , add one to the count and divided the new total by the count to get the average. Doing it this way your worksheet is going to be the size of 2 days worth of values and never grow any bigger.
 
Upvote 0
Tupe77, no I really do appreciate it. I know that you're right, but it's the overall frustration in how what I understood to be a simple project has turned out to be a Pandora's Box of issues which you're seeing come out. And the person who pitched this to me came off like he knew what would be entailed, only to find out that he has no clue whatsoever and doesn't even understand when I'm trying to explain about the difference in initial data between this project and my earlier automated Excel sheets which he fell in love with. I apologize if my response offended you, as that was not my intent. I'm super-pissed right now, and I'm just taking a step back today, and taking a deep breath, and tonight I'm going to regroup, and try out some different things. You've always been nothing but helpful, and I really appreciate you taking any of your valuable time to help out a guy who you don't even know.
 
Upvote 0
Offthelip, thanks That was actually close to where my brain was at as soon as I opened my eyes this morning. I initially didn't do it that way because I was told that there would be more info that would be requested to be added in later, so I wanted to keep it all right there so that I could easily just go into PQ and add columns back as needed. But that plan hasn't worked out too well. Lol. I want to look a little more into RoryA's Pivot Table suggestion first, but this solution is totallly doable and I can see how it would virtually eliminate my data probem 100%.
 
Upvote 0
It sounds like you just need pívot tables for the analysis so you don’t need the data in a worksheet at all. If you’re loading the data with PQ already, you can use that data directly from the connection.
RoryA, unless I'm mistaken I would still need to have all that initial data (data that the PT's are extracting from) in my WB, right? The data portion growing to be so large is a large part of the problem. The more I've been looking at Pivot Tables, the more it seems to be a somewhat easy solution. But I'd still have to have all of that data located in the sheet (I think... I'm still learning it). Unless PT's could just access the files in the XLSX /XLSM docs located in a folder on my computer and place the resulting PT into the sheet in my WB without the need for having all of that info also be in the WB. But I haven't gotten that far yet.

Do you think that this would be a better solution than offthelip's suggestion, given the enormity of the data involved and that I seem to be starting to scrape the top of the data cap right already.
 
Upvote 0
The data does not have to ever go near a worksheet. You can create a connection only query and use that as the data source for pivot tables. You can also load it to the data model and use Power Pivot, which may be a better option if you are using time intelligence functions.
 
Upvote 0
Solution
RoryA, you just blew my mind a little. I am going to look into all of this tonight, including Power Pivot which I don't have a clue about. When you say "time intelligent functions", is this something in Power Pivot? Or are you referring to the fact that many of the data points I need are encased in time-related needs... like "yesterday", "month to date", etc?
 
Upvote 0
Both. :) Power Pivot uses a language called DAX which has lots of built-in time intelligent functions for doing calculations like Year to date, Same period last year and so on. It is very powerful, but also therefore can be very complicated. :)
 
Upvote 0
I'd also consider taking a srep back and reviewing your data. I know you have said they need millions of rows of ever increasing information, but given the ability of Excel and Other forms of data storage mentioned by @RoryA, I'd be looking at massive blocks of DAta reduced to summation then stored in a seperate workbook. Once that is done each lot of new data can be added to the summation and stored elsewhere.
Also look at the data structure, is it 1 column, 5 columns 500 columns ?? Can columns be removed as redundant ?
We don't know what the data does, or what you have to do with it ....Is it imported, is it manually input( I doubt).
We're all here to give input....if we can help !!
 
Upvote 0

Forum statistics

Threads
1,214,559
Messages
6,120,208
Members
448,951
Latest member
jennlynn

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top