Time series, medium size numeric data (lite to none RDBMS/relationships) visualization and reporting

[pavanb]

I work with generators and 100MB time series CSV data files are generated every day (250K rows), as shown here: https://www.dropbox.com/sh/d57qcdkx6087u72/AACv4MqXDmpVrxN-Qci0Pjida?dl=0

I am interested in zooming in/out of chart that has 4 variables plotted against time. Then consolidating these daily logs & charts into 2-3 week summaries. Max, Min, Average on 10 mins or X mins, from the 1 second data above.

The reporting tool should find the minimum & maximum points for these 4 variables and plot all 4, +- 1 minute about the minimum & maximum point.

The above scenario can happen at 3 other sites so I have 3 separate sets of data weekly.

Would like to work in the windows 7 64b environment. A tool needs to demonstrate handling an equivalent 1GB CSV file.
OR 300MB XLSB or the tool’s native/binary format
OR 2 million rows and 10 columns, at a minimum

I realized that my data isn't large but medium/small sized from a hdf5 standpoint. Responsiveness to pan/zoom & reports are important. Pandas, SqlLite, TDMS TDM, Microsoft BI Excel VBA, power pivot, Amazon AWS have been suggested. I don’t need Hadoop, Spark, mapreduce.

A i7 laptop with 200 GB ssd and 20GB memory would be used. Time series data, just numbers, no text or much of RBMS like relationships. Willing to invest in servers & tools.

What are some right tools, resources, plans you suggest? Thank you.

 

[TimRodman]

Do you want the maximum and minimum points to be plotted on the chart or just summarized off to the side?

 

[pavanb]

Plotted.

 

[miguel.escobar]

Power Query = for consolidating your Files
Power Pivot = to create an analytical model and compress the data (also so it can seat in memory)
Power View = depending on how many points and series you need to plot it might fit your needs


In short, what you're doing might become possible with Power BI. Go ahead and visit the official www.powerbi.com website for more information about the tool and their resources.