Visual type shifting between two datasets on basis of filter

Q&A
, , ,
1

I wanted to have my visual type make use of dataset. But it turns out the dataset is exceeding the number of rows limit which is 1B. Due to this I have to divide it into two datasets. So is it possible for me to create a visual type by using two datasets ?

2

Hi @akarroth, welcome to the Quick Sight Community. It sounds like you are working with a very large dataset that exceeds the 1 billion row limit in Amazon Quick Sight. Unfortunately, Quick Sight does not support direct creation of a visual type from multiple datasets in a single analysis. However, here are some best practice strategies you can consider:

  1. Data Preprocessing: Preprocess your data outside of Quick Sight using AWS services such as AWS Glue or Amazon EMR to reduce the size of the dataset. This can include aggregating data, filtering out unnecessary rows, or splitting the dataset by logical partitions (e.g., by date or category).
  2. Data Partitioning: Since you have to divide the dataset, you can split it based on a logical key, such as date range or specific dimensions, that aligns with your analysis requirements. Ensure each dataset is optimized for the specific visual you need to create.
  3. Union and Join: If your analysis can be divided into parts, you can create separate analyses for each dataset and use joins within Quick Sight for combining smaller chunks of data. You can also use calculated fields to create combined metrics that make sense for your visualization.
  4. RDS or Redshift: Use Amazon RDS (Relational Database Service) or Amazon Redshift to handle large datasets. You can store your large datasets in these databases and create views or queries to aggregate or filter data as needed for Quick Sight. Amazon Redshift, in particular, is optimized for handling large-scale data warehousing.
  5. Aggregated Data: Use aggregated data for your visualizations. Aggregated datasets are smaller and can provide the necessary insights without overwhelming Quick Sight. For example, you can aggregate data monthly instead of daily if detailed granularity is not required.
  6. Data Source Optimization: Optimize your data source for Quick Sight by ensuring it is indexed and partitioned properly. This can significantly improve the performance and manageability of your datasets.

By following these strategies, you can manage large datasets effectively and still achieve meaningful visualizations in Amazon Quick Sight.

From @emilyzhu, the Quick Sight products team:

Hi All, I would like to clarify, 1TB is not total SPICE capacity limit, 1TB and 1B row is the current limitation for each dataset (as of Feb. 2024)

SPICE capacity is configured at account and region level, (meaning for each account in each region, customer can purchase their SPICE capacity to share with all the users). QS has implicit limit for 50TB per account per region, but it is a soft limit, customers can request to increase the capacity limit from QS.

QS also offer auto-purchase capability, customers don’t need to purhcase manually everytime they need more capacity, they can turn on the option and QS will automatically bump the capacity up to match customer usage. Details please see Amazon Quick Sight supports SPICE capacity auto-purchase

Did this solution work for you? I am marking this reply as, “Solution,” but let us know if this is not resolved. Thanks for posting your questions on the Quick Sight Community!