I have two datasets, 1 with a few hundred rows (dataset A), 1 with 49+ million rows (dataset B). My goal is to create a new dataset where these two datasets are joined on a common field.
Dataset B is from Athena query and in SPICE. Any sort of work to join on this dataset has failed due to timeout.
My thought was to create a filter initially to minimize dataset B prior to the join.
Any suggestions on how to do this?
Hi @jackattack6800
i guess dataset 1 isnt a Athena, right?
What join option do you use?
You could try you set a filter on B, do the join, and remove the filter on B.
BR
Hi @jackattack6800
any updates on your side?
BR
The joins fail due to timeout.
Does issue still exist and you still need help ? Can you provide additional details as where is the timeout happening/screenshots ?
If both datasets are already in SPICE : Dataset B (Bigger table ) join Dataset A(smaller table ) should work.
Kind regards,
Koushik
Hi @jackattack6800,
It’s been awhile since we last heard from you, are you still having a problem in regards to your original request or were you able to find a work around? If so, please share with the community what worked for you.
If you’re still having issues, feel free to share what additional tests you’ve run since your last post to try and work out the solution. If we do not hear back from you within the next 3 business days, I’ll go ahead and archive this topic. Past that though, if there are remaining questions, feel free to create a new topic and link this topic for relevant information.
Thank you!
I was not able to get the join to work. Luckily I was able to shrink dataset A to about 20 rows and I simply implemented a calculated field to achieve my goals. This question can be closed.