Photo by Wynand Uys on Unsplash
Streamlining Your Analytics Pipeline: Exporting GA4 Data to BigQuery, Running DBT Models, and Visualizing with Looker Studio
GA4 has issues, luckily there's a straightforward solution
Table of contents
Problem
GA4 is great until it isn't. There are 2 big issues with standard (free) GA4, especially if you use Looker Studio:
GA4 API Quota Limits
14-Month Data Retention Policy
GA4 API Quota Limits
If your organization accesses GA4 data through Looker Studio you have probably encountered this error:
Accessing GA4's API via Looker Studio is limited by tokens, there is a certain allowable amount. Tokens are consumed each time a chart or visualization is loaded, pivoted or filtered.
There is some caching that is happening behind the scenes but if you are a power user of Looker Studio and or have multiple stakeholders accessing reports, you can very quickly run into these limits, halting your analytic analyses.
Standard GA4 Limits
Frequency | Quota Name | Allowed Tokens |
Hourly | Per Property Per Project | 1,750 |
Hourly | Per Property | 5,000 |
Daily | Per Property | 25,000 |
In other words, a single GA4 property is only allowed 5,000 tokens per hour. Loading 1 page of 5 visualizations with basic complexity in Looker consumes about ~500 tokens. Your token usage will vary greatly with the number of page objects and the complexity of those queries.
14-Month Data Retention Policy
Standard GA4 has a data retention policy of 14 months. That's to say only 14 months' worth of data is stored on your GA4 instance.
After that period, unless exported, the data is deleted forever...
What About Analytics 360..
Upgrading to 360 has several advantages but in regards to these 2 issues:
All Quotas are increased by 10x (1,750 tokens per hour to 17,500)
Data Retention is increased from 14 to 50 months
However.. Analytics 360 is a whopping $50,000 per year. So onto another option for most of us..
Solution - High-Level Steps
Exporting GA4 data to BigQuery
Set up a BigQuery project and dataset.
Link your GA4 property to your BigQuery project.
Choose which GA4 data you want to export to BigQuery.
Schedule regular data exports to keep your data up-to-date.
Running DBT Models
Create a Cloud DBT account and connect it to your BigQuery project.
Copy or Write DBT models in SQL to transform your GA4 data into a format that is more conducive to analysis.
Use DBT to automate the data transformation process.
Test your DBT models to ensure they are working correctly.
Visualizing with Looker Studio
Connect Looker Studio to your BigQuery project.
Create custom dashboards and visualizations to display your data.
Disclaimers
1 Million Event Export Limit
Google allows standard GA4 properties to export 1M events per day, if you are below this threshold, the export process is free.
Costs
There will be incremental storage costs and querying costs associated with Big Query. However, costs should be fairly nominal, a few hundred dollars a month, for your average-sized organization. See Google Cloud's Cost Estimator and my other [future] article about estimating Big Query costs.
Google - Subject to Change
A lot of folks are upset about the token limitation. Google is subject to change its terms and product offerings at any time. They have already increased the token limit once with the initial outcry and I believe their native solution to this, the Extract Data Source, could get better in time.
But.. not a bad idea to get your exports going regardless so you can retain a longer history of your web analytics.
Other Solutions
There are certainly other solutions to GA4's quota limits. Some involve paying another vendor like Power My Analytics, using Looker's Extract layer or transitioning to native GA4 reports.
All are valid solutions. However, unless you pay for 360, there is no other solution to retain your data. With that, adding a DBT layer isn't too much work. If you're already exporting data to Big Query you might as well do something with it.
All that said, I do believe this may not fit everyone's use case or technical aptitude. I plan on doing another article that lays out other options.
Why DBT?
The reason we need to use DBT is because the native export that comes out of GA4 is incredibly nested. The data needs to go through another step to be usable in analysis.
There can be costs associated with DBT, but you can get away with the free tier depending on your needs. The process can all be done within the free DBT tier, the advantages to an upgraded plan are more organizationally related (more user accounts, access control etc.) at least for this process specifically.
Conclusion
We didn't get too into the weeds on setting up DBT or a GA4 export. Those detailed steps deserve another dedicated write-up. However, I hope this enlightened you and at the very least, got you thinking about taking action with your GA4 data.
More to come! Please leave a comment if you found this helpful or would like to know more!