diff options
| author | Martin Kagamino Lehoux <martin@lehoux.net> | 2025-04-07 22:01:31 +0200 |
|---|---|---|
| committer | Martin Kagamino Lehoux <martin@lehoux.net> | 2025-04-07 22:01:31 +0200 |
| commit | 6f1dfe6e21e3fc735769995be95d8a495695f6ce (patch) | |
| tree | 97e40cdb11653c95afd6b2f1d2a7f2a026804a9f /content | |
| parent | 3ac17dce5d38c0050fd6a347d8515822744fb722 (diff) | |
feat(analytics): Draft of new post
Diffstat (limited to 'content')
| -rw-r--r-- | content/posts/analytics/index.md | 29 |
1 files changed, 29 insertions, 0 deletions
diff --git a/content/posts/analytics/index.md b/content/posts/analytics/index.md new file mode 100644 index 0000000..268dff7 --- /dev/null +++ b/content/posts/analytics/index.md @@ -0,0 +1,29 @@ +--- +title: "Managing analytical workloads in startups" +draft: true +categories: + - PostgreSQL +--- + +## Current issues + +- Issues with querying large dataset in online postgres (90GB) +- The dataset is analytics events and some business models. +- Often used for AB testing purpose, for instance measuring conversion rates grouped by some experiment values. +- We a have a readonly database, the queries are slow + +## Hypothesis/Solutions + +- Writing better queries: this also needs to be maintainable (e.g. we add more and more queries, it should not need to be carefully handcrafted) +- Using another engine + - Postgres like but OLAP oriented + - Pure OLAP (Clockhouse, Duckdb) + - AWS for simplicity (Athena or Redshift Spectrum) + - Scripts on parquet files: maximum adaptability +- How to get the data? + - it seems that it is already parquets + - it looks too slow to download all the data periodically: process in the cloud or incremental sync + +## Resources + +- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html |