Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> "Generating Parquet"

It is also useful for moving data from Postgres to BigQuery! ( batch load )

https://cloud.google.com/bigquery/docs/loading-data-cloud-st...

Thanks for the "ogr2ogr" trick! :-)

I hope the next blog post will be about GeoParquet and storing complex geometries in parquet format :-)

https://github.com/opengeospatial/geoparquet




GeoParquet just embeds the geometry as WKB, so getting PostGIS geometry is as simple as going "SELECT geom::geometry FROM my_parq_fdw_table". The spec doesn't yet have anything to say about how to get a good spatial sort or carry out a spatial filter, so it's all pretty ad hoc on that end. Thanks to ogr2ogr, generating a GeoParquet file is now pretty straightforward.


Cloud SQL has BigQuery connections that can be leveraged. But yea, this seems like a nice solution if you have a postgres instance outside of Cloud SQL. Another approach would be to write the CDC to a message queue and archive that to parquet.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: