javier ramirez
supercoco9.bsky.social
javier ramirez
@supercoco9.bsky.social
Developer Advocate at QuestDB and all around happy person. Fan of Open Source,Tech Communities,Data&ML.He/him.Ex-AWS,Ex Google Developer Expert
Some fast databases are already implementing some of these ideas, but to the best of my knowledge there is still none implementing them all. At #questdb we are working to make this happen. Stay tuned!
October 30, 2024 at 10:35 AM
Lastly, compute and storage need to be decoupled, so the divide between recent and batch data is blurred. Deleting or downsampling old data might be acceptable, but it should not be mandatory. The query engine should be able to query over petabytes of storage if needed.
October 30, 2024 at 10:34 AM
ngesting massive amounts of very fast data, querying quickly, and outputting small result sets to dashboards or apps is still useful; but you also need to support streaming out massive query result sets to other apps (e.g., ML apps).
October 30, 2024 at 10:34 AM
The query engine should be able to efficiently query data produced by third parties just by pointing the engine to the parquet files and registering some metadata. Queries should support the whole range of features and perform as if data was ingested through the database.
October 30, 2024 at 10:34 AM
Third parties wanting to use the data ingested via the database should have the option to skip completely the query engine and just read from the parquet files.
October 30, 2024 at 10:34 AM
Fast databases should stop being gatekeepers of your data, and should be able to both store and consume data in open (parquet and friends) formats.
October 30, 2024 at 10:34 AM