'Data Mesh - A Contrarian View' by ProgRockRec datamesh datalake
3) self-serve data infrastructure as a platformI appreciate that Zhamak has given this so much thought and acted on it, however, I think she is ignoring something, and maybe I’ve missed her talking about it, but how did we get to the point where we’re thinking about this? I think there are a couple of main contributors, one goes much further back than the other.
1) Many of the systems we are using today like Facebook or Uber or one of these hyper-scale types of companies, were often developed quickly. They were building something that hadn’t necessarily been done before, so they were anticipating behaviors without knowing exactly what would resonate. The speed at which the systems had to be developed also meant that the amount of planning was more a seat of the pants activity. Every group was building their part, often in isolation, with different people that had different favorite tools. Going back to MySpace, if I recall correctly, they wrote the original version in less than a month. So, that led to real spaghetti systems that then required the development of other tools to deal with the mess the system was to avoid having to rewrite the system. That gave us things like Hudi, Presto, etc., so the initial problem was a lack of planning to begin with. 2) The rise of cloud computing, while super convenient, has created an entire ecosystem around how to keep your costs down. The data lake is in part a response to that since storage is cheaper than computing, so by leaving it out of a database, you have a lower cost. Data egress fees are also a big deal, so you want to push down your queries to resolve as much as possible before the data comes back to you. Leaving it in the data lake gave rise toSo, now we have another mess to deal with and are inventing clever tools to work with it instead of rewriting the core systems. And that is the rub right there. To move to a data mesh philosophy, you have to rewrite everything. No one rewrote it to deal with the original mess; they came up with clever technology to manage the messes, so are they going to rewrite it for this philosophy? Probably not.I think the closest you get at the moment would be her point 4, “federated computational governance”. I seeThe company is founded by the inventors of Presto at Facebook, which they have evolved into Trino, but a big advantage to their tech is their data connectors for federated queries. You can join all sorts of things, all over the place, on-prem, on the cloud, you name it.Summary Let’s be honest though, most companies aren’t these massive organizations with high-velocity data and a need for near real-time analysis. They don’t have sophisticated data analysts that are going around to their company-published data stores to find what they need. The companies that are in that space, are not in a position to rewrite their systems. It’s possible that you are building something new and want to design around that concept, but again, there aren’t necessarily the tools to do it. I think you should spend your time doing a really solid job of designing your system, to begin with. Understand it as best you can and take the time to build it so it will be extensible, flexible, and reliable. It’s worth being aware of what is going on in the space, but don’t overcomplicate your systems with a bunch of black boxes that can fail and no one understands.
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Veterans committing suicide at rate 2 times higher than VA data shows: studyA new study from a veterans group shows that veteran suicides in the United States are over 30% higher than data reported by the Department of Veterans Affairs.
Read more »
Maternal health advocates in the dark as Texas stalls on new mortality dataThe last-minute delay has infuriated maternal health experts who have spent years...
Read more »
Report: Some census takers who fudged data didn't get firedA watchdog group has determined that some census takers who falsified information during the 2020 census didn’t have their work redone fully, weren’t fired in a timely manner and in some cases even received bonuses
Read more »
The first drone to collect weather data in the U.S. may launch this fallPending government approval, the “Meteodrone” will launch this fall and be the first drone to record weather data used operationally in the United States.
Read more »
Report: Some Census Takers Who Fudged Data Didn't Get FiredA watchdog group has determined that some census takers who falsified information during the 2020 census didn’t have their work redone fully, weren’t fired in a timely manner and in some cases even received bonuses. The report by the U.S. Commerce Department’s Office of Inspector General raises concerns about possible damage to the quality of the once-a-decade head count that determines political power and federal funding. The report released Friday also says that off-campus students at colleges and universities were likely undercounted since the census started around the same time students were sent home to stop the spread of COVID-19 in March 2020.
Read more »