Thanks for the detailed analysis.
I’m wondering if you factored in the cost of engineering time invested in this analysis, and how that affects the payback time (if at all).
Author here: this probably took a 2.5 days to put together, all in.
The first day was spent hacking together a new hot reloaded but this also fixed a lot of issues we’d had with the previous loader such as restarting into stale code, which was really harming people’s productivity. That was well worth even several days of effort really!
The second day I was just messing around with OpenAI to figure out how I’d do this analysis. We’re right now building an AI assistant for our actual product so you can ask it “how many times did I get paged last year? How many were out-of-hours? Is my incident workload increasing?” Etc and I wanted an excuse to learn the tech so I could better understand that feature. So for me, well worth investing a day to learn.
Then the article itself took about 4hrs to write up. That’s worth it for us given exposure for our brand and the way it benefits us for hiring/etc.
We trust the team to make good use of their team and allowing people to do this type of work if they think it’s valuable is just an example of that. Assuming I have a £1k/day rate (I do not) we’re still only in for £2.5k, so less than a single MacBook to turn this around.
I gave a talk about this at the Packaging Summit during Pycon which was well received, so the team is definitely aware of the problem.
However, the sense I got was that it was going to be a lot of work to “fix Python packaging” which wasn't feasible with an all-volunteer group.
At work, we're migrating away from pip as a distribution mechanism for this reason; I don't expect to see meaningful improvements to the developer experience anytime soon.
This is especially true because pip today is roughly where npm was in 2015, so there's a lot of fundamental infrastructure work (including security) that still needs to happen.
An example of this is that PyPI just got the ability to namespace packages.
That article is about the packaging summit talk on introducing namespaces, not about organizations. In fact, when talking about organizations, it explicitly says:
> But support for namespaces is not part of the new feature.
Not the parent but pipenv is decent, poetry is even better:
- clear separation of dev and production dependencies
- lock file with the current version of all dependencies for reproducible builds (this is slightly difference than the dependency specification)
- no accidental global installs because you forgot to activate a virtual environment
- (not sure if supported by pip) allows installing libraries directly from a git repo, which is very useful if you have internal libraries
- easier updates
I'm curious if you're planning to turn Meltano Cloud into a managed ELT solution.
The FAQ at the bottom of the pricing page[1] describes Meltano Cloud as an EL solution, but the docs for open-source Meltano[2] highlight that it's an ELT tool.
Thanks! Meltano is primarily focused on EL, but we let you run any Python data tool or custom script as well, including dbt (Core) for transformation. So we occupy a space somewhere between EL tools like Fivetran and workflow orchestrators like Airflow/Astronomer, and depending on what you add to your Meltano projects, you can use it for either EL, ELT, just T, or anything you can come up with.
