Great article! Just yesterday I watched a Devoxx talk by Andrei Pangin [1], the creator of async-profiler where I learned about the new heatmap support. To many folks it might not sound that exciting, until you realise that these heatmaps make it much easier to see patterns over time. If you’re interested there’s a solid blog post [2] from Netflix that walks through the format and why it can be incredibly useful.
Great that you had the time to be curious and dig into what was going on. QEMU is quite an amazing tool.
I'm kind of surprised there isn't a fairly robust kernel test around this issue, since it locks the machine down and I think the fix was to prevent a stuck CPU last time as well?
It's also vaguely surprising that this hasn't been encountered more often, particularly by the https://news.ycombinator.com/user?id=everlier talking in links to this HN post about "20-30 containers" running simultaneously and occasionally locking up the machine.
If you're still thinking about the bug a little, you could look over how other kernel tests work and implement a failing test around it....?
I imagine the tests have some way of detecting a locked up kernel... I don't know exactly how they'd do it, but they probably have a technique. Most likely since the kernel is literally in a loop it won't respond to anything.. so starting any process, something as simple as creating any process, even one as simple as printing "Hello World!!" would fail and indicate the machine is locked.
Perhaps this is one of those cases where something like UserModeLinux would allow a test to be easily put together, rather than spawning complete VMs via some kind of VM software. Again, would be interesting to know what Linux does with this kind of test.
Definitely not the first AI generated font. One can find an enormous amount of research in AI font generation on https://scholar.google.com/ going back many years. This could possibly be the first one that used Nano Banana though, and the result is impressive for sure!
I believe there is no contradiction with the definition from the linked article?
> A system is said to be real-time if the total correctness of an operation depends not only upon its logical correctness, but also upon the time in which it is performed. Real-time systems, as well as their deadlines, are classified by the consequence of missing a deadline:
> Hard – missing a deadline is a total system failure.
> Firm – infrequent deadline misses are tolerable, but may degrade the system's quality of service. The usefulness of a result is zero after its deadline.
> Soft – the usefulness of a result degrades after its deadline, thereby degrading the system's quality of service.
> I guess I just have to accept that the term has lost it's meaning at this point and can be used for whatever whoever wants to use it for
It's maybe more like you point out: realtime in the OS context vs realtime in an event processing context. The latter is certainly not defined as strictly and often just means push-based. It has been a popular moniker, e.g. in kafka-land, for a while. I'm not sure it intrinsically takes away from the OS context - it doesn't need to be a deep dish pizza situation.
The highest level of cringe you can feel is when you see people you know well in real life post on LinkedIn. The contrast between the way they speak in real life and on LinkedIn is often immense, you don't feel that level of contrast with random internet strangers.
On the other hand, people have commented (in real life to me) that my linkedin comments are bold, hilarious and entirely unprofessional- earning me a sort of credibility in their eyes for being authentic and having integrity.
(and probably more privately, they believe I am too outspoken..)
Pro’s/Con’s; just like with all public broadcast information.
Also, its always embarrassing when someone talks about a linkedin comment I have made, not because I am ashamed but because I am sort of used to a semi-anonymous shouting into the void style forum like hackernews.
Reminds me of a blog post I once read from a manager writing about all the qualities of being a good manager. I read it nodding along that they all seemed like good traits. Then in the comment section there was a post from someone saying something like "You were my manager at one point and honestly you were one of the worst managers I've had in my career. I didn't see many of these behaviours from you". The author responded with something like "I don't disagree. There's sometimes a gap between knowing and doing"
In my professional network, people mostly just reshare and like things their peers are doing or that they want to boost engagement for (mainly job postings, which they also post occasionally).
I _do_ have acquaintances I made outside of working life on LinkedIn, though - the only two that are really active are a mechanical engineer who mostly just posts about robotics and someone in marketing. I don't know if it's because I'm just really good friends with the latter person, but I've never felt annoyed reading their posts; they mostly seem to just talk about enjoying conferences or new externally facing projects - ad campaigns, large-scale promotions, etc - wherever they are currently working. I don't know if part of that is they're in the EU and the culture for marketers there is different?
I have a friend who behaves similarly on linkedin and in real life, and he's very blunt. I like how he calls out some crap on linkedin posts, and nobody dares to like his comments, even though I'm sure everybody approves.
Overall, I don't see anyone I know being a cringe bootlicker on LinkedIn. These people are very visible, but probably a small minority of users.
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency.
Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did
Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.
We're in a Live Fast Die Young karma world. If you can't get a TikTok ready with 2 minutes of the post modem drop, you might as well quit and become a barista instead.
> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.
I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.
I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.
How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
Probably because he could check legalities of a release himself without council. It is probably equivalent to educating yourself on your rights and laws so if you get pulled over by a cop who may try to do things that you can legally refuse, you can say no.
The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.
I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0]
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
Why give this sort of content more visibility/reach?
I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.
Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit.
The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.
I'm not the person you were replying to, but there is a rule I often see about not directly replying/quote tweeting because "engagement" appears to boost support for the ideas expressed. The recommendation then, would be to screenshot it (often with the username removed) and link to that.
FWIW it seems pretty obvious that this was ragebait. OP's profile is pretty much non-stop commentary on politics with nearly zero comments or submissions pertaining to the broader tech industry.
Posts like that deserve to be flagged if the sum of their substance is jingoist musing & ogling dumb people on Twitter.
> Let me save you fifteen minutes, or the rest of your life: They aren’t.
Knowing that all profilers aren't perfectly accurate isn't a very useful piece of information. However, knowing which types of profilers are inaccurate and in which cases is indeed very useful information, and this is exactly what this article is about. Well worth 15 minutes.
> And that often involves ignoring the fancy visualization and staring at the numbers.
Visualisations are incredibly important. I've debugged a large number [1] of performance issues and production incidents highlighted by the async profiler producing Brendan Gregg's flame graphs [2]. Sure, things could be presented as numbers, but what I really care about most of the time when I take a CPU profile from a production instance is – what part of the system was taking most of the CPU cycles.
Isn’t not that they’re “not perfectly accurate”, it’s that you can find half an order of magnitude of performance after the profiler tells you everything is fine.
That’s perfectly inaccurate.
Most of the people who seem to know how to actually tune code are in gaming, and in engine design in particular. And the fact that they don’t spend all day every day telling us how silly the rest of us are is either a testament to politeness or a shame. I can’t decide which.
> Isn’t not that they’re “not perfectly accurate”, it’s that you can find half an order of magnitude of performance after the profiler tells you everything is fine.
> That’s perfectly inaccurate.
That's a very strong claim, and it's not true in my experience as I've showed above.
My read is that it's easy to be quite negative on Java features when you're not the person they were designed for. For example, the main "customer" of the module system is the JDK itself. The main customer of NIO/2 is the low-level libraries like Netty.
I highly recommend the Growing the Java Language talk by Brian Goetz to anyone who's interested in the philosophy behind evolving the modern Java language [1]. And Don’t be misled by the title, it’s not just about Java, it’s about software design.
> So yeah, why expose it to those who are not the "main customer"?
How did modules affect you as a user? I'd guess that you had to add `--add-opens`/`--add-exports` during one of the JDK migrations at some point. And the reason you had to do it was that various libraries on your classpath used JDK internal APIs. So modules provided encapsulation and gave you an escape hatch for when you still have to use those libraries. How else would you do it while still achieving the desired goal?
What this post might be missing is that it’s not just Google that can block your website. A whole variety of actors can, and any service that can host user-generated content, not just html (a single image is enough), is at risk, but really, any service is at risk. I’ve had to deal with many such cases: ISPs mistakenly blocking large IP prefixes, DPI software killing the traffic, random antivirus software blocking your JS chunk because of a hash collision, even small single-town ISPs sinkholing your domain because of auto-reports, and many more.
In the author’s case, he was at least able to reproduce the issues. In many cases, though, the problem is scoped to a small geographic region, but for large internet services, even small towns still mean thousands of people reaching out to support while the issue can’t be seen on the overall traffic graph.
The easiest set of steps you can do to be able to react to those issues are:
1. Set up NEL logging [1] that goes to completely separate infrastructure,
2. Use RIPE Atlas and similar services in the hope of reproducing the issue and grabbing a traceroute.
I’ve even attempted to create a hosted service for collecting NEL logs, but it seemed to be far too niche.
I own a 12 mini, and I'm planning to upgrade my phone this year, it's time. If there were an iPhone 17 mini, I'd buy it, but because there isn't one, I'll probably go for the Pro to get a bigger battery. Apple knows that many folks like me would buy a cheaper mini if there were one, and not spend as much on Pro.
Apple still has 12 mini batteries in stock, and will for a while.
Might be worth trying to get the battery replaced at Apple.
Extra bonus: while it does cost money in theory, every time I've gotten Apple to replace the battery they end up breaking the screen, so I get a battery and phone replacement for free. 12 mini battery replacements _might_ be de facto free.
Just had my 12 mini battery replaced around 1 month ago, the screen survived but it was still definitely worth the money. It had degraded to around 70% capacity which was just barely preventing me from only charging overnight.
This reminds me of users getting blocked for asking an LLM how to kill a BSD daemon. I do hope that there'll be more and more model providers out there with state-of-the-art capabilities. Let capitalism work and let the user make a choice, I'd hate my hammer telling me that it's unethical to hit this nail. In many cases, getting a "this chat was ended" isn't any different.
I think that isn’t necessarily the case here. “Model welfare” to me speaks of the models own welfare. That is, if the abuse from a user is targeted at the AI. Extremely degrading behaviour.
Thankfully, current generation of AI models (GPTs/LLMs) are immune as they don’t remember anything other than what’s fed in their immediate context. But future techniques could allow AIs to have a legitimate memory and a personality - where they can learn and remember something for all future interactions with anyone (the equivalent of fine tuning today).
As an aside, I couldn’t help but think about Westworld while writing the above!
[1]: https://www.youtube.com/watch?v=u7-S-Hn-7Do
[2]: https://netflixtechblog.com/netflix-flamescope-a57ca19d47bb