The point that the trinity of logs, metrics and traces wastes a lot of engineering effort to pre-select the right metrics (and labels) and storage (by having too many information triplicate), is a good one.

> We believe raw data based approach will transform how we use observability data and extract value from it. Yep. We have built quuxLogging on the same premise, but with more emphasis on "raw": Instead of parsing events (wide or not), we treat it fundamentally as a very large set of (usually text) lines and optimized hard on the querying-lots-of-text part. Basically a horizontally scaled (extremely fast) regex engine with data aggregation support.

Having a decent way to get metrics from logs ad-hoc completely solves the metric cardinality explosion.

Many companies are having trouble to even keep Prometheus running without it getting OOM killed though.

I understand and agree with the problem this is trying to solve; but the solution will rival the actual business software it is observing in cost and resource usage. And hence, just like in quantum mechanics, observing it will drastically impact the event.

> in cost and resource usage

Nah, it's fine. Storage of raw logs is pretty cheap (and I think this is widely assumed). For querying, two problems arise:

1. Query latency, i.e. we need enough CPUs to quickly return a result. This is solved by horizontal scaling. All the idle time can be amortized across customers in the SaaS setting (not everyone is looking at the same time).

2. Query cost, i.e. the total amount of CPU time (and other resources) spent per data scanned must be reasonable. This ultimately depends on the speed of the regex engine. We're currently at $0.05/TB scanned. And metric queries on multi-TB datasets can usually be sampled without impacting result quality much.

It's not the storage cost; it's the computational load (memory, CPU, sometimes network) of gathering thousands and thousands of metrics by default, most of which go unused.

> observing it will drastically impact the event

this presumes 'metrics' are 'cheaper' than 'traces' / observability 2.0 from a setup standpoint; purely from an implementation perspective?

Wide events seems like it would require more memory and CPU to combine and more bandwidth due to size.

I've implemented services with loggers that gather data and statistics and write out just one combined log line at the end. It's certainly more economical in regard to dev time, not sure how "one large" compares to "many small" in reality resource-wise.

> having a decent way to get metrics from logs ad-hoc completely solves the metric cardinality explosion.

last i checked, the span metrics connector[1] was supposed to "solve" this in otel; but i'm not particularly inclined, as configurations are fixed.

any data analytics platform worth it's money should be able to do this at runtime (for specified data volume constraints, in reasonable time).

in general, structured logging should also help with this; as much as i love regex, i do not think extracting "data" from raw logs is lossless.

[1] https://github.com/open-telemetry/opentelemetry-collector-co...

I'd be curious how you think about my favorite version: https://pastebin.com/raw/dXdZXnux

I think the license has a loophole where it forgot to include the blood signature on my computer screen, extracted with a carbon-fiber blade made from Stallman's mane.

That's some hardcore licensing terms btw.

I had basically the same problem as OP some years back. Then I wrote down, for each and every task, how much I'd value having already completed it (including how much I'd value having experienced doing it), and how long I estimated it'd take. This immediately gives value / hour. Add some categorization (e.g. to only work on work tasks during the week), and voila: Auto-prioritization. Helped a lot with getting the actually important things done and get reminded to do useful long term stuff when idle time arose.

FWIW, I'm currently rebuilding it as a web-app, if you want to try: https://quuxtodo.com/

I usually try leave work-time flexible in contracts. Would be cool to have "up to X hours/week" as availability option.

That is a concept I've thought about for a future iteration. A little more flexibility in time blocks etc.

My school used the products from https://iserv.eu/ They sell a complete software solution for servers at schools.

;) Yes. It works exactly as you envisioned it. You can do it with pretty much everything, actually:

  |dump =*clap
  [ "pa" |dump /tsch < > 0.0 ] ==slow slow clap
  # [
  #   "pa"
  #   <function: 00006000001CCD00>
  #   "tsch"
  #   <scope: 00006000004B3360>
  #   +0.0e0
  # ]

The storage is exactly the same, the difference is only in what will happen if the name is encountered somewhere.

{ 2 } /putTwoOnStack deff

{ 2 } /putClosureOnStack defv

putTwoOnStack dump # 2

putClosureOnStack dump # <function 0006002514AE5>

putClosureOnStack * dump # 2

I'm not convinced the distinction is valuable. Since you already have * and {} you don't really need two distinct variable types.

I think it's a good thing (tm) that { x sin } could be valid code with x pushing the value of x whereas sin is executing the value of sin.

Theoretically speaking, I need at least the deff and defq distinction, otherwise { } could not be invoked during "parse" time. The defv case could in principle be removed, but as * needs to execute (otherwise nothing ever will), execution would need to be the default or I would need to decide based on dynamic type. The first option would mean a lot of superfluous {} (or | actually) around normal data variables, the second one would destroy the nice similarities between arrays, strings and functions with integer domains.

It is self targeted snark allright. Not just not doing it with LISP, but also ignoring pretty much everything I learned about compiler building in university.

I may only be a few years out of university, but over the last couple years I've learned that what I studied in university falls into three categories:

1) Incredibly out of date.

2) Far newer than what my job requires me to use.

3) Existing in some ideal world that matches nothing anyone actually created or uses in industry.

And yes, it's possible (and not rare at all) for both (1) AND (2) to be true at the same time.

All of above - at the same time - ends up in the middle of Gartner's hype cycles: a solution has been found 20 years ago and people are searching for problem it solves ever since.

The README claims easy Domain-specific-language creation. Here is a concrete example how code which utilizes these features looks like: https://gist.github.com/Drahflow/c463717d5c72e114ac63

For the love of brackets...

:) Why does this comment remind me of the abbreviation (I heard) of LISP, Lots of Irritating and Superfluous Parenthesis. :)

I miss semicolons ;)

There are semicolons :) They do sequential function composition:

{ /foo dump } /f defv { /bar dump } /g defv

f g ; /h deff

h # "foo" "bar"

Define "inspired". I became aware of Forth while implementing Elymas. But the decision to make it stack based was actually because I was too lazy (after some attempts) to implement a correct LALR-parser-generator.

If you want a more conventional syntax, I'd suggest trying with a recursive-descent/precedence-climbing parser.

You might like Pop11:

https://en.wikipedia.org/wiki/POP-11