cirego

I've noticed that there's another problem with microservices as well. People tend to tie microservices and multi-repo into the same strategy.

Multi-repo appears to make teams faster (builds are faster! fewer merge conflicts!) but, like micro-services, they push complexity into the ether. Things like updating service contracts, library updates, etc. all become more complicated.

cirego

4 ay öncediscuss

First thing I noticed too.

cirego

geçen yıldiscuss

It sounds like this framework is susceptible to head of line blocking. In my experience, that significantly reduces the utility of any applications written choosing this framework. What’s the benefit being delivered?

cirego

geçen yıldiscuss

I noticed the same thing. I would have expected an Arc<Mutex<…>> or something similar for safe concurrency. Not sure what value is delivered by a single threaded, blocking web server.

cirego

geçen yıldiscuss

You bring up an interesting question -- I think Tilt works best for those that find themselves in an environment where product is delivered using a service-oriented architecture deployed to Kubernetes. It's also easy to get started within a small team in a big company.

To your other point, Tilt is to development as ArgoCD is to deployment. Tilt enables on-demand, reproducible development environments that are sufficiently high-fidelity that you can often replace your shared and/or long-lived testing clusters.

With Tilt, I test my application using the same Kubernetes specs / Kustomizations / Helm Charts that you use to deploy into production. When it comes time to deploy my application, I supply these same specs / kustomizations / charts to ArgoCD.

Because I can reuse the specs for both testing and production, I enjoy far greater testability of my application, improving quality and time to market.

cirego

geçen yıldiscuss

I think Tilt Extensions highlights the combined power and composability that using Starlarks brings to Tilt: https://docs.tilt.dev/extensions.html.

Want to create a Kubernetes secret? It's as simple as:

    load('ext://secret', 'secret_yaml_generic')
    k8s_yaml(secret_yaml_generic(...))

Want to create that secret from Vault instead?

    load('ext://vault_client', 'vault_read_secret', 'vault_set_env_vars')
    vault_set_env_vars('https://localhost:8200','mytoken')
    my_foo = vault_read_secret('path/myfoo', 'value')
    my_bar = vault_read_secret('path/mybar', 'foobar')

cirego

geçen yıldiscuss

Thank you for sharing this! I think your Tiltfile just showed me how to solve something that's been bugging me for a while!

I see that you also have docker-compose files -- are those for different tasks or for developer preference?

I'm also curious to understand why you have different build scripts for CI (`buildx`) vs local (regular docker build)? In our team, we use the same build processes for both.

cirego

geçen yıldiscuss

I've always appreciated that Tilt chose Starlark instead of YAML. Makes things so much cleaner!

cirego

geçen yıldiscuss

I think that's a fair point -- you're making a tradeoff. And the best part is that you don't need to choose one or the other.

In my case, I find that I prefer having higher fidelity and simpler service code by using Tilt to avoid mocks. It's also nice for frontend development because, using a Kubernetes ingress, you can avoid the need for things like frontend proxies, CORS and other development-only setup.

cirego

geçen yıldiscuss

My understanding is that dev containers are more about configuring your development environment with the right toolchains to build and run services.

Tilt is a monitor process that builds and starts your services, with a hot-reload loop that rebuilds and restarts your services when the underlying code changes. The hot reload loop even works for statically compiled languages.

cirego

geçen yıldiscuss

Yes, Tilt really shines when you’re testing interactions with Kubernetes, such a APIs. But also things like your services’ ingress configuration and metrics scraping.

By default, Tilt is actually intended for local development using kind, minikube or other similar tooling. It supports developing against a multi-node cluster but it requires extra configuration and slows down iteration time.

cirego

geçen yıldiscuss

Tilt can also help validate things like:

- Service discovery

- Volume mounts

- Ingress and Certificates

- Metrics scraping and configuration

- Dashboards

It’s really quite powerful and replaces the need to mock things out with docker compose. If you’re deploying to Kubernetes, Tilt gives you the option to avoid “development-only” setups like docker compose.

cirego

geçen yıldiscuss

Interesting to see this pop up here! I’ve been using Tilt for multiple years now but the pace of development seems to have slowed down after the Docker acquisition.

I love how Tilt enables creating a local development environment that lets my services run the same in production , test and development. Greatly simplifies my service code and improved my quality.

In particular, I’d love to see Tilt be better around handling things like CRDs (there’s no way to mark a k8s_yaml as depending on a CRD being available, a frequent source of broken tilt up invocations).

Having said that, the first thing I do, when working on any new project, is to get “tilt up” working.

Things I’ve used for testing include: eBPF-based collectors for security and observability, data pipelines, helm chart development, and Kubernetes controllers. It’s very flexible and powerful for a wide range of development.

cirego

geçen yıldiscuss

Sorry, I should have added my second thought. Your original comment about isolating MCP servers is also good!

These are tools where the AI may tell you it’s doing one thing and then accidentally do another (I had an LLM tell me it would make a directory using mkdir but then called the shell command kdir (thankfully didn’t exist)). Sandboxing MCP servers is also important!

cirego

geçen yıldiscuss

I believe echoangle’s concern is about the security and privacy of the LLM using the data, not the MCP server itself.

cirego

geçen yıldiscuss

I think you might be off by a factor of 10. Alphanumeric would be at least 36 characters, which would imply 2.8 trillion combininations (36 ^ 8).

cirego

5 yıl öncediscuss

At the moment, we are focused on scaling up with a single instance. While there are a couple of strategies for scaling out, none of our current methods support scaling a single view beyond a single instance. This isn't really a technical limitation but rather more of a testing and supportability limitation. Timely and Differential (the incremental computation frameworks used by Materialize) suport scale-out and have been extensively tested in scale-out scenarios.

Do you have a limit in mind on how large you're willing to go? A brief bit of napkin math shows that, at 1KB per record, your largest view should fit within 100GB of memory. This is easily supported by Materialize and I can certainly understand if this exceeds your appetite!

If you're interested in reducing the size of the in-memory dataset, does your fact table have a temporal dimension to it? By default, Materialize stores data for all time (like a database) and you can write views in such a way that it will only materialize recent data (like a stream processor). Our co-founder Frank wrote a blog post[1] detailing how to do this.

I'm honestly not sure how TAIL-ing a non-materialized view would perform. Sounds like something that would be fun to test!

Happy to chat about this further in our community Slack channel if you have more questions.

[1] https://materialize.com/temporal-filters/

cirego

5 yıl öncediscuss

You are correct! Updates are a expressed as a retraction and an insert that happen within the same timestamp.

An example may not be necessary but it might also help clarify. Assuming you're using the psql client to run "TAIL WITH (PROGRESS)", the logical grouping for a single update will be a set of rows like the following:

  ...
  1608081358001 f -1 ['Lockal', '4590']
  1608081358001 f 1 ['Epidosis', '4595']
  1608081358001 f -1 ['Matlin', '5220']
  1608081358001 f 1 ['Matlin', '5221']
  1608081359001 t \N ['\\N', '\\N']
  ...

All of these occur at the same timestamp, meaning that they should be applied atomically to maintain consistency of your dataset. In this case, my query is a top-10 query and Epidosis has now entered the top10 while Lockal has dropped out of the top10. Matlin remains in the top10 but their total has gone from 5220 to 5221. The final example record is produced when you run with PROGRESS enabled and serves as an indicator that 1608081359001 is now closed and no further updates will ever happen at timestamp 1608081359001.

I find that this stream of rows is very easy to convert to a data structure "{timestamp, inserts[], deletes[]}" and this, in turn, maps naturally onto reactive APIs, such as React or D3. My blog post, linked above, delves into this in more detail. Hope this explanation helps!

cirego

5 yıl öncediscuss

> In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.

I see that now -- makes sense!

> Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.

I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.

[1] https://github.com/TimelyDataflow/timely-dataflow [2] https://github.com/TimelyDataflow/differential-dataflow

cirego

5 yıl öncediscuss

> Yep. I meant it was easy to do it the inefficient way where you just refresh the entire query when any table mentioned in the query changes. You would just have to also check if something was a view and recursively parse the SQL that is used in the view. Just use Postgres `LISTEN` and triggers or the WAL for change monitoring.

Unfortunately, it turns out that recursively refreshing views still leads to surprising behavior. I think post summarizes the problem quite nicely: https://scattered-thoughts.net/writing/internal-consistency-.... If you cannot refresh all of the views, at a single point in time, then there will be internal inconsistencies in your dataset.

When looking at automatic refreshing, simple triggers and `LISTEN/NOTIFY` don't scale, as was mentioned in the comment regarding Hasura's multiplexing. I think, in the absence of incrementally maintained views, their multiplexing strategy is a good compromise for databases like postgres. However, it should be noted that continuous query / subscription of views is the exact scenario under which incremental computation will provide both lower latency and greater resource efficiency.

Submissions

DOJ Drops Challenge to California Net Neutrality Law

A Simple and Efficient Real Time Application Powered by Materialize

comments