Airflow helped my team out a lot a couple years ago mainly for the simplicity of the topdown UI-based view of a complicated ETL AND the ability to retry parts of the ETL.
We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.
Totally agree on hoping there is at least some movement forward on async--particularly async ORM. I love Django and DRF. It is my go-to for a lot of things, but the lack of async support is increasingly a problem for more organizations. Honestly, it would not surprise me that if a solid, kitchen sink web framework emerges in golang, Django will be abandoned rather quickly.
Rapid7 | Senior Software Engineer | REMOTE or Arlington, VA | Full time
The Rapid7 Labs team uses our experience, expertise, and passion for cyber security to establish a leading understanding of worldwide emerging attacker methodologies.
In this senior role, you will cover multiple disciplines. This includes Data Engineering to enable and enhance our petabyte-scale internet scaling platform (Project Sonar) and our global honeypot network (Project Heisenberg) which drive research, enhance products, and empower the community. This role requires being a seasoned Software Engineer too as we build out proof of concept projects that may later become Rapid7 products. As a strong software engineer, you have experience in writing maintainable code that is properly instrumented and tested. You should love being always on the lookout for better solutions and keeping technical debt at bay.
I've messaged you guys a couple times, but never heard anything! I'm a seasoned Django developer, led some teams, and have a bit of experience in public health (mainly disease surveillance) platforms. I'm located in Austin.
Just fyi, depreciating the property isn't a choice in the US. When you sell, the government assumes you did depreciate on your previous tax returns. If you didn't, you missed out on those deductions and depreciation will be reclaimed regardless.
This is true, and I love DRF but I find it unfortunate then that the concept of Serializer exists. Tom Christie essentially says that Serializers can and should be used outside of views (REST) to decouple and encapsulate business logic. He seems to believe models are database operations-only. This view is fine (e.g., data mapper) but then it seems the scope of DRF is beyond REST and should be part of core.
I'm not sure if this is exactly where al2o3cr was going, but a Ruby (or whatever) application would normally include such things as caching and queueing. Despite the slowness of Ruby et al, I actually would expect a modern application with these layers to handle load much better than straight database calls. So I guess I would consider the PostgREST implementation a bottleneck because you can't add any of these layers.
That's a good set of pitfalls to watch out for. Mainly, I just find it odd that in Django 1.7, they are using frozen models to mitigate some of these problems, but don't remove or freeze the base classes, therefore leaving the same problem. I'm sure the migrations system will get better over time.
Awesome that migrations are now in Django core. I do see some pitfalls regarding Historical Models though.
The problem is that application code usually changes over time. Consider a deployment that hasn't been migrated in a long time. When migrations are eventually run, the older migrations might assume application code behaved a certain way. This is why you should never use your regular application models in migrations. Both South and Django 1.7 migrations copy "shallow" versions of your models (no custom methods, save methods, etc), which helps to encourage developers to keep the migration isolated from application models that might change. The problem with Django 1.7 though is that these shallow versions retain any inheritance they had at the time the migration was written. So if that class goes away in the future, I presume the migration will break. Worse yet, if the base class behavior changes, the migration run later in time might behave differently than you intended.
For this reason, custom methods even from inheritance should not be saved in the historical model. Only the fields should be saved off from the base classes, and then merged into one class (remove any inherited classes). Any other app code needed for the migration should be directly copied into the migration itself.
Thank you. This is closer to a much better upgrade path. The official documentation is really not a good way to have non-technical customers move from South to Django 1.7. With the new version of South, you could package both old South (perhaps some of them not yet run) as well as the new migrations. It could be better though. With this approach, they would still have to:
1) ./manage.py migrate
2) pip uninstall south
3) ./manage.py migrate
Better yet would be to have either South or Django 1.7 migrations check for the other's existence and run both sets of migrations and not force South to be removed. This would make it a truly seamless transition for the system administrator.
Yes. I think saying "you can't sacrifice correctness because of an inability to scale your database" is perhaps conveying the wrong message though. I mean, your very first point is about database scaling issues and the advantages of using something like RabbitMQ to avoid expensive SQL queries.
If you are processing a lot of data in Celery, you really want to try to avoid performing any database queries. This might mean re-architecting the system. You might for example have insert-only tables (immutable objects) to address this type of concern.
Passing objects to Celery and not querying for fresh objects is not always a bad practice. If you have millions of rows in your database, querying for them is going to slow you way down. In essence, the same reason you shouldn't use your database as the Celery backend is the same reason you might not want to query the database for fresh objects. It depends on your use case of course. Passing straight values/strings should be strongly considered too since serializing and passing whole objects when you only need a single value is not good either.
What happens if more traffic is coming in than Packetbeat can process? Either you are dropping that traffic or you are (perhaps unintentionally) blocking it, right?
Maybe I'm not following your response. What I'm saying is that there are reasons to use /dev/random at least after boot (to block for seed entropy). Saurik gives a real world example of the "cold boot" problem as other articles have done in the past. Are you saying we should be able to simply stop there and then just use /dev/urandom? If so, then why is the CSRNG re-seeded every 60 seconds? It would seem that there are reasons for doing so.
As people have pointed out in this thread and other articles, clearly there can be issues with using /dev/urandom on boot where there is a lack of entropy. What I would love to know more than anything on this topic though, is whether it might ever make sense to use /dev/random after boot--after the CSPRNG has been properly seeded with good entropy. Specifically, I would love to hear of any real attacks. Anyone?