And I'm saying you can't have fast rescans in all cases - it very much depends on the filesystem and directory structure.
I'm not trying to downplay nnn - I meant it when I said it's a cool project! I'm saying each project has its strengths and weaknesses, but your marketing doesn't reflect that (or I missed it).
ncdu's memory usage is definitely its weak point - that's not news to me - but it's because I chose the other side of the trade-off: No rescans. If you're curious where that 60MB goes to, it's 400K of 'struct dir's: https://g.blicky.net/ncdu.git/tree/src/global.h?id=d95c65b0#...
I appreciate your work, but you're not being very honest with your claims.
nnn is not keeping information about 400K files in memory in that benchmark. As a result, the rescan is necessary when changing directory. The rescan may be fast in many cases and in some cases it may even be what you'd want, but I can also name many cases where you certainly won't want it (large NFS mounts being one example).
Sorry for the pedantry. I spent a fair amount of time optimizing ncdu's memory usage, so I tend to have an opinion on this topic. :)
This looks incredibly cool, I love the speed and minimalism. It doesn't seem as configurable and feature-packed as vifm[1], so I'll probably stick with that, but it certainly fills a good niche.
> nnn vs. ncdu memory usage in disk usage analyzer mode (400K files on disk):
I assume that's because nnn isn't keeping the entire directory structure in memory, which means that browsing to a subdirectory involves rescanning the entire directory. That's a fair trade-off, but an unfair comparison.
1. https://vifm.info/ - I'm surprised this hasn't been mentioned yet in this thread.
There was a similar talk[1] at FOSDEM, where the speaker describes how, as an experiment, he replaces a full ELK stack plus other monitoring tools with PostgreSQL. He even goes as far as implementing a minimal logstash equivalent (i.e. log parsing) into the database itself.
It wasn't an "we do this at scale" talk, but I'd love to see more experiments like it.
For the impatient: Skip to 17 minutes into the video, where he describes the previous architecture and what parts are replaced with Postgres.
I'm surprised that even the fastest implementation needs 20+ms to search a 1000-element array. I would expect even a linear search to finish within 1 or 2ms with such a small data set. How large are the array elements? How were the times measured?
EDIT: Oh the time measured is the total of 1,000,000 random lookups? Nevermind my confusion then, that would certainly explain it.
Recently finished my Master's degree in Embedded Systems, and still looking into the exact direction I'm heading for. Experienced (non-professionally) as (embedded) software dev, back-end web dev, and Linux sysadmin (websites/email). I have a broad interest in technology and am a quick learner, so I'm open to any offer that is relevant to the above listed technologies. :)
SEEKING WORK - Remote or near Enschede, Netherlands
Available for any C (not C++, sorry) development where performance and efficiency matter. Love working with low level stuff, embedded systems, network protocols, security, and making it all work together.
Hmm? The stack buffer in yxml will never cause parsing to block, and there's no added dependencies on... anything? I don't think that buffer causes any problems even on a size-restricted microcontroller that doesn't have malloc(). As long as you can find ~512 bytes or so of free memory you can parse a lot of files.
The only situation in which that buffer would cause an error is when the application used a too small buffer, or when the document is far too deeply nested or has extremely long element/property names. Both the maximum nesting level name lengths should, IMO, be limited in the parser in order to protect against malicious documents. Most parsers have separate settings for that, yxml simplifies that by letting the application control the size of a buffer.
The stack buffer in yxml is also used to make the API a bit easier to use. With the buffer I can pass element/property names as a single zero-terminated C string to the application, without it I would have to use the same mechanism as used for attribute values and element contents, and that mechanism isn't all that easy to use. (This is the one case where I chose convenience over simplicity, but I kinda wanted the validation anyway so that wasn't really a problem)
It's new, yes. I wrote it about two months ago because I needed a parser and found the existing solutions too bloated.
> It doesn't allocate or buffer, so when it says that it verifies proper nesting, I assume that doesn't mean verifying that start tags and end tags have matching names. I think that would be impossible.
It does match start tags and end tags, and in order to do so it does, indeed, need a small buffer. However, the only thing that needs to be stored in this buffer is a stack of element names, and that rarely needs much space - a simple fixed-size buffer is sufficient. It can be safely allocated on the stack if the application wants to. See the documentation for the yxml_init() for more details. The "mostly-bufferless" part of the homepage refers to the lack of a complicated input buffer and the avoidance of complex buffer management for output where possible.
> It looks like you pass the parse function a character at a time. To be fast, you'd want to somehow encourage that to get inlined.
The function is quite large. You can probably get it inlined when put inside a header file and when it's only called in a single place, and you may indeed see a performance boost. I haven't measured how significant such an improvement may be.
However, if performance is the primary goal (it wasn't for yxml), a completely different design would probably be better. Yxml needs to manage an explicit state object for each input character. For each new character it has to look at the state to know how it's handled, handle it, and then update the state for the next character. I was honestly surprised to see that yxml performed quite well in my benchmarks.
A more efficient design is probably the goto-based state machine approach that, e.g. ragel[1] can output. The reason I didn't go for that approach is that it complicates buffering. You either need to have the full file mapped into memory (not possible when reading from a socket or from a compressed stream), or you need to implement your own buffering and provide hooks so that the application can fill the buffer when it is depleted. The latter approach is used by most other XML parsers.
Agree with the article; Linda is great, both the name and the project. It was one of the more interesting topics of a Distributed Systems course that I followed, but I never ended up using it (or its concepts) in practice due to the performance overhead.
Please do use autotools! You only need two files: configure.ac and Makefile.am, both at the top-level of your project. The autogenerated stuff can be ignored, you don't have to learn M4 to use autoconf. And (if you are a bit careful, but that's not too hard) you'll have many nice features such as out-of-source builds, proper feature checking, amazing portability, 'make distcheck', and acceptable cross-compiling (still hard to get right, but the alternatives tend to be even harder). Don't switch to another build system purely based on the idea that it's "more elegant".
> Firstly you need a way to incrementally verify a file
Following the proposal of the article, you don't really need that. If you don't have the hash in your local cache, you should assume that the link provided by the site is correct and resume as you normally would without the hash present. When it's downloaded, you verify the downloaded contents with the hash, and add the file to the cache if the verification succeeds.
If you already had the hash in the local cache, then you already have the entire file and there's no need for incremental verification.
As I understand it, the point of the hash is purely for caching purposes, not strictly for validating that the downloaded file matches it.
Odd-sounding request: Is it possible to have an option to not use gpg, or any encryption, at all? I only store my passwords on a LUKS loopback file, which I mount/umount as I need to access some information on it. The double-encryption does not really add any better security, but does add another password to type. I'd love to make use of the nice looking pass interface: A password generator, copy-to-clipboard, and simple management operations.
Minor downside: You have to ensure that the contents of a {code ..} block have all their open-brackets matched with close-brackets. This can be quite tricky with larger code examples. The bbcode [/code] equivalent is slightly less problematic.
If the /robots.txt does not mention a Crawl-delay, one page per 3 seconds is often a safe value. Of course this rather heavily depends on the site. In any case, if you have any specific need, always contact the people responsible for the site. I occasionaly run custom queries against the database on request, for example.
> I managed to crawl [..] more than 300k movies from IMDB in just a few hours
I suppose IMDB already has a pretty good architecture to handle that load, but please, if you're crawling from a single site, be careful. I host a similar database myself, and the CPU/load graphs of my server can tell me exactly when someone has a crawler active again. That's not fun if your goal is to keep a site responsive while keeping the hosting at low cost.
I'm not trying to downplay nnn - I meant it when I said it's a cool project! I'm saying each project has its strengths and weaknesses, but your marketing doesn't reflect that (or I missed it).
ncdu's memory usage is definitely its weak point - that's not news to me - but it's because I chose the other side of the trade-off: No rescans. If you're curious where that 60MB goes to, it's 400K of 'struct dir's: https://g.blicky.net/ncdu.git/tree/src/global.h?id=d95c65b0#...