manticore, earlier sphinx search, has been rock solid for us for the past 16 years. now serving searches across nearly 300M short documents. we're using it in the old mode - where full index is re-created every 24h.
it's great to see that the project is alive and adding embeddings-related functions needed for semantic search.
aspect worth noting: up to my knowledge HE's tunnel will work only if you're assigned public IPv4 by your ISP. if you're behind a carrier grade NAT - too bad, you'll need to use another solution to get IPv6 to your home.
environment: KVM VMs running on physical hardware managed by us.
we have a belt & suspenders approach:
* backups of selected files / database dumps [ via dedicated tools like mysqldump or pg_dumpall ] from within VMs
* backups of whole VMs
backups of whole VMs are done by creating snapshot files via virsh snapshot-create-as, rsync followed by virsh blockcommit. this provides crash-consistant images of the whole virtual disks. we zero-fill virtual disks before each backup.
all types of backups later go to borg backup[1] [ kopia[2] would be fine as well ] deduplicated, compressed repository.
this approach is wasteful in terms of bandwidth, backup size but gives us peace of mind - even if we miss to take file-level backup of some folder - we'll have it in the VM-level backups.
Allegro has amazing metadata allowing you to precisely filter out the results. Search experience on amazon is an utter abomination compared to Allegro.
We're using BTRFS to host PostgreSQL and MySQL replication slaves. We're snapshoting drives holding data for both every 15 minutes, 1h, 8h and 12h and keep few snapshots for each frequency.
Those replicas are not used for any workload, besides nightly consistency checks for MySQLs via pt-table-checksum to ensure we don't have data drift.
Snapshots are crash consistent. Once in a while they give us ability to very quickly inspect how data looked like few minutes or hours ago. This can be life-saver in case of fat-fingering a production data and saved us from lenghty grepping of backups when we needed to recover few records from a specific table.
Yes, I know soft deletes, audit logs - all of those could help and we do have them, but sometimes that's not enough or not feasible.
Due to it's COW nature BTRFS is far from perfect for data that changes all the time [ databases busy with writes, images of VMs with plenty of disk write activity ]. There's plenty of write amplification, but that can be solved with NVMe drives thrown on the problem.
Also - thanks to that FIDO2 does not seem to be usable with Microsoft's MS365 services [ Teams, Outlook, Excel etc ] on Android or iOS. there's no way to provide pin for the security key, regardless if it's plugged in via the USB port or used via NFC.
coincidentally - we've been using first sphinx and then manticore for over 15 years as well. in our case it's fed each night with XML generated by Java code from data stored in MySQL databases. we index over 294M pseudo documents.
borg is great. we've been using it for the past 3 years to archive hundreds of file-level backups of servers, database dumps and VM images. average size of each borg repo is few GB but there are few outliers up to few hundreds of GB. most of backups are done daily, with 7-24 past days preserved in borg archive. borg repos are verified, copied to external disks, verified again and rotated offline each week.
borg replaced https://rdiff-backup.net/ for us and gave:
* nice speedup of backups/backup tests,
* decent saving in the disk space thanks to compression and deduplication,
* decreased backup replication time [ borg repo tends to have much less, larger files compared to rdiff which has in its repo at least as many files as your source data; rsync likes it ].
to finish backups in a reasonable time we had to parallelize backup gathering [ each server / vm goes to separate borg repo; this limits a failure domain in case of corrupted repo, but denies us benefit of deduplication on larger scale - across servers ] and borg archiving. without that - we would be a limited by a single cpu core performance [ borg is not multithreaded yet ].
it's worth testing the backups - we're doing it each day by using borg's repo self test and by extracting few key files and checking their checksums and content... just in case.
echoing other comments - https://kopia.io/ looks interesting but we have not tried it yet.
if you're running your test over the internet [ fluctuating latency, some packet losses ] - try enabling BBR [1] tcp congestion control algorithm on the sender side to utilize the available bandwidth more efficiently.
I use it at work. Sphinx search has been powering our text search since the inception in 2008 and was scaling greatly. In 2021 we've moved from to Manticore without any hiccups.
It's now handling index over 280M+ documents and has been drama-free - just works.
i've been using BTRFS since 2014 to store backups. there is a noticeable performance penalty when rsync'ing hundreds of thousands of files to a spinning-rust disk connected to USB-SATA dock when BTRFS is used instead of EXT4. i'm accepting it in exchange for ability to run scheduled scrub of the data to detect potential bitrot.
since 2017 i'm also using BTRFS to host mysql replication slaves. every 15 min, 1h, 12h crash-consistent snapshots of the running database files are taken and kept for couple of days. there's consensus that - due to its COW nature - BTRFS is not well suited for hosting vms, databases or any other type of files that change frequently. performance is significantly worse compared to EXT4 - this can lead to slave lag. but slave-lag can be mitigated by: using NVMe drives and relaxing durability of MySQL innodb engine. i've used those snapshots few times each year - it worked fine so far. snapshots should never be the main backup strategy, independently of them there's a full database backup done daily from masters using mysqldump. snapshots are useful whenever you need to very quickly access state of the production data from few minutes or hours ago - for instance after fat fingering some live data.
during those years i've seen kernel crashes most likely due to BTRFS but i did not lose data as long as the underlying drives were healthy.
https://manticoresearch.com/ is the lively, open source, fork of Sphinxsearch. that's where some of the earlier developers from the project moved to. it's used as a text-search backend on craigslist.
probably related news from Dell: "NOTE 1: 13G, select 12G, and select DSS server BIOS files have been pulled from http://dell.com/support. This note and article will be updated as soon as more information is available" [1]
the pulled out BIOS update files for 13th gen were released on 5th of Jan.
That's what I've gathered based on the online discussions:
Possible reasons why the original project stagnated: [0].
Mention of the manticoresearch as a fork project was removed from the sphinx forum [1] - so I can guess that developers who moved to the new project did not part on good terms with Andrew - the original author.
that's the last reply in that thread from person involved in Manticore, posted on the 23rd of Oct 2017:
"
aditirex just replied to 'Sphinx search fork':
===cut===
> But your the people who are already using sphinx, why we should change?
The open-source version of Sphinx received 5 code commits since November last year, from which 3 are related to building stuff. Last release was 12 months ago.
There are also a lot of unresolved reported bugs (many of them are crashes) in the bug tracker.
Andrew said a while ago that the open-source version would only receive fixes (which doesn't seem to happen either).
No one wanted to do the fork, it was the only way several big users saw it in order to continue using Sphinx. Don't ask me how we got into this situation, I'm not the right person to answer to that.
> What are the main benefits rather than using Sphinx?
Manticore is pretty much continuing Sphinx. Last year we had 4 developers + Andrew working on the code, 3 of them are working now on Manticore.
If Sphinx just works for you there is no reason to switch. But we're adding new features, fix existing bugs, the software is tested by some big users before getting released, you get a software that has support from it's developers.
> Is Foolz\SphinxQL\SphinxQL working?
Everything works as before, it's a fork, not a total new software.
"
It's worth mentioning that the original Sphinxsearch project has been stagnant for the past year. There's a new lively fork - https://manticoresearch.com/
Beyond being a user of both I don't have affiliations with either of them.
as far as i understand apparent death of sphinx and demand for continued development/support from big users of it led to creation of manticore.