olviko

vor 10 Jahrendiscuss

Maybe. Don't think any of this existed when I need it...

1. The code is tiny with 90% of it dealing with RSS parsing and filtering. Using RX.NET wouldn't really simplify anything.

2. I wanted a library that I can integrate into my apps and run locally to avoid throttling, robots.txt and other BS Yahoo Pipes was suffering from.

I am also not a huge fan of RX... to put it mildly

olviko

vor 10 Jahrendiscuss

Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time…

But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine, in C# it can be nicely abstracted with “async/await”, not sure how it is handled in Python, I stopped using it around 2.5

olviko

vor 10 Jahrendiscuss

Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc

Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps.

[1] https://github.com/olviko/RssPercolator

[2] https://github.com/olviko/RssPercolator/blob/master/RssPerco...

olviko

vor 11 Jahrendiscuss

Just curious, why not Redis?

olviko

vor 11 Jahrendiscuss

Same here. InfluxDb has a pretty nice DSL. I wish they switched their backend to something usable and mature instead of re-inventing the bicycle. Operationally, it is expensive to support specialized databases like Influx, unless it is your core business, I guess...

olviko

vor 11 Jahrendiscuss

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

olviko

vor 11 Jahrendiscuss

Checkout https://github.com/olviko/RssPercolator

Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.

No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.

olviko

vor 11 Jahrendiscuss

Checkout https://github.com/olviko/RssPercolator.

RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.

- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)

olviko

vor 12 Jahrendiscuss

If you like Yahoo Pipes as a concept this library is for you ;-)

  - RSS and Atom feed formats
  - Asynchronous multi-feed download
  - Multiple sources and multiple destinations
  - Feed filters (string match, wildcards, and Regex

Submissions

Show HN: RssPercolator – A .NET library for downloading and filtering RSS feeds

comments