How does one determine which user agent to use to retrieve feeds?

[touwys]

This discussion is in reference to this closed topic: [BUG]: Newsmax feeds not getting added... · Issue #1716 · martinrotter/rssguard

Question:

1. How does one get to determine which User Agent to use with which to successfully retrieve a new news feed, and where does one eventually get to locate such user agents to try out?
2. Is it at all possible to provide support for a standardised internal list of user agents from which RSS Guard can automatically pick in succession to try and retrieve the feed data?

For example, this is another new news feed that opens in Firefox and other feed readers, but it fails getting added to RSS Guard:

https://dailyinvestor.com/feed/

[DawEdhel]

1. How does one get to determine which User Agent to use with which to successfully retrieve a new news feed, and where does one eventually get to locate such user agents to try out?

By brute-forcing a list of latest UAs like this one.

[Ac314]

1. How does one get to determine which User Agent to use with which to successfully retrieve a new news feed, and where does one eventually get to locate such user agents to try out?

Open the feed with the "normal" browser (where the feed is loading correctly) with the developer tools enabled. You will see there what UA is used by the browser to access the feed URL.

[touwys]

@DawEdhel , @Ac314

1. By brute-forcing a list of latest UAs like this one.

2. Open the feed with the "normal" browser (where the feed is loading correctly) with the developer tools enabled. You will see there what UA is used by the browser to access the feed URL.

Thank you, to the both of you, for these valuable pointers. I'm on to it.

[touwys]

I am still failing to add this feed: https://dailyinvestor.com/feed/

I'd gone ahead and added a list of User Agents — assuming that I had to add them to the HTTP Headers section of the "Add a new feed" dialogue, as shown in the following screenshot.

Obviously, I am not doing it the way it should be done, notably:

1. How/where exactly, is "brute force" applied?
2. Also, where can one gain improved insight to the additional information supplied in this dialogue piece?

Enter each key/value HTTP header pair on a separate line. Note that all spaces are
significant and that header names are case-sensitive. Also, make sure to separate
key from value with like the example below:
HeaderKey=HeaderValueValue

Thank you.

[Ac314]

This site (including the feed page) is protected by CloudFlare so unfortunately this feed cannot be added to RSS Guard. Currently there is no way to pass this protection from the application.

The only way now is to ask the site owner to disable the CloudFlare protection for the feed URL (https://dailyinvestor.com/feed).

[touwys]

@Ac314

I appreciate your help very much, thanks.

The same feed works fine with my Android feed reader. Therefore, I am curious to know how that one is capable of bypassing the CloudFlare limitation.

[Ac314]

Generally CloudFlare works the following way: when you try to access the page on the protected site it performs the series or redirects (to determine if you are human or not) and, if success, set the specific cookie (named "cf_clearance"). Then when you access the site in the future it just reads this cookie and allow you to pass if it is valid.

My guess is your current feed reader kind of "emulate" the real web browser to perform this series of redirects and finally set the required cookie.

I also guess that RSS Guard was never intended to do such thing, it just downloads the feed from the specified URL. Making it act like "real browser" will be a very huge revamp, with a lot of work required.

What can be (theoretically) done besides that:

1. RSS Guard includes its own browser. So in case of such "protected" feed one approach is to try to open it in the internal browser and wait for the protection passed. Then the browser will hopefully set the cookies required and these cookies will be respected by the whole application when accessing the pages on this site (including feeds subscriptions).
   Unfortunately in case of CloudFlare it does not work because CloudFlare allow only the "real" browsers to pass their protection. Which means we should emulate it with the RSS Guard by using the correct User Agent. Sadly it is now impossible in RSS Guard, it only allow to set UA (and other headers) for feeds, not for the browser.

2. Another way could be to find out what cookies are used by the real browser (to check if user is allowed to access the site or not), and then set them manually for the feed in RSS Guard. Unfortunately it does not work too because RSS Guard does not allow to set the cookies for the feeds. They are set (and used) by the internal browser automatically and cannot be changed manually.

So without additional work there is no way to pass the CloudFlare protection now.

I should also add then even if something will be implemened it does not guarantee that everything will work for the CloudFlare. It will be just a step forward to allow further (maybe successful) experiments.

[martinrotter]

Custom UA can be set app-wide in settings, it is then used for internal browser as well.

[DawEdhel]

@touwys

I'd gone ahead and added a list of User Agents — assuming that I had to add them to the HTTP Headers section of the "Add a new feed" dialogue, as shown in the following screenshot.

Obviously, I am not doing it the way it should be done, notably:

1. How/where exactly, is "brute force" applied?
2. Also, where can one gain improved insight to the additional information supplied in this dialogue piece?

About the UA part: @martinrotter has already answered in your closed issue ticket on how to supply its custom value here. Notice that first you prepend a line with the "user-agent=" part, then you add a custom UA value after it.

Another issue from your screenshot is that I'm not sure if RSS Guard will be happy with multiple UA values provided, as usually there can be only one value at given moment (or maybe the app will only take the first or last value, perhaps @martinrotter can give an insight on that). That's what I meant by brute-forcing: you provide only a single UA value and try fetching with it; if denied, replace it with another value, until a website accepts it.

I am still failing to add this feed: https://dailyinvestor.com/feed/

As @Ac314 mentioned, the website is hidden behind Cloudflare, which only accepts HTTP/2 or above connections, while by default RSS Guard sends older HTTP/1.* ones. To circumvent this, you need to enable HTTP/2 on this feed.

Here is how I did it:

1. in 'add new feed' window press the 'switch to advanced mode' button;
2. in opened window you should be in a 'General' tab, where you need to select a 'RSS 2.0' feed type, provide some title, make sure that your URL source is correct (which should be "https://dailyinvestor.com/feed" per your example);
3. then go to the experimental tab and there for the 'Enable HTTP/2' option select the 'enabled' value, which means you will enable HTTP/2 only for this feed;
4. press 'ok' to close the window and try fetching it; if everything went right, you should be able to fetch 10 articles from this feed as I did.

P.S. for @martinrotter: for some reason this mentioned feed (https://dailyinvestor.com/feed) neither can't be discovered in a 'adding new feed' window nor its metadata can't be fetched even with HTTP/2 enabled. Not sure if the feed is invalid or metadata fetching ignores the HTTP/2 option, if you're willing to look into it.

[Ac314]

My guess is this site does not have the CloudFlare protection enabled for every IP address. Some countries / IP ranges can be excluded/included. So maybe you are lucky and CloudFlare is disabled for your IP on this site. No other explanations so far...

[DawEdhel]

If only, but I'm getting cloudflare here as well so far.

[Ac314]

Can you check what cookies RSS Guard is sending to the site while fetching the feed?

[martinrotter]

I personally managed to fetch it successfully as I mentioned how in my post. And in my understanding, a metadata fetching should behave the same as a normal feed fetching minus discarding its content/body part.

Which it does. Exact same C++ method for fetching is called.

[DawEdhel]

Can you check what cookies RSS Guard is sending to the site while fetching the feed?

I don't think the app was sending any cookies, when I managed to fetch successfully on a first try with a clean new setup/profile.
And I'm not sure if the app provides such info, unless you meant using an external sniffer.

Which it does. Exact same C++ method for fetching is called.

Long story short, I finally managed to figure out, why I had different behaviors here between old/current and new setups:
as a portable mode user, my update process consists of downloading a new version and unpacking/overwriting an app root folder on top of the older one. So when I decided to compare folders' structures (except a data4 one), I'd found a single difference I had was an extra file "tls/qopensslbackend.dll" in my current setup, and moving out/deleting it fixed my problem, as now I can properly discover this feed on the current setup, getting the same result as on the new one.

Looks like this file was an leftover from previous versions, and my updating process can't mitigate such situation. Can you give more info to sate my curiosity, what was this file for (I understand it's a part of QT) and why it was removed, if it were present in the past? As per my understanding, having this file makes the app going a different route somewhere, concluding to an error in some cases.

Also I'm not sure completely deleting an old version manually before updating is the right way. Maybe there should be a process during a new version's first launch to validate current root folder's structure.

[touwys]

@DawEdhel

#1717 (comment)

Your recommendations worked fine for me. Also learned a lot from you all in this discussion — thanks, again.

Settings: Daily Investor

•

•

•

•

•

•

•

[touwys]

@OnlineProxy-io

Thank you.