An interesting experiment with this board's content

[G0ddard B0lt]

This is a technical experiment with the content from this board.

As you guys may know, this forum software (SMF) has a "Who's Online" display. (It's a link at the bottom of the home page of the forum in the "Info Center" block under a heading labeled Who's Online.)

Ordinary users see only a cut down version of this display without IP addresses (for privacy's sake).

A moderator or admin sees something that looks like the following:



Mostly this displays bots and search engines constantly visiting the forum. It's also always changing. But if you look at this image you'll see a couple of lines in the "Action" column labeled "Viewing the topic".

I've noticed that during the day these "topic" lines can be fairly numerous - perhaps 8-10 on each viewing.

Today I realized something significant: the visitors viewing these topics MAY be an expression of interest from real visitors, which may be funnelled through some search engine that is spidering the site.

I've noticed that when I click through these linked articles, they're often fairly classic, lengthy threads we had here a few years ago.

This might be a key to understanding how to attract more users to this site: monitor which threads are being requested the most.

So, I just built a simple shell script that is running every 10 minutes on my Linux desktop as a cron job:

1) It requests the URL of "Who's Online". (using wget)
2) The text from that page is run through grep and sed, in order to locate all lines containing the "viewing the topic" text and then strip out the URL.
3) The URLs being collected on each pass are appended, along with a time stamp, to a file containing the collected links.

I could put the script on one of my internet hosting servers to run 24 hours a day,  but at night the activity dies down and therefore having it run while my PC is active is just fine.

I'm noticing that almost every thread being viewed by bots, search engines or whatever is pretty high quality and interesting to revisit.

If anyone is interested I'll post a cleaned up list to the private section.

[unix]

Hm, interesting. I didn't realize that.

[G0ddard B0lt]

Here's an example of the output. It's not live because I manually uploaded it but it's for my own amusement so no need to make it evergreen.

You can go to any of the URLs in this file.

http://posteddocs.nfshost.com/links.txt