miscellaneous

I've decided to make my own website like Eternals. Where it shows your hourly, daily, etc of your wins, losses and such.

It's pretty basic so far, but it doesn't need to be over the top. The way it works, it goes through a list of clans in the database (only 5 right now), and then, goes to each profile, pulls the 20 members that are scoring, and saves the data for later.

Main Page : http://i.imgur.com/n9F5Tjk.png
Individual Clan page : http://i.imgur.com/a6tvaKc.png

Lots left to do, stats to show, etc. Only a couple of hours of data have been stored so far. It's pretty smart, Ed could change around the layouts and it should still pick up the information as it relies on the "classes" of each element instead of brute force line counting like http://botsunauthorized.net does. I may put more clans, I may only have mine. Not sure how I want to go about it as it takes about 3-4 seconds per clan to go through the info. I may be able to optimize the code even more. Not sure where I'll host the site either as I don't want to buy another domain just for this. I have to many already at $16 a year.

Ideas/suggestions are always welcome, thanks!

the pic showing eternal is off - seems like the stats are shifted down by one row

you could make it so every clan has to sign up to get there data collected, seen it work like that with similar sites for other games.

stats, always +1 :)

the pic showing eternal is off - seems like the stats are shifted down by one row

I'm trying to figure that out, thanks for the catch Shoegazer.

you could make it so every clan has to sign up to get there data collected, seen it work like that with similar sites for other games.

I like this idea. :)

Cool, thanks for setting this up! It's been my hope for awhile that someone else would step up as the bots4 historian. In my long-term vision of the game, it would have this sort of data built-in, but that's a very long ways off. And I apologize in advance for when I inevitably break your scraping, but it sounds like it will at least be pretty easy to update if it's just referencing CSS classes.

Not sure how I want to go about it as it takes about 3-4 seconds per clan to go through the info. I may be able to optimize the code even more.

I assume it's the part that talks over the network that's slow, not the actual parsing of the page and updating of your database? If so, this is the classic crawler concurrency problem: If you want to optimize it, launch each of the crawlers in a separate thread and have them put what they find onto a work queue, then have a separate thread for processing that queue. Just please use a sane limit on the number of simultaneous crawlers as to not DoS the game. :)

That's the general concept. I assume you're using PHP for this and though I haven't used it myself, I came across the curl_multi_* set of functions which seem to do what you want. This library may be a useful abstraction on top of them.

Not sure where I'll host the site either as I don't want to buy another domain just for this. I have to many already at $16 a year.

Back when I was experimenting with stuff more, I had a single domain and just threw each project into its own directory, e.g. /bots or /game. You can always move something to a standalone domain later on if it gets popular and you decide that's worth it.

Also as the general concept is still;

• Botrow
  • botrow-wins
  • botrow-losses *etc

It should work just fine.

I'd assume it's the network because I pulled a source of the clan page and saved it on my localhost. It ran under a second. I'll see about dividing it up.

I've heard of Curl, but haven't played with it. Thanks for the info!