Basilisk current data wiped?

Discuss galaxy specific issues.

Moderator: Forum Moderator

User avatar
Slyvampy
Lead Developer
Posts: 321
Joined: Sat Mar 24, 2007 10:00 pm
Location: England, UK
Contact:

Re: Basilisk current data wiped?

Post by Slyvampy » Wed Oct 17, 2018 8:58 am

The parser hasnt changed at all. I will look into this, the job that collects information from GH is once an hour. I have increased this to every two hours to see if it resolves the issue. (This might be as the job could be taking longer than one hour to complete - there are that many resources to check).
Slyvampy
Founder, SWGCraft.org
http://www.swgcraft.org

divory
Novice Crafter
Posts: 15
Joined: Tue Jan 26, 2016 5:42 pm

Re: Basilisk current data wiped?

Post by divory » Thu Oct 18, 2018 10:07 am

didn't help parser is still wiping all data.

User avatar
Savacc
Architect & Shipwright Forum Moderator
Posts: 1182
Joined: Mon Mar 26, 2007 9:45 pm
Location: Central Oregon

Re: Basilisk current data wiped?

Post by Savacc » Fri Oct 19, 2018 8:07 am

I thought we were only drawing data from GH once per day as they only update once per day and our hourly enquiries were overloading their system?

Also Divory is correct. The problem with Basilisk's current resources is still there. It only lists the "new resources" that spawn since the last time the parser runs and marks all other resources as "unavailable" even though they still have many days yet to go.

User avatar
Slyvampy
Lead Developer
Posts: 321
Joined: Sat Mar 24, 2007 10:00 pm
Location: England, UK
Contact:

Re: Basilisk current data wiped?

Post by Slyvampy » Fri Oct 19, 2018 10:18 pm

Ok, I have changed it to twice a day see if that makes a difference. I am confused as the the file was changed was 22nd of July 2016. Nothing has changed our end, unless GH changed anything we are not aware of.
Slyvampy
Founder, SWGCraft.org
http://www.swgcraft.org

User avatar
Savacc
Architect & Shipwright Forum Moderator
Posts: 1182
Joined: Mon Mar 26, 2007 9:45 pm
Location: Central Oregon

Re: Basilisk current data wiped?

Post by Savacc » Sat Oct 20, 2018 4:54 am

Yesterday and today's resources are there, for Basilisk's current resources. I guess we wait and see what happens tomorrow?

User avatar
Savacc
Architect & Shipwright Forum Moderator
Posts: 1182
Joined: Mon Mar 26, 2007 9:45 pm
Location: Central Oregon

Re: Basilisk current data wiped?

Post by Savacc » Sun Oct 21, 2018 5:32 am

We've got resources from three days now. Give us two more weeks of this and we will be back to normal :roll:

divory
Novice Crafter
Posts: 15
Joined: Tue Jan 26, 2016 5:42 pm

Re: Basilisk current data wiped?

Post by divory » Sun Oct 21, 2018 4:01 pm

should we mark available the stuff that has been inappropriately mark unavailable?

User avatar
Savacc
Architect & Shipwright Forum Moderator
Posts: 1182
Joined: Mon Mar 26, 2007 9:45 pm
Location: Central Oregon

Re: Basilisk current data wiped?

Post by Savacc » Mon Oct 22, 2018 6:56 am

Can you do that? If you can, sure, it's worth a try. If it takes an admin, let me know and I'll do it, or it least some resources to track.

Thing is, this is not how it is supposed to work. we should get a full report of all that GH has, each day. If it is not on GH's feed, it gets marked "unavailable". Slyvampy seems convinced it is a fault on GH's feed. He could be right. I have not read anything on the Emu forums saying anyone besides us has noticed this problem. That tends to make me think it is our error, but maybe not. Maybe we are the only ones using GH's feed and the only ones that will notice?

We are really waiting for Sobuno to look at this. He responded to an email with a request for more info, but that was week ago. He last logged on here a week ago Sat. So real life must be keeping him away :(

divory
Novice Crafter
Posts: 15
Joined: Tue Jan 26, 2016 5:42 pm

Re: Basilisk current data wiped?

Post by divory » Sun Oct 28, 2018 11:37 pm

the parser data seems to be 12 hours behind Galaxy Harvester. Also some spawns like the new phrik alum says its suppose to be available on talus and its not. Otherwise aside from 293 listed and SWGAide showing 291 and 12 hours off it seems to be mostly accurate. Anyone else noticing discrepancies?

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Basilisk current data wiped?

Post by Sobuno » Wed Oct 31, 2018 7:44 pm

Our GH Parser has not changed since 2016-07-22 as Slyvampy correctly has identified.

What follows is an outline of the logic it performs:

Preparation:
  1. Find all servers that have a GH ID associated
  2. Find all planets and make some changes to match the naming on GH (Space becomes underscore, the 4 in Yavin 4 becomes IV)
  3. Find all resources previously imported that are available and keep a list of them (Called the "missing" list - At the end, all resources still listed here are marked unavailable).
  4. Generate a list of files to fetch from Galaxy Harvester on the format:

    Code: Select all

    http://galaxyharvester.net/exports/current<GH id>.csv
A file looks like this:

Code: Select all

"name","galaxy_id","galaxy_name","enter_date","type_id","type_name","group_id","CR","CD","DR","FL","HR","MA","PE","OQ","SR","UT","ER","unavailable_date","planets"
"heapefe",14,"Basilisk","Mon, 29 Oct 2018 03:58:45 -0800","aluminum_agrinium","Agrinium Aluminum","aluminum",110,491,253,0,349,436,0,633,371,457,0,,"Corellia|Dantooine|Lok|Naboo|Rori|Talus|Tatooine|Yavin 4"
"meagio",14,"Basilisk","Tue, 30 Oct 2018 04:17:55 -0800","aluminum_agrinium","Agrinium Aluminum","aluminum",132,428,302,0,314,538,0,4,516,485,0,,"Dantooine|Dathomir|Lok|Naboo|Rori|Talus|Tatooine|Yavin 4"
"onaic",14,"Basilisk","Sat, 27 Oct 2018 04:03:18 -0800","iron_axidite","Axidite Iron","iron",315,192,584,0,609,135,0,11,642,542,0,,"Rori|Tatooine"
...
...
...
For each file fetched:
  1. Retrieve file
  2. Delete first line (It contains the column names, e.g. name, type, stats)
  3. For each following line, Check if a resource with the given name for that line already exists on the server:
    1. If it doesn't exist, check if the resource is planet-specific and add it to the appropriate planet. If not planet-specific, parse the planet names at the end of the line and add it to the planets we recognize.
    2. If it does exist, some different logic applies:
      1. Remove the resource from the "missing" list if present
      2. Add stats that are missing on a resource if GH has stats for fields we are blank on.
      3. Mark resource as imported
      4. If we already have the resource marked as available, we parse GH's list of planets it is available on and mark it available there as well. If we do not already have the resource marked as available, we do no such thing. This is in order to trust our own availability data more than GH's, otherwise any changes made to Availability on SWGCraft would be reverted the next time the Parser runs.
And finally, at the end, every resource still on the "missing" list will be de-spawned.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Basilisk current data wiped?

Post by Sobuno » Wed Oct 31, 2018 8:00 pm

Looking at the logs it generates, I do see that it sometimes had a fatal error (meaning the Parser stops that iteration of its run) while performing despawns - Our resource functionality was refering to some mailing client we used a long time ago that is no longer maintained. I have stripped that from our resource-related code now.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Basilisk current data wiped?

Post by Sobuno » Wed Oct 31, 2018 8:03 pm

Slyvampy wrote:
Wed Oct 17, 2018 8:58 am
This might be as the job could be taking longer than one hour to complete - there are that many resources to check.
I just started a run by hand - It took about 30 seconds to complete for all servers, so this shouldn't be the issue.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Basilisk current data wiped?

Post by Sobuno » Wed Oct 31, 2018 8:24 pm

divory wrote:
Sun Oct 28, 2018 11:37 pm
the parser data seems to be 12 hours behind Galaxy Harvester. Also some spawns like the new phrik alum says its suppose to be available on talus and its not. Otherwise aside from 293 listed and SWGAide showing 291 and 12 hours off it seems to be mostly accurate. Anyone else noticing discrepancies?
Regarding Oimoium and Talus, I see in the history on GH that it was added to Talus there (https://galaxyharvester.net/resource.py/14/oimoium) and then cleaned up the next morning. There is no special behaviour relating to the individual planets and removing faulty markings - I'll try taking a quick look at that situation as it should be somewhat easily fixed.

Update: The Parser now removes availability for planets marked as available by itself under the assumption that it was a mistake from someone at GH.

divory
Novice Crafter
Posts: 15
Joined: Tue Jan 26, 2016 5:42 pm

Re: Basilisk current data wiped?

Post by divory » Wed Oct 31, 2018 8:57 pm

think it just canned again. All the yavin stuff just went unavailable....Only on the SWGAIde tool. There was an update pushed to the tool about an hour ago and it took 30 of the flora and creature resources and marked them as unavailable. I would have never noticed but i have a few harvesters (heavy flora) on the wild corn Kausnili. When i checked the website the resource is still available.
SWG AIDE shows 327 total resources available and the web site shows 358 available.
I was able to "revive" the yavin resources with SWGAIDE and the numbers are back to 357/358 Although the spawn timer was reset its at least accurate on both.

User avatar
Slyvampy
Lead Developer
Posts: 321
Joined: Sat Mar 24, 2007 10:00 pm
Location: England, UK
Contact:

Re: Basilisk current data wiped?

Post by Slyvampy » Thu Nov 01, 2018 1:21 am

divory wrote:
Sun Oct 28, 2018 11:37 pm
the parser data seems to be 12 hours behind Galaxy Harvester.
This was as I changed the timer to be updated once every 12 hours as I changed it to see if it would solve the problem (see above posts). If someone at Galaxy Harvester or anyone can confirm how frequent they update their XML export we can change the timer to match this.
Slyvampy
Founder, SWGCraft.org
http://www.swgcraft.org

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest