Finding and Fixing "Duplicate" Resource Entries

Submit ideas and suggestions on how we display, catalogue and export the resources.

Moderator: Forum Moderator

User avatar
DeathTrooper
Novice Crafter
Posts: 45
Joined: Mon Mar 08, 2010 1:22 am

Finding and Fixing "Duplicate" Resource Entries

Post by DeathTrooper » Fri Apr 09, 2010 7:13 pm

It was suggested I bring this to Sobuno's attention, and if others feel like doing some "spring cleaning" this might be a good place to list "goofs" you find. It started when I openly admitted to the one (and only) lousy attempt I made at entering a resource, in which I had all info right except the name was off by like 1 letter. I did some "Find Resource" searches...

http://www.swgcraft.org/dev/find_resources.php

...for Bloodfin chemicals in hopes of finding my own goof, and came across a few others in the process. The "clues" I looked for were: 2 entries with the EXACT same stats but only TINY differences in naming, suggestive of a simple typo. Like I had done with my goof, it looks like some other errors were set to "unavailable" which, as pointed out to me, is not the correct way of "fixing" a bad resource entry.

So, I'm starting this thread so I (and others) can list the "goofy" looking entries so others can help confirm which (if any) is actually in error. Is there really a chance 2 resources could have names that are only 1 letter apart, AND have ALL of the EXACT same stats? I guess with "millions" of resources it is possible, but to me these look very suspicious.

Sobuno, if this catches your eye, maybe you can make a suggestion as to how to deal with these, but for now I'm playing it safe and just posting what I found in hopes that it helps.


User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Tue Apr 13, 2010 5:03 pm

I can't think of any way I can deal with this, this is mostly an issue for the players at the various galaxies.

All I might be able to do here is query the database for similar names with identical stats. Then again, I might not even be able to do that.

User avatar
Monty Burns
Master Crafter
Posts: 549
Joined: Sat Mar 08, 2008 9:26 am
Location: New Zealand

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Monty Burns » Tue Apr 13, 2010 8:18 pm

Would it be possible to run a query on identical stats by resource type (I mean Steel rather than each type of steel) for a server then post the list for each server and let people sort them out?

I would be more than happy to give it a shot for Sunrunner but I am less enthusiastic about going through the database item by item myself.

Despite the amount of data there I cant imagine it is a huge list.

User avatar
DeathTrooper
Novice Crafter
Posts: 45
Joined: Mon Mar 08, 2010 1:22 am

Re: Finding and Fixing "Duplicate" Resource Entries

Post by DeathTrooper » Wed Apr 14, 2010 12:37 am

In the search screen i just chose the type, server, and then sorted by the first stat. Without looking at names I could easily pick out identical stats, and it did not take me long to go through the Bloodfin chemicals. All the different metals would indeed be a chore. One thing that would have made it easier, would be to get a longer list of search results shown on screen. I did not see an option to set number of results shown, but I may have overlooked it. I don't have the 30k resource crate, but I can at least search BF vendors for some of the listings and confirm which ones do exist. I guess a question I have, is it possible that 2 similar resources could have identical stats?


User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Wed Jul 21, 2010 3:17 am

I have set my own computer to compute a list of resources that have identical stats and are of the same type (Specific type, not general type, that would take even longer).

Suffice to say it will be quite a few hours before it finishes; the server "SWGCraft.co.uk" took 22 seconds. Phew, then it's only going to take 22 seconds times the number of servers, right? Well, SWGCraft.co.uk only had 900 resources, the real servers vary between 15.000 and 65.000 resources, and I very much doubt that the time increases linearly...

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Wed Jul 21, 2010 10:50 am

7,5 hours later and it's still running, good thing this machine is multi-cored, the database server has been using 100% of one CPU the entire morning.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Wed Jul 21, 2010 1:38 pm

10 hours and it's done! I'll clean up the results and post them later, see if we can squash some duplicates :)

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Wed Jul 21, 2010 8:16 pm

There are 4023 pairs of duplicates in the database for the active servers, of which less than a hundred have multiple duplicates. The worst ones exist in pairs of 4, example:
Omnireosrebaine
Omnieosrebaine
Omineosrebaine
Omnineosrebaine

I am considering what the best way to publish these duplicates is, any suggestions?

User avatar
Monty Burns
Master Crafter
Posts: 549
Joined: Sat Mar 08, 2008 9:26 am
Location: New Zealand

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Monty Burns » Wed Jul 21, 2010 11:48 pm

Sobuno wrote:There are 4023 pairs of duplicates in the database for the active servers, of which less than a hundred have multiple duplicates. The worst ones exist in pairs of 4, example:
Omnireosrebaine
Omnieosrebaine
Omineosrebaine
Omnineosrebaine

I am considering what the best way to publish these duplicates is, any suggestions?

Would it work to post them by server and let people go through them and sort them out?

Maybe do it like the schematics updating process.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Thu Jul 22, 2010 12:21 am

This is all I can offer at the moment: http://swgcraft.org/dev/dup_res.php?server=ID

Where ID is

1: Ahazi
2: Bloodfin
3: Bria
4: Chilastra
5: Chimaera
7: Eclipse
8: FarStar
9: Flurry
10: Gorath
17: Radiant
19: Shadowfire
20: Starsider
21: Sunrunner
100: TCPrime
101: Test Center

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Thu Jul 22, 2010 12:23 am

Actually, wait a minute and I'll develop something that will at least allow you to return a list of ID's that need to be deleted.

User avatar
Sobuno
Developer
Posts: 2589
Joined: Sun Mar 25, 2007 2:17 am
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Sobuno » Thu Jul 22, 2010 12:34 am

Okay, the link above now has a checkbox next to each individual resource as well as a submit button at the bottom. Check all resources that do not exist and press the submit button when you have done all you want; this will result in a list of ID numbers to you that you should then give me in some way or another. I'll then do a mass-delete of resources at some point.

User avatar
Zimoon
Forum Moderator
Posts: 4817
Joined: Mon May 14, 2007 6:55 am
Location: Stockholm, SE
Contact:

Re: Finding and Fixing "Duplicate" Resource Entries

Post by Zimoon » Thu Jul 22, 2010 9:25 am

The page with check-boxes is GREAT ........... :mrgreen:


However, please add resource class to the right of each name-blob, that way we can sit with the 30k kit and tick them off without clicking to-and-fro that much.

And add the ID and names to the index page too, perhaps.

If I had been really nasty I would ask for them sorted the same way as in the resourcefile.xml, that way even further have them aligned with the 30k kit, but I am fine with this and want you to enjoy the summer, the sun and the girls 8)

/Zimoon

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests