Misc NFT - Anybody here provide (or know of ) data scrubbing services? Referral $$$ paid

**Trivers** · 10-17-2014, 03:02 PM

Would prefer to spend $$$ with fellow CPers.

We have 43 databases averaging 50K each that need to be cleaned up.

Seeking consultant.

Referral $$$ paid to your paypal account

Thanks

58-4ever · 10-17-2014, 03:08 PM

Let me know what kind of services you need. Are you in the KC Metro?

stumppy · 10-17-2014, 03:08 PM

I got a gal that comes by when I call and cleans my pipes. She's pretty talented, I'm sure she could clean your databases too.......while wearing a french maid outfit.

58-4ever · 10-17-2014, 03:14 PM

Shoot me a PM

stonedstooge · 10-17-2014, 03:15 PM

Can we keep your porn?

**Rain Man** · 10-17-2014, 04:13 PM

Cleaned up how?

CaliforniaChief · 10-17-2014, 04:15 PM

TLO · 10-17-2014, 04:17 PM

This thread had potential.... but alas.

**srvy** · 10-17-2014, 05:09 PM

Better call Saul.

**srvy** · 10-17-2014, 05:12 PM

If its really bad The Wolf.

**AustinChief** · 10-17-2014, 05:14 PM

Gonna need more details. Are you looking at something that could be automated or something that requires a manual once-over?

**Trivers** · 10-24-2014, 03:01 PM

Thank you for your responses.

Here are the details:

We are sending emails to list of insurance agents in all fifty states.

The databases are public records.

The first list we need cleaned up is from WisCONsin. (How the natives pronounce it.)

http://oci.wi.gov/agentlic/agntlist.shtml

Scroll down to bottom of page:

Format 2 - Agents by Company Appointments

There are 17 files. First one is al_1st-al.exe. We figured out how to separate columns, and remove the records without email addresses.

1) We need all the first and last name duplicates removed; and 2) words turned into lower cases.

We would prefer a way to automate this whole process.

If interested, please PM.

Thanks

unlurking · 10-24-2014, 04:20 PM

*1) uniq file > newfile
2) tr '[:upper:]' '[:lower:]' < file > newfile

*(Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?)

**Trivers** · 10-24-2014, 04:34 PM

Quote:

Originally Posted by unlurking

*1) uniq file > newfile
2) tr '[:upper:]' '[:lower:]' < file > newfile

*(Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?)

Good catch! Yes.

Thank you!

unlurking · 10-24-2014, 04:51 PM

If you have a bash shell, this will work...

Code:

cat infile | cut -c101-135,136-160,350-420 | tr '[:upper:]' '[:lower:]' | sed 's/ \s\+/,/g' > outfile
uniq infile outfile

EDIT: This was tested using the "al_1st-al.exe" file from the site you linked to. Only 8 dupe lines are dropped.

10-17-2014, 03:02 PM
Trivers Always ask 'Why?" Join Date: Oct 2011 Location: Two Rivers, Wisconsin Casino cash: $2342400	NFT - Anybody here provide (or know of ) data scrubbing services? Referral $$$ paid Would prefer to spend $$$ with fellow CPers. We have 43 databases averaging 50K each that need to be cleaned up. Seeking consultant. Referral $$$ paid to your paypal account Thanks
Posts: 2,929

10-17-2014, 03:08 PM	#2
58-4ever The Collective Unconscious Join Date: Sep 2005 Location: The OP Casino cash: $9520422	Let me know what kind of services you need. Are you in the KC Metro?
Posts: 10,425

10-17-2014, 03:08 PM	#3
stumppy New and Improved Join Date: Mar 2002 Location: Springfield, Mo. Casino cash: $1508590	I got a gal that comes by when I call and cleans my pipes. She's pretty talented, I'm sure she could clean your databases too.......while wearing a french maid outfit. __________________ “As a nation, we can endure damaging policies for a four-year term. But we cannot survive a president willing to terminate our Constitution”
Posts: 21,082

10-17-2014, 03:14 PM	#4
58-4ever The Collective Unconscious Join Date: Sep 2005 Location: The OP Casino cash: $9520422	Shoot me a PM
Posts: 10,425

10-17-2014, 03:15 PM	#5
stonedstooge Arf! Arf! Arf! Join Date: Dec 2007 Location: North Casino cash: $6074173	Can we keep your porn? __________________ 2013 ADOPT A CHIEF Eric Kush "Often is a word I seldom use"--JP
Posts: 25,283

10-17-2014, 04:13 PM	#6
Rain Man NFL's #1 Ermines Fan Join Date: Jul 2001 Location: My house Casino cash: $3338491 VARSITY	Cleaned up how? __________________ I'm putting random letters here as a celebration of free speech: xigrakgrah misorojeq rkemeseit.
Posts: 141,338

10-17-2014, 04:15 PM	#7
CaliforniaChief #RIPAce Join Date: Jan 2009 Location: So Cal Casino cash: $12422899
Posts: 17,319

10-17-2014, 04:17 PM	#8
TLO Life is changing.. Join Date: Jul 2012 Location: NW Missouri Casino cash: $1934900	This thread had potential.... but alas.
Posts: 38,902

10-17-2014, 05:09 PM	#9
srvy Fight, build, win! Join Date: Sep 2004 Location: KC Casino cash: $1691250	Better call Saul.
Posts: 24,353

10-17-2014, 05:12 PM	#10
srvy Fight, build, win! Join Date: Sep 2004 Location: KC Casino cash: $1691250	If its really bad The Wolf.
Posts: 24,353

10-17-2014, 05:14 PM	#11
AustinChief Administrator Join Date: Aug 2000 Location: Austin Casino cash: $2689112	Gonna need more details. Are you looking at something that could be automated or something that requires a manual once-over?
Posts: 19,495

10-24-2014, 03:01 PM	#12
Trivers Always ask 'Why?" Join Date: Oct 2011 Location: Two Rivers, Wisconsin Casino cash: $2342400	Thank you for your responses. Here are the details: We are sending emails to list of insurance agents in all fifty states. The databases are public records. The first list we need cleaned up is from WisCONsin. (How the natives pronounce it.) http://oci.wi.gov/agentlic/agntlist.shtml Scroll down to bottom of page: Format 2 - Agents by Company Appointments There are 17 files. First one is al_1st-al.exe. We figured out how to separate columns, and remove the records without email addresses. 1) We need all the first and last name duplicates removed; and 2) words turned into lower cases. We would prefer a way to automate this whole process. If interested, please PM. Thanks
Posts: 2,929

10-24-2014, 04:20 PM	#13
unlurking MVP Join Date: Aug 2003 Casino cash: $7737309	1) uniq file > newfile 2) tr '[:upper:]' '[:lower:]' < file > newfile (Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?) __________________
Posts: 10,620

10-24-2014, 04:51 PM	#15
unlurking MVP Join Date: Aug 2003 Casino cash: $7737309	If you have a bash shell, this will work... Code: cat infile \| cut -c101-135,136-160,350-420 \| tr '[:upper:]' '[:lower:]' \| sed 's/ \s\+/,/g' > outfile uniq infile outfile EDIT: This was tested using the "al_1st-al.exe" file from the site you linked to. Only 8 dupe lines are dropped. __________________ Last edited by unlurking; 10-24-2014 at 06:27 PM..
Posts: 10,620