01-09-2003 21:37									#1
	Programmers please read. Help yourself to an AD-FREE life

I have an annoying problem with my latest piece of research, which is an empirical study (please don't stop reading - I'm getting to the programming part ).

I have a main dataset which attaches a postcode (like zip code) district to each individual. I then have other datasets which I will map into the main one. Unfortunately many of the postcodes (which are saved as strings) in the main set have spaces at the start, and since [(space)B1] is not equal to B1 I can't merge my sets.

Now I have converted the string to an ascii file, and of course I can remove the spaces and reconvert back into my econometric package. But there are about 100,000 offending spaces (stop laughing, it's not funny!). Glancing at the rest of the data it seems there will be further space related problems.

Attached is a .txt file, with a small sample of the observations on individual lines between quote marks. Essentially I need an automated way to remove all spaces within each set of quote marks, so that "(space)(space)B1" becomes "B1"

It's such a trivial problem I'm sure a programmer can suggest a solution straight away. I hope so anyway.

Attachment: juvos.txt
This has been downloaded 10 time(s).

	01-09-2003 21:40									#2
	Browse Apolyton AD-FREE

I was going to say 'how are you going to learn if you just want solutions straight up', but then again I realized I might get into a problem I can't solve at all, and I might ask help here, so you know.. I'd be digging my own grave .

	01-09-2003 21:47									#3
	Support Apolyton, buy Galactic Civilizations: Deluxe Edition

Don't most word processors have a "Find and Replace" feature? Would that help at least a little?

	01-09-2003 21:48									#4
	Support Apolyton, buy Galactic Civilizations

Post the entire thing with a fixed example line. I'll use Emacs's query string replace to remove the spaces. I just tested on juvos.txt and it worked.

	01-09-2003 21:53									#5
	Support Apolyton, buy Alpha Centauri

Find and replace I did think of, but there are still 3,000 different postcodes in each set. Geeslaka's comment looks more promising.

Geeslaka: As I said the dataset is huge (even the text file with all the observation on this variable is 2.5MB) so I can't attach it. Also I will face more space related problems with the sets I am mapping in, I am sure.

Hence can you explain this method of yours? Or is it something a humble economist will not be able to do?

	01-09-2003 21:55									#6
	Increase Your PM Length

quote: Originally posted by mrmitchell Don't most word processors have a "Find and Replace" feature? Would that help at least a little?

That would be the simplest way - just import the file into word, trim out spaces (replace "(space) with " ) and be done with it quicker than you could write a chunk of code to do the same thing.

If this was going to be a constant problem with many new data sources, and the program had reusability, then I'd code something to clean up those files.

	01-09-2003 21:56									#7
	Got spare money?

If you wish I can write a short pascal prog to fix it.

	01-09-2003 21:56									#8
	Help yourself to an AD-FREE life

quote: Originally posted by DrSpike Find and replace I did think of, but there are still 3,000 different postcodes in each set. Geeslaka's comment looks more promising. Geeslaka: As I said the dataset is huge (even the text file with all the observation on this variable is 2.5MB) so I can't attach it. Also I will face more space related problems with the sets I am mapping in, I am sure. Hence can you explain this method of yours? Or is it something a humble economist will not be able to do?

What language / DB are you using?

	01-09-2003 21:57									#9
	Inflate your Upload Space

Or one could use emacs if he has such a thing.

	01-09-2003 22:04									#10
	Increase Your PM Length

M-x query-replace-regexp
Is each line simple " B2", or is there more after that? If you want to remove all spaces, not just the leading ones, MtG's suggestion will work.

	01-09-2003 22:05									#11
	Increase the size of your Attachments

quote: Originally posted by MichaeltheGreat What language / DB are you using?

I should reiterate I am an economist/econometrician.

What is emacs?

	01-09-2003 22:10									#12
	Suffering from ads?

quote: Originally posted by geeslaka M-x query-replace-regexp Is each line simple " B2", or is there more after that? If you want to remove all spaces, not just the leading ones, MtG's suggestion will work.

Some have 2 leading spaces, and some 1. Other have spaces in the middle, which I may or may not want to remove, I'm not sure yet.

	01-09-2003 22:10									#13
	Tired of ads?

it's allows for very advanced and configuarble search and replace queries.

But you have to have unix or some unix-like tool to use it.

	01-09-2003 22:15									#14
	Support Apolyton, buy GURPS/ Alpha Centauri

Since you do not have Emacs and it would take longer to install it, email the text file to me with one line fixed. I just emptied my box so it should fit.

	01-09-2003 22:27									#15
	Put an end to popups!

Geeslaka: Thanks, I will certainly take up that offer if a find and replace will not work. However there will probably be more requests later.

Let's assume for a second I want to remove all the spaces, no matter where they are. It has been suggested that MtG's idea would work in this case, but how do you remove the spaces with one find and replace?

Example:

"(space)(space)B1" must become "B1"
"(space)B12" must become "B12"

There may be spaces in the second/third position or both as well for some codes.

I can see how to replace "(space)(space)B1" with "B1" but I have thousands of finds and replace to do in that case, right?

	01-09-2003 22:33									#16
	Support Apolyton, buy Civilization: The Boardgame

If you can find something with regex find and replace, use this for the regex.
(space)*B.*
* is a wildcard that means "zero or more of the previous character"
. is a wildcard that means "any character"
That will match both of those cases.

	01-09-2003 22:34									#17
	Put an end to popups!

Just expot it into Access,
write a small VBA Solution and then reconvert it into Text

If it is in a compatible Format to convert it (i.e. all Fields are separated, for example by Commas or by Tabs) there should be no Problem

	01-09-2003 22:35									#18
	Put an end to popups!

Hold that..........I can see how 2 separate replace "(space) with " can solve the problem for the main 2 cases. I can also see now that removing all spaces with find and replace is easy (I didn't think before it woudl work).

If there is a more complex problem I'll bug Geeslaka.

Cheers all.

Doc

Last edited by DrSpike on 01-09-2003 at 22:43

	01-09-2003 22:45									#19
	Enter the AD-FREE zone

* waits to be bugged *
It's the monday after Labor Day so I have no school and lots of better things to do that need ignoring.

Edit: typo

Last edited by geeslaka on 01-09-2003 at 23:17

	01-09-2003 22:50									#20
	Lose 30 kilos (of popups)

Well I've finished for the day now. I'll check tomorrow and see if I need a "clever" solution. Right now I think the blunt method will work. I had no idea find and replace would recognise and remove (space) without any unwanted side-effects, which of course there would be in a normal setting - I was envisaging doing a find and replace for each separate code.

And I said "bug". There are other people around here who can help you with your other problem.

	01-09-2003 23:10									#21
	Dr. Spike Support Apolyton buy from Amazon

If you need to remove leading spaces only, then you will need to install some more software on your machine. There are a number of choices:

1) Emacs for PC
2) Perl or some other scripting language (one of us can send you an appropriate script to run)
3) Customer program (one of us can write a simple program and compile it to a windows executable and send it to you - this may require installation of a dll too).

	01-09-2003 23:22									#22
	Support Apolyton, buy Civilization III: Complete

Open file in Word, select all, turn into list, turn list off again... Tadaa! All leading spaces are gone.

Or get a decent program that can do regular expressions.

Edit: major crosspost, and I didn't see there was a quotation mark before it... oh well

Last edited by Mercator on 01-09-2003 at 23:35

	01-09-2003 23:58									#23
	Support Apolyton, pre-order Civilization IV

Spike: What you are looking for is just one line of code. I would ship it off to geeslaka, because of what pchang mentions. It shouldn't take him but 5 minutes to do.

	02-09-2003 03:24									#24
	Suffering from ads?

As I said thanks all, but I think I can use MtG's suggestion. If I need something more clever I will ask one of the people who have kindly offered to help out.

	02-09-2003 15:04									#25
	Support Apolyton, buy Galactic Civilizations: Deluxe Edition

In order of preference:

1. Perl, awk or any similar tools that support regex
2. BASIC (nice built-in string handling capabilities)
3. Turbo Pascal/Delphi/Klylix (very handy, not quick-and-dirty for this situation, though)

	02-09-2003 15:06									#26
	Got spare money?

If you are interested, you can investigate Cywin for future uses