eftychia: Me in kilt and poofy shirt, facing away, playing acoustic guitar behind head (Default)
Add MemoryShare This Entry
posted by [personal profile] eftychia at 12:12pm on 2003-06-01

Every so often when I'm working with a large spreadsheet or database under Windows, I'm tempted to export the whole thing to a comma-delimited text file, whack it with a few 'vi' commands on a Linux box, then re-import it, because it'd be so many fewer keystrokes. Or I just find myself wanting to 'grep' an Excel spreadsheet. Slightly less often, I find myself wanting to use 'uniq' or other tools. Even 'sort', if I want to sort on a fragment of a column. Sometimes I want to do things that there's really no good way to do in the fancy package I'm using; other times there's a perfectly good way that only feels like too much work because I know how easy it'd be in my favourite text editor. (I'm sure Emacs users (the heretics!) say pretty much the same thing.)

For all the data manipulation and calculation features of these modern packages, sometimes I still need the very slick Raw Power of thirty-year-old "simple" text-based utility programs. There's so much to be said for tools that just let you concisely tell them what to do, and let you build complex operations out of a bunch of very simple commands. I don't want a tool that tries to "do everything" because there's something I'll want to do that the designers won't have thought of. I want tools that let me do everything as long as I can describe what I want to do.

(So why don't I export-to-text/whack-with-vi/re-import? Sometimes I do. Other times I don't want to lose my formatting, or I've got a bunch of formulas or lookups into other tables, which wouldn't survive the export/import process well. Now a version of Excel that understood 'vi' commands would be wicked cool.)

Mood:: distracted
There are 6 comments on this entry. (Reply.)
 
posted by [identity profile] silmaril.livejournal.com at 09:00pm on 2003-06-01
You just basically outlined why I do all the grading/spreadsheet processing in MATLAB, even if they pass the data to me in EXCEL.
 
posted by [identity profile] dglenn.livejournal.com at 09:47pm on 2003-06-01
Yup. Actually, the same reasoning has to do with why I've got a few relational databases implented as Excel spreadsheets linked with hlookup() and vlookup() functions instead of being in a proper RDBMS -- there were just enough things I wanted to do that Excel handled more easily than the database managers I had handy at the time I started keying data in. (And then I hit the things that Excel doesn't do well either, and write a mini-rantlet like the one above.) Eventually I'll write my own data entry/manipulation front end and make it talk to MySQL or Postgress or something so I can have the efficiency of a proper DBMS, the power of SQL queries, and the "doesn't piss Glenn off" factor of a comfortable and powerful user interface for the data entry. But that's been on my to-do list for a while now.

Is MATLAB a collection of small, powerful tools you can link together in the UNIX cli tradition, or a big do-everything tool whose user interface involves a whole lot of small, powerful commands you can link together or script conveniently?
 
posted by [identity profile] silmaril.livejournal.com at 09:38am on 2003-06-02
Is MATLAB a collection of small, powerful tools you can link together in the UNIX cli tradition, or a big do-everything tool whose user interface involves a whole lot of small, powerful commands you can link together or script conveniently?

The latter. It's more a interpreter-based language than anything else, actually. Typical small MATLAB program (actually, excerpt from the grade processing program I have):



load gradelist
a=gradelist;

% calculating totals
for k=1:sno
t=0;
for l=1:qno
if l==1 t=t+0.2*a(k,l);
elseif l<=3 t=t+0.01*a(k,l);
elseif l<=11 t=t+0.005*a(k,l);
elseif l==12 t=t+a(k,l);
else t=t+0.35*a(k,l);
end
end
total(k)=t;
end

%writing expanded matrix, including totals
for k=1:sno
a2(k,1:qno)=a(k,1:qno);
a2(k,qno+1)=total(k);
end

%create average and std vectors
noans=a(sno+1,:);
noans(qno+1)=sno-empty;
for k=1:qno+1
x=a2(1:sno,k);
ave(k)=sum(x)/noans(k);
stds(k)=std(x);
end



[snip writing results to file parts]



netgrades=a2(:,qno+1);

edges=[0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100];
[myhist,mybin]=histc(netgrades,edges);
figure(1)
clf
bar(edges,mhist3,'histc')
grid on
axis([1 100 0 20])


It reads very easily. It's based on C, and you can incorporate C programs into it. It's got tools for almost every numerical mathematics thing you might want to do, and extra toolboxes for control theory, image processing, signals/systems etc.
 
posted by (anonymous) at 01:57pm on 2003-06-02
Despite the fact that I've loaded my machine at work
(running WinXP) with Emacs, tcsh, cut, awk, tr, grep, diff, and all that, it somehow still doesn't "feel" link Unix. Then I go home, and confident I can now paste by clicking my middle mouse button,
I feel truely "at home". I guess that's why Unix (in its many flavors) has been around 30+ years and isn't going away any time soon.


- Vicky

 
posted by [identity profile] dglenn.livejournal.com at 01:13pm on 2003-06-03
I used to make frequent use of the MKS Toolkit tools under MS-DOS (eventually using Desqview X, which I rather liked, back when the alternative was Windows 3.1, which wasn't worth my time). I found that they made my life under MS-DOS a whole lot easier, but I concur that they weren't enough to make the DOS box feel "like home". (Familiar enough, sure, since I was using MS-DOS well before I first got ahold of UNIX, but still...) For me, it's not stuff like using the middle button to paste, since I spend relatively little time in X and almost none of it sitting in front of a Linux or UNIX machine.

(The UNIX and Linux boxes are downstairs in the half of the living room I refer to as "the server room", because they don't much care whether I'm accessing them from their own keyboards or over the LAN. So most of what I do in X is using the Exceed X server under Windows NT (and I'm still looking for a copy of MacX). Yes, I get middle-button paste under Exceed, but I'm usually copying to or from a Windows app, so first I have to pick "copy clipboard to X selection" or vice-versa off a menu. The "feels like home" aspect of UNIX for me is mostly the command-line interface. All my favourite tools, and tcsh or csh. Perhaps part of my problem with MKS was that their shell was ... uh, either ksh or bash, I'm not sure -- ksh, I think.)

Fortunately, since I keep damn near everything on the file server, which the Windows boxes see via Samba, the Mac sees via Netatalk, and the other UNIX/Linux machines see via NFS, switching to UNIX just means clicking the mouse in one of the open Telnet windows.
 
posted by [identity profile] madbodger.livejournal.com at 06:16pm on 2003-06-02
I know how you feel. I've been seen using BBEdit to export a text file with Unix linebreaks so I can attack it with vi (under GLTerm). I did write a utility to convert CSV into XML, allowing me to attack excel data with XSLT (and then graphviz or OmniGraffle or something else). I have a lovely little PostScript wrapper that lets me make presentation slides (which I then convert to PDF, complete with clickable URLs). Seems to me I just don't need M$.

Links

January

SunMonTueWedThuFriSat
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24
 
25
 
26
 
27
 
28
 
29
 
30
 
31