eftychia: Me in kilt and poofy shirt, facing away, playing acoustic guitar behind head (Default)
Add MemoryShare This Entry

Simultaneously frustrated/relieved/worried. Head still in tech-mode. Not bothering with English until brain starts doing English by default again. Laugh, sympathize, or just skip to English part.

Got up, fed the cat, went to check my mail ... no response from tipton (main Lunix box). Went downstairs, plugged a monitor in, and tipton was dark. No response to anything I did to the keyboard. Pinged tipton from wood (name server); nothing. LED on Ethernet card was lit. Other machines could see each other. Make "ew yuck" face and power-cycle tipton. During boot, fsck comes up complaining -- expected. Then fsck complains about errors it can't fix in automagic mode, so I have to come up single-user and manually fsck root. Short reads! %wince% Badness. Bunch of inodes trashed -- what else to do? Why bad blocks now? (And on fourth machine in house to develop them.) Feh. Boot. Some services fail to start. No NFS. Why no NFS? Critical binaries broken in root? NOOOOOO! Try manual mount. "RPC: Port mapper failure. Unable to receive." WTF? Ping richards (file server). Can see richards. Richards can see tipton. Can telnet to ISP from richards. Check mail. Realize if quarter-hour mail cron job doesn't run soon, will exceed quota on ISP. Can wood see NFS dirs on richards? Yep. NFS ok on server then, right? Still RPC error message from mount on tipton. Check /etc/exports, good. Check ps ... seem to be processes missing (on richards). Try /etc/rc.d/init.d/nfs stop (& start). Some items fail to stop. Some items fail to start. Confused. Confusedconfusedconfused. Annoyed. Go upstairs to boygeorge and rupaul (WinNT, Win95) to Google for error message. (Because 'man -k' unhelpful.) Meantime run mail download cron job manually on richards to make sure quota doesn't overflow. Note more missing processes on richards ... but why is wood still able to see /home, /shared, and /var/spool/mail if richards broken? WTF? Bang head on kybd. Check /var/log/messages on richards for clue; figure skip down to approx. time of last time cron fetched mail on tipton (3:09 this AM). NOTHING SINCE NOVEMBER IN MESSAGES?!?!?! Badbadbad (what's the point of syslogd then?) Do ps ... syslogd running! Why no messages? Kill syslogd, restart syslogd, messages from half hour ago suddenly appear, but no messages from 3:09 AM. Data not clue-worthy. Badword. Try stuff on richards. Get SAME error msg (Port mapper unable to receive) on RICHARDS when trying to restart a daemon. WTFingF? Google finds mailing list archives w/similar questions but less mysterious; says to make sure expected stuff (lockd, mountd, nfsd, statd) running, but those are what complain. Try to find commandname/filename of port mapper, since nothing apropriate is in ps list. Finally restart portmapper on richards. Try 'mount -a' again on tipton. Silent. Look at mounted filesystems: success. *whew* Still worried LAN is getting "fragile", concerned about badblocks on tipton:/dev/hda7 ... wondering what I can do about upgrades etc. in next few months. Has morning's head-against-keys-banging broken NFS connections from wood? Check. No, still mounted. Didn't touch samba or netatalk, so maybe no Windows problems? Boygeorge still sees server ... rupaul still sees server ... carlos (Mac in bedroom) still sees server (have to use directory other than $HOME to check 'cause OS9 finder takes forever to read folder with that many files in it). *Whew* Really did not want to bring richards all the way down and reboot -- really fugly kludge for that box; PITA to boot. (Ok, boots okay; getting it back to serving everybody is the PITA.)

Fine. Can start day now. Still grumbly. (And fingers so cold ... *pout*) Shift gears. Think about other tasks. English. Yeah, right. Do I feel like dealing with email yet? Just want to complain about how I didn't need a surprise like this (but realize that it turned out to be not as bad as it looked when I first noticed tipton was dark, so I'm as upset about the fear as anything else). So I post to LiveJournal to get it out of my system and put off what else I'd planned to write, and try to figure out what to eat for breakfast and what I need to do outside the house while I still have a useful amount of energy left.

Good morning afternoon.

There are 3 comments on this entry. (Reply.)
 
posted by [identity profile] juuro.livejournal.com at 09:43am on 2003-12-18
You write admirable prose even when it is in the mentalese-variant-English.
 
posted by (anonymous) at 10:52am on 2003-12-18
Hi Glenn,

Our main server (hosts www.steeds.com, our email, file server, etc.)
recently choked and had serious fsck issues when I rebooted. /var/log
was no longer a directory... Ack! I was able to find most of the files
in /vat/lost+found, but I decided it was time to upgrade to a version
which supported journaling (in my case ext3 filesystems, though I imagine
that rieserfs, jfs, xfs would all work as well). I also switched
to more stable (but slower) hardware (Athlons have given me tons of heat
problems).


Also, a central logserver is a good thing. Make it the most secure and
stable host you've got, and a bit of disk space helps too (a GB or two
is plenty!). You'll need to change the default syslogd command-line in
your rc script to allow incoming syslog messages from certain hosts,
open the appropriate port in your firewall (if running local to that host), and change the /etc/syslog.conf on the other hosts (there's even a util for Windows that will forward it's log events to a syslog
server).


- Vicky

 
posted by [identity profile] badgerthorazine.livejournal.com at 03:14am on 2003-12-19
*big hugs* Crazy, man. Stay warm, y'hear?

Links

January

SunMonTueWedThuFriSat
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24
 
25
 
26
 
27
 
28
 
29
 
30
 
31