sheepy.org2011-12-11T23:39:26+00:00http://sheepy.org/sheepy.orgcornet@sheepy.orgMunin and Graphite, using bunny this time.2011-12-11T00:00:00+00:00http://sheepy.org/2011/12/11/munin-graphite-bunny<p>After reading <a href='http://www.devco.net/archives/2011/12/11/common-messaging-patterns-using-stomp-%E2%80%93-part-2.php'>this</a> post by <a href='https://twitter.com/#!/ripienaar'>@ripienaar</a> I decided I would re-visit my munin-graphite setup to make it somewhat more elegant.</p>
<p>While the article advocates the use of stop I would prefer to stick with Graphite’s built in AMQP support if possible but I agree that the amqp gem is far from elegant. Not what you want if you’re a sysadmin needing to solve a problem quickly.</p>
<p>I’ve seen <a href='https://github.com/ruby-amqp/bunny/wiki'>bunny</a> in passing but never used it before. Turns out it’s quite nice. Also, again from a previous article from <a href='http://www.devco.net/archives/2011/10/02/interact-with-munin-node-from-ruby.php'>@ripienaar</a>, the <a href='https://github.com/sosedoff/munin-ruby'>munin-ruby</a> gem caught my eye.</p>
<p>Not too long later I’d replaced my original implementation with something a lot simpler. As ever it needs some work whch I’ll hopefully get round to this week however here is the new code:</p>
<script src='https://gist.github.com/1463505.js?file=munin-graphite.rb' />Munin and Graphite2011-12-02T00:00:00+00:00http://sheepy.org/2011/12/02/munin-graphite<p><a href='http://munin-monitoring.org/'>Munin</a> is a great piece of software for monitoring servers. However it’s well known that it’s central poller sucks at scale.</p>
<p><a href='http://graphite.wikidot.com/'>Graphite</a> looks like just the ticket for collating stats so I spent of my spare time recently investigating. After some research I found <a href='https://github.com/adamhjk/munin-graphite'>munin-graphite</a> which was what I wanted to punt data from Munin to Graphite. However I wanted to decouple the two by using AMQP.</p>
<p>Graphite now supports AMQP nateively although doesn’t admit to it. If you check out the latest version you’ll find carbon.amqp.conf.example which contains all you need to get started.</p>
<p>So now I’ve got Graphite consuming from AMQP I just needed to modify <a href='https://github.com/adamhjk/munin-graphite'>munin-graphite</a> submit it’s data how I wanted.</p>
<p>The result of which is my own <a href='https://github.com/cornet/munin-graphite'>fork</a>. Just customise munin-graphite-daemon.rb to your needs. I’ve hhad it running for around a month now with no problems. YMMV as always.</p>Rake for config deployment2011-12-02T00:00:00+00:00http://sheepy.org/2011/12/02/Rake-for-config-deployment<p>I’m a great advocator of <a href='http://puppetlabs.com/'>puppet</a> or any configuration management tool. However sometimes it’s an overkill. We have a number of servers which are not puppet controlled for various reason but we push new configurations out to them regularly.</p>
<p>For some things we have scripts to push the configs out and of course, some we don’t. This occasionally leads to problems.</p>
<p>Taking the example of a DNS server, easy to configure, easy to mess up and easy not to notice when you’ve messed up (until the TTL runs out).</p>
<p>The more commands you have to issue, the more likely a mistake is to be made.</p>
<p>There are many hammers for this nail but ruby is my weapon of choice these days so I’ve knocked up a quick Rakefile to do everything. So all I need do is rake deploy and it’ll do the following:</p>
<ul>
<li>Sync the config out</li>
<li>Set the correct permissions</li>
<li>Run named-checkzone against all the zonefiles</li>
<li>Reload the config if the checks pass</li>
</ul>
<p>To avoid creating loads of SSH connections I’ve employed the <a href='https://github.com/jamis/net-ssh-multi'>Net::SSH::Multi</a> gem which is definitly worth a look.</p>
<div class='highlight'><pre><code class='ruby'><span class='c1'>#</span>
<span class='c1'># Configuration</span>
<span class='c1'>#</span>
<span class='vg'>$ssh_user</span> <span class='o'>=</span> <span class='s2'>"foo"</span>
<span class='vg'>$ssh_host</span> <span class='o'>=</span> <span class='s2'>"example.com"</span>
<span class='n'>remote_root</span> <span class='o'>=</span> <span class='s2'>"/etc/bind"</span>
<span class='n'>rsync</span> <span class='o'>=</span> <span class='s1'>'rsync -a -e "ssh" --rsync-path="sudo rsync"'</span>
<span class='n'>remote_dir</span> <span class='o'>=</span> <span class='s2'>"</span><span class='si'>#{</span><span class='vg'>$ssh_user</span><span class='si'>}</span><span class='s2'>@</span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>:</span><span class='si'>#{</span><span class='n'>remote_root</span><span class='si'>}</span><span class='s2'>/zones/"</span>
<span class='n'>local_dir</span> <span class='o'>=</span> <span class='s2'>"zones/"</span>
<span class='k'>def</span> <span class='nf'>remote</span>
<span class='vi'>@remote</span> <span class='o'>||=</span> <span class='k'>begin</span>
<span class='nb'>require</span> <span class='s1'>'net/ssh/multi'</span>
<span class='n'>session</span> <span class='o'>=</span> <span class='no'>Net</span><span class='o'>::</span><span class='no'>SSH</span><span class='o'>::</span><span class='no'>Multi</span><span class='o'>.</span><span class='n'>start</span>
<span class='n'>session</span><span class='o'>.</span><span class='n'>use</span> <span class='s2'>"</span><span class='si'>#{</span><span class='vg'>$ssh_user</span><span class='si'>}</span><span class='s2'>@</span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>"</span>
<span class='n'>session</span>
<span class='k'>end</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Get config from </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>"</span>
<span class='n'>task</span> <span class='ss'>:get</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Getting config from </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'> ***"</span>
<span class='nb'>system</span> <span class='s2'>"</span><span class='si'>#{</span><span class='n'>rsync</span><span class='si'>}</span><span class='s2'> </span><span class='si'>#{</span><span class='n'>remote_dir</span><span class='si'>}</span><span class='s2'> </span><span class='si'>#{</span><span class='n'>local_dir</span><span class='si'>}</span><span class='s2'>"</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Put config to </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>"</span>
<span class='n'>task</span> <span class='ss'>:put</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Deploying config files to </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'> ***"</span>
<span class='nb'>system</span> <span class='s2'>"</span><span class='si'>#{</span><span class='n'>rsync</span><span class='si'>}</span><span class='s2'> </span><span class='si'>#{</span><span class='n'>local_dir</span><span class='si'>}</span><span class='s2'> </span><span class='si'>#{</span><span class='n'>remote_dir</span><span class='si'>}</span><span class='s2'>"</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Set correct permissions"</span>
<span class='n'>task</span> <span class='ss'>:set_perms</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Setting correct file permissions ***"</span>
<span class='n'>remote</span><span class='o'>.</span><span class='n'>exec</span><span class='p'>(</span><span class='s2'>"sudo chmod 640 /etc/bind/zones/db.*"</span><span class='p'>)</span><span class='o'>.</span><span class='n'>wait</span>
<span class='n'>remote</span><span class='o'>.</span><span class='n'>exec</span><span class='p'>(</span><span class='s2'>"sudo chown bind:bind /etc/bind/zones/db.*"</span><span class='p'>)</span><span class='o'>.</span><span class='n'>wait</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Test config on </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>"</span>
<span class='n'>task</span> <span class='ss'>:test</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Testing zone files ***"</span>
<span class='n'>errors</span> <span class='o'>=</span> <span class='nb'>Array</span><span class='o'>.</span><span class='n'>new</span>
<span class='no'>FileList</span><span class='o'>.</span><span class='n'>new</span><span class='p'>(</span><span class='s2'>"zones/*"</span><span class='p'>)</span><span class='o'>.</span><span class='n'>each</span> <span class='k'>do</span> <span class='o'>|</span><span class='n'>zone</span><span class='o'>|</span>
<span class='n'>output</span> <span class='o'>=</span> <span class='nb'>Array</span><span class='o'>.</span><span class='n'>new</span>
<span class='n'>channel</span> <span class='o'>=</span> <span class='n'>remote</span><span class='o'>.</span><span class='n'>exec</span><span class='p'>(</span><span class='s2'>"sudo named-checkzone </span><span class='si'>#{</span><span class='n'>zone</span><span class='o'>.</span><span class='n'>sub</span><span class='p'>(</span><span class='sr'>/zones\/db./</span><span class='p'>,</span> <span class='s1'>''</span><span class='p'>)</span><span class='si'>}</span><span class='s2'> </span><span class='si'>#{</span><span class='n'>remote_root</span><span class='si'>}</span><span class='s2'>/</span><span class='si'>#{</span><span class='n'>zone</span><span class='si'>}</span><span class='s2'>"</span><span class='p'>)</span> <span class='k'>do</span> <span class='o'>|</span><span class='n'>ch</span><span class='p'>,</span> <span class='n'>stream</span><span class='p'>,</span> <span class='n'>data</span><span class='o'>|</span>
<span class='n'>output</span> <span class='o'><<</span> <span class='s2'>"[</span><span class='si'>#{</span><span class='n'>ch</span><span class='o'>[</span><span class='ss'>:host</span><span class='o'>]</span><span class='si'>}</span><span class='s2'> : </span><span class='si'>#{</span><span class='n'>stream</span><span class='si'>}</span><span class='s2'>] </span><span class='si'>#{</span><span class='n'>data</span><span class='si'>}</span><span class='s2'>"</span>
<span class='k'>end</span>
<span class='n'>channel</span><span class='o'>.</span><span class='n'>wait</span>
<span class='k'>if</span> <span class='n'>channel</span><span class='o'>.</span><span class='n'>any?</span> <span class='p'>{</span> <span class='o'>|</span><span class='n'>c</span><span class='o'>|</span> <span class='n'>c</span><span class='o'>[</span><span class='ss'>:exit_status</span><span class='o'>]</span> <span class='o'>!=</span> <span class='mi'>0</span> <span class='p'>}</span>
<span class='n'>errors</span> <span class='o'><<</span> <span class='s2'>"</span><span class='si'>#{</span><span class='n'>zone</span><span class='si'>}</span><span class='s2'> failed to validate"</span>
<span class='n'>errors</span> <span class='o'><<</span> <span class='n'>output</span><span class='o'>.</span><span class='n'>join</span>
<span class='k'>end</span>
<span class='k'>end</span>
<span class='k'>unless</span> <span class='n'>errors</span><span class='o'>.</span><span class='n'>empty?</span>
<span class='nb'>abort</span> <span class='s2'>"</span><span class='si'>#{</span><span class='n'>errors</span><span class='o'>.</span><span class='n'>join</span><span class='p'>(</span><span class='s2'>"</span><span class='se'>\n</span><span class='s2'>"</span><span class='p'>)</span><span class='si'>}</span><span class='s2'>"</span>
<span class='k'>end</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Reload bind on </span><span class='si'>#{</span><span class='vg'>$ssh_host</span><span class='si'>}</span><span class='s2'>"</span>
<span class='n'>task</span> <span class='ss'>:reload</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Reloading Bind ***"</span>
<span class='n'>channel</span> <span class='o'>=</span> <span class='n'>remote</span><span class='o'>.</span><span class='n'>exec</span><span class='p'>(</span><span class='s2'>"sudo rndc reload"</span><span class='p'>)</span> <span class='k'>do</span> <span class='o'>|</span><span class='n'>ch</span><span class='p'>,</span> <span class='n'>stream</span><span class='p'>,</span> <span class='n'>data</span><span class='o'>|</span>
<span class='nb'>puts</span> <span class='s2'>"[</span><span class='si'>#{</span><span class='n'>ch</span><span class='o'>[</span><span class='ss'>:host</span><span class='o'>]</span><span class='si'>}</span><span class='s2'> : </span><span class='si'>#{</span><span class='n'>stream</span><span class='si'>}</span><span class='s2'>] </span><span class='si'>#{</span><span class='n'>data</span><span class='si'>}</span><span class='s2'>"</span>
<span class='k'>end</span>
<span class='n'>channel</span><span class='o'>.</span><span class='n'>wait</span>
<span class='k'>end</span>
<span class='n'>desc</span> <span class='s2'>"Put config, test and reload"</span>
<span class='n'>task</span> <span class='ss'>:deploy</span> <span class='o'>=></span> <span class='o'>[</span><span class='ss'>:put</span><span class='p'>,</span> <span class='ss'>:set_perms</span><span class='p'>,</span> <span class='ss'>:test</span><span class='p'>,</span> <span class='ss'>:reload</span><span class='o'>]</span> <span class='k'>do</span>
<span class='nb'>puts</span> <span class='s2'>"*** Zonefiles deployed ***"</span>
<span class='k'>end</span>
</code></pre>
</div>
<p>Yes it’s hacky and could do with some improvement but then I’m a sysadmin and not a developer. I might have a go at writting a gem to make these sort of Rakefiles cleaner but it’s not too bad.</p>Fairwell Drupal2011-09-27T00:00:00+01:00http://sheepy.org/2011/09/27/fairwell-drupal<p>Fairwell <a href='http://www.drupal.org/'>Drupal</a>. It has been fun but things have changed and it’s time to move on to something <a href='https://github.com/mojombo/jekyll/'>new</a>.</p>Arduino fun2010-11-09T00:00:00+00:00http://sheepy.org/2010/11/09/arduino-fun<p>So after all the weekend plans got cancelled I decided to do a bit of tidying… right up until the point I discovered a female RJ45 socket on the bottom of my energy meter.</p>
<p>Doing what any self-respecting geek would do I plugged in a piece of cat5 and waved a DHCP lease at it. No dice, time for some research,</p>
<p>The energy meter in question is a re-branded <a href='http://www.currentcost.com/product-theclassic.html'>current cost classic</a> of which, at least on initial research, there are 2 versions. (I’ll come back to this in a bit).</p>
<p>Now the basic principle as to how these devices work is thus: You clamp an induction meter round your main electric feed connected to a wireless transmitter which talks to a unit with an LCD which has this handy RJ45 socket. The RJ45 socket actually turns out to speak 3.3V TTL for which you can acquire a data cable from <a href='http://stores.ebay.co.uk/Current-Cost-Ltd'>here</a>.</p>
<p>But it’s Saturday and I want this working now! Enter, stage left, Arduino. Some reading of <a href='http://e.inste.in/2008/06/16/interfacing-the-currentcost-meter-to-an-arduino/'>this</a> and <a href='http://mungbean.org/blog/?p=477'>this</a> there are 2 versions: <ul>
<li>4200 Baud over pins 4 & 8</li>
<li>9600 Baud over pins 6 & 7</li>
</ul></p>
<p>Mine obviously has some personality issues and decided that it would operate using pins 4 & 8 at 9600 Baud.</p>
<p>A quick Ardunio sketch later I had it spewing out XML over the hardware serial port using <a href='http://arduiniana.org/libraries/newsoftserial/'>NewSoftSerial</a> to do the conversion.</p>
<p>All going well I decided to punt this data out using the <a href='http://www.arduino.cc/en/Main/ArduinoEthernetShield'>Arduino Ethernet Shield</a>.</p>
<p>Cue timing problems…</p>
<p>It appears the poor little Arduino couldn’t cope and was missing/corrupting the output and was missing quite a lot of characters whitest trying to parse and punt data out over Ethernet. Throwing some buffers into the mix appears to have resolved the issue and it now appears to get 90%+ of the readings using this <a href='https://gist.github.com/667488'>sketch</a></p>
<p>All in all a productive weekend and hopefully I should have some graphs soon to show what sort of power I’m consuming.</p>
<p>Current TODO list:</p>
<ul>
<li>Tidy up the code</li>
<li>Write agent to connect to the Arduino and get current usage</li>
<li>Stick this into my <a href='http://munin-monitoring.org/'>munin</a> install <a href='https://secure.sheepy.org/munin'>here</a></li>
</ul>JBoss woes2009-11-07T00:00:00+00:00http://sheepy.org/2009/11/07/jboss-woes<p>JBoss, on the whole, does hold up surprisingly well. This can probably be attributed to our skilled developers who, to be fair, I don’t always give enough credit to.</p>
<p>However every so often JBoss plays up and the symptoms presented seem to point to obvious problems. However all is not as it appears.</p>
<p>It’s release day and shiny new software is just itching to be deployed. The sysadmin gets up early and rocks up to the office to go through the standard deployment procedure. If only things always went to plan!</p>
<p>We have a cluster of a number of servers and use the <a href='http://www.jboss.org/community/wiki/JBossFarmDeployment'>JBoss Farm Deployment</a> service to deploy applications. It’s fairly straight forward, you build and deploy .ear, .war, .spring, etc… files to the <em>$JBOSS_HOME/server/default/farm/</em> directory and all the nodes pick up the new code.</p>
<p>Here comes the first gotcha, which we have known about for quite a while now. If you re-farm an already running package then by default it won’t free itself from the PermGen heap so continuous redeploying will eventually mean you run out of PermGen memory.</p>
<p>The solution we have is to make sure we restart every node after deployment to clear out this memory.</p>
<p>What I’ve found out recently is this issue can be resolved by setting the following Java options: <code>
-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
</code> which will free up the PermGen memory.</p>
<p>This will be going into testing out or dev and test environments shortly and hopefully on live (once we are sure there are no adverse affects). This should mean no restarting required in most cases.</p>
<p>I say in most cases as we do have some applications we can’t “hot deploy”. We have been instructed to do the following:</p>
<ul>
<li>Shutdown the build node</li>
<li>Build and farm the application</li>
<li>Shutdown all other nodes</li>
<li>Bring up the build node</li>
<li>Bring up the other nodes</li>
</ul>
<p>This obviously leads to a complete outage lasting a few minutes, but we can live with that for the most part.</p>
<p>Once such morning a co-worker followed this procedure and all appeared to go fine. However not long after some of our application started throwing “Broken Pipe” exceptions. The applications in question were communicating with our JBoss clusting using RMI. From the exceptions this initially looked like some network issue. The load balancers (LVS ones) were checked but no issues. More investigation required…</p>
<p>The nodes throwing the exception were part of a 6 node Tomcat cluster communicating to a 3 node JBoss cluster. On closer exception only nodes 1 and 4 of the Tomcat cluster were throwing exceptions. These were restarted but to no avail.</p>
<p>Then I remembered that we we do Source Hashing on our LVS nodes. Source hashing is used to make sure the same clients hit the servers, normally for session tracking purposes, this helped with diagnosis.</p>
<p>I found that 1 JBoss cluster node was at fault, but there were no exceptions in the logs. Further more most transactions were working fine. Just to be safe JBoss was restarted on the offending node but no difference. On with investigation I guess…</p>
<p>Eventually I found something that didn’t make sense. <em>$JBOSS_HOME/server/default/tmp/deploy</em> had a timestamp older than I expected.</p>
<p>This directory is used to hold the expanded files from <em>$JBOSS_HOME/server/default/farm/</em> and <b>should</b> disappear when JBoss is shut down. I shutdown JBoss again and, for whatever reason, it still remained. So I deleted the directory by hand and started up JBoss. Sure enough the “Broken Pipe” exceptions disappeared.</p>
<p>I shutdown JBoss on offending node again and this time it removed the directory. Started up again and all fine.</p>
<p>After much playing around I’ve no idea what causes this. I know that if JBoss doesn’t shut down correctly then this directory can remain causing clustering issues (which really don’t make sense to me) but I’ve seen a number of occasions on our Test Environment where this directory has remained after a successful shutdown.</p>
<p>To make sure this doesn’t happen again I’ve modified out start scripts to check for the presence of this directory and refuse to start up if it exists.</p>
<p>Fingers crossed we won’t see this issue again.</p>Committing to Git2009-10-22T00:00:00+01:00http://sheepy.org/2009/10/22/committing-to-git<p>Unfortunately for me I still have to use and administer CVS at work for our developers. Personally I ditched it a long time ago in favour of SVN and then Bazaar.</p>
<p>Lately I’ve been looking because, well….just because really (us tech’s have a habit of trying out things for the hell of it).</p>
<p>So I’ve decided to commit to using it. I’ve also gone and got myself a <a href='http://github.com/cornet'>GitHub account</a> which I will use to store configurations, scripts and any public dev work that might be of use.</p>
<p>Will see how it all pans out, I’ll hopefully get round to adding more repositories soon.</p>
<p>For now probably the only one worth a look is my <a href='http://github.com/cornet/dotvim/tree/'>vim configuration</a>. Of course you can get a copy by just doing: <code>
git clone git://github.com/cornet/dotvim.git
</code></p>Microsoft's BPOS Using Postfix2009-06-10T00:00:00+01:00http://sheepy.org/2009/06/10/microsofts-bpos-using-postfix<p>Nice to see MS using opensource software :)</p>
<p>Email received today from someone using BPOS:</p>
<pre><code> Received: from mail187-tx2-R.bigfish.com (10.9.14.251) by
TX2EHSOBE009.bigfish.com (10.9.40.29) with Microsoft SMTP Server id
8.1.340.0; Wed, 10 Jun 2009 10:33:02 +0000
Received: from mail187-tx2 (localhost.localdomain [127.0.0.1]) by
mail187-tx2-R.bigfish.com (Postfix) with ESMTP id 520131100037 for
<******@*******.***>; Wed, 10 Jun 2009 10:33:02 +0000 (UTC)</code></pre>
<p>Looks like bigfish.com are using the <a href='http://www.postfix.org/'>Postfix</a> mail server.</p>
<p>Quick whois on bigfish.com reveals:</p>
<pre><code>Registrant:
Microsoft Corporation
Domain Administrator
One Microsoft Way
Redmond, WA 98052
US
Email: domains@microsoft.com</code></pre>
<p>Most interesting ;)</p>Failover hosts using Xen, DRBB and Heartbeat2009-04-08T00:00:00+01:00http://sheepy.org/2009/04/08/failover-hosts-using-xen-drbb-and-heartbeat<p>After quite a lot of reading and a morning playing I managed to get failover Xen hosts working.</p>
<p>The idea was to have 2 physical servers to run 2 (or more) Xen hosts between them. If one server was to die or needed some work doing on it then the domU would automatically move to the other node.</p>
<div style='text-align:center'>
<img alt='Failover Diagram' height='169' src='/images/failover.png' width='499' />
</div>
<p>I’ve done some testing and all appears to work fine. However let me stress that this is not live migration so you would suffer about a minute or so outage (not really a big deal in the grand scheme of things).</p>
<p>Click the “Read More” button for full details on the setup. <!--break--> <br /></p>
<h3>Requirements</h3><ul>
<li>Debian Etch</li>
<li>Xen Installed</li>
</ul><br /><h3>Installing DRBD</h3><em>To be done on both Xen nodes</em>
<p>DRBD is only at version 7 in Debian Etch, so we pull in version 8 from <a href='http://www.backports.org'>backports.org</a></p>
<p>Version 8 comes with some helper scripts for Xen which make the failover a lot easier to configure.</p>
<p>These are the packages we need: <ul>
<li>[drbd8-utils](http://packages.debian.org/etch-backports/drbd8-utils)</li>
<li>[drbd8-source](http://packages.debian.org/etch-backports/drbd8-source)</li>
</ul></p>
<p>If you are unsure on how to install stuff from backports then please read the documentation located <a href='http://backports.org/dokuwiki/doku.php?id=instructions'>here</a></p>
<p>After installing the packages we need to build the DRBD kernel module. The easiest way to achieve this is to do: <pre>module-assistant auto-install drbd8</pre></p>
<p>Then load the module by doing <pre>modprobe drbd</pre></p>
<h3>Installing Heartbeat</h3><em>To be done on both Xen nodes</em>
<p>This can be installed straight from the Debian repos: <pre>aptitude install heartbeat-2</pre></p>
<p>Next there is some basic configuration to do. First we need to create <em>/etc/heartbeat/ha.cf</em> with the following contents: <pre>
# Enable new cluster manager
crm on
# Specify interface to send bcast packets out on
bcast eth0
# Specify nodes in cluster, these must correspond with "uname -n"
nodes xen-1 xen-2
</pre></p>
<p>Then we create <em>/etc/heartbeat/authkeys</em> with the following contents: <pre>
auth 1
1 sha1 SomeLongStringWithRandomChars
</pre> Note that <em>SomeLongStringWithRandomChars</em> should be randomly generated and the same on both nodes.</p>
<h3>Setting up DRBD resources</h3><em>To be done on both Xen nodes</em>
<p>I generally use LVM for my Xen hosts, so first off I create all the paritions required for my 2 hosts: <pre>
lvcreate -L10G -n host1-root xenhostfs
lvcreate -L1G -n host1-swap xenhostfs
lvcreate -L10G -n host2-root xenhostfs
lvcreate -L1G -n host2-swap xenhostfs
</pre></p>
<p>Next we need to create <em>/etc/drbd.conf</em> and configure our resources (DRBD devices):</p>
<pre>
#
# Global Parameters
#
global {
# Participate in http://usage.drbd.org
usage-count yes;
}
#
# Settings common to all resources
#
common {
# Set sync rate
syncer { rate 10M; }
# Protocol C : Both nodes have to commit before write
# is considered successful
protocol C;
net {
# Xen tests that it can write to block device
# before starting up. Not allowing this causes
# migration to fail.
allow-two-primaries;
# Split-brain recovery parameters
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
}
#
# Resource Definitions
#
resource "host1_root" {
on xen1 {
# The block device it will appear as
device /dev/drbd0;
# The device we are mirroring
disk /dev/xenhostfs/host1-root;
# Store DRBD meta data the above disk
meta-disk internal;
# Address of *this* host and port to replicate over
# You must use a different port for each resource
address 10.0.0.1:7790;
}
on xen2 {
device /dev/drbd0;
disk /dev/xenhostfs/host1-root;
meta-disk internal;
address 10.0.0.2:7790;
}
}
resource "host1_swap" {
on xen1 {
device /dev/drbd1;
disk /dev/xenhostfs/host1-swap;
meta-disk internal;
address 10.0.0.1:7791;
}
on xen2 {
device /dev/drbd1;
disk /dev/xenhostfs/host1-swap;
meta-disk internal;
address 10.0.0.2:7791;
}
}
resource "host2_root" {
on xen1 {
device /dev/drbd2;
disk /dev/xenhostfs/host2-root;
meta-disk internal;
address 10.0.0.1:7792;
}
on xen2 {
device /dev/drbd2;
disk /dev/xenhostfs/host2-root;
meta-disk internal;
address 10.0.0.2:7792;
}
}
resource "host2_swap" {
on xen1 {
device /dev/drbd3;
disk /dev/xenhostfs/host2-swap;
meta-disk internal;
address 10.0.0.1:7793;
}
on xen2 {
device /dev/drbd3;
disk /dev/xenhostfs/host2-swap;
meta-disk internal;
address 10.0.0.2:7793;
}
}
</pre>
<p>Now we have to initialise the metadata on the DRBD resources. This takes some time so I advise a coffee or lunch at this point ;) <pre>
drbdadm create-md host1_root
drbdadm create-md host1_swap
drbdadm create-md host2_root
drbdadm create-md host2_swap
</pre></p>
<p>Now that we have done that we can start up drbd: <pre>
/etc/init.d/drbd restart
</pre></p>
<p>Right now all that is set up we now need to select one node as the primary, (say xen1).</p>
<p><em>On xen1 only</em> we now do the following which does 2 things:</em></p>
<ul>
<li>Sets xen1 as the primary for the DRBD devices</li>
<li>Syncs all the data from xen1 to xen2</li>
</ul>
<pre>
drbdadm -- --overwrite-data-of-peer primary host1_root
drbdadm -- --overwrite-data-of-peer primary host1_swap
drbdadm -- --overwrite-data-of-peer primary host2_root
drbdadm -- --overwrite-data-of-peer primary host2_swap
</pre>
<p>You can check the process of his by doing <pre>
cat /proc/drbd
</pre></p>
<p>which will give you output something like <pre>
version: 8.0.14 (api:86/proto:86)
GIT-hash: bb447522fc9a87d0069b7e14f0234911ebdab0f7 build by phil@fat-tyre, 2008-11-12 16:40:33
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:52120 nr:1016 dw:503652 dr:203481 al:121 bm:872 lo:0 pe:0 ua:0 ap:0
[>...................] sync'ed: 1.0% (10380903/10485760)K
finish: 0:07:43 speed: 10,836 (10,836) K/sec
resync: used:0/61 hits:2296 misses:10 starving:0 dirty:0 changed:10
act_log: used:0/127 hits:120616 misses:128 starving:0 dirty:7 changed:121
1: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:0 dw:4 dr:1348 al:1 bm:12 lo:0 pe:0 ua:0 ap:0
...
</pre></p>
<p>Note that you do not have to wait for this to finish, you can carry on using your /dev/drbdN devices but preformance will be reduced until it has completed the sync.</p>
<h3>Creating Xen DomU</h3>
<p>I’m not going to go through the complete details of this as its a reasonable assumption that if you’re reading this then you have used Xen before, or at least are capable of reading around.</p>
<em>This is done only on the primary host</em>
<p>However here are the edited highlights. Firstly create the filesystems: <pre>
mkfs.ext3 /dev/drbd0
mkswap /dev/drbd1
mkfs.ext3 /dev/drbd2
mkswap /dev/drbd3
</pre></p>
<p>Now mount and use debootstrap to create the base installs: <pre>
mkdir /mnt/host1 /mnt/host2
mount /dev/drbd0 /mnt/host1
mount /dev/drbd2 /mnt/host2
debootstrap --arch i386 etch /mnt/host1 http://ftp.uk.debian.org/debian/
debootstrap --arch i386 etch /mnt/host2 http://ftp.uk.debian.org/debian/
</pre></p>
<p>Create <em>/mnt/host1/etc/fstab</em> and <em>/mnt/host2/etc/fstab</em> with the following content: <pre>
proc /proc proc defaults 0 0
/dev/xvda / ext3 defaults,errors=remount-ro 0 1
/dev/xvdb none swap sw 0 0
</pre></p>
<p>Then unmount <em>/mnt/host1</em> & <em>/mnt/host2</em> and we are ready to configure Xen.</p>
<h3>Xen Domain Config</h3><em>To be done on both Xen nodes</em>
<p>Create <em>/etc/xen/host1.dom</em> with the following contents:</p>
<pre><code># -*- mode: python; -*-
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.6.18-6-xen-686"
ramdisk = "/boot/initrd.img-2.6.18-6-xen-686"
# Initial memory allocation (in megabytes) for the new domain.
memory = 1024
# A name for your domain. All domains must have different names.
name = "host1"
# Filesystems
disk = [ 'drbd:host1_root,xvda,w', 'drbd:host1_swap,xvdb,w']
# Network
vif = ['bridge=xenbr0']
# Set root device.
root = "/dev/xvda ro"
# Sets runlevel 4.
extra = "4"</code></pre>
<p>and</p>
<em>/etc/xen/host2.dom</em>
<pre><code># -*- mode: python; -*-
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.6.18-6-xen-686"
ramdisk = "/boot/initrd.img-2.6.18-6-xen-686"
# Initial memory allocation (in megabytes) for the new domain.
memory = 1024
# A name for your domain. All domains must have different names.
name = "host2"
# Filesystems
disk = [ 'drbd:host2_root,xvda,w', 'drbd:host2_swap,xvdb,w']
# Network
vif = ['bridge=xenbr0']
# Set root device.
root = "/dev/xvda ro"
# Sets runlevel 4.
extra = "4"</code></pre>
<p>Now on ONE of the nodes ONLY you can start the xen instances by doing: <pre>
xm create /etc/xen/host1.dom
xm create /etc/xen/host2.dom
</pre></p>
<h3>Configuring Heartbeat</h3>
<p>We need to create 3 XML files: <ul>
<li><strong>bootstrap.xml</strong>: This sets up the defaults for heartbeat Cluster Resource Manager (CRM)</li>
<li><strong>host1.xml</strong>: Heartbeat config for host1</li>
<li><strong>host2.xml</strong>: Heartbeat config for host2</li>
</ul></p>
<p>Now the documentation for these XML files is somewhat thin on the ground. The only real thing to go on is the DTD which, on Debian, lives here:<em>/usr/lib/heartbeat/crm.dtd</em></p>
<p>You can also refer to the online version <a href='http://hg.clusterlabs.org/pacemaker/dev/file/tip/xml/crm-1.0.dtd'>here</a> which is always the latest version.</p>
<p>Anyway here are the contents of the 3 files (just save them to your home directory for now).</p>
<em>bootstrap.xml</em><em>To be done on one Xen node only</em><pre>
<cluster_property_set id="bootstrap">
<attributes>
<nvpair id="bootstrap01" name="transition_idle_timeout" value="60"/>
<nvpair id="bootstrap02" name="default_resource_stickiness" value="0"/>
<nvpair id="bootstrap03" name="default_resource_failure_stickiness"
value="-500"/>
<nvpair id="bootstrap04" name="stonith_enabled" value="false"/>
<nvpair id="bootstrap05" name="stonith_action" value="reboot"/>
<nvpair id="bootstrap06" name="symmetric_cluster" value="true"/>
<nvpair id="bootstrap07" name="no_quorum_policy" value="stop"/>
<nvpair id="bootstrap08" name="stop_orphan_resources" value="true"/>
<nvpair id="bootstrap09" name="stop_orphan_actions" value="true"/>
<nvpair id="bootstrap10" name="is_managed_default" value="true"/>
</attributes>
</cluster_property_set>
</pre><em>host1.xml</em><pre>
<resources>
<primitive id="host1" class="ocf" type="Xen" provider="heartbeat">
<operations>
<op id="host1-op01" name="monitor" interval="10s" timeout="60s" prereq="nothing"/>
<op id="host1-op02" name="start" timeout="60s" start_delay="0"/>
<op id="host1-op03" name="stop" timeout="300s"/>
</operations>
<instance_attributes id="host1">
<attributes>
<nvpair id="host1-attr01" name="xmfile" value="/etc/xen/host1.dom"/>
<nvpair id="host1-attr02" name="target_role" value="started"/>
</attributes>
</instance_attributes>
<meta_attributes id="host1-meta01">
<attributes>
<nvpair id="host1-meta-attr01" name="allow_migrate" value="true"/>
</attributes>
</meta_attributes>
</primitive>
</resources>
</pre><em>host2.xml</em><pre>
<resources>
<primitive id="host2" class="ocf" type="Xen" provider="heartbeat">
<operations>
<op id="host2-op01" name="monitor" interval="10s" timeout="60s" prereq="nothing"/>
<op id="host2-op02" name="start" timeout="60s" start_delay="0"/>
<op id="host2-op03" name="stop" timeout="300s"/>
</operations>
<instance_attributes id="host2">
<attributes>
<nvpair id="host2-attr01" name="xmfile" value="/etc/xen/host2.dom"/>
<nvpair id="host2-attr02" name="target_role" value="started"/>
</attributes>
</instance_attributes>
<meta_attributes id="host2-meta01">
<attributes>
<nvpair id="host2-meta-attr01" name="allow_migrate" value="true"/>
</attributes>
</meta_attributes>
</primitive>
</resources>
</pre>
<p>Then we need to load them: <pre>
cibadmin -C -o crm_config -x bootstrap.xml
cibadmin -C -o resources -x host1.xml
cibadmin -C -o resources -x host2.xml
</pre></p>
<h3>Monitoring</h3>
<p>Basic monitoing is done with <em>crm_mon</em>. Issuing this command will result in the following which is updated every 15 seconds:</p>
<pre>
============
Last updated: Wed Apr 8 21:08:56 2009
Current DC: xen2 (1b3dbdb3-f9ca-4d16-bf7d-8a57167b85ed)
2 Nodes configured.
2 Resources configured.
============
Node: xen2 (1b3dbdb3-f9ca-4d16-bf7d-8a57167b85ed): online
Node: xen1 (ea043f8c-5afc-4aee-9ffc-d17cb3cfed06): online
host1 (heartbeat::ocf:Xen): Started xen1
host2 (heartbeat::ocf:Xen): Started xen1
</pre><h3>Basic Operations</h3><strong>Migrating</strong>
<p>To move host2 from xen1 to xen2 you do: <pre>
crm_resource --migrate --resource host2 --host-uname xen2
</pre></p>
<strong>Stopping/Starting</strong>
<p>To stop host2: <pre>
crm_resource --resource host2 --set-parameter target_role \
--property-value stopped
</pre> …and to start it again: <pre>
crm_resource --resource host2 --set-parameter target_role \
--property-value started
</pre></p>
<h3>Credits</h3>
<p>Thanks to <a href='http://www.netdotnet.net/'>Doug</a> for pointing out all my lame mistakes.</p>MySQL AUTO_INCREMENT Madness2009-04-02T00:00:00+01:00http://sheepy.org/2009/04/02/mysql-auto_increment-madness<p>Can anyone explain this behaviour ?</p>
<pre>
mysql> select version();
+--------------+
| version() |
+--------------+
| 5.1.30-2-log |
+--------------+
1 row in set (0.00 sec)
mysql> create database foo;
Query OK, 1 row affected (0.01 sec)
mysql> use foo;
Database changed
mysql> CREATE TABLE test (
id bigint(20) NOT NULL AUTO_INCREMENT,
stuff varchar(10) DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB;
Query OK, 0 rows affected (0.04 sec)
mysql> SHOW CREATE TABLE test \G
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`stuff` varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> INSERT INTO test (id, stuff) VALUES (-1,'Hello!');
Query OK, 1 row affected (0.04 sec)
mysql> SELECT * FROM test;
+----+--------+
| id | stuff |
+----+--------+
| -1 | Hello! |
+----+--------+
1 row in set (0.00 sec)
mysql> INSERT INTO test (id, stuff) VALUES (0,'Hello!');
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine
mysql> SHOW CREATE TABLE test \G
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`stuff` varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=18446744073709551615 DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
</pre>
<p>We have found a workaround (sort of):</p>
<pre>
mysql> DROP TABLE test;
Query OK, 0 rows affected (0.04 sec)
mysql> CREATE TABLE test (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
stuff varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
Query OK, 0 rows affected (0.10 sec)
mysql> INSERT INTO test (id, stuff) VALUES (-1,'Hello!');
Query OK, 1 row affected (0.03 sec)
mysql> SHOW CREATE TABLE test \G
*************************** 1. row ***************************
Table: test
Create Table: CREATE TABLE `test` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`stuff` varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
mysql> INSERT INTO test (id, stuff) VALUES (0,'Hello!');
Query OK, 1 row affected (0.03 sec)
mysql> SELECT * FROM test;
+----+--------+
| id | stuff |
+----+--------+
| -1 | Hello! |
| 1 | Hello! |
+----+--------+
2 rows in set (0.00 sec)
</pre>
<p>For those that missed it, the difference is the backticks in the CREATE TABLE line. Also I’ve yet to test with other versions.</p>
<h4>Update</h4>
<p>Looks like this is a bug that was fixed in 5-1-31</p>
<p><a href='http://bugs.mysql.com/bug.php?id=41841'>http://bugs.mysql.com/bug.php?id=41841</a> <a href='http://bugs.mysql.com/bug.php?id=36411'>http://bugs.mysql.com/bug.php?id=36411</a></p>This morning, in #mysql2008-10-02T00:00:00+01:00http://sheepy.org/2008/10/02/this-morning-in-mysql<p>Some people are just too transparent :)</p>
<pre><code>05:23 <MyName> if i have a table of 3 columns, and 3 rows, and i do a cartesian
product of the table with itself, do i get 6 columns and 9 rows?
05:24 <MyName> if the table originally have column names (A B C), do i get ( A
B C A B C) as the column names after the cartesian product?
05:24 <MyName> then all the entries are just each entry of the original tables
with each of the entreis again?
05:25 <MyName> or do i eliminate any duplicate columns?</code></pre>
<p>… after no response for some time</p>
<pre><code>06:21 <MyName> hello?
06:22 <pc_> hello
06:22 <Gm4n> hello!
06:27 <MyName> ok
06:27 <MyName> if i have two tables , one is movies(title, studio), and the other is stars(name, title)
06:28 <MyName> i want to find all stars who played in ALL movies by studio "ABC" for example
06:28 <MyName> not all stars who played in movies by studio ABC, but all stars who played in ALL movies by ABC
06:28 <pc_> MyName, when is it due?
06:28 <MyName> tomorrow
06:28 <pc_> lol
06:28 <Gm4n> haha</code></pre>Sun Java on Debian/Ubuntu2008-09-18T00:00:00+01:00http://sheepy.org/2008/09/18/sun-java-on-debian-ubuntu<p>It would appear that even if you install ONLY the sun java package it drags in some GNU java stuff as well.</p>
<p>You can go through everything in /etc/alternatives and update it manually but that is somewhat time consuming.</p>
<p>Instead just run:</p>
<code>sudo /usr/sbin/update-java-alternatives --set java-1.5.0-sun</code>
<p>and this will sort everything :)</p>Dropbox2008-09-18T00:00:00+01:00http://sheepy.org/2008/09/18/dropbox<p>Somone appears to have created a online storage and sync thing that actually works:<br /><br /><a href='http://www.getdropbox.com/'>Dropbox - Home - Secure backup, sync and sharing made easy.</a><br /><br />Few grips thou:<br /><ul><li>It requires nautilus<br /></li><li>Not obvious how to create an account (it's not on the website, the client does it)<br /></li><li>The initial dialog box (which you use to sign up for an account) takes an age to appear</li></ul><br />All that aside it works wonderfully, copy file into ~/Dropbox and it syncs automatically :)<br /><br /><br />… oh and it versions files too<br />… and only sends binary diffs<br /><br /></p>Another blog added2008-08-07T00:00:00+01:00http://sheepy.org/2008/08/07/another-blog-added<p>Another one I’ve just added to my feedlist,</p>
<p><a href='http://blog.kovyrin.net/'>http://blog.kovyrin.net/</a></p>
<p>some interesting stuff, especially on the MySQL front.</p>
<p>He is also the author of the MySQL Master-Master Replication Manager which can be found here:</p>
<p><a href='http://code.google.com/p/mysql-master-master/'>http://code.google.com/p/mysql-master-master/</a></p>
<p>definitely something I’ll be looking at.</p>Nokia MMSC Server2008-07-25T00:00:00+01:00http://sheepy.org/2008/07/25/nokia-mmsc-server<p>Nokia’s MMSC is a stunning example of how to make problems as hard as possible to debug.</p>
<p>The MMSC can speak a number of protocols over HTTP including EAIF and MM7. As you would expect you submit your POST and get back a HTTP Error Code and Message. However the Codes and Messages leave a lot to be desired:</p>
<p>So lets take the 499 Error code which, according to their spec, can mean: <ul>
<li>Address Hiding - basically reject all messages to this Number</li>
<li>Desired Delivery Time Error - aka delivery time set too far in the future</li>
<li>No Credit</li>
<li>Recipient Barred</li>
<li>Sender Barred</li>
<li>Rejected - or "We ain't gonna tell you why"</li>
</ul></p>
<p>No surprises for guessing which one I’m seeing at the moment :)</p>
<p>Next comes the legendary 599 Error Code, again lets look at the spec. This can mean: <ul>
<li>CCR Error - Rejected as the MMSC has exceeded the number of messages licensed</li>
<li>Capacity Exceeded - Number of msgs/second exceeded</li>
<li>Kernel Overloaded</li>
<li>DB Error - Message could not be stored in database</li>
</ul> although in our case we normally see “No reason given or reason given not included in EAIF specification”.</p>
<p>If the above wasn’t fun enough, Nokia obviously doesn’t think it is, if you use the Nokia MMSC Client libs they will generate their own 666 error code if they don’t get anything back from the MMSC. Great!</p>
<p>Seriously, how do people manage to come up with this crap. If you’re going to define Error codes and Messages then FFS use them and don’t have a <em>"Oh we can't be bothered to tell you why"</em> message.</p>XMLStarlet2008-06-24T00:00:00+01:00http://sheepy.org/2008/06/24/xmlstarlet<p>I’m not the greatest fan of XML for much the same reasons as <a href='http://www.codinghorror.com/'>Jeff Atwood</a> who wrote a controversial <a href='http://www.codinghorror.com/blog/archives/001114.html'>blog post</a> on the subject</a>.</p>
<p>Sometimes though I’m forced to deal with it. <a href='http://xmlstar.sourceforge.net/'>XMLStarlet</a> looks like it might relieve some of my frustration.</p>
<p>Thanks to <a href='http://bashcurescancer.com/'>BASH Cures Cancer</a> for <a href='http://bashcurescancer.com/the-best-in-command-line-xml-xmlstarlet.html'>flagging</a> this.</p>Log Levels2008-06-18T00:00:00+01:00http://sheepy.org/2008/06/18/log-levels<p>As a sysadmin I spend much of my day trawling though log files looking for problems. Unfortunately many developers don’t appear to know how to use log levels effectively which can make a sysadmin’s life a small hell when it comes to debugging.</p>
<p>As a general rule I advise the following: <blockquote>
Be quiet about successful operations and verbose about unsuccessful ones.
</blockquote></p>
<p>Yes I like to see what an application is doing but I don’t need to know every function call that it’s making, yes this is useful when developing an application but can make isolating a problem in production a real pain.</p>
<p>Generally I use 4 different log levels:</p>
<ul>
<li>DEBUG</li>
<li>INFO</li>
<li>WARN</li>
<li>ERROR</li>
</ul>
<p>The definitions for these go something as follows:</p>
<table>
<tr>
<td><b>DEBUG</b></td>
<td>Things that should happen but you don't want to see in production. This includes function tracing, SQL statements, variable values etc...</td>
</tr>
<tr>
<td><b>INFO</b></td>
<td>Things that should happen that give you just enough idea about what's going on. This should be limited to 1 or 2 lines per transaction,</td>
</tr>
<tr>
<td><b>WARN</b></td>
<td>Things that shouldn't happen but the end user should see no difference. For example $foo failed so falling back to $bar instead</td>
</tr>
<tr>
<td><b>ERROR</b></td>
<td>Houston we have a problem! This is a serious error that has caused a problem for the end user. If you want to dump variables to the logs then now is a good time to do it.</td>
</tr>
</table>
<p>Some applications do logging in a completely different way, they have more log levels. Generally in production you want to know just enough to know what the application is doing + all the warnings and errors.</p>
<p>If something goes wrong then you need enough information to be able to replicate it. If all is well then, as sysadmins, we don’t care.</p>MySQL 5.1 on Debian Etch2008-06-04T00:00:00+01:00http://sheepy.org/2008/06/04/mysql-5-1-on-debian-etch<p>Here is a quick guide to getting MySQL installed on Debian Etch without pulling in anything from testing or unstable.</p>
<p>Now there are a number of different ways you could do this but I wanted a debian package (as I have banned installing anything from source on new servers at work).</p>
<p>After some google action I found that <a href='http://tretkowski.de/blog/'>Norbert Tretkowski</a> had submitted packages to <a href='http://packages.debian.org/experimental/misc/mysql-server'>experimental</a> However after checking these out they wanted to pull things in from testing and/or unstable and I didn’t want that (especially considering it wanted to upgrade libc6)</p>
<p>That said it turned out that this package would build fine on Etch by doing the following:</p>
<p>Add the following to <em>/etc/apt/sources.list</em>:</p>
<pre><code>deb http://ftp.uk.debian.org/debian/ experimental main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ experimental main contrib non-free</code></pre>
<p>Then create <em>/etc/apt/preferences</em> with the following in:</p>
<pre><code>Package: *
Pin: release o=Debian,a=stable
Pin-Priority: 900
Package: *
Pin: release o=Debian
Pin-Priority: -1</code></pre>
<p>which basically says “only install stuff from stable”.</p>
<p>Next we get the dev tools and the build dependencies:</p>
<pre><code>aptitude install dpkg-dev devscripts
apt-get build-dep mysql-server-5.1
apt-get source mysql-server-5.1</code></pre>
<p>Then build the .debs:</p>
<pre><code>cd mysql-dfsg-5.1-5.1.24rc
debuild </code></pre>
<p>Once that has finished get the run time deps:</p>
<pre><code>aptitude install libdbd-mysql-perl libdbi-perl</code></pre>
<p>And finally install the .debs we created earlier:</p>
<pre><code>cd ..
dpkg -i *.deb</code></pre>
<p>…and there you go, MySQL 5.1 on Debian Etch :)</p>gnome-do2008-05-29T00:00:00+01:00http://sheepy.org/2008/05/29/gnome-do<p>So after a mate a work started raving about <a href='http://do.davebsd.com/'>Gnome Do</a> I decided to give it another go, not quite remembering which I gave up with it last time.</p>
<p><a href='ttp://docs.blacktree.com/quicksilver/what_is_quicksilver'>Quicksilver</a> is a program I’ve always installed first when using any Mac and Gnome Do is a very similar thing for Linux.</p>
<p>It wasn’t long before I worked out why I had ditched it. I keep switching between window managers and seem to always end up back with <a href='http://pekwm.org'>pekwm</a>. Launching Gnome Do in pekwm for some reason causes the “window” to end up behind all the other windows.</p>
<p>A quick RTFM later and I found the answer. In my ~/.pekwm/autoproperties I added:</p>
<pre><code>Property = "Do,Do" {
ApplyOn = "Start New Reload"
Layer = "Menu"
Title = "^Do"
}</code></pre>
<p>which forces it above all other windows.</p>
<p>With that sorted the only thing left is the occasional erratic behavior of the window placement, but I can live with that for now :)</p>irssi + screen + UTF-82008-05-27T00:00:00+01:00http://sheepy.org/2008/05/27/irssi-screen-utf-8<p>I thought I’d sorted this once but apparently not :(</p>
<p>I’d managed to get things sorted so that I could input pound signs and they would appear fine to me and everyone else but if someone else typed a pound sign I got a question mark.</p>
<p>I <em>think</em> what was happening was that it was taking UTF-8 input and converting it to latin1… not entirely sure.</p>
<p>Anyway for those suffering the same issues take a look at <a href='http://www.iovene.com/the-ultimate-guide-for-utf-8-in-irssi-and-gnuscreen/'>The ultimate guide for UTF-8 in irssi and GNU/Screen</a></p>Debian 'fixes' OpenSSL2008-05-14T00:00:00+01:00http://sheepy.org/2008/05/14/debian-fixes-openssl<p>So it would appear that Debian ”<a href='http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516"'>fixed</a>” a problem in OpenSSL a few years ago. Unfortunately this “fix” has meant that they have had to release <a href='http://lists.debian.org/debian-security-announce/2008/msg00152.html'>this</a> security announcement.</p>
<p>Now this vulnerability is quite bad, so much so that Debian have stated the following:</p>
<blockquote>
<p>It is strongly recommended that all cryptographic key material which has been generated by OpenSSL versions starting with 0.9.8c-1 on Debian systems is recreated from scratch. Furthermore, all DSA keys ever used on affected Debian systems for signing or authentication purposes should be considered compromised; the Digital Signature Algorithm relies on a secret random value used during signature generation.</p>
</blockquote>
<p><a href='http://www.links.org'>Ben Laurie</a> has posted a great <a href='http://www.links.org?p=327'>blog entry</a> as to why this so stupid. This quote from that entry sums the problem up nicely: <blockquote>
Secondly, if you are going to fix bugs, then you should install this maxim of mine firmly in your head: never fix a bug you don’t understand.
</blockquote></p>Temporary Solutions2008-04-25T00:00:00+01:00http://sheepy.org/2008/04/25/temporary-solutions<p>I’m a firm believer that temporary solutions become permanent very quickly however this quote from <a href='http://mysqldba.blogspot.com/2008/04/avoid-storing-markup-html-in-database.html'>mySQL DBA</a> is by far the best way of putting it.</p>
<p>Remember kids <blockquote>There is nothing more Permanent then a temporary solution</blockquote></p>Upgraded2008-04-05T00:00:00+01:00http://sheepy.org/2008/04/05/upgraded<p>Upgraded to Drupal 6.1</p>
<p>No major problems other than I forgot to set my site back to the default theme before updating.</p>Railscasts2008-04-05T00:00:00+01:00http://sheepy.org/2008/04/05/railscasts<p>So I was pointed in the direction of <a href='http://railscasts.com/'>Railscasts.com</a> and duly leeched the site:</p>
<pre><code>curl "http://feeds.feedburner.com/railscasts_ipod" \
| grep -Eo "http://media.railscasts.com/ipod_videos/.*m4v" \
| sort | xargs -n1 wget -c</code></pre>
<p>did the trick.</p>
<p>Now I need to find time to watch them all :)</p>Bash Cures Cancer2008-02-24T00:00:00+00:00http://sheepy.org/2008/02/24/bash-cures-cancer<p>Another site that I’ve just added to my feed list.</p>
<p>Quite a few things on here I wasn’t aware of such as:</p>
<pre><code>find . -name 'file-*' -delete</code></pre>
<p>is much faster than</p>
<pre><code>find . -name 'file-*' -exec rm {} \;</code></pre>
<p>although I need to test how it compares to:</p>
<pre><code>find . -name 'file-*' | xargs rm</code></pre>
<p><a href='http://bashcurescancer.com/'>BASH Cures Cancer</a></p>Drupal 6 Released2008-02-23T00:00:00+00:00http://sheepy.org/2008/02/23/drupal-6-released<p>Apparently the Drupal 6.0 passed me by completely..<br /><br />Will look at upgrading soon to see what’s new<br /><br /><br /><a href='http://drupal.org/drupal-6.0'>Drupal 6.0 released | drupal.org</a></p>XKCD Bug2008-01-30T00:00:00+00:00http://sheepy.org/2008/01/30/xkcd-bug<a href='http://xkcd.com/376/'>
<img src='http://imgs.xkcd.com/comics/bug.png' />
</a>Hacking BT Home Hubs2008-01-15T00:00:00+00:00http://sheepy.org/2008/01/15/hacking-bt-home-hubs<p>So first we have the BT Voyger that gave out your username and password if you <a href='http://www.securityfocus.com/bid/10589/exploit'>asked it nicely</a>.</p>
<p>Then, yet again, BT ship a router which another <a href='http://www.gnucitizen.org/blog/bt-home-flub-pwnin-the-bt-home-hub-5'>flaw</a> in it.</p>
<p>I’ve always had this gut feeling that <a href='http://en.wikipedia.org/wiki/Universal_Plug_and_Play'>UPnP</a> was bad. It allows machines on your network to modify, for example, settings on your router without any authentication.</p>Check Email Script2008-01-13T00:00:00+00:00http://sheepy.org/2008/01/13/check-email-script<p>So I’ve been playing with <a href='http://conky.sf.net/'>conky</a> again and wanted it to check all my email accounts for new mail.</p>
<p>I found <a href='http://gentoo-wiki.com/TIP_conky#IMAP_Email_Checking_Script'>this</a> script but I wanted to check multiple accounts and show the total mail count in my inbox.</p>
<p>After some hacking here is what I came up with… and yes I’ve removed the base64 encoding as it doesn’t add any security what so ever!</p>
<pre><code>#!/usr/bin/env ruby
require "optparse"
require "net/imap"
#
# Parse options
#
def parse_options(args)
# Hash to hold options
options = Hash.new
parser = OptionParser.new do |opts|
opts.on('-H host', '--host host', 'Host') do |v|
options[:host] = v
end
opts.on('-u user', '--user user', 'User') do |v|
options[:user] = v
end
opts.on('-p password', '--password password', 'Password') do |v|
options[:password] = v
end
opts.on('-h', '--help', 'Displays usage information') do
puts opts
exit 1
end
end
# Parse Parameters
begin
parser.parse!(args)
rescue OptionParser::ParseError => e
puts "Parse Error: " + e
puts parser.to_a
end
# Check for required params
if (options.has_key?(:host) && options.has_key?(:user) && options.has_key?(:password))
#
# Return options
#
options
else
puts parser.to_a
exit 1
end
end
options = parse_options(ARGV)
#
# Lets check some mail!
#
imap = Net::IMAP.new(options[:host])
imap.login(options[:user], options[:password])
#
# Get total mail
#
status = imap.status("inbox", ["MESSAGES", "UNSEEN"])
puts "#{status['UNSEEN']}/#{status['MESSAGES']}"
imap.disconnect</code></pre>DVD Ripping2007-12-19T00:00:00+00:00http://sheepy.org/2007/12/19/dvd-ripping<p>Just found a few scripts that look like they will make the process of ripping a DVD somewhat easier (for backup purposes of course!)</p>
<ul>
<li><a href='http://www.matusiak.eu/numerodix/blog/index.php/2007/01/30/undvd-dvd-ripping-made-easy/'>undvd</a></li>
<li><a href='http://wonkabar.org/archives/373'>dvd2mkv</a> Although the 2nd one rips to <a href='http://www.matroska.org/'>Matroska</a> format which can play on your phone :)</li>
</ul>Wombat Vim Theme2007-10-16T00:00:00+01:00http://sheepy.org/2007/10/16/wombat-vim-theme<p>Might have actually found a theme I like for gvim :)</p>
<p><a href='http://dengmao.wordpress.com/2007/01/22/vim-color-scheme-wombat/'>http://dengmao.wordpress.com/2007/01/22/vim-color-scheme-wombat/</a></p>Sysadmin tools2007-03-01T00:00:00+00:00http://sheepy.org/2007/03/01/sysadmin-tools<p>There are a number of tools sysadmins should be looking at in order to make their life less hellish when it comes to configuration management.</p>
<p>Some tools that I have found so far are:</p>
<ul>
<li>[cfengine](http://www.cfengine.org/)</li>
<li>[Puppet](http://reductivelabs.com/projects/puppet/)</li>
<li>[Bcfg2](http://trac.mcs.anl.gov/projects/bcfg2)</li>
</ul>Password Analysis2007-02-21T00:00:00+00:00http://sheepy.org/2007/02/21/password-analysis<p>After a recent phishing scam, which resulted in someone gaining over 55,000 passwords for my space, someone has posted some analysis on the passwords.</p>
<p>See the results <a href='http://www.schneier.com/blog/archives/2006/12/realworld_passw.html'>here</a></p>Kernel Sorted2007-02-12T00:00:00+00:00http://sheepy.org/2007/02/12/kernel-sorted<p>Upgrading my kernel to 2.6.18 appears to have fixed the issue I was having with aircrack.</p>
<p>Now just need to write a script to make things a bit easier.</p>Aircrack2007-02-08T00:00:00+00:00http://sheepy.org/2007/02/08/aircrack<p>Finally got a 1/2 decent wireless card that will do packet injection.</p>
<p>All I need to do is wait for my kernel to compile again to remove SMP support. At which point I can get on with <a href='http://www.aircrack-ng.org/doku.php?id=how_to_crack_wep_with_no_clients'>this</a> tutorial.</p>Ophcrack2007-02-01T00:00:00+00:00http://sheepy.org/2007/02/01/ophcrack<p>I’ve yet <em>really</em> understand Rainbow Tables but <a href='https://www.isc2.org/cgi-bin/content.cgi?page=738'>this article</a> gives a good overview on the subject.</p>
<p>Using this concept <a href='http://ophcrack.sourceforge.net/'>Ophcrack</a> allows you to recover passwords from Windows based machines very quickly (we are talking single figure minutes at the most here) by exploiting the fact that, by default, Windows uses saltless hashes for passwords.</p>
<p>The iso is downloading at the moment and will be put to good use ;)</p>MySQL Monitoring2007-02-01T00:00:00+00:00http://sheepy.org/2007/02/01/mysql-monitoring<p>Database monitoring is a must if unless you like living on the edge.</p>
<p>Here are some tools to help monitor MySQL so you don’t get any nasty surprises:</p>
<h3>
<a href='http://forums.cacti.net/about6108.html'>
The MySQL Graph Collection for cacti
</a>
</h3>
<p><a href='http://www.cacti.net'>Cacti</a> is a great tool for monitoring, giving you pretty graphs allowing you to easily spot sudden and long term trends.</p>
<p>It takes some setting up but well worth it.</p>
<p>I do not currently know how much extra load this will put on a database but I don’t expect it to be much. Expect and updated post once I’ve done some testing.</p>
<h3>
[mytop](http://jeremy.zawodny.com/mysql/mytop/)
</h3>
<p>mytop is a console based monitoring tool by <a href='http://jeremy.zawodny.com/'>Jeremy Zawodny</a> who helps look after <a href='http://www.yahoo.com/'>Yahoo’s</a> MySQL databases.</p>
<p>It is basically a clone of <em>top</em> for MySQL and is useful for seeing, in real time, what is happening on you MySQL server.</p>
<h3>
[innotop](http://www.xaprb.com/blog/2006/07/02/innotop-mysql-innodb-monitor/)
</h3>
<p>innotop is much like mytop with but specifically for InnoDB tablespace. It can show loads of things such as <a href='http://www.xaprb.com/innotop/innotop-screenshot-T-mode.png'>transactions</a>, <a href='http://www.xaprb.com/innotop/innotop-screenshot-D-mode.png'>deadlocks</a> and <a href='http://www.xaprb.com/innotop/innotop-screenshot-V-mode.png'>statistics</a></p>
<h3>
[MySQL Report](http://hackmysql.com/mysqlreport)
</h3>
<p>This tool does nothing other than take the output of SHOW STATUS and puts it into a more readable form. The main thing is not necessarly the tool itself but the <a href='http://hackmysql.com/mysqlreportguide'>documentation</a> that goes through a sample report and explains it line by line.</p>
<p>This tool is extreamely use for for quickly diagnosing problems related to high load on a database.</p>
<h3>
Other Notes
</h3>
<p>You really should monitor the size of your InnoDB table space and also individual table sizes. Its often the case that tables which grow rapidly contain redundant or old data eating up space and reducing the performance of the server.</p>
<p>I have yet to find a nice script that will do this for you, especially when you have 100s or 1000s of tables, although it shouldn’t be that tricky to hack one together.</p>Vista: The most secure operating system avaliable2007-01-31T00:00:00+00:00http://sheepy.org/2007/01/31/vista-the-most-secure-operating-system-avaliable<p>So apparently <a href='http://news.bbc.co.uk/1/hi/technology/6313981.stm'>Windows Vista is “dramatically more secure than any other operating system released”</a></p>
<p>While I suspect my definition of “secure” is wildly different that of Mr Gates, there are many other <a href='http://arstechnica.com/articles/paedia/hardware/hdcp-vista.ars'>worrying</a> <a href='http://www.theregister.co.uk/2006/10/29/microsoft_vista_eula_analysis/'>things</a> to consider.</p>
<p>I don’t think i’ll be leaving my little Linux/BSD/OS X world just yet.</p>Bash Scripting Guide2007-01-24T00:00:00+00:00http://sheepy.org/2007/01/24/bash-scripting-guide<p>I’m not convinced on some of the coding style but definitly looks useful.</p>
<p><a href='http://freeos.com/guides/lsst/'>http://freeos.com/guides/lsst/</a></p>Suspect Nation2007-01-23T00:00:00+00:00http://sheepy.org/2007/01/23/suspect-nation<p><a href='http://youtube.com/watch?v=Qo0a23YDXLM'>Watch on youtube</a></p>
<p>Well worth watching if you are at all interested in security.</p>
<p>Its a documentary on the “Security” measures that the british goverment are putting in place, making you think if they are really a “good thing”.</p>
<p>For hard core tech people, features include: <ul>
<li>RFID Exploits (including UK Passports)</li>
<li>CCTV Security</li>
<li>Wireless Security</li>
<li>Mobile Phone exploits</li>
</ul></p>
<p>Enjoy!</p>Bluetooth Obex DoS2007-01-23T00:00:00+00:00http://sheepy.org/2007/01/23/bluetooth-obex-dos<p>DoS your mobile phone with lots of Obex pushes. It would seem that most mobiles force you to either accept or deny the push, this disables any other usage of the phone including the ability to turn off bluetooth.</p>
<p>This has apparently been confirmed on the following phones: <ul>
<li>Sony Ericsson K700i</li>
<li>Nokia N70</li>
<li>Motorola MOTORAZR V3</li>
<li>Sony Ericsson W810i</li>
<li>LG Chocolate KG800</li>
</ul></p>
<p>No doubt many other phones are effected.</p>
<p>Attached is the current release of the <a href='http://www.xmailserver.org/ussp-push.html'>ussp-push</a> program which uses the <a href='http://www.bluez.org/'>Bluez</a> bluetooth stack to do a Obex push.</p>
<p>All that is required is a wrapper script of the form: <pre>
while true
do
./ussp-push $MAC@$OCHAN $FILENAME $FILENAME
done
</pre></p>
<p>You can work the rest out yourself…</p>Cafepress2006-12-09T00:00:00+00:00http://sheepy.org/2006/12/09/cafepress<p>Started a cafepress store.<br /><br />Find goodies at <a href='http://www.afepress.com/cowjuice'>http://www.afepress.com/cowjuice</a><br /></p>GnuPlot2006-11-23T00:00:00+00:00http://sheepy.org/2006/11/23/gnuplot<p><a href='http://t16web.lanl.gov/Kawano/gnuplot/index-e.html'>Excellent site</a> on <a href='http://gnuplot.info'>GNUPlot</a></p>tar exploit in the wild2006-11-21T00:00:00+00:00http://sheepy.org/2006/11/21/tar-exploit-in-the-wild<p>Just when you thought you were safe…</p>
<p><a href='http://lists.grok.org.uk/pipermail/full-disclosure/2006-November/050812.html'>http://lists.grok.org.uk/pipermail/full-disclosure/2006-November/050812.html</a></p>
<p>Have tested this and it works as described:</p>
<pre>
nathan@holly /tmp $ tar --version
tar (GNU tar) 1.15.1
nathan@holly /tmp $ ~/tmp/tarxyz > foo.tar
nathan@holly /tmp $ mkdir -p xyz/home/foo
nathan@holly /tmp $ echo "Hello" > xyz/home/foo/hello.txt
nathan@holly /tmp $ tar -rf foo.tar xyz/home/foo
nathan@holly /tmp $ rootdo mkdir /home/foo
nathan@holly /tmp $ rootdo chown nathan /home/foo
nathan@holly /tmp $ rm -rf xyz
nathan@holly /tmp $ tar -xf foo.tar
nathan@holly /tmp $ ls -l xyz
lrwxrwxrwx 1 nathan users 1 Nov 22 00:03 xyz -> /
nathan@holly /tmp $ cat /home/foo/hello.txt
Hello
nathan@holly /tmp $
</pre>DrySQL2006-11-09T00:00:00+00:00http://sheepy.org/2006/11/09/drysql<p>Currently investigating this <a href='http://www.rubyonrails.org/'>Ruby on Rails</a> thing that everyone is raving about and so far I’m liking it.</p>
<p><a href='http://drysql.rubyforge.org/'>DrySQL</a> is one of the many examples as to why Rails rocks :)</p>
<blockquote>
DrySQL is a Ruby plug-in that extends ActiveRecord and applies the DRY principle to Object-Relational Mapping.
The idea behind DrySQL is that you define columns, keys, constraints, and relationships on your database, and you shouldn't need to re-define any of these things in your application code.
</blockquote>
<p>Shame about this thou… <blockquote>has_and_belongs_to_many association generation is not implemented at this point </blockquote></p>
<p>… hopefully it’ll be supported soon.</p>Bogofilter2006-11-04T00:00:00+00:00http://sheepy.org/2006/11/04/bogofilter<p>I, and other people on my server, are having SPAM issues again. Spamassassin is great but it eats RAM for breakfast about the same rate it eats SPAM.</p>
<p>After some hunting I think I have found a solution: <blockquote>Bogofilter is a mail filter that classifies mail as spam or ham (non-spam) by a statistical analysis of the message's header and content (body). The program is able to learn from the user's classifications and corrections.
[Bogofilter Home Page](http://bogofilter.sourceforge.net/)</blockquote></p>
<p>Happy SPAM munching.</p>Hard disk monitoring2006-05-18T00:00:00+01:00http://sheepy.org/2006/05/18/hard-disk-monitoring<p>IDE drives are basically crap. However they are cheap so most people use them.</p>
<p>I’ve recently discovered smartmontools which monitors most hard disks using S.M.A.R.T which allows the disk to do self testing.</p>
<p>Setting up on gentoo is fairly easy and should hopefully give me some advanced warning of disk failure. Together with mdadm , which monitors raid arrays, then you can minimise the risk of loosing data.</p>
<p><strong>Big fat hairy warning</strong> This is NOT any reason not to back up your data.</p>
<p>On with the install and config. <pre>
emerge -v smartmontools
</pre> Now create /etc/smartd.conf and have it contain something like the following: <pre>
/dev/hda -a -m nathan@domain -s (S/../.././02|L/../../4/05)
/dev/hdb -a -m nathan@domain -s (S/../.././03|L/../../5/05)
/dev/hdc -a -m nathan@domain -s (S/../.././04|L/../../6/05)
</pre> The usual start on bootup. <pre>
rc-update add smartd default
/etc/init.d/smartd start
</pre></p>
<p>This config checks all supported attributes of the disks performs a short self test once a day at 2am, 3am and 4am for hda,b and c respectivly. It also does a long self test once a week at 5am on Thurs, Fri and Sat, again respectivly. If a test produces an error then it emails nathan@domain telling me so.</p>
<p>Also check the man page for smartctl for more options and details.</p>
<p>Next up - mdadm This is fairly trivial to set up and get working. Again it emails you when there is a problem.</p>
<p>Install emerge -v mdadm</p>
<p>Create the mdadm.conf with details of our RAID array <pre>
mdadm --detail --scan > /etc/mdadm.conf
</pre></p>
<p>Set your email address <pre>
echo "MAILADDR nathan@domain" >> /etc/mdadm.conf
</pre> Make the monitor start on boot, and start it now. <pre>
echo "mdadm --monitor --scan --daemonise --config=/etc/mdadm.conf" >> /etc/conf.d/local.start
/etc/init.d/local restart
</pre></p>Dell 2005fpw2006-04-06T00:00:00+01:00http://sheepy.org/2006/04/06/dell-2005fpw<p>My Dell 2005fpw 20” widescreen TFT arrived today.</p>
<p>Hunting arround there appears to be a few different modelines that people use for it.</p>
<p>After some more digging the most popular seemed to be the following:</p>
<pre>
ModeLine "1680x1050" 146.2 1680 1960 2136 2240
1050 1080 1086 1089 -hsync +vsync
</pre>
<p>So the relevent part of /etc/X11/xorg.conf looks like:</p>
<pre>
Section "Modes"
Identifier "16:10"
ModeLine "1680x1050" 146.2 1680 1960 2136 2240
1050 1080 1086 1089 -hsync +vsync
EndSection
Section "Monitor"
Identifier "dell"
UseModes "16:10"
HorizSync 30-83
VertRefresh 56-75
EndSection
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "dell"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1680x1050" "1280x1024"
"1024x768" "800x600"
Viewport 0 0
EndSubSection
EndSection
</pre>...and so it begins2006-03-13T00:00:00+00:00http://sheepy.org/2006/03/13/and-so-it-begins<p>Powered by <a href='http://www.drupal.org/'>Drupal</a> this time.</p>