Lotus Support is clearly overwhelmed

My friend Ninke logs PMR's with Lotus Support on an almost weekly basis. He checked one of his PMR's today and found this note:

Called client and explained the reason for the delay. Enormous
amount of PMR's currently opened and ongoing, impacting
unfortunately all, especially lower prior/sev 4/4.

There are two things this brings to mind. First, Ed Brill keeps saying that there have been fewer issues reported for R8 than previous releases. That being the case it's odd that both I and the four Domino admins I talk to almost daily, including Ninke, have all opened more PMR's for R8 than they did for any others. I'm sure Ed's statistics are more representative than my limited sample, but it's still hard for me to reconcile the official reports with what I see firsthand.

Second, why would one customer having a large number of PMR's open slow down resolution on all their PMR's? I didn't think IBM dedicated staff to resolving individual customers' problems. Even if they did that should speed things up, not slow it down. If they don't, why would the volume of open PMR's have any impact on resolution? Something seems to be broken besides Notes and Domino 8.x.

VMWare ESX, virtualized DNS and an ISCI SAN

Since the fire we had last year we have replaced all our old servers with a new virtualized infrastructure. We're running VMware ESX 3.5, an HP BLc-3000 blade chassis with six blades, and an HP AiO1200R ISCSI SAN. It is working great and I have a writeup about that decision-making process that I will be publishing shortly.

Today I wanted to bring up one of the potential pitfalls when you're creating a fully virtualized environment. This past weekend we had to cut building power for an extended period of time, so the network administrator brought down everything in our server room. As he brought everything back online he realized that Virtual Center, the control console for VMware ESX, could not talk to the SAN because it required DNS resolution.

The Problem

Our DNS servers are virtualized with storage on the SAN. He ran into a chicken-and-egg situation where he had dependent services that relied on each other.

It took him a while to realize that DNS was the issue. The logs on the SAN side simply said "Could not connect ISCSI LUN". On the VMware side the virtual machines said "storage not available". Figuring out why the two were unable to connect took some careful analysis. Solving it proved difficult because our departmental wiki also used SAN storage, so he had no access to our documentation. In a flash he found himself back in the same situation he was in after the fire, when he could not access critical documentation because the servers with it were not available.

The Solution

So how did he solve it? Luckily he still had the old primary domain controller hanging out, which had all the DNS information. He was extremely lucky, and he knows it. To keep from having to rely on luck, how should you configure your VMware environment so this doesn't happen to you? There are a couple of ways to tackle it.

Use local storage for your virtualized name servers.

Pros

• Name servers will load without SAN access.
• Resilient to SAN outages.

Cons

• Cannot mix guest VM's that require SAN storage. The ISCSI initiator in VMware ESX loads when ESX boots. By having your DNS server on the same physical host as another VM that requires SAN storage, the guest on SAN storage will not be able to start.

Use a non-virtualized DNS server.

Pros

• Resilient to SAN outages.

Cons

• If using a Windows server, also requires you run Active Directory services.

Use hosts files.

Pros

• Resilient to SAN outages.
• May improve performance slightly since lookups will always be from local cache.

Cons

• Requires you add hosts files to the Virtual Center server, SAN server, and every ESX host server.
• Can be a maintenance burden if your environment changes frequently and you have to constantly add/remove ESX hosts.

We have opted for the last option. Our VMware host environment is fairly static, so maintaining hosts files will be a minimal maintenance issue. The resilience we gain from it make it very worthwhile. Oh, and we printed a copy of our wiki page that has all the hostnames and IP addresses of every server we have, and put it in the safe. :-) You do have a similar list, and a fireproof safe... right?

A script to check remote computers for directories

At work we needed a way to check servers to see if certain software had been installed. The easiest way was to check for the software's installation directory. There isn't an easy way to do this remotely, though, so I wrote a script to take care of it: dircheck.vbs.

While I was writing this tool I learned a lot about VBScript. For starters, you can't interact with stdin or stdout using the default VBS command interpreter. If you try to write information to the user's console it will display everything in a popup. To fix this, you can use the special cscript interpreter:

cscript dircheck.vbs

If you execute the above command you will get command line help for the utility. Full source code is obviously included, so please feel free to use it however you need to.

How to copy SQL Server DBMail configuration to another server

I'm setting up a new SQL Server from scratch and wanted to copy the existing DBMail configuration from the old server. I did some searches and the best I could find were pointers to the msdb.dbo.sysmail_* system tables. I did some trial and error and got everything copied over, so here's how I did it.

1. Log into the new server
   
2. Create a server link from the new server to the old server
3. Copy the DBMail configuration
   

I had to log into the new server and do the server link there. From my workstation SQL Server considered it a redirection, and that is a security violation. Save yourself some headaches and just start at the new server. Note that the following SQL script will delete any existing DBMail configuration in the target SQL Server. If you want to keep the existing configuration you'll need to take out the DELETE and SET IDENTITY_INSERT statements and manipulate the account_id and profile_id in the related tables.

SET IDENTITY_INSERT sysmail_account ON
INSERT INTO sysmail_account (account_id, [name], [description], email_address, display_name, replyto_address, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_account
SET IDENTITY_INSERT sysmail_account OFF
GO

DELETE sysmail_configuration
GO
INSERT INTO sysmail_configuration (paramname, paramvalue, [description], last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_configuration
GO

DELETE FROM sysmail_profile
GO
SET IDENTITY_INSERT sysmail_profile ON
INSERT INTO sysmail_profile (profile_id, [name], [description], last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_profile
SET IDENTITY_INSERT sysmail_profile OFF
GO

DELETE FROM sysmail_principalprofile
GO
INSERT INTO sysmail_principalprofile (profile_id, principal_sid, is_default, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_principalprofile
GO

DELETE FROM sysmail_profileaccount
GO
INSERT INTO sysmail_profileaccount (profile_id, account_id, sequence_number, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_profileaccount
GO

DELETE FROM sysmail_servertype
GO
INSERT INTO sysmail_servertype (servertype, is_incoming, is_outgoing, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_servertype
GO

DELETE FROM sysmail_server
GO
INSERT INTO sysmail_server (account_id, servertype, servername, port, username, credential_id, use_default_credentials,
enable_ssl, flags, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_server
GO

free alternative to defrag.nsf

My friend Adam asked me recently about defrag.nsf. In case you don't know, this is a Windows-only tool for Domino that will do a file-level defragmentation of Domino databases. According to the product page the theory is this will increase performance. I haven't tried the product so I can't state whether this is true or not, but I do know that you can get the same results for free.

defrag.nsf is using the Windows defragmentation API to do file-specific defragmentation. Microsoft baked this into Windows NT 4 and the same API has been in Windows 2000, XP and Server 2003 and 2008. It is robust, stable, and has been proven over time. Because all the necessary libraries are included with Windows you could write this application yourself if you wanted to.

But you don't have to. Sysinternals, a division of Microsoft, has the free contig tool that does the exact same thing. You can defrag a single file, a directory, or recurse directories. And it accepts wildcards, so you could defrag "c:\program files\ibm\lotus\domino\data\mail\*.nsf".

If you decide you want to find out if file fragmentation is an issue for your Domino server it wouldn't hurt to try out the free contig tool and compare it to the results from defrag.nsf.

P.S. While I was researching this I came across Ulrich Kraus' write up of contig. The comments there include links to more free defragmentation utilities.

blueberry lime sauce

You can use this with anything that needs a slightly sweet, fruity, and citrusy pop of flavor. I served this with braised pork belly.

1 C Riesling wine
1 C unsweetened blueberry juice
2 cardamom pod, or 1/4 teaspoon ground cardamom
2 kaffir lime leaves, or the zest of 2 limes
juice from 1 lime
sugar to taste
salt to taste
2 small saucepans
strainer

Put the Riesling in a small saucepan. Heat over medium-high heat until it reduces to 1/4 to 1/3 of a cup.
Remove from the heat
Thinly slice the kaffir lime leaves and stir the ribbons into the Riesling reduction, or add the lime zest
Add a pinch of salt, stir, and let sit.

Put the blueberry juice and cardamom pod (if using, ground cardamom later would be added later) into a second sauce pan. Heat over medium-high heat until it reduces by about 1/2.
Add the Riesling reduction to the blueberry reduction. Add the ground cardamom now, if you're not using a whole pod.
Reduce the entire mixture to about 1/2 a cup
Stir in sugar to taste. How much you need depends on how sweet the Riesling and blueberry juice was to begin with.
Add salt to taste
Continue cooking until sugar and salt are fully dissolved, about 1 minute

Allow to completely cool, then strain into a storage container. Stir in the lime juice.

buttermilk cake with spiced vanilla icing

This is the buttermilk cake I have been making for dinner parties recently. It's delicious, easy, and a little unexpected with the buttermilk and butternut squash. I organized the ingredients into the groups you will need to prepare this recipe.

For the cake 10 cup bundt pan 1T unsalted butter, softened 1/4 cup unbleached all-purpose flour 4 oz (1/2 cup) unsalted butter, softened 1 1/2 cups granulated sugar 1/2 cup canola oil 2 large eggs 1 Tbs distilled white vinegar 2 tsp pure vanilla extract 13 1/2 oz (3 cups) unbleached all-purpose flour 1 tsp baking soda 1 tsp table salt 1/2 tsp ground ginger 1/4 tsp freshly grated nutmeg 3/4 cup buttermilk 2 1/4 cups peeled and grated butternut squash (about 8 oz)

For the icing and garnish 9 oz (2 1/4 cups) confectioners' sugar 3 Tbs buttermilk; more as needed 1 tsp pure vanilla extract 1/4 tsp freshly grated nutmeg 1/4 tsp table salt 1/4 cup finely chopped crystallized ginger

Make the cake

• Position a rack in the center of the oven and heat the oven to 325. Butter and flour a 10 cup bundt pan, tap out excess flour
  
• Using a hand mixer or a stand mixer with a paddle attachment beat the butter and sugar on medium speed in a large bowl until well combined, about 1 minute
  
• Add the oil and beat until combined, about 15 seconds
  
• Add the eggs one at a time, mixing well on low speed
  
• Add the vinegar and vanilla and mix until just combined
  
• Add half the flour and the baking soda, salt, ginger and nutmeg, mixing on low speed until just combined
  
• Add half the buttermilk and mix until just combined
  
• Add the remainder of the flour and buttermilk, mixing until combined
  
• Stir the squash into the batter
  
• Transfer the batter into your prepared bundt pan and smooth the top with a rubber spatula
  
• Bake the cake until a tester comes out clean, about 1 hour
  
• Remove from the oven and cool the cake in the pan for 30 minutes
  
• Carefully invert the cake onto a wire rack. You want to do this while the cake is still slightly warm to minimize sticking.
  
• When the cake is completely cool transfer to a serving plate
  

Make the icing

• In a medium bowl using a whisk or hand mixer on low speed blend the sugar, buttermilk, vanilla, nutmeg and salt until smooth
  
• Continue mixing and add more buttermilk a few drops a time until the icing is still quite thick but pourable
  
• Pour the icing back and forth over the cake in thick ribbons, or drizzle using a spatula
  
• Sprinkle with crystallized ginger
  
• Let the iced cake sit at room temperature for about 45 minutes before serving

Rufous, the red-lored Amazon

In September 1989 Myron got a new parrot. It was a red-lored Amazon that he named Rufous. Myron got Rufous when he was so young Myron had to hand feed him three times a day. Over the next twenty years Rufous learned to speak, saying "what are you doing", "bye", "hello birdie". In the last year or so he started mimicking my laugh.

Tonight as we were getting our dinner party underway we went into the kitchen and Rufous was in the bottom of his cage with his wings spread and his beak open. Myron picked him up and Rufous bit him, which Myron took as a good sign. Myron gave Rufous some water with a spoon, which he drank, and Myron returned him to his cage. Within a few minutes he was sprawled with his wings spread, panting.

Myron called the emergency vet, which did not have an avian vet on staff but said they would do what the could. By the time he got there Rufous was barely breathing. Rufous died shortly afterwards. The cause of death is unknown.

I can't imagine what Myron is going through right now. Having a pet for 20 years and losing it so suddenly is an unthinkable horror.

R.I.P Rufous, 1989 - 2009

Dining in the Vail area

In Colorado we stayed in Avon, which is about 8 miles from Vail proper. This is a playground for the super rich and the dining scene follows suit. That's not to say that the old Colorado cowboy culture has been completely subverted, though. During our stay we enjoyed everything from rustic cowboy fare at The Gashouse to the ultimate in fine dining at Kelly Liken's eponymous restaurant. In between we had soft-shelled crab po boy's, tenderloin sandwiches, amazingly good Chinese, and a spectacular dinner at a slopeside mountain cabin. Our least expensive meal was breakfast at Daylight Donuts in Breckenridge ($17) and the most expensive was at Kelly Liken in Vail ($440).

The Good

Our first night in town we looked through the dining guides and came up with a few ideas, then asked the front desk staff which one they liked. They all raved about Fiesta Jalisco's, which you may have guessed is a Mexican restaurant. It was a short (but very cold) walk to the restaurant, where we had to wait about 15 minutes for a table. The restaurant was packed and insanely crowded, and we heard several locals who were leaving say they had never seen it that busy.

The (American) traditional chips and salsa was served with a coleslaw made from shredded cabbage dressed in lime juice. It was incredibly good, and our waiter said it was a traditional accompaniment with fish tacos in Puerto Vallarta, a coastal town in the Mexican state of Jalisco.

The food was very good but the standout was the Original Margaritas. They consisted simply of tequila, cointreau, and lime juice. That's it. They were also shaken with ice and served in a martini glass, not on the rocks or frozen. It was a delicious concoction and one I'll be sure to work hard to perfect. Do I have any taste testers who will volunteer?

The Better

Colorado is known for wild game. They hunt everything that walks, flies, swims or crawls, and you can find it on a menu somewhere. Many restaurants specialize in wild game, so we sought one out. We were very happy to find The Gashouse in Edwards, about 15 minutes away from Avon. The decor features mounted animal heads of everything from deer and antelope to cape buffalo.

The must-have dishes were the buffalo carpaccio, which was lightly smoked but still served raw, and the truffle and parmesan cheese fries. Both were superb.

The Even Better

One night Myron booked us at Zach's Cabin, which is located behind the Ritz Carlton in Bachelor's Gulch, a part of the Beaver Creek ski resort. We arrived a few minutes early and had wonderful pre-dinner drinks at Spago's bar. At the designated time we went out to a sled that was drawn by a snow cat (for those unfamiliar think bulldozer, but lower to the ground and wider), which took us up the mountain to a cabin in the woods.

The restaurant is a classic log cabin, complete with a double-sided fireplace. We were seated in a cozy corner table and the magic unfolded. The decor and ambience were wonderful and the service spectacular. The elk tenderloin was butter tender, and baklava cheesecake was to die for. It was a dollop of delicious cheesecake filling in a fillo dough shell, drizzled with honey. Sublime.

The only disappointment was that the wine list was shockingly overpriced. Bottles I have bought for $20 were over $100. The least expensive wines were still over $30 and I wouldn't even buy them at $6, which is what they are in Charleston.

The Best

Kelly Liken has been called rising star among female chefs and she has been featured in magazines ranging from Bon Appetit to Sky Magazine. I'm not sure where I first learned of her, but I was reminded of her presence in Vail by the in-flight magazine. We had done dinner at Zach's Cabin, which was fairly pricey, so we weren't sure we wanted to do something even higher end. Finally I decided that since we were there I was going to splurge and worry about paying for it later. That's the American way, right?

One of the options was a custom tasting menu with custom wine pairings. We put ourselves in the hands of Chef Liken and the very capable sommelier, Jeremy, for a completely blind five course tasting menu. We started with the cobia crudo, continued with braised pork belly, honey marinated duck breast, Colorado rack of lamb, and finished with an Earl Grey tea infused chocolate truffle cake. It was a nearly orgasmic progression. All five courses were delicious, the wine pairings were perfect, and the service superb. The only complaint I could offer is it was a little loud and sometimes hard to hear, but even that is highly subjective and dependent on who is dining that particular night. This was one of the most memorable dining experiences of my life, rating up there with our dinner at La Pergola.

The most striking thing I came away with was being introduced to banyuls, which is essentially a French port made from grenache. It is lighter and brighter in fruit than port and pairs wickedly well with chocolate (we had it with the chocolate truffle cake). If you like port you owe it to yourself to seek this out.

advertising and marketing

There has been much talk in the Yellowverse about advertising, marketing and how the two interrelate -- or not. Most of the people I know still seem to correlate the two, and (ironically) equate the attempts at making a distinction just marketing spin. To be honest, I was one of them until I had an epiphany today. Before we get into that, let's start with some definitions:

Marketing is defined by the American Marketing Association as the activity, set of institutions, and processes for creating, communicating, delivering, and exchanging offerings that have value for customers, clients, partners, and society at large. The term developed from the original meaning which referred literally to going to market, as in shopping, or going to a market to sell goods or services.

Advertising is a form of communication that typically attempts to persuade potential customers to purchase or to consume more of a particular brand of product or service.

Okay... that still sounds about the same, right? First, advertising is a subset of marketing. Therefore all advertising is marketing, but not all marketing is advertising. Second, the distinction lies in the intent. Marketing is about disseminating information; advertising is about persuasion.

Still not clear? Suppose you're at a farmer's market and you want tomatoes. You see someone with tomatoes displayed, and you go buy them. He doesn't have to tell you to buy them or announce their availability, him simply presenting them is all the marketing you need. It would be advertising if he were yelling out loud "Get your fresh tomatoes here!" since he is trying to influence you to purchase his tomatoes. The announcement would be marketing, the persuasion (via the command "get") is advertising.

So, circling back to where I started, what led me to finally grasp the distinction between advertising and marketing was a blog post by Seth Godin titled "which comes first, the product or the marketing". In it he points out that most of the time you actually need the marketing first since that will drive product creation. Once I read that I recalled a Facebook status update by Matt White and a blog post by Ben Langhinrichs illustrating how this has started happening. IdeaJam is marketing because a community is exchanging information in an attempt to achieve something of value to them. Some entrepreneurial types are taking this and capitalizing on it by delivering products to meet the market demand.