I said, “Honey, you know I’ve been in IT way too long to do some rookie thing like that…” and so it stayed.

New York City Taxi Cab Software Kiosk Windows Crashed Application

Click For Large Image Diagram

I serve a large organization with multiple branch offices in remote places. Typically, these offices are staffed with from one to fifty employees. Most of the offices have a local Windows Domain Controller, which doubles as a file/print server, DNS, and DHCP server.

The larger of the offices are usually connected to the Internet via T1, or DSL via a local service provider. In addition, the offices have a firewall that is connected back to headquarters via IPSec VPN tunnel.

This arrangement has provided a good solution for several years, however there are limitations. Recently, many of our offices have begun providing Internet access for clients – this added network load, in addition to increased usage of high-bandwidth services like You Tube, have placed new demands on us to manage the bandwidth. In addition, configuring servers and firewalls per individual office – and getting local support who can help us on site – is challenging.

A major point of our current initiative is to make our network locations more homogeneous, and more under the control of IT staff at headquarters. Virtualization has become an attractive option for several reasons: it eliminates the cost of a separate hardware firewall, and it allows us to configure a hardware-agnostic server “image” for use on any local hardware.

My “One Box Solution” allows for the firewall, bandwidth management, and Windows Domain Controller to exist on a single, portable, server.

I recently began piloting such a solution in one of our offices. Not having a current budget for my project, I took advantage of an unused Dell workstation at HQ. I added an additional 10/100 NIC we had lying in our closet. I installed Ubuntu 8.10 server (any version of Linux will work) and VMWare’s latest version of free server for Linux.

If you haven’t been exposed to VMWare yet, go to www.vmware.com and download the free player and one of the free virtual appliances (pre configured workstations and computers). VMWare server is also free and will allow you to build and configure your own virtual machines.

Having built the Linux box and installed VMWare server, I configured one of my NICS as an internal nic, with an address on my local subnet, and the other NIC as an external NIC with one of my assigned Internet IP addresses.

Inside VMWare server, I configured three virtual networks. One network connected to my external interface of my Linux server. The second connected to a host only virtual network, and the third connected to the internal NIC of my Linux server.

The first virtual server I built was my M0n0wall firewall. Note that M0n0Wall is available PRE-BUILT! AS A VIRTUAL APPLIANCE! This means you don’t have to compile or build it. Just download the Virtual Appliance files and open them in VMWare Server.  I choose M0n0wall for several reasons. It is free. It is easy to configure. It allows for QoS, Traffic Shaping, and most importantly the IPSec tunnels that connect back to Headquarters and our DR NOC. The external WAN interface of M0n0Wall was connected to the external virtual network. The internal LAN interface of M0n0Wall was connected to the HOST ONLY virtual nework (we’ll see why in sec…). The WAN and LAN interfaces were configured with appropriate network settings, NTP server settings, DNS, etc. The WAN IP will be on our Internet subnet, and the Gateway will point to our ISP’s router or gateway IP address. The LAN interface will become the internal default gateway for our local network. M0n0wall is a powerful firewall solution for a small office. With the addition of a third NIC, you can easily set M0n0wall up to provide a DMZ, or a Captive Portal for your Wireless users. A Captive Portal will allow you to plug in a wireless device, authenticate users in a browser, and/or use RADIUS for advanced authentication.

The second virtual server is Untangle – also available Pre-built as a virtual appliance! Untangle can install as a bridge – meaning there is no routing involved. It sits between your firewall and your internal network. Untangle can also function as the firewall, but since it lacks the IPSec function for our tunnel back to Headquarters, we choose M0n0Wall. Where Untangle really excels is in Internet filtering and management. Untangle provides a suite of free modules for management and OpenVPN. There are also paid and supported modules available. The free version provides for very granular reporting and a powerful degree of access control right out of the box.

The third server is our Windows Domain Controller. This is the only commercial device which requires a paid-for license. Our virtual domain controller runs inside VMWare and connects via one interface to our internal network.

This arrangement has allowed us to provide a one-box all-in-one appliance to our remote offices that can be built and exchanged as needed, with a minimum of configuration. The Windows server can easily be promoted to a domain controller on site. IP Addresses and other site specific information can all be easily configured through graphical utilities.

Using online backup, such as Mozy Pro, in conjunction with our single box, we have discovered a way to provide highly-available network services to our smaller, budget and staff challenged offices in the field.

Options include using ESXi, which is VMWare’s free version of ESX server. The downside of ESXi is that it requires more expensive hardware. It will not run on a workstation with a SATA disk drive. However, if you have a true server that is on the hardware compatibility list, ESXi will provide a better platform. It installs as it’s own OS. Linux and Windows are not required. The management tools and options are also much nicer.

One additional thing we’ve looked at – and likely something we’ll be hearing more about in the coming months – are WAN optimization appliances such as Riverbed that run as a virtual machine. This will likely become a solution in our most remote offices where slow satellite connectivity is the norm.

The problem is that you are using an XP build using nLite. NLite removes uncessary Windows components to make a compact and speedy image. Unfortunately, it is not officially supported by Microsoft. When you install Windows XP Service Pack 3, the sysoc.inf file is overwritten. The deleted configuration information for the components is put back into the new sysoc.inf file written by Service Pack 3. When you attempt to open Windows Compoents in Add Remove Programs, it will look for the missing pieces due to sysoc.inf.

To fix this, I did the following: as each error comes up in Add/Remove programs, simply delete the corresponding line from sysoc.inf. Make a backup of sysoc.inf first. Sysoc.inf is located in your windows\inf folder (maybe hidden, so enabling viewing hidden files or be a real man/woman and use the command line <giggle>.

The error will also result in errors for the following files (components):

xsocm.inf
fxsocm.dll
setupqry.inf
setupqry.dll
fp40ext.inf
fp40ext.dll
msgrocm.dll
msnmsn.inf
rootau.inf
games.inf
communic.inf
optional.inf
pinball.inf
igames
zoneoc.dll

Some things are just plain stupid.

I work in linux a lot as well these days. I’m just finding after years of Microsoft that I really like linux. It is my base OS and I run VMWare or RDP sessions to manage my Microsoft shop.

I’ve been wanting to use a simple POP or IMAP client to check my mail when I’m traveling, have poor connectivity, etc. My company has many people overseas on very slow and highly contended VSAT connections. I don’t like passing my credentials in plain text.

I applied a secure certificate to the virtual POP3, IMAP, and SMTP servers on our Exchange 2003 front ends. I opened the appropriate firewall ports, and voila we had secure IMAP and POP3. SMTP was a little weird.

Exchange does not use a separate port for secure SMTP. It uses port 25 for everything. This works out fine. When you configure your client, choose TLS and not SSL. Make sure it is port 25. You can confirm your email was sent encrypted by looking at the header (send yourself a test). It will say the email was received by your smtp server in an encrypted session.

One last gotcha – the secure SMTP worked inside the firewall and not outside. When using telnet into the mail server on port 25 from the inside, an EHLO issued a full string of options, including STARTTLS. Outside the firewall, these options were only a string of XXXX’s. Cisco firewalls using inspect ESMTP statements filter out the STARTTLS option. This also causes the client to fail with an error stating the STARTLS is not offered. Remove the ip inspect esmtp statement and all will be well.

Outlook and Outlook Express use SMTP 25 and the SSL option, not TLS. Any other client, choose TLS.

Clients tested: Thunderbird, Evolution, Outlook Express

Note: if you are using Outlook or Evolution you might consider using the rpc/https built into these two clients.

The answer was to uninstall Microsoft Exchange Tools and run the command fixmapi.

Now, it works.

AppName: outlook.exe      AppVer: 12.0.6316.5000     AppStamp:4833a470
ModName: oladd.fae      ModVer: 12.0.4518.1014      ModStamp:45417457
fDebug: 0       Offset: 00008c24

Windows XP SP 3, Outlook 2007 SP 1

Here’s why.

I work for a very large nonprofit humanitarian organization. In all fairness, we have received generous gifts from Microsoft in the past. However, we have been involved in a massive project to roll out a company intranet portal for about two years. Large sums of money were invested – primarily on development – and the product was unleashed on the user base (1000 domestically and several thousand overseas) about two months ago.

We built the platform on an infrastructure consisting of a SAN based clustered SQL back end, an application server, and two load balanced front end servers. This has proven to be more than enough horsepower. Most of the overseas users are stuck with very slow VSAT connections and thus the demand on our end to support simultaneous transactions is low.

Soon after we released the product, we began to experience several bizarre issues. The environment is highly customized – it is not an out of the box build by any means. However, the vendor who did the development had every Microsoft blessing a vendor could have.

Among the issues we have experienced are that suddenly new users are unable to access the site. Old users are still able to access it. The permissions, policies, groups, ou’s and every other Active Directory attribute of the new users is identical to the old users.

In addition to this issue, we are unable to modify the existing groups in SharePoint.

The most recent issue involved adding a third front end (WFE in Microsoft parlance) web server to the farm, which we were unable to do. This is the point where we decided to open a $249 ticket with Microsoft to solve the issue. This is the experience I shall discuss.

Our IT Director opened the case which I promptly took over. I was put on the phone with a support ‘engineer’ who was clearly in a country that begins with the letter ‘I’. I assure you that I do not have anything against a person’s ethnicity, country of origin, or anything else – especially when they are trying to help me solve a problem! However, the connection sounded like he was on 1960’s Apollo mission circling the dark side of the Moon, or perhaps calling from the Titanic as it was about to bubble beneath the sea. In addition to the horrid quality of the connection which kept dropping out, his accent was so severe I could barely understand him.

I was on the phone, quite patiently, for 8 1/2 hours that Friday. My overseas friend ran through every wizard, every permission and registry key. When he had scraped the bottom of his barrel of tricks he began taking shots in the dark – none of which proved fruitful. Eventually he was able to make the ‘wizard’ say that the server had been successfully joined to the farm – at which point he tried to say that he met his obligations according to the original scope of the ticket. However, when we tried to browse the site from the new web server we received only errors.

Finally I told him I had to go and we put the case on hold until Monday morning. He was supposed to call at 10AM but did not. I called in and was put through to another ‘engineer’ in the same country. The connection was again so bad and his accent so severe that I refused to repeat the eight and a half hour torture session I had been through on Friday. This resulted in me having to call back and get in the queue two more times until I finally got through on a usable connection with another overseas engineer with a much clearer enunciation of the English language. I spent another five hours on the phone with him which proved fruitless and again reached the point of taking shots in the dark. At this point I really didn’t have much patience left and turned the call back over to my Director. He echoed my sentiments that clearly we weren’t getting anywhere and the case needed to be escalated. It appeared that the issue most likely was coming from a corrupted configuration database and not from the actual install of SharePoint. However, these “enterpise level support” engineers had no idea how to troubleshoot beyond the wizards and graphical tools of IIS and SharePoint Central Management. It seemed fairly obvsious that the troubleshooting would likely have to move into the realm of tracing SQL transactions, looking at database tables, etc – or perhaps even restoring an old backup of the configuration database if that was the culprit. We didn’t want to attempt these things ourselves because we are not SharePoint people. Our department is lean – I manage everything from switching, routing, VMWare and SAN administration to Exchange. Digging into the guts of SharePoint was something we wanted to leave to the ‘experts’.

When my boss explained this to the engineer he told us he was the last line of support and the case could go no further than he. That seemed a bit difficult to believe. I’ve been involved in other such issues that eventually went to the software engineering team responsible for building it. So my friend overseas is the last line of support for one of the largest and richest corporations in the world. Pretty impressive.

Essentially we cut our losses and tried to contact someone else at Microsoft who could direct us on a more fruitful support path. We failed. We were lied to, given attitude, and again told we had no other choice. We have contracted a private SharePoint engineer responsible for some very large corporate projects and hoping he will be able to help.

Tell me how a corporation the size of Microsoft with such a near global ‘monopoly’ in the business world, omnipresent and all powerful, can not provide a decent quality of support? The connections were unusable or barely usable, the engineers were unable to be understood, in some cases they were rude and indignant, and ultimately were unable to solve our issue. Their understanding of the product was limited to a very narrow scope and they clearly did not understand things like clustering, or load balancing. One engineer tried to tell me that all users on a domain by default have permission to log onto a server. Can you imagine?

I can not express how deeply disturbed, angry and disappointed I am. In the mean time I’m cozying up to Linux. I can assure you my next out of the box computer will be a Mac.

This would be a great time to buy Apple…

I managed to tweak that HP into submission so it worked pretty well on XP Pro; but when a new opportunity came up for an Intel Dual Core 2 processor HP 6910p laptop I jumped at it. A little more weight, but the full size/speed hard drive, the bigger screen, and opportunity to run 64 bit OS’s were more than enough to motivate me.

 The laptop came with Vista, of course. There was a choice between 64 and 32 bit versions. Normally, I wipe all of that off the laptop, repartition and install everything myself from scratch, just the way I want it. But in this case, we don’t have Vista 64, or a volume license – and there are all these little features on the laptop I decided to go ahead and give it a go. Time to jump in and get the feet wet and all of that. What the hell, Vista x64 here we come.

As I expected, I ran into a lot of walls. I’ve spent the past 72 hours fussing about and figured out the following. I may not have it all quite exactly right, but I believe you will find solutions to most Vista issues can be solved by researching around the issues mentioned below.

Most of my issues involve networking and permissions which are fairly different from anything prior. In many ways it seems they have tried to incorporate linux or Mac-like features; but it is such a hybrid atmosphere that it becomes totally non intuitive. I’m constantly second guessing: could this be a Mac type solution, could this be a Linux type solution, is this plain old Windows, or is this some new Vista ‘feature’.

A few things I’ve come across:

1. UAC – user account control: similar to linux or Mac, the logged on user (even one with administrative rights) operates with reduced rights. Access to any settings, features, and many applications must be confirmed by clicking ‘OK’. A great idea in theory, but presents ALL kinds of issues since older management tools were not designed with this in mind. Example: this feature seemed to cause issues opening management tools in an mmc console. I ended up turning it off, which solved a lot of issues. In linux you run as root, and everything underneath (any processes run) runs as root automatically. It works quite well – but this has been a ‘feature’ of Unix/linux forever and so my theory is that it just doesn’t really work in Vista. Not if you’re in a mixed  environment, and who isn’t?

2. Authentication – Vista forces the highest level of authentication which is NTLM v.2 – unfortunately many non Microsoft systems, older Microsoft systems, and XP based shares (even on a domain) do not support it. The result is you can’t access shares or printers on many linux/samba systems or Microsoft workstations. The solution is to edit the local system policy on Vista to fail back to the older authentication methods. This will likely come up as an issue in some of our field offices.

3. Accessing shares on an XP Pro domain workstation required creating a local user on the workstation with the same username - userxyz – as the username I am logged onto my Vista workstation with. This is common in peer to peer environments but not domain environments where you would be prompted for a username and password. Accessing my Windows servers does result in a prompt to authenticate.

4. Exchange 2007 and 2003 management tools are not supported on Vista

5. Cisco does not have a VPN client for Vista x64 and does not plan to release one.

6. Adding Vista to a domain that has GPO’s enabled can result in issues not seen on XP or 2000 workstations. This primarily centers around the LOCAL SERVICE and NETWORK SERVICE accounts on the Vista workstation. If security rights like “adjust memory quotas” or anything related to “process level tokens” are set in GPO, they will defeat the local rights of service accounts on the workstation. The fix is to add these two accounts into your domain group policy objects so they get applied (and there are a number of new entries for Vista)

All in all, I’m not seeing the WHY of Vista, except that it is kind of pretty… One guy described it as a “pig with lipstick” but I won’t go so far. The potential exists for some real security improvements, but mostly I’ve had to defeat all of them to be compatible in my existing environments.

As for x64 – most of my 32bit apps work, with the exception of apps that involve any direct type of hardware/network involvement (here it is 50/50). More apps work on XP-64 than work on Vista-64. There are a number of 64bit solutions for many applications, but it is very spotty. I find the lack of management tools for Microsoft’s flagship Exchange 2007 to be quite puzzling.

Problem: When connected to my internal LAN via VPN client, or through IPSec tunnel I can’t upload FTP files to my internal FTP server. Additionally, when trying to access external FTP servers through my ISA 2006 server, I can not upload.

I scoured my FTP server looking for an issue, when it occured to me I ought to try access from inside the firewall. It worked. Then I searched every access list, cache setting, rule, and property I could think of in ISA 2006.

Finally I found the problem on Tom’s Blog at Lanlogic - Find the rule that applies between your client and the server. This might be the second to last rule, or it might be “allow outbound access to the Internet”. It will be the same rule that sits between you and the FTP server. It probably says “allow all outbound access”. Guess what? It lies.

Right click the rule and LOOK at the drop down, past properties… You will see this. Click the “configure FTP” option. You will see a checked check box that says “Read Only”. Un check it. FTP problem with ISA 2006 solved.

These are the things I love about Microsoft.