Continous Installation : knowing the version

With the setup I have made to install the software at our clients server, there was a small problem : how to know which version was installed where? Now you can keep logs in excel/word or whatever, but my experience is that these get outdated very easily. So I wanted to know which version was installed, and the ability to get an overview of this information of ALL software on ALL servers. Turns out that this is not so hard to do : just create a custom labeler, and use the dashboard to view all the servers.
Normally a 'build' label is generated when you check something out of source control, a continuous increasing number, a date based number, ... but always something related to a source control. Luckily CCNet already made abstraction of this, making it easy to create labels with custom logic. Since this logic is very dependant on our setup, I wrote it as a plugin, adding it to the core off CCNet is not a good choice for this.
Writing a labeler plugin is not more difficult than writing a publisher or so, it means implementing ThoughtWorks.CruiseControl.Core.ILabeller

A bit of of background on our deployment setup :
° whenever a program passes QA, it gets zipped
format is version_number_program name
° it is placed in a folder hierarchy so programs belonging to the same 'family' stand together (all finance programs are together)
° we always install the latest version, (should the need arise that we need to install a previous version, we just delete the newer zip files from the folder)

So suppose I do an install of bookkeeping, the labeler must just get the zip file with the highest number from the finance/bookkeeping folder. One caveat : 1_2_0_10 is higher than 1_2_0_3 so plain sorting on the file names is a no-go. Anyway, here's the code of the labeler :

using Exortech.NetReflector; using ThoughtWorks.CruiseControl.Core; [ReflectorType("releaseLabeller")] public class ReleaseLabeller : ILabeller { private string _ProgramtoInstall; [ReflectorProperty("programToInstall")] public string ProgramToInstall { get { return this._ProgramtoInstall; } set { this._ProgramtoInstall = value; } } public string Generate(IIntegrationResult integrationResult) { return GetLatestVersionOfDeployedProgram("\Install", ProgramToInstall, integrationResult); } public void Run(ThoughtWorks.CruiseControl.Core.IIntegrationResult result) { result.Label = Generate(result); } private string GetLatestVersionOfDeployedProgram(string installFolder, string program, IIntegrationResult integrationResult) { string[] DeployedFiles = null; string FolderToScan = System.IO.Path.Combine(installFolder, program); string Dummy = null; System.IO.FileInfo fi = default(System.IO.FileInfo); string[] VersionArray = null; string Version = null; DeployedFiles = System.IO.Directory.GetFiles(FolderToScan, "*.zip"); if (DeployedFiles.Length == 0) { return integrationResult.LastSuccessfulIntegrationLabel; } Array.Sort(DeployedFiles, new ReleaseZipFileSorter()); fi = new System.IO.FileInfo(DeployedFiles(DeployedFiles.Length - 1)); Dummy = fi.Name; VersionArray = Dummy.Split('_'); if (VersionArray.Length < 4) { throw new Exception(string.Format("Unsupported zip file {0} found in {1} for determining latest version", Dummy, FolderToScan)); } Version = string.Format("{0}.{1}.{2}.{3}", VersionArray(0), VersionArray(1), VersionArray(2), VersionArray(3)); return Version; } }

The programToInstall contains which program is being installed (duh)
so in the ccnet.config I fill this with finance/bookkeeping which is the same argument as I pass to the install program.

Here's the code for the sorting of the zip files :
Basically, I format the numeric parts to length 5 and use the string sorting on it.
So 1_0_3_0 becomes 00001_00000_00003_00000 for the sort procedure.

public class ReleaseZipFileSorter : IComparer, IComparer<string> { public int Compare(object x, object y) { return FormatCCNetLabel(x.ToString).CompareTo(FormatCCNetLabel(y.ToString)); } public int Compare1(string x, string y) { return FormatCCNetLabel(x).CompareTo(FormatCCNetLabel(y)); } int System.Collections.Generic.IComparer<string>.Compare(string x, string y) { return Compare1(x, y); } private string FormatCCNetLabel(string label) { Text.StringBuilder Result = new Text.StringBuilder(); string Dummy = label.Substring(label.LastIndexOf("\\") + 1); string[] parts = Dummy.Split('_'); if (parts.Length != 5) { throw new Exception("Invalid zip file name : " + label); } Result.Append(Convert.ToInt32(parts(0)).ToString("00000")); Result.Append("."); Result.Append(Convert.ToInt32(parts(1)).ToString("00000")); Result.Append("."); Result.Append(Convert.ToInt32(parts(2)).ToString("00000")); Result.Append("."); Result.Append(Convert.ToInt32(parts(3)).ToString("00000")); return Result.ToString; } }

Buildscript Update

Today I had a problem with a certain program, and could not find an immediate solution. So I went for help to a colleague. He came, and the first thing he said was : 'Hey, you do not have Option Strict On'!. The bad thing was : he was right! How could I make such a stupid mistake? After fixing the first problem, I went looking into that Option Strict thing. Now it seems that Visual Studio sometimes saves this setting in the .suo file iso in the project file!

So in order to make sure all projects have this setting to on, I decided to make an extra task in our buildscripts : Before every compile scan all the projects for the setting Option Strict and Option Explicit. This is easy, from the top of the source, scan down all project files, and scan each of them for the involved string.

Now the fun part : I checked this in and got lots of email about me breaking the build! According to CCNet, it was me who broke all these builds. 50+ projects with these bad settings : AARGH
Another lesson learned : when you do stuff like this, foresee a way that it is filtered out using a filtertrigger on a specific username or so. So the people maintaining the projects get the mail on their next check-in ;-)

Showing the progress of buildscripts in the dashboard.

For my buildscripts I hardly use batch files. These are ok for the short and quick stuff, but when it gets more complex, or takes a while to execute, I write a .Net Console program. The reasons :

• string handling is a lot easier in .Net
• you can easily obscure passwords in a .Net console program
• if it needs be, you can call webservices
• make the progress visible in the dashboard
• ...

Calling external programs from .Net is very easy, you can do it with System.Diagnostics.Process. This gives very flexible control over the called program. Below is an example for calling Mage :

static void CallMage(string mageFolder, string mageArguments) { const Int32 TimeOutInSeconds = 20; System.Diagnostics.Process MageProcess = new System.Diagnostics.Process(); string MageError = ""; string MageResult = ""; // setting up mage MageProcess.StartInfo.UseShellExecute = false; MageProcess.StartInfo.RedirectStandardError = true; MageProcess.StartInfo.RedirectStandardOutput = true; MageProcess.StartInfo.Arguments = mageArguments; MageProcess.StartInfo.FileName = String.Format("{0}{1}{0}", "\"", mageFolder); // running mage MageProcess.Start(); MageProcess.WaitForExit(TimeOutInSeconds * 1000); if (!MageProcess.HasExited) { MageProcess.Kill(); throw new Exception(String.Format("Mage has timed out after {0} seconds ", TimeOutInSeconds)); } // checking result MageError = MageProcess.StandardError.ReadToEnd(); MageResult = MageProcess.StandardOutput.ReadToEnd(); MageProcess.Close(); if (MageError.Length > 0) { throw new Exception(String.Format("Mage Error : \n {1}", MageError)); } Console.WriteLine(" {0}", MageResult); }

Now a more conrete example, below is a piece of the deployment script I use, I stripped a lot the functionality, but the core is the same. Also a short simple example shows more than a full blown one. Basically it unzips a file, and calls mage to resign an assembly. This call to mage is the procedure above.

static void Main(string[] args) { const string MageProgramLocation = @"C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\mage.exe"; Int32 ReturnValue = 0; ConsoleColor OriginalBackgroundColor = Console.BackgroundColor; try { string tempFolder = @"d:\temp\unzip"; Console.WriteLine(" Un-Zipping to temp folder"); Ionic.Utils.Zip.ZipFile zip = new Ionic.Utils.Zip.ZipFile(@"d:\safe\programs\examples\1_0_1_256_FunkyStuff.zip"); zip.ExtractAll(tempFolder); zip.Dispose(); string MageArguments; MageArguments = String.Format(" -Sign {0}{1}{0} -CertFile {0}{2}{0} -Password {3}", "\"", @"d:\temp\unzip\x.manifest", @"d:\temp\keys\deployManifestKey", "DeploymentKeyPWD"); CallMage(MageProgramLocation, ""); } catch (Exception e) { Console.BackgroundColor = ConsoleColor.Red; Console.WriteLine(e.ToString()); Console.BackgroundColor = OriginalBackgroundColor; ReturnValue = 2; } Environment.ExitCode = ReturnValue; }

Now the ones paying attention will notice that there is nothing special about the above code. There is nothing foreseen that will show any output to the dashboard of CCNet. And this is correct. However, the adjustment needed for this is very small. I made a small helper class that I use, just for this kind of things. The only thing I need to change to any console program is I have to make an instance of this class, and replace every Console.Writeline call to x.Writeline.
For example add static private CCNetExecListener CCNetListener = new CCNetExecListener(); just above static void Main, so it is known in the entire program, and replace every Console.Writeline call to CCNetListener.Writeline. This will make the above program to output the buildprogress when ran from CCNet, but it also remains usable when ran from the commandline. Nice and clean ;-)

Below is the code of the class.

using System; using System.Collections.Generic; public class CCNetExecListener { private string CCNetListenerFile = string.Empty; private int AmountOfLinesToKeep = 10; private Queue<string> Messages = new Queue<string>(); #region "Constructors" public CCNetExecListener() { this.CCNetListenerFile = Environment.GetEnvironmentVariable("CCNetListenerFile"); } public CCNetExecListener(int amountToKeep) : this() { this.AmountOfLinesToKeep = amountToKeep; } #endregion #region "Write Information" public void WriteLine() { HandleNewInformation(""); } public void WriteLine(bool value) { HandleNewInformation(value.ToString()); } public void WriteLine(string value) { HandleNewInformation(value); } public void WriteLine(char value) { HandleNewInformation(value.ToString()); } public void WriteLine(decimal value) { HandleNewInformation(value.ToString()); } public void WriteLine(double value) { HandleNewInformation(value.ToString()); } public void WriteLine(float value) { HandleNewInformation(value.ToString()); } public void WriteLine(int value) { HandleNewInformation(value.ToString()); } public void WriteLine(uint value) { HandleNewInformation(value.ToString()); } public void WriteLine(long value) { HandleNewInformation(value.ToString()); } public void WriteLine(object value) { HandleNewInformation(value.ToString()); } public void WriteLine(string value, params object[] arg) { string info = string.Format(value, arg); HandleNewInformation(info); } #endregion #region "Private functions" private void HandleNewInformation(string value) { string Data = string.Format("<Item Time={0}{1}{0} Data={0}{2}{0} />", "\"", GetTimeStamp(), value); if (this.Messages.Count >= this.AmountOfLinesToKeep) { this.Messages.Dequeue(); } this.Messages.Enqueue(Data); Console.WriteLine(value); WriteQueueData(); } private void WriteQueueData() { if (this.CCNetListenerFile == string.Empty) return; System.IO.StreamWriter TraceFile = default(System.IO.StreamWriter); try { TraceFile = new System.IO.StreamWriter(this.CCNetListenerFile, false); } catch (Exception) { return; } TraceFile.AutoFlush = true; TraceFile.WriteLine("<data>"); foreach (string s in this.Messages) { TraceFile.WriteLine(s); } TraceFile.WriteLine("</data>"); TraceFile.Close(); } private string GetTimeStamp() { return System.DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss"); } #endregion }

Continuous Installation and ClickOnce setup

When you read my previous posts on the Continuous Installation setup, and know how ClickOnce works, you know that I had to do some stuff to get this working. A small introduction into ClickOnce, for the ones that are not familiar with it.
From the site of MS :

---

Simply stated, a ClickOnce application is any Windows Presentation Foundation, Windows Forms, or console application published using ClickOnce technology. You can publish a ClickOnce application in three different ways: from a Web page, from a network file share, or from media such as a CD-ROM. A ClickOnce application can be installed on an end user's computer and run locally even when the computer is offline, or it can be run in an online-only mode without permanently installing anything on the end user's computer.
ClickOnce applications can be self-updating; they can check for newer versions as they become available and automatically replace any updated files. The developer can specify the update behavior; a network administrator can also control update strategies, for example, marking an update as mandatory. Updates can also be rolled back to an earlier version by the end user or by an administrator.
Because ClickOnce applications are isolated, installing or running a ClickOnce application cannot break existing applications. ClickOnce applications are self-contained; each ClickOnce application is installed to and run from a secure per-user, per-application cache. By default, ClickOnce applications run in the Internet or Intranet security zones. If necessary, the application can request elevated security permissions.

How ClickOnce Deployment Works

The core ClickOnce deployment architecture is based on two XML manifest files: an application manifest and a deployment manifest.

The application manifest describes the application itself. This includes the assemblies, the dependencies and files that make up the application, the required permissions, and the location where updates will be available. The application developer authors the application manifest by using the Publish Wizard in Visual Studio 2008 or the manifest generation tool (Mage.exe) in the Windows Software Development Kit (SDK).

The deployment manifest describes how the application is deployed. This includes the location of the application manifest, and the version of the application that clients should run. An administrator authors the deployment manifest using the manifest generation tool (Mage.exe) in the Windows SDK.

---

Now this sounds rather logical, but there are a few catches, the deployment manifest is secured/signed with a key, so far so good. One part of the security is a hash of the needed files (crc or something) and the deployment manifest itself, which contains from which servername the ClickOnce is authorised to run. This is problem 1.
When you develop your application, you deploy it to your companies test server, and when it is ok, you can deploy it again to a production server. But when you must have this ClickOnce setup on 100+ servers, you're not going to do this by hand ;-) So I must do this within a script.
You can create these 2 manifests totally from scratch with mage, but this is more difficult than it needs be(you must specify all dependent dll's + location, ...) My solution is a mix of MSBuild and Mage. To generate a ClickOnce setup via MSBuild, the help says compile your solution with msbuild /t:publish. There are a few properties that you can/must set. You can find some (incomplete) information about them here.

Problem 2 : when you do msbuild /t:publish /p:(new properties), the values of the new properties are not taken if there are already assemblies created! you must do msbuild /t:clean /t:publish /p:(new properties)
I use the following msbuild command (we use web based installations) :
msbuild /t:clean /t:publish
/p:ApplicationVersion=${CCNetLabel}
/p:PublishVersion=${CCNetLabel}
/p:PublishUrl=${publishUrl}
/p:InstallUrl=${publishUrl}
/p:GenerateManifests=true
/p:IsWebBootstrapper=true
/p:TargetZone=LocalIntranet
/p:PublisherName=TheCompany
/p:Install=true
/p:InstallFrom=Web
/p:UpdateEnabled=true
/p:UpdateRequired=false
/p:BootstrapperEnabled=true
/p:MapFileExtensions=true
/p:UseApplicationTrust=false
/p:BootstrapperComponentsLocation=Relative

Now we finally have deployed to a test server. Problem 3 : This does NOT generate the html file showing the version and so, this is done in the VS interface, and not by MSBuild. But the html page is an easy one, so I took one generated by VS as a template, and adjusted my buildscript so it also creates one.
PS.: is you want to install via a fileshare, you must adjust the following properties :
/p:InstallFrom=Unc
/p:IsWebBootstrapper=false
/p:PublishUrl=${publishUNC}
/p:InstallUrl=${publishUNC}

Now I zip the generated ClickOnce folder, so it is easier to update this specific version later on.

Problem 4 :getting this click once to run from a different folder.
When you read the pages about Mage, it just says that you only have to update the deployment manifest and resign it. WRONG
If you do it this way, you'll end up in trouble. The ClickOnce installation made by MSBuild did come clever tricks, if you take a look in the 'Application Files' folder, you see that there is a subfolder for every build(which is ok). But the contents of these subfolders do NOT contain any .exe or .dll file, it only contains .deploy files. This is done because IE makes trouble in downloading .exe or .dll files. That's why a simple update of the deployment manifest with Mage does not work. The deployment manifest can not find the files anymore! I hacked around this with the following procedure :

• rename every .deploy file to the correct name (strip the .Deploy extention)
• update the app.config files if needed
• update the application manifest with mage (because the app.config file could be changed)
• re-sign the application manifest with mage
• update the deployment manifest : update the url from where to install
• update the deployment manifest with mage
• resign the deployment manifest with mage
• re-add the deployment extention to the files where it was removed from

Command for updating the application manifest with mage:
mage -Update ApplicationManifestFile -FromDirectory BinFolder (Binfolder is where the application manifests resides.)
Command for re-signing the application manifest with mage:
mage -Sign ApplicationManifestFile -CertFile applicationManifestKey -Password ThePassword
Command for updating the deployment manifest with mage :
mage -Update DeployManifestFile -appm ApplicationManifestFile -appc ApplicationCodeRelativeBasePath (ApplicationCodeRelativeBasePath is the relative location of the ApplicationManifest from the DeploymentManifest.)

Now you have a Web enabled ClickOnce installation. And with this procedure it is very easy to adjust the settings for 100+ servers. Keep in mind that the copying is not done by MSBuild, this must still be done!

Continous Installation observations

I set up a dashboard on a separate PC, so I can keep an eye on the situation on all the servers of the customers. This works, but I found a small issue : speed.
For the moment there are 20 servers installed, and they are all remote. And off course some of them do not have a fast connection :-(
I'll look in to some items to speed things back up. Things I encountered so far :

• Changing the sort order seems to refetch all data, which is not needed
• Navigating back from a buildserver in the farm grid, causes this grid to refetch the data slowing down navigation
• when a second person also opens a browser on his pc, the data is again fetched

A fix for all this would be to cache the grid data at the server(dashboard) level, according to a setting controlling how long this data must be cached. This will improve the speed tremendously and lesson the burden on the network. The refresh data button will re-query all the buildsevers, and so update the information. A small remark here, when you use listeners,( extending the default detail data of the grid) the refresh button is sometimes placed off screen. Maybe a fix for this is to locate the refresh button at the left side of the screen.

When I have ironed out all the small stuff with the Continuous Installation setup, I'll make a post on that. I've encountered some things that could be usefull. Stay tuned for it .

CI : Continous Installation

In my first post I said that my company also wanted to use CCNet to place our programs at the servers of our customers (110+). Now this is possible because we also supply the servers (the customers lease them). Today we did our first test, and it looks very promising. For the moment we have about 15 servers installed (each customer has 1), so the sooner we get this procedure sorted out, the better. These 15 were manually updated, and it took about 20 minutes to install / upgrade 1 program per server. This is including the download of the software.
Now with CCNet installed at these servers, it costs us no time anymore. Well only a few minutes actually. This is how we set things up :

• Whenever a program passes QA, we zip it and upload to an external FTP server
• each server at the customers site monitors this ftp server, and downloads the new software : CCNet Project 1
• the installation itself (unzipping and the like) is CCNet project 2, which responds only to a force build
• and there is also a CCNet Update project (project 3) which contains a ccnet config file, so updating the CCNet servers at the customers is also automated

The only time we now have to 'spent' on installing new software at our customers site, is the time needed to upload it to the FTP server. From that point on, all the servers update themselves. Now this is a massive speed gain. For the moment we have 5 programs (Click-Once apps + WCF services + SQL Databases). So suppose we have to update these manually at once for all customers, it would take us : 5 * 20 * 110 minutes : 11.000 minutes, this is 183 hours corresponding to 7.6 days work (24 hour day). So even trying to do this by hand is ludicrous.
The positive (and also downside) is that this setup forces us to test every step extremely. If there is something wrong with the upgrade script : 110 customers on the phone, not good ;-) We do test the setup at a local spare server, before we press the force build button, just to make sure we do not commit suicide.
Another benefit of having CCNet at our customers server, is that scheduling is now very easy, suppose they want to have a certain report printed/mailed at 07:00, we just update the ccnet config file, upload to the ftp server, and presto : done.
This setup will also be a nice opportunity to check the dashboard, CCTray and BVC2 with a very large amount of CCNet projects. If these apps can handle this load, CCNet is a very stable program. If all goes well, I'll try to place the FTP stuff in the trunc, this will benefit other people as well.

Current Work load ...

Merging the security branch into the trunc :
The project admin said that there should be at least 2 people who know this part of the code, before it can be merged, so I've been busy the last week with the security mergure. This is a very big change to the code see the blog of Craig.

I hope to get this part done in the following days/week(s), so this great addition can be used by everybody, and the work is not lost. Later the other parts of the security branch will also be merged. The biggest problems I encountered :

• very big changes in a long time frame(both in trunc and branch)
• I never had any real intrest into security so the terminology used (roles, permissions, assertions, ...)was new and sometimes confusing
• the security branch also has other functionality (translation, messaging), which is not fully finished yet
• I never did a branch merge, so this was a challenge ;-)