Tuning CCNet to your whishes : a new trigger

There was a request on the user list to run a nightly "publish" build, which takes the output of the other "child" projects and copies them to a new output directory. Since the output is quite large (> 1 GB), this should only be done each night if something changed in any of the other projects, but only if all of them have run successfully.
You can read the full details also on Stack overflow. Now one should be able to do this using the existing functionality, but that is not the case :-(. The reason is mainly that CCNet has grown from a simple CI server to a more feature rich one over the years, but here and there are still leftovers from the beginning, read the simple CI server. Triggers are just events that occur, and do not know a state, there is the project trigger, but that is more of a fire and forget one. So restarting a server poses a problem, combining multiple project triggers into one trigger poses a problem with timings over a day and so on.

To handle this situation, the only option is to write some code that does it. Thankfully CCNet has a plugin architecture that works quite well, see my previous creating a plugin about it for more details.

As an example of this request, I will create a new trigger, because this will be the fastest way to get a result with just a few lines of code adjustments needed. For starters I just copied the code of the existing schedule trigger into a new class, named ScheduleTriggerExtended.

Changes done :
° added property SubProjectsToWatch
The project names of the child projects to watch. For simplicity only the project name, meaning that it has to be on the same build server and has browse rights (or no security setup)
° added property ServerUri
the uri of the ccnet server, example : tcp://localhost:21234/CruiseServerClient.rem
° added property SubProjectsSavePath
A path reachable by the CCNet server (preferably local). This is used to store the timestamps of the subprojects. So we can check if the timestamp changed between the integration of a subproject and the build of the involved subproject.
° updated public IntegrationRequest Fire
Whenever the scheduled time is reached, also check if the subProjects are changed, by comparing the current last build times with the saved ones.
° updated public void IntegrationCompleted
Save the new buildtimes of the subprojects to the SubProjectsSavePath for later reference.

When reading this all over, the best approach would be to create a 'CCNet source control', that can scan other CCNet projects for changes, and filter them according to some filters (state, build time, ...)

Anyway, below is the trigger class, all changes are highlighted with comment : added code. Copy entire source into an assembly with name something like : CCNet.whatever.plugin
Example : CCNet.RuWi.plugin
And set references to NetReflector.dll, ThoughtWorks.CruiseControl.Core.dll, ThoughtWorks.CruiseControl.Remote.dll


A config is also provided below the code.

using System; using System.Globalization; using Exortech.NetReflector; using ThoughtWorks.CruiseControl.Core.Config; using ThoughtWorks.CruiseControl.Core.Util; using ThoughtWorks.CruiseControl.Remote; namespace ThoughtWorks.CruiseControl.Core.Triggers { [ReflectorType("scheduleTriggerExtended")] public class ScheduleTriggerExtended : ITrigger, IConfigurationValidation { private string name; private DateTimeProvider dtProvider; private TimeSpan integrationTime; private DateTime nextBuild; private bool triggered; private Int32 randomOffSetInMinutesFromTime/* = 0*/; Random randomizer = new Random(); /// <summary> /// Initializes a new instance of the <see cref="ScheduleTrigger"/> class. /// </summary> public ScheduleTriggerExtended() : this(new DateTimeProvider()) { } /// <summary> /// Initializes a new instance of the <see cref="ScheduleTrigger"/> class. /// </summary> /// <param name="dtProvider">The dt provider.</param> public ScheduleTriggerExtended(DateTimeProvider dtProvider) { this.dtProvider = dtProvider; this.BuildCondition = BuildCondition.IfModificationExists; WeekDays = (DayOfWeek[])DayOfWeek.GetValues(typeof(DayOfWeek)); } /// <summary> /// The time of day that the build should run at. The time should be specified in a locale-specific format (ie. H:mm am/pm is acceptable for US locales.) /// </summary> /// <version>1.0</version> /// <default>n/a</default> [ReflectorProperty("time")] public virtual string Time { get { return integrationTime.ToString(); } set { try { integrationTime = TimeSpan.Parse(value); } catch (Exception ex) { string msg = "Unable to parse daily schedule integration time: {0}. The integration time should be specified in the format: {1}."; throw new ConfigurationException(string.Format(CultureInfo.CurrentCulture, msg, value, CultureInfo.CurrentCulture.DateTimeFormat.ShortTimePattern), ex); } } } /// <summary> /// Adds a random amount of minutes between 0 and set value to the time. This is mainly meant for spreading the load of actions to a central server. /// Value must be between 0 and 59. /// </summary> /// <version>1.4</version> /// <default>0</default> [ReflectorProperty("randomOffSetInMinutesFromTime", Required = false)] public int RandomOffSetInMinutesFromTime { get { return randomOffSetInMinutesFromTime; } set { randomOffSetInMinutesFromTime = value; if (randomOffSetInMinutesFromTime < 0 || randomOffSetInMinutesFromTime >= 60) throw new ConfigurationException("randomOffSetInMinutesFromTime must be in the range 0 - 59"); } } /// <summary> /// The name of the trigger. This name is passed to external tools as a means to identify the trigger that requested the build. /// </summary> /// <version>1.1</version> /// <default>ScheduleTrigger</default> [ReflectorProperty("name", Required = false)] public string Name { get { if (name == null) name = GetType().Name; return name; } set { name = value; } } /// <summary> /// The condition that should be used to launch the integration. By default, this value is <b>IfModificationExists</b>, meaning that an integration will /// only be triggered if modifications have been detected. Set this attribute to <b>ForceBuild</b> in order to ensure that a build should be launched /// regardless of whether new modifications are detected. /// </summary> /// <version>1.0</version> /// <default>IfModificationExists</default> [ReflectorProperty("buildCondition", Required = false)] public BuildCondition BuildCondition { get; set; } /// <summary> /// The week days on which the build should be run (eg. Monday, Tuesday). By default, all days of the week are set. /// </summary> /// <version>1.0</version> /// <default>Monday-Sunday</default> [ReflectorProperty("weekDays", Required = false)] public DayOfWeek[] WeekDays { get; set; } //added code /// <summary> /// The project names of the child projects to watch. For cimplicity only the project name, /// meaning that it has to be on the same build server and has browse rights (or no security setup) /// </summary> [ReflectorArray("subProjectsToWatch", Required = true)] public string[] SubProjectsToWatch { get; set; } /// <summary> /// the uri of the ccnet server, example : tcp://localhost:21234/CruiseServerClient.rem /// </summary> [ReflectorProperty("serverUri", Required = true)] public string ServerUri { get; set; } /// <summary> /// A path reachable by the CCNet server (preferably local). This is used to store the timestamps of the subprojects. /// So we can check if the timestamp changed between the integration of a subproject and the build of the involved subproject. /// </summary> [ReflectorProperty("subProjectsSavePath", Required = true)] public string SubProjectsSavePath { get; set; } /// <summary> /// Looks at the timestamp saved in the reference file of the subprojects. If that timestamp differs from the last /// integration timestamp of the subproject, a re-build is needed. /// </summary> /// <returns></returns> private bool SubProjectsAreChanged() { // nothing to watch if (SubProjectsToWatch == null || SubProjectsToWatch.Length == 0) return false; //the save path does not exist, so we must do the first build if (!System.IO.Directory.Exists(SubProjectsSavePath)) return true; //get the last build info from the build server var cm = new RemoteCruiseManagerFactory().GetCruiseManager(ServerUri); var buildInfos = cm.GetCruiseServerSnapshot().ProjectStatuses; foreach (string subprojectName in SubProjectsToWatch) { string locationOnDisk = System.IO.Path.Combine(SubProjectsSavePath, subprojectName); if (! System.IO.File.Exists(locationOnDisk)) return true; DateTime LastTimeBuildInIntegration; string data = System.IO.File.ReadAllText(locationOnDisk); if (DateTime.TryParse(data, out LastTimeBuildInIntegration)) { var lastbuildinfo = GetSubProjectLastBuildInfo(subprojectName, buildInfos); if (lastbuildinfo.BuildStatus != IntegrationStatus.Success) return false; if (LastTimeBuildInIntegration < lastbuildinfo.LastBuildDate ) return true; } else { // file is empty, the contents is not a date, is corrupted, .... throw new Exception(string.Format("Contents of {0} --{1}-- is not convertable to a datetime", locationOnDisk,data)); } } return false; } private Remote.ProjectStatus GetSubProjectLastBuildInfo(string projectName, Remote.ProjectStatus[] serverInfo) { foreach (ProjectStatus ps in serverInfo) { if (ps.Name == projectName) return ps; } throw new Exception(string.Format("Project {0} not found on server with uri {1}", projectName, ServerUri)); } private void SaveSubProjectsLastBuildTimes() { // nothing to watch if (SubProjectsToWatch == null || SubProjectsToWatch.Length == 0) return ; if (!System.IO.Directory.Exists(SubProjectsSavePath)) System.IO.Directory.CreateDirectory(SubProjectsSavePath); DateTime now = dtProvider.Now; foreach (string subprojectName in SubProjectsToWatch) { string locationOnDisk = System.IO.Path.Combine(SubProjectsSavePath, subprojectName); System.IO.File.WriteAllText(locationOnDisk, now.ToLongDateString()); } } //end added code private void SetNextIntegrationDateTime() { if (integrationTime.Minutes + RandomOffSetInMinutesFromTime >= 60) throw new ConfigurationException(string.Format(System.Globalization.CultureInfo.CurrentCulture, "Scheduled time {0}:{1} + randomOffSetInMinutesFromTime {2} would exceed the hour, this is not allowed", integrationTime.Hours, integrationTime.Minutes, RandomOffSetInMinutesFromTime)); DateTime now = dtProvider.Now; nextBuild = new DateTime(now.Year, now.Month, now.Day, integrationTime.Hours, integrationTime.Minutes, 0, 0); if (randomOffSetInMinutesFromTime > 0) { Int32 randomNumber = randomizer.Next(randomOffSetInMinutesFromTime); nextBuild = nextBuild.AddMinutes(randomNumber); } if (now >= nextBuild) { nextBuild = nextBuild.AddDays(1); } nextBuild = CalculateNextIntegrationTime(nextBuild); } private DateTime CalculateNextIntegrationTime(DateTime nextIntegration) { while (true) { if (IsValidWeekDay(nextIntegration.DayOfWeek)) break; nextIntegration = nextIntegration.AddDays(1); } return nextIntegration; } private bool IsValidWeekDay(DayOfWeek nextIntegrationDay) { return Array.IndexOf(WeekDays, nextIntegrationDay) >= 0; } /// <summary> /// Integrations the completed. /// </summary> /// <remarks></remarks> public virtual void IntegrationCompleted() { if (triggered) { // added code SaveSubProjectsLastBuildTimes(); // added code SetNextIntegrationDateTime(); } triggered = false; } /// <summary> /// Gets the next build. /// </summary> /// <value></value> /// <remarks></remarks> public DateTime NextBuild { get { if (nextBuild == DateTime.MinValue) { SetNextIntegrationDateTime(); } return nextBuild; } } /// <summary> /// Fires this instance. /// </summary> /// <returns></returns> /// <remarks></remarks> public IntegrationRequest Fire() { DateTime now = dtProvider.Now; //added code //if (now > NextBuild && IsValidWeekDay(now.DayOfWeek)) if (now > NextBuild && IsValidWeekDay(now.DayOfWeek) && SubProjectsAreChanged()) // end added code { triggered = true; return new IntegrationRequest(BuildCondition, Name, null); } return null; } void IConfigurationValidation.Validate(IConfiguration configuration, ConfigurationTrace parent, IConfigurationErrorProcesser errorProcesser) { string projectName = "(Unknown)"; var project = parent.GetAncestorValue<Project>(); if (project != null) { projectName = project.Name; } if (integrationTime.Minutes + RandomOffSetInMinutesFromTime >= 60) { errorProcesser.ProcessError("Scheduled time {0}:{1} + randomOffSetInMinutesFromTime {2} would exceed the hour, this is not allowed. Conflicting project {3}", integrationTime.Hours, integrationTime.Minutes, RandomOffSetInMinutesFromTime, projectName); } } } }

Example Config

<cruisecontrol xmlns:cb="urn:ccnet.config.builder"> <project name="Child1" > <artifactDirectory>D:\temp\Integration\Child1</artifactDirectory> <workingDirectory>D:\temp\Integration\Child1</workingDirectory> <triggers /> <tasks > <nullTask/> </tasks> <publishers> <xmllogger /> </publishers> </project> <project name="Child2" > <artifactDirectory>D:\temp\Integration\Child2</artifactDirectory> <workingDirectory>D:\temp\Integration\Child2</workingDirectory> <triggers /> <tasks > <nullTask/> </tasks> <publishers> <xmllogger /> </publishers> </project> <project name="Child3" > <artifactDirectory>D:\temp\Integration\Child3</artifactDirectory> <workingDirectory>D:\temp\Integration\Child3</workingDirectory> <triggers /> <tasks > <nullTask/> </tasks> <publishers> <xmllogger /> </publishers> </project> <project name="Integrator" > <artifactDirectory>D:\temp\Integration\Intgrator</artifactDirectory> <workingDirectory>D:\temp\Integration\Integrator</workingDirectory> <triggers> <scheduleTriggerExtended time="14:08" buildCondition="IfModificationExists" name="Scheduled" serverUri="tcp://localhost:21234/CruiseServerClient.rem" subProjectsSavePath="d:\temp"> <subProjectsToWatch> <name>Child1</name> <name>Child3</name> </subProjectsToWatch> <weekDays> <weekDay>Saturday</weekDay> </weekDays> </scheduleTriggerExtended> </triggers> <tasks > <nullTask/> </tasks> <publishers> <xmllogger /> </publishers> </project> </cruisecontrol>