Can I change a build aggregation policy ? #220

takyoshi · 2 decades ago

I'm trying to make a build configuration that is triggered by Java API. The API is called by Perforce trigger, and changelist number is passed to build as a variable.
When triggered frequently, multiple builds are queued. I set "Aggregate adjacent builds" flag to yes, so queued builds are aggregated and executed when a running build has completed.
Aggregation is expected, but I found that the first build in queue was executed, and remaining builds were discarded.
For example, if builds for changelist 100,101,102 are queued and aggregation has occured, I expect that build 102 is executed and build 100,101 are discarded. But build 100 is executed and build 101,102 are discarded in practice.
If build 102 is discarded, I can't know whether the current source code in a perforce repo is available or not.

So can I configure that the last build in queue is executed and the remaining builds are discarded when aggregating builds ?

replies 7
views 11309
stars 0

robinshen ADMIN · 2 decades ago

QB does not support such policy. However, I think current aggregation mode should work for your case as long as you build against latest change list number (this is the default behavior if change list number is not specified) when triggers QB from perforce. Let me explained as below:

1. Developer A checks code into Perforce with change list 100, and triggers a build, say build1. Suppose there is already a build (suppose it to be build0) running there, and build1 will be queued.
2. While build1 is waiting, developer B checks code into Perfoce with change list 101, and triggers the build2 consequently. Of course build2 will be queued.
3. Now build0 finished, and build1 is picked up for running. At the same time build2 is discarded. Please be noted that build1 will take care of 100 and 101 now as it is configured to build against latest change list number. So although build2 is discarded, developer A and B can both receive notification emails about the build errors if there is any.

Please let me know if this meets your needs.

Regards.
Robin

takyoshi · 2 decades ago

Please let me explain what I want to do.
My principal goal is to execute one build per one change list number, and include the change list number into build version like 'Build-12345'.
So I usually set "Aggregate adjacent builds" flag to no in order to execute all builds even if more than one build is queued.
But I hope to change the behavior so that queued builds could be aggregated if change occurs frequently.
So I'm happy if I could easily switch or control the behavior of aggregation of queued builds.

robinshen ADMIN · 2 decades ago

In case of too frequent builds, even if using your suggested aggregation strategy, some change list numbers may not get built, which violates your goal of "one build per change list number".

So I think the aggregation strategy you suggested will have the same effect as current one if you are building against latest change list number.

As to include change list number into build version, it is possible even if you build against latest change list number. To do so, just insert below OGNL expression into the build version:

${repository["nameOfYourPerforceRepository"].headRevision}

Regards.
Robin

takyoshi · 2 decades ago

> In case of too frequent builds, even if using your suggested aggregation strategy,
> some change list numbers may not get built, which violates your goal of "one build per change list number".
> So I think the aggregation strategy you suggested will have the same effect as current one
> if you are building against latest change list number.

My Java program executed by Perforce triggers a build and pass a change list number as a variable 'changelistnum'.
Then the 'changelistnum' variable is used to specify a label name of all modules of Perforce repository as '${var["changelistnum"]}'. So I thought that I could build per change list number if "Aggregate adjacent builds" flag is set to no.
Is this idea wrong ?

> As to include change list number into build version, it is possible even if you build against
> latest change list number. To do so, just insert below OGNL expression into the build version:
>
> ${repository["nameOfYourPerforceRepository"].headRevision}

Oh, it's very useful for me.
Thanks.

Let me explain about my build configurations and build sequence for the purpose of reference.

1. Perforce executes Java program when new change list is submitted.
2. Java program triggers 'triggerAll' on central build server with 'changelistnum' variable.
3. 'triggerAll' build triggers 'config1-3(proxy)' with 'changelistnum' variable.
(It does not check whether Perforce repo is changed or not. It only triggers all configurations.)
4. 'config1-3(proxy)' build triggers config1-3 on remote build server respectively, and checks out
build artifacts from QuickBuild repository for remote build server.
('config1-3(proxy)' does not check whether Perforce repo is changed or not too. So central build
server never access perforce server.)
5. 'config1-3' build executes synchronization of Perforce repo with the specified change list number,
and builds artifacts.

The reason of setting up 'config1-3(proxy)' is to show all build status and to access all artifacts
from central build server. Developers don't care which build server executes build actually.

========================
<Central Build Server>
root.productA
- triggerAll
- config1(proxy)
- config2(proxy)
- config3(proxy)

<Remote Build Server #1>
root.productA
- config1
- config2

<Remote Build Server #2>
root.productA
- config3

========================

robinshen ADMIN · 2 decades ago

Thanks for the detailed information provided, I understand your scenario now. The enhancement for aggregation policy is submited as JIRA issue http://jira.pmease.com/browse/QB-244.

The obvious two policies I am thinking of is:
1. Pickup first build in the queue and discard all followings.
2. Pickup last build in the queue and discard all previous.

Policy 2 should work for your scenario. Please comment if you've got any other concerns.

Thanks again for your patience.

Robin

drdt · 2 decades ago

In addition to these options, I would like to see:
- discard all 'scheduled build' tasks (not discarding 'manual build' tasks).

Really, though, what I need is an option that allows tasks not to be queued at all, if a task is already running in the same configuration. This would enable builds not to queue up at all, and prevent problems where queued tasks are runnning outside of the desired build execution window.

My scenario is as follows: I run continuous integration builds throughout the day, except between midnight and 6am, when our Perforce system is potentially offline. The builds can take up to four hours to complete; however, I trigger them every ten minutes so that builds will start as soon as possible after the checkin occurs. When a build completes, there are a dozen or so accumulated builds waiting to start, which get aggregated into one, which then starts, if there have been any checkins while the previous build was running.

BUT. If a build starts at 11:30pm, and finishes at 3:30am, the scheduled task which triggered at 11:40pm will start immediately... and will fail, because it cannot access Perforce, even to find out if it should run!. My workaround is to only schedule tasks up until 7:30pm, so that the last possible time for a queued build to start is 11:30pm. But, this potentially loses me at least one continuous integration run every night, if someone checks in after 7:30pm and there isn't a queued task waiting at that time.

If I could configure this queue to discard all scheduled build tasks that come in while another build is running, I could avoid this problem.

drdt · 2 decades ago

Another (perhaps prettier) solution would be to have a time/date range on the queue itself, so that the queue shuts down and does not process builds in its queue outside of the specified time window.