ReleaseEngineering/Maintenance

This page is to track upcoming changes to any part of RelEng infrastructure; buildbot masters, slaves, ESX hosts, etc. This should allow us keep track of what we're doing in a downtime, and also what changes can be rolled out to production without needing a downtime. This should be helpful if we need to track what changes were made when troubleshooting problems.

ReleaseEngineering:BuildbotBestPractices describes how we manage changes to our masters.

Relevant repositories

• buildbot
• buildbot-configs
• buildbotcustom
• tools
• mozharness
• puppet

Non-releng repositories involved in the CI

TBPL

• Chief self-deploy recent history (needs VPN; more readable than raw logs, but is cleared if the Chief service is restarted)
• Chief self-deploy logs archive (needs VPN; use if the history page is empty)
• tbpl production TBPL production push bugs (Approximation, summaries may occasionally vary)

Mozharness deployments

Merges to the production branch of the mozharness repository are live immediately. Please see that repo for push history.

Puppet deployments

Merges to the production branch of the puppet repository are automatically deployed by Puppet within a 1/2 hour. Please see that repo for push history.

Mobile talos webhost

When	Bug #(s) - Description(s)
20/11/2013	Bug 940968 - archive talos data on bm-remote web hosts before move to scl3 <- updated talos rev to ad927d1aa867

Reconfigs / Deployments

This page is updated by the person who does a reconfig on production systems. Please give accurate times, as we use this page to track down if reconfigs caused debug intermittent problems. (This includes config changes not requiring a "reconfig", such as mozharness changes.)

Outcome should be 'backed out' or 'In production' or some such. Reverse date order pretty please.

Older changes are archived in: 2013 | 2012 | 2011 | 2010 | 2009

Outcome	When	Bug #(s) - Description(s)
in production	2014-03-12 02:09 PT	bug 962187 - Tracking bug for 17-mar-2014 migration work. r=jhopkins bug 975131 - migrate remaining w64-ix-slaves to rev2. r=bhearsum bug 956631 - enable tarako-eng build on mozilla-b2g28_v1_3t, r=catlee Note: bug 980120 was landed and then backed out and should be a no-op.
in production	2014-03-12 10:12 PT	bug 967704 - Use android23.json for Android 2.3 mochitests; r=armenzg bug 979962 - Make Gi tests get node modules from git mirror, r=gaye bug 956631 - [tarako][build]create "tarako" build, r=nthomas bug 981777 - Stop running mochitest-browser-chrome on B2G branches. r=bhearsum bug 981777 - Simplify our code while we deprecate Fedora jobs + merge day note. r=bhearsum bug 981777 - Reduce mochitest browser-chrome timeouts. r=jmaher bug 837017 - Disable debug mochitest browser chrome for Elm. r=rail bug 982225 - Run side-by-side debug mochitest-browser-chrome on m-i on EC2 test machines. r=rail bug 975131 - day8batch1 -> rev2. r=bhearsum bug 977298 - Use python -u for Android 2.2 Talos tests; r=jmaher bug 975131 - remove disabled_branches, mixed_branches logic for w64-ix-slave rev2's. r=bhearsum
in production	2014-03-10 13:30 PT	bug 979450 enable emulator-kk builds r=mgerva bug 980976 - buildbot-master82 doesn't have rev2 machines enabled for a bunch of branches. r=rail bug 962187 - Tracking bug for 17-mar-2014 migration work. r=nthomas bug 837017 - Run debug-mochitest-browser-chrome on Elm and streamline Jetpack in our configs. r=rail bug 752836 - Don't build Thunderbird for Instantbird pushes. r=catlee bug 980890 disable skip-spot logic to get builds going again. r=bustage nobug: don't classify slaves that don't have names (or connected slave objects). bug 979440 - we can't call get_output_from_command() from inside setup_mock(), r=catlee bug 978512 - prep buildbot.py for desktop builds in mozharness, r=armenzg
in production	2014-03-10 07:38 PT	bug 968200 - Clean up the download directory before running tests on Android
in production	2014-03-07 09:05 PT	bug 980976 - buildbot-master82 doesn't have rev2 machines enabled for a bunch of branches
in production	2014-03-06 12:10 PT	Backed out changeset 55fec886d16d Update release config for Thunderbird-28.0b1-build1 backout 86027af6a98f: * bug - Bug 979440 merge backout 02c74aa31b51 commit -> defaul bug 977579 - Please schedule linux jit-tests in two chunks bug 979440 - we can't call get_output_from_command() from inside setup_mock() bug 979450 - Add emulator-kk to bumper. bug 979584 - Increase timeout for Gaia UI emulator builds on buildbot. bug 979963 - Fix binary path for gaia integration tests bug 979963 - Fix binary path for gaia unit tests bug 979963 - Install B2G using mozinstall bug 979963 - binary -> binary_path bug 980091 - , Increase free space requirement for linux debug builds to 15G
in production	2014-03-04 14:40 PT	bug 793989 - Self-serve should be able to request arbitrary builds on a push Backout bug 846104 for breaking new nightly requests
in production	2014-03-04 13:36 PT	bug 976822 - Define config file for gaia-integration tests to use on OSX, bug 976822 - Update config to work on OSX, bug 977420 - Change birch config and add jacuzzis. bug 979076 - b2g bumper should check tree status. bug 979080 - Clear cache of device manifests if we fail to complete the push loop. Update release config for Fennec-28.0b8-build1 Update release config for Firefox-28.0b8-build1
in production	2014-03-03 10:20 PT	bug 977579 - Please schedule linux jit-tests in two chunks; r=jlund <- mozharness bug 978233 - emulator-jb should use --manifest instead of --test-manifest, r=jgriffin <- mozharness bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas bug 977702 - Remove --no-update, r=ahal bug 977420 - remove nightlies and b2g builds from birch. r=rail bug 966257 - Enable periodic b2g device builds on mozilla-inbound r=bhearsum nobug - Update release config for Firefox-28.0b7-build1 bug 977420 - remove tests from birch. r=rail bug 969988 - ASan is the only Linux platform running the separate jittest suite on try; r=bhearsum bug 972835 - use tbirdbld account to submit thunderbird data to balrog. r=nthomas <- buildbotcustom bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas <- buildbotcustom
in production	2014-02-27 08:08 PT	bug 942167 - Enable non-unified android{-debug} builds bug 961314 - Switch to in-tree mozconfig and switch to tooltool. bug 964438 - Update mozharness scripts with new marionette errors, bug 965588 - Remove --no-update from B2G mozharness scripts, bug 976158 - Tweak buildername for Android l10n nightlies to make consistent with other multi-part jobs; bug 976822 - Schedule gaia-integration on OSX on cedar, bug 977298 - Use -u when calling Talos runtests.py; bug 977420 - Change birch config and add jacuzzis. bug 924476 - s/[email protected]/[email protected]/g' per QA request.
in production	2014-02-26 0654 PT	bug 973063 - Use proper manifest path in emulator-jb mochitests, r=armenzg bug 965588 - Remove --no-update from B2G script invocation, r=ahal nobug - Update release config for Fennec-28.0b6-build1 nobug - Update release config for Firefox-28.0b6-build1 bug 947426 - Remove LINUX_VMS - r=armenzg bug 976629 - Remove linux-opt arm-sim builder, r=bhearsum bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas bug 970738 - URL encode builder names. r=bhearsum bug 748800 - replace TuxedoEntrySubmitterFactory with a mozharness script. r=nthomas bug 976251 - Use 'make node_modules', r=gaye bug 910689 - Add comment for original fetch url. r=rail bug 973545 - Add -timeout 10000 to gaia-ui-tests, r=zac bug 972659 - Add git dict to b2g_bumper, r=catlee bug 973545 - s/append/extend bug 967704 - Enable blobber for Android 2.3; r=armenzg bug 966441 - Set -b option for mozbase unit tests; r=jlund
in production	2014-02-25 0820 PT	bug 973004 - deploy newer startTalos.bat for Windows machines
in production	2014-02-24 1113 PT	bug 975511 - Replace dev-master01 for dev-master1. r=bhearsum bug 975496 - Use MozillaPulse from pypi. r=bhearsum bug 974166 - Allow more flexible control of project try_by_default setting, r=bhearsum bug 974166 - Allow the possibility of spidermonkey builds on more platforms, r=bhearsum bug 965447 - Run arm-sim build by default on try, r=bhearsum bug 970738 Integrate jacuzzi allocator with prioritizeBuilders r=bhearsum bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar. r=Callek bug 974166 - Make spidermonkey try_by_default into a dict, r=bhearsum bug 970738 - Jacuzzi implementation for nextSlave r=bhearsum bug 973491 - port MockReset to mozharness, r=rail bug 973492 - allow for multiple basedirs to be passed to PurgeMixin.purge_builds(), r=armenzg bug 973493 - allow mozharness's tbox_print_summary() to be used outside of unittests, r=armenzg bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness, r=rail bug 974777 - add --dump-config and --dump-config-hierarchy to mozharness -- nosetests, r=rail
in production	2014-02-21 0715 PT	bug 974622 - Configure holly like Aurora properly bug 803764 - Run fuzzers on newer CentOS 6.x slaves instead of CentOS 5 ones bug 973063 - Schedule multimedia b2g mochitests for emulator-jb on Cedar bug 972415 - Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock bug 970432 - Do nightly builds with profiling disabled bug 899969 - mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia bug 974093 - Save qemu.log to blobber bug 974082 - Don't reset mock environments if we don't have to
in production	2014-02-19 0640 PT	bug 795793 - decomm w64-ix-slave30 bug 960569 - Please decomm talos-r3-fed64-058 bug 753367 - decomm talos-r3-fed-057 bug 851270 - Don't clobber the source checkout bug 974140 - release builds not using space requirement of platform config bug 972415 - Intermittent Linux spot builder "command timed out: 1800 seconds without output, attempting to kill" while trying to install mock bug 966441 - Run mozbase unit tests from test package bug 974093 - Save qemu.log to blobber
in production	2014-02-14 07:45 PT	bug 971687 - Add config file specifications for mozbase tests; r=bhearsum bug 971841 - [mq]: ant-apache-regex bug 967704 - Chunk more the Android 2.3 test jobs. r=armenzg
in production	2014-02-13 14:00 PT	bug 970552 - Do not use spot instances for some builders bug 972422 - Bump mock timeout
in production	2014-02-13 07:30 PT	bug 965447 - Add a new "arm-sim" spidermonkey variant, r=bhearsum bug 969360 - Disable Firefox nightlies on b2g26_v1_2. r=aki bug 971687 - Please schedule mozbase unit tests on Cedar; r=bhearsum bug 971841 - install ant on try builders. r=bhearsum bug 970552 - Do not use spot instances for some builders. r=catlee
in production	2014-02-11 07:30 PT	bug 967337 - Only run non-unified builds on trunk. r=catlee bug 968545 - Remove nanojit support r=bhearsum bug 969930 - Disable PGO builds and Talos on UX, and reduce priority to 5. r=aki bug 969542 - Change b2g device builds to run periodically. r=rail bug 970491 - Reset Fig and Elm. r=hwine bug 818968 - Configure Elm to run b2g emulator jobs on Fedora and Ubuntu EC2 machines. r=rail bug 968545 - remove nanojit support r=bhearsum bug 969542 - Add periodic build support by platform r=rail
in production	2014-02-10 06:21 PT	no bug - Update release config for Firefox-28.0b1-build2 bug 967704 - (1) Run more mochitest and crashtest chunks on Android 2.3; r=armenzg bug 967084 - Make Pine use mozharness production