Cloud-init bi-weekly status

Posted on Mon 19 March 2018 in status-meeting-minutes • 11 min read

Meeting information

Meeting summary

Recent Changes

The discussion about "Recent Changes" started at 16:05.

In-progress Development

The discussion about "In-progress Development" started at 16:12.

Office hours (next ~30 mins)

The discussion about "Office hours (next ~30 mins)" started at 16:43.

Vote results

Done items

  • (none)

People present (lines said)

  • blackboxsw (85)
  • smoser (57)
  • stanguturi (16)
  • ubot5` (8)
  • ajorg (5)
  • rharper (5)
  • dpb1 (4)
  • meetingology (3)

Full Log

16:02 <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting

16:02 <meetingology> Meeting started Mon Mar 19 16:02:30 2018 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:02 <meetingology>

16:02 <meetingology> Available commands: action commands idea info link nick

16:03 <blackboxsw> ok, let's kick off this cloud-init bi-weekly meeting. welcome all!

16:04 <dpb1> o/

16:04 <blackboxsw> it's been a busy couple weeks for a few of us w/ planning meetings and vacation, but let's see what progress we've made on cloud-init.

16:05 <blackboxsw> #topic Recent Changes

16:05 <dpb1> smoser vacation specifically

16:05 <blackboxsw> hehe. Generally we're tracking high-points of what lands in our trello board

16:05 <blackboxsw> #link http://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:06 <blackboxsw> but from changelogs folks have made progress on azure, vmware and FreeBSD deployment targets

16:06 <blackboxsw> - netplan: render bridge port-priority values (LP: #1735821)

16:06 <blackboxsw> - net: recognize iscsi root cases without ip= on kernel command line.

16:06 <blackboxsw> (LP: #1752391)

16:06 <blackboxsw> - util: Fix subp regression. Allow specifying subp command as a string.

16:06 <blackboxsw> (LP: #1755965)

16:06 <blackboxsw> - This commit fixes get_hostname on the AzureDataSource.

16:06 <blackboxsw> [Douglas Jordan] (LP: #1754495)

16:06 <ubot5>` Launchpad bug 1735821 in nplan (Ubuntu Artful) "netplan needs bridge port-priority support" [Medium,Fix committed] https://launchpad.net/bugs/1735821

16:06 <blackboxsw> - shellify: raise TypeError on bad input.

16:06 <blackboxsw> - FreeBSD: Set hostname to FQDN. [Dominic Schlegel] (LP: #1753499)

16:06 <blackboxsw> - Make salt minion module work on FreeBSD.

16:06 <blackboxsw> [Dominic Schlegel] (LP: #1721503)

16:06 <ubot5>` Launchpad bug 1752391 in cloud-init "cloud-init does not recognize initramfs provided network config in all cases" [Medium,Fix committed] https://launchpad.net/bugs/1752391

16:06 <blackboxsw> - set_hostname: When present in metadata, set it before network bringup.

16:06 <blackboxsw> (LP: #1746455) VMWare

16:06 <ubot5>` Launchpad bug 1755965 in cloud-init "util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965

16:06 <blackboxsw> - cc_snap: Add new module to install and configure snapd and snap

16:06 <blackboxsw> packages.

16:06 <blackboxsw> - doc: fix all warnings issued by 'tox -e doc'

16:06 <blackboxsw> - tests: Make pylint happy and fix python2.6 uses of assertRaisesRegex.

16:06 <ubot5>` Launchpad bug 1755965 in cloud-init "duplicate for #1754495 util.subp regression: no longer accept commands as string" [High,Fix committed] https://launchpad.net/bugs/1755965

16:06 <blackboxsw> - tests: fix run_tree and bddeb

16:06 <blackboxsw> - tests: Fix some warnings in tests that popped up with newer python.

16:06 <ubot5>` Launchpad bug 1753499 in cloud-init "hostname in FreeBSD should prefere FQDN" [Undecided,Fix committed] https://launchpad.net/bugs/1753499

16:06 <blackboxsw> - tests: fix flakes warning for unused variable

16:06 <blackboxsw> - tests: patch leaked stderr messages from snap unit tests

16:06 <ubot5>` Launchpad bug 1721503 in cloud-init "salt module not able to be used on FreeBSD" [Medium,Fix committed] https://launchpad.net/bugs/1721503

16:06 <blackboxsw> - tests: Centralize and re-use skipTest based on json schema presense.

16:06 <ubot5>` Launchpad bug 1746455 in cloud-init "cloud-init vSphere cloud provider DHCP unique hostname issue" [High,Fix committed] https://launchpad.net/bugs/1746455

16:08 <blackboxsw> a big thanks to dojordan (Azure) and Dominic Schlegel (FreeBSD) for patching some gaps in support as cloud-init master progresses

16:10 <blackboxsw> On the ubuntu side of the house we got tip of tree published into Bionic thusday & friday, we are awaiting cloud-image builds which look like they are stale at 03-15-2018. once those builds are published, all clouds should be getting latest cloud-init on Bionic

16:11 <blackboxsw> I think that's probably it for 'done' work. We have a few things in flight at the moment

16:12 <blackboxsw> #topic In-progress Development

16:13 <blackboxsw> Ubuntu is getting a number of new cloud-config modules:

16:13 <blackboxsw> - new cc_snap module (deprecated cc_snappy and cc_snap_config modules) the ability to install and manage snap package

16:14 <blackboxsw> - new cc_ubuntu_drivers: support to install 3rd party drivers on install

16:15 <blackboxsw> - new cc_ubuntu_advantage: manage Ubuntu Advantage subscriptions for services such as Extended Security Mainenance (trusty), canonical livepatch and FIPS PPAs

16:15 <blackboxsw> these should be landing in the week(s) to come

16:15 <blackboxsw> and cc_snap landed already

16:17 <blackboxsw> also there are a couple of branches that we are trying to wrap up for first class chrony support (per rharper, inspired by robjo's work)

16:17 <rharper> blackboxsw: smoser: on the lander emails, the subject could include the git hash (or branch name); it's currently only in the body;

16:17 <smoser> rharper: i had suggested to blackboxsw that it should acutally change to not send a subject. so it threads in your email reader with the other MP mails.

16:18 <rharper> heh

16:18 <blackboxsw> maybe we can toggle between the two modulus 2 :)

16:18 <rharper> sorry, didn't meant to disturb the flow

16:19 <rharper> continue

16:19 <blackboxsw> yeah, we've also touched a little bit of our code landing automation this last week. powersj also is working on a git lander plugin that we might be able to use to automate landing of approved branches w/ tox test runs

16:19 <blackboxsw> anything to free up developer time will give us more time for reviews/code

16:19 <ajorg> should vendor-specific modules be shipped in a separate package?

16:20 <smoser> vendor specific modules ?

16:20 <ajorg> ubuntu_advantage

16:22 <blackboxsw> good question/point. I hand't thought about that separation as a lot of the modules cloud-init delivers support a subset of distros

16:23 <blackboxsw> each module has a distro attribute defined as to whether or not it will even run

16:23 <blackboxsw> so we have spacewalk, zypper_repos etc

16:24 <smoser> and cloud.cfg is rendered based on knowledge of the distro

16:24 <smoser> so ubuntu_advantage wont even be in the list of config modules

16:24 <smoser> having a static config module list is WIN in this case (but pain elsewhere)

16:25 <smoser> at some point whe may have a more dynamic config module list.

16:25 <smoser> but anyway... at the moment the only cost to non-ubuntu of that module being shipped is bytes on disk.

16:26 <blackboxsw> if/when we do define that more dynamic config module list, I'd like us also to look at having configurable/separate plugin directories defined for folks providing vendor-specific content.

16:26 <ajorg> agree that having a more dynamic config module list is prerequisite to being able to parcel out modules to other packages

16:26 <blackboxsw> so that we don't expect folks to add plugins directly into /usr/lib/python3/dist-packages/cloudinit/config/ for instance

16:26 <smoser> yeah. at the point when it is dynamic, the module would still declare its support for a list of distros and would be filtered out.

16:29 * ajorg is satisfied

16:30 <blackboxsw> :). the only other thing I can think of in progress two more datasources softlayer cloud support by smoser and hetzner cloud

16:30 <blackboxsw> so cloud-init is getting it's grubby hands into a couple of more clouds shortly.

16:31 <blackboxsw> s/it's/its/

16:31 <blackboxsw> it's nice to see the adoption continue to grow

16:31 <smoser> looks like someone followed up on hetzner

16:31 <smoser> so that hopefully is ready to land

16:31 <blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init

16:32 <blackboxsw> rharper: also has a couple of branches to allow cloud-init to work a bit better when rendering netplan configuration

16:33 <blackboxsw> I think that's all for in-progress development at the moment.

16:33 <smoser> man we need to fix that pylint thing.

16:33 <smoser> did you mention ?

16:33 <rharper> blackboxsw: yeah, I just pushed a fix for v1 global dns entries to get rendered under interfaces without any dns configuration

16:33 <blackboxsw> Anything else that should be noted by anyone?

16:33 <blackboxsw> #link https://code.launchpad.net/~lp-markusschade/cloud-init/+git/cloud-init/+merge/338439

16:33 <blackboxsw> oops

16:34 <smoser> cloudinit/config/cc_puppet.py:143: [W1505(deprecated-method), handle] Using deprecated method readfp()

16:34 <blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/341662

16:34 <smoser> that needs fixing. it has come to us due to a new version of some of our tox environemnts. we do not fully pin the versions , only the top level packages. Ie, pylint's dependencies changed, but we only pin pylint version.

16:35 <blackboxsw> yeah how much should we freeze our deps?

16:35 <blackboxsw> it's kindof annoying to have your branch locally pass ci, and a fresh build of CI deps fail when you try to land

16:36 <blackboxsw> but I don't really know whether it's worth us 'pinning' everything

16:36 <smoser> i think pinning everything is generally the best practice for this sort of thing now.

16:37 <smoser> oh my.

16:37 <blackboxsw> so if we pin the world, should we also just make it a habit to occasionally tox -e tip-pylint?

16:37 <smoser> blackboxsw: did you know you accidently fixed that in trunk ?

16:38 <blackboxsw> smoser: I know I intentionally added a pylint ignore on that to come back and address it today.

16:38 <ajorg> i tend to believe it's better to stay current and take your punches a few at a time so you don't have a major upset when you have to upgrade.

16:38 <smoser> oh ko. i see.

16:39 <smoser> ajorg: well, sor tof. if you have c-i that you want to be green, and consider it bad when it is not, then you dont want dude-on-the-internet to break you

16:40 <smoser> there is the "good" break, where new upload to pypi identifies some lingering bug

16:40 <blackboxsw> +1 ajorg, but I'm good (on avoiding a avalanche) if we agree to run tip-pylint target fairy regularly to avoid the landslide

16:40 <smoser> but also the "bad" break where some upload breaks your c-i for invalid reason.

16:42 <smoser> one huge advantage to pinning is the ability to re-create things.

16:42 <blackboxsw> it definitely felt like last week was a lot of c-i breaks for changes unrelated to the code up for review

16:42 <smoser> ie, if you were looking to it bisect something...

16:42 <smoser> git bisect

16:43 <blackboxsw> I should transition to the office hours topic so we can continue discussion

16:43 <smoser> you can't really do that if trunk from a point in the past does not pass C-I because an external dependency changed.

16:43 <smoser> sure we can transition to office hours.

16:43 <blackboxsw> #topic Office hours (next ~30 mins)

16:43 <smoser> but yeah... i want c-i on tip to not just start failing.

16:44 <stanguturi> @blackboxsw, I have couple of requests. First, https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1724128 discussed this in last meeting as well. Any help is greatly appreciated

16:44 <ubot5>` Ubuntu bug 1724128 in open-vm-tools (Ubuntu) "Need a Success / Failure notification mechanism when cloud-init finishes." [Undecided,New]

16:44 <blackboxsw> that's fair. smoser can we maybe add a jenkins job to run tip-pylint then weekly. So, we don't have a huge backlog of lint failures against tip?

16:45 <smoser> blackboxsw: i'mi fine with that... thats why we added the tip-* targets. so it was easy enough to keep current.

16:45 <blackboxsw> +1 smoser I'll put a card up for that

16:46 <smoser> stanguturi: your suggestion there is not a bad idea at all.

16:46 <blackboxsw> #link https://trello.com/c/vD1em9WP/698-jenkins-job-to-run-tox-tip-pylint-weekly-pin-all-lint-versions-otherwise

16:46 <smoser> the desire to have cloud-init tell the platform/datasource that it failed or succeeded is valid.

16:47 <smoser> with MAAS, that his done via reporting

16:47 <blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exists

16:47 <blackboxsw> stanguturi: hiya. I think we talked after that meeting about trying to allow the datasource to subscribe to a callback when cloud-init exits

16:47 <smoser> cloud-init reports status and results via a Reporter.

16:47 <smoser> dojordan (i think) had also put up a request for a reproter module on azure.

16:47 <smoser> so we do kind of have the function you're after in place.

16:48 <stanguturi> smoser: ok. Any inputs / examples of using it will be really great.

16:50 <smoser> stanguturi: well, the reporter interface is pretty simple. you can cloudinit/reporting/handlers.py

16:51 <smoser> blackboxsw: http://paste.ubuntu.com/p/6KjDX8WHQH/

16:51 <smoser> did you intend boht of those changes ?

16:51 <stanguturi> smoser: ok. Then do I need to write a new handler for our DataSource?

16:52 <smoser> stanguturi: well you write a reporting Handler, ankd then either system confi or optionally datasource config would turn that reporter on.

16:52 <blackboxsw> wow smoser, no

16:52 <blackboxsw> wow, ok, I'll put up a branch to fix that

16:52 <stanguturi> smoser: ok. Will work on that.

16:52 <smoser> ok.

16:53 <stanguturi> I have another quick request about https://code.launchpad.net/~sankaraditya/cloud-init/+git/cloud-init/+ref/set_hwclock_module

16:53 <smoser> stanguturi: do you find this is actually needed ?

16:53 <smoser> to my knowledge the only time anyone would ever set their hardware clock to something other than UTC would be dual booting with windows.

16:54 <smoser> which i can't seem to believe is all that a common situation in VMs

16:54 <stanguturi> smoser: Yeah. We need this for 'VMware guest customization workflow'.

16:54 <dpb1> smoser: oh man, I hope not

16:54 <stanguturi> smoser: if you think, this is not worth for the base cloud-init modules, I can modify to do this change in our datasource specific modules.

16:54 <smoser> stanguturi: does it actually solve a current problem for you ?

16:55 <stanguturi> smoser: For 'VMware managed VMs', customers can specify in the specification file if they want UTC or localtime for the hardware clock.

16:55 <smoser> or one that originally came in from a decade ago

16:55 <stanguturi> smoser: Our existing customization (non cloud-init) engine does it. If we want to move to cloud-init, we want to port all our changes from our engine to our datasource in cloud-init.

16:55 <smoser> hm... so my sugestion is really to stop allowing that :)

16:56 <stanguturi> smoser: Oh ok. Can you please add a comment to that merge request just for the record.

16:56 <smoser> i very well could be wrong

16:56 <smoser> but the only time that i ever had to deal with this was when dual booting

16:56 <smoser> with windows specifically

16:57 <stanguturi> smoser: ok.

16:57 <smoser> am i wrong there ?

16:57 <smoser> i really could be.

16:58 <stanguturi> smoser: I can discuss this within our team. But to be on par with our existing engine, want to port the changes.

16:58 <smoser> and even if you get it wrong, generally speaking you havhe some sort of ntpdate or ntp that will fix your system clock anyway.

16:58 <smoser> stanguturi: yeah. i understand that.

16:58 <stanguturi> I have another request. For Ubuntu 18.04, we are planning to set 'disable_vmware_customization' flag to False by default in /etc/cloud/cloud.cfg file.

16:59 <stanguturi> Want to know your opinion, shall we set it in cloud-init installation phase or request Ubuntu maintainers to set it in 18.04

17:00 <stanguturi> smoser: And when is the cloud-init 18.2 scheduled for release? 3/22?

17:01 <blackboxsw> probably a good time for us to bring that up

17:01 <smoser> yeah. whoops.

17:01 <smoser> :)

17:01 <smoser> 18.2 is scheduled for 3/22 (thursday)

17:02 <blackboxsw> We cloud-init 18.2 have it scheduled for an arbitrary 3/22 date, we'd like to slip that out to next week Tuesday 3/27

17:02 <smoser> but amoung our internall team we decided to push that to 3/27

17:02 <stanguturi> smoser: ok. Thanks for the update.

17:02 <blackboxsw> there a a few in flight branches, azure, softlayer etc that we'd like to get in and get tested before 18.2

17:02 <smoser> we will send an email today or tomrrow witih "pending release" like subject like we've done before.

17:03 <blackboxsw> dpb1: others any objections to cutting the 18.2 release on Tuesday 3/27?

17:03 <dpb1> none

17:04 * blackboxsw adds the upcoming date to the topic

17:05 <blackboxsw> ... ok, folks interested in discussing today?

17:08 <blackboxsw> #link https://cloud-init.github.io

17:09 <blackboxsw> The above link will have our captured logs for this meeting.

17:09 <blackboxsw> thanks again for tuning in

17:09 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)