Cloud-init bi-weekly status

Posted on Mon 30 April 2018 in status-meeting-minutes • 12 min read

Meeting information

Meeting summary

Recent Changes

The discussion about "Recent Changes" started at 16:07.

#cloud-init Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/14 16:00 UTC | cloud-init 18.2 released (03/28/2018)

The discussion about "#cloud-init Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/14 16:00 UTC | cloud-init 18.2 released (03/28/2018)" started at 16:08.

In-progresss Development

The discussion about "In-progresss Development" started at 16:21.

Office Hours (next ~30 mins)

The discussion about "Office Hours (next ~30 mins)" started at 16:31.

Vote results

Action items, by person

Done items

  • (none)

People present (lines said)

  • blackboxsw (113)
  • mgerdts (21)
  • ubot5 (12)
  • akik (5)
  • stanguturi (4)
  • meetingology (4)
  • robjo (3)

Full Log

16:04 <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting

16:04 <meetingology> Meeting started Mon Apr 30 16:04:15 2018 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:04 <meetingology>

16:04 <meetingology> Available commands: action commands idea info link nick

16:04 <blackboxsw> hi folks, sorry for the mis-representation of when our cloud-init status meeting date. It's time for another episode/update of the happenings in cloud-init.

16:05 <blackboxsw> Next meeting will be in two weeks: May 7th

16:05 <blackboxsw> at 16:00 UTC

16:06 <blackboxsw> The last couple weeks on the upstream side of the house has been a big push to get testing and stability into master for the Ubuntu Bionic release freeze

16:06 <blackboxsw> ... I'd better start with the topic

16:07 <blackboxsw> #topic Recent Changes

16:07 <blackboxsw> The last couple weeks on the upstream side of the house has been a big push to get testing and stability into master for the Ubuntu Bionic release freeze.

16:07 <robjo> May 7th would be 1 week from today that should be May 14th

16:07 <blackboxsw> robjo: gah, I did it again. Thank you... glad someone's listening. Next cloud-init status meeting Monday May 14th 16:00 UTC

16:08 <blackboxsw> #topic #cloud-init Reviews: http://bit.ly/ci-reviews | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting: Monday 5/14 16:00 UTC | cloud-init 18.2 released (03/28/2018)

16:08 <blackboxsw> ok topic agrees in channel now, so I don't botch it at the end of meeting

16:09 <blackboxsw> Along with a blitz for stability in Bionic the following changes have been shepherded into tip of master

16:09 <blackboxsw> - Add reporting events and log_time around early source of blocking time

16:09 <blackboxsw> [Ryan Harper]

16:09 <blackboxsw> - IBMCloud: recognize provisioning environment during debug boots.

16:09 <blackboxsw> (LP: #1767166)

16:09 <blackboxsw> - net: detect unstable network names and trigger a settle if needed

16:09 <blackboxsw> [Ryan Harper] (LP: #1766287)

16:09 <blackboxsw> - IBMCloud: improve documentation in datasource.

16:09 <ubot5> Launchpad bug 1767166 in cloud-init (Ubuntu) "IBMCloud datasource does not recognize provisioning in debug mode." [Medium,Confirmed] https://launchpad.net/bugs/1767166

16:09 <blackboxsw> - sysconfig: dhcp6 subnet type should not imply dhcpv4 [Vitaly Kuznetsov]

16:09 <blackboxsw> - packages/debian/control.in: add missing dependency on iproute2.

16:09 <ubot5> Launchpad bug 1766287 in cloud-init (Ubuntu) "18.04 minimal images on GCE intermittently fail to set up networking " [Undecided,In progress] https://launchpad.net/bugs/1766287

16:09 <blackboxsw> (LP: #1766711)

16:09 <blackboxsw> - DataSourceSmartOS: add locking of serial device.

16:09 <blackboxsw> [Mike Gerdts] (LP: #1746605)

16:09 <blackboxsw> - DataSourceSmartOS: sdc:hostname is ignored [Mike Gerdts] (LP: #1765085)

16:09 <ubot5> Launchpad bug 1766711 in cloud-init (Ubuntu Bionic) "cloud-init missing dependency on iproute2" [Medium,Fix committed] https://launchpad.net/bugs/1766711

16:09 <blackboxsw> - DataSourceSmartOS: list() should always return a list

16:09 <blackboxsw> [Mike Gerdts] (LP: #1763480)

16:09 <ubot5> Launchpad bug 1746605 in cloud-init "DataSourceSmartOS needs locking" [Medium,Fix committed] https://launchpad.net/bugs/1746605

16:09 <blackboxsw> - schema: in validation, raise ImportError if strict but no jsonschema.

16:09 <blackboxsw> - set_passwords: Add newline to end of sshd config, only restart if

16:09 <blackboxsw> updated. (LP: #1677205)

16:09 <ubot5> Launchpad bug 1765085 in cloud-init "DataSourceSmartOS ignores sdc:hostname" [Medium,Fix committed] https://launchpad.net/bugs/1765085

16:09 <blackboxsw> - pylint: pay attention to unused variable warnings.

16:09 <blackboxsw> - doc: Add documentation for AliYun datasource. [Junjie Wang]

16:09 <blackboxsw> - Schema: do not warn on duplicate items in commands. (LP: #1764264)

16:09 <ubot5> Launchpad bug 1763480 in cloud-init "DataSourceSmartOS list() should always return a list" [Medium,Fix committed] https://launchpad.net/bugs/1763480

16:09 <ubot5> Launchpad bug 1677205 in cloud-init "cloud-init eats final EOL of sshd_config" [Medium,Fix committed] https://launchpad.net/bugs/1677205

16:09 <ubot5> Launchpad bug 1764264 in juju 2.3 "bionic cloud-init 18.2 WARNING Juju's 'runcmd' stanza" [High,Triaged] https://launchpad.net/bugs/1764264

16:10 <blackboxsw> the general theme has been: new IBMCloud datasource support for cloud-init, SmartOS datasource work by mgerdts, and some json schema improvements

16:12 <blackboxsw> so background on IBM, is that their support used to be ConfigDrive based datasource only, but there is now some additional support for different IBM boot/provisioning stages, hence a new datasource that can support different boot modew

16:12 <blackboxsw> *boot modes

16:14 <blackboxsw> over the last two weeks we've landed an SRU into xenial and artful: 18.2-4-g05926e48-0ubuntu1~16.04.1 and bionic sits at 18.2-14-g6d48d265-0ubuntu1

16:15 <mgerdts> On the SmartOS side, my changes are driven by our adoption of bhyve (moving away from kvm/qemu). qemu provides a dhcp server VMs could fall back to if could-init was missing or misbehaving. bhyve doesn't have that, so I've been working on getting cloud-init to be more stable with the bhyve serial metadata service.

16:15 <blackboxsw> Also, to our continuous integration on jenkins we now have an additional test for proposed packages in ubuntu for the bionic release to make sure ubuntu doesn't break across pending upgrades

16:15 <blackboxsw> #link https://jenkins.ubuntu.com/server/job/cloud-init-integration-proposed-b/

16:16 <blackboxsw> that integration tests hits the suite of platforms lxd, kvm and ec2

16:16 <blackboxsw> excellent mgerdts, and thanks for the blitz on these branches

16:17 <blackboxsw> looks like there are a few still in our review queue that we'll be able to get through once the dust settles on the bionic release (which should be this week)

16:17 <blackboxsw> #link https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/master/+activereviews

16:19 <mgerdts> Is now the right time to discuss bug 1765801, or is that later?

16:19 <ubot5> bug 1765801 in cloud-init "network should be optionally reconfigured on every boot" [Undecided,Confirmed] https://launchpad.net/bugs/1765801

16:20 <blackboxsw> I think over the last 2 weeks there have been a couple of requests in channel for how someone goes about getting newer cloud init into RHEL7, if anyone on the line today knows the contact point or process for that it'd be helpful. larsks doesn't seem to be around

16:20 <blackboxsw> mgerdts: probably in about 10 mins. thanks for brining it up

16:20 <blackboxsw> hopefully less.

16:20 <blackboxsw> ok I think that's it for recent changes, next topic (in-progress dev, then office hours (and bug discussion))

16:21 <blackboxsw> #topic In-progresss Development

16:21 <blackboxsw> We'll make this one short:

16:22 <blackboxsw> for ubuntu : bionic just went feature freeze last week, our team has a couple of IBM-related cheanges that we are pulling together for a quick SRU into xenial/artful to handle upgrade path from configdrive ->` IBMCloud that we are working on the beginning of this week

16:22 <blackboxsw> we are also trying to wrap up validation of a Bionic SRU per the following bug

16:22 <blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1767412

16:22 <ubot5> Launchpad bug 1767412 in cloud-init (Ubuntu Bionic) "SRU cloud-init 18.2-27-g6ef92c98-0ubuntu1" [Medium,Fix committed]

16:23 <blackboxsw> which grabs a number of the updates I listed in the last topic

16:23 <blackboxsw> since Ubuntu tends to sync all changes from tip into each release stream

16:23 <mgerdts> Is there any chance the SmartOS changes can piggy back on that IBM SRU

16:24 <mgerdts> asked too soon - I see they are mentioned in that bug.

16:25 <blackboxsw> mgerdts: no worries. good ask. probably not for this IBM SRU into xenial/artful which is going to be an exception to our update rule and only be a single cherry pick, but planning a folllowup SRU in about 2 weeks which will pull all changes from tip into artful/xenial/bionic/chunky releases

16:25 <mgerdts> ok

16:25 <blackboxsw> the cherry pick is to fasttrack it for IBM into xenial with minimal risk.

16:25 <blackboxsw> and we want to pull in all your changes if we can (and perform additional validation)

16:25 <blackboxsw> so the next SRU is our target

16:26 <blackboxsw> Also inprogress is some more Azure work on pre-provisioning that should land shortly:

16:26 <blackboxsw> #link https://code.launchpad.net/~jocha/cloud-init/+git/cloud-init/+merge/344192

16:27 <blackboxsw> as well as some builddeb fixes and network configuration printout fixes from smoser

16:29 <blackboxsw> smoser and rharper also worked out some issues on specific google regions where cloud-init was getting hit by a race condition. Cloud-init started up before the kernel/udev was able to rename network devices to stable names like ens4 etc, so cloud-init's network configuration written ended up breaking because it represented devices like eth0 etc.

16:29 <blackboxsw> there are a couple of branches in flight to fix this issue:

16:29 <blackboxsw> #link https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/344181

16:30 <blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/344198

16:30 <blackboxsw> ok I think that's it for in-progress work. So we'll head to office hours so we can chat bugs, branches reviews etc

16:31 <blackboxsw> #topic Office Hours (next ~30 mins)

16:31 <blackboxsw> We'll be hanging out here for anyone who wants more eyes on a review, feature discussions or bug triage....

16:32 <blackboxsw> well, some of us will be :) a couple of us are at a feature planning conference for the week.

16:33 <mgerdts> In https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 smoser said that he was concerned about how this would interact with eventual network hotplug

16:34 <mgerdts> There doesn't seem to be a timeline for network hotplug and the lack of network autoreconfig on reboot is has popped up a couple times in the past week. This is just with a couple early adopters and internal users.

16:34 <blackboxsw> #link https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712

16:35 <blackboxsw> just to track it in the meeting

16:35 <mgerdts> So coming up with some mechanism to make this work soon is pretty important to us.

16:35 <mgerdts> gotcha, will be sure to do that in the future.

16:35 * blackboxsw reads up on that link

16:35 <blackboxsw> no worries, I'm pedantic :)

16:36 <mgerdts> That's how you got chosen to run the meeting, I suppose. :)

16:36 <blackboxsw> yeah network hotplug will have a long tail as far as feature develpment (agreed). I believe it's on our charter for this next quarter. but that's what is being discussed this week

16:37 <blackboxsw> heh on meeting comment ;) too true

16:39 <blackboxsw> so mgerdts your branch allows metadata to set maintain_network to allow cloud-init to control network configuration each reboot with a True value

16:39 <blackboxsw> ?

16:39 <mgerdts> yes

16:39 <mgerdts> if it's not set to true in our metadata, the traditional behavior stays.

16:40 <mgerdts> That is, in the default path, any customization that someone does in the guest will not get whacked.

16:41 <robjo> cloud-netconfig handles hotplug https://github.com/SUSE/Enceladus/tree/master/cloud-netconfig contributions for other distros welcome

16:42 <blackboxsw> nice reference robjo

16:42 <blackboxsw> #link https://github.com/SUSE/Enceladus/tree/master/cloud-netconfig

16:43 <robjo> We currently have no GCE specific information but that is easy enough to add. The GCE guest environment handles this and we use the GCE guest environment code in our images in GCE

16:43 <blackboxsw> mgerdts: so can a user turn off that feature on an instance once they've already deployed, or is it create-time only

16:43 <mgerdts> It can be flipped at any time, in the current implementation.

16:44 <mgerdts> current implementation is only in a development branch

16:47 <blackboxsw> mgerdts: the only things I can see being an issue with the maintain network in cloud-init is that we are adding the cost of another function call && metdata dict parse to look for a signal about maintaining the network. I agree that cloud-init having granularity between is_new_instance vs just re-do network, is something that cloud-init should have.

16:48 <blackboxsw> we probably need to discuss this too with rharper about what short-term vision we can get to while we await our network hotplug support in cloud-init proper

16:49 <blackboxsw> I'd tend to agree that waiting on fully baked hotplug solution is probably too long in this case

16:49 <blackboxsw> as that runway will be at least 2 months I'd think

16:50 <blackboxsw> ok, I'll take an action item to resolve this if we can by next meeting

16:50 <mgerdts> Not only that, but support for it will likely require changes in the host as well. We tend not to do host updates very often, so it could be a year or more after the feature is available in images before it will be useful.

16:51 <blackboxsw> #action blackboxsw to have discussions w/ team on datasource maintaining network on each reboot per https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712

16:51 * meetingology blackboxsw to have discussions w/ team on datasource maintaining network on each reboot per https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712

16:51 <mgerdts> thanks

16:51 <blackboxsw> good topic.

16:52 <mgerdts> Is there another place that is good to catch up with larsks or other people that can offer guidance on for redhat/centos?

16:52 <blackboxsw> let's see, anything else folks want to chat about? stagnant reviews, bugs of interest etc?

16:53 * blackboxsw looks at the last cloud-init community summit attendees list to see if rhel folks have another contacts that was supposed to replace larsks

16:53 <stanguturi> Chad, Is it possible that someone from cloud-init team can take a look at https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538

16:53 <ubot5> Launchpad bug 1766538 in cloud-init (Ubuntu) "network customization with cloud-init does not work on Ubuntu18.04 Beta2 Server" [Undecided,Confirmed]

16:55 <blackboxsw> mgerts, ryan mccabe is a potential contact too, looks like he's not here either today.

16:56 <mgerdts> ok, thanks

16:56 <blackboxsw> hrm, yeah not certain what mechanism is used to get cloud-init updated into RedHat mgerdts. Maybe filing a redhat bug about the request

16:57 <blackboxsw> mgerdts: https://bugzilla.redhat.com/ maybe

16:58 <blackboxsw> stanguturi: yes we can, we are trying to sort and understand any bugs against Bionic that we can

16:58 <mgerdts> ok, I can try that.

16:58 <blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1766538

16:58 <ubot5> Launchpad bug 1766538 in cloud-init (Ubuntu) "network customization with cloud-init does not work on Ubuntu18.04 Beta2 Server" [Undecided,Confirmed]

16:58 <stanguturi> blackboxsw: Thanks

17:00 <blackboxsw> stanguturi: ok, so this is netplan + cloud-init related right?

17:00 <stanguturi> blackboxsw: Yes.

17:01 <akik> what does network hotplug mean in cloud-init context?

17:02 * blackboxsw tries to remember what vmware datasource does, (like writing files direct to network /etc/network/interfaces.d)

17:02 <blackboxsw> akik: https://hackmd.io/M1Tae41PQBC7a9qMsurTJw?both is a shared document for comment on hotplug in cloud-init

17:02 <blackboxsw> #link https://hackmd.io/M1Tae41PQBC7a9qMsurTJw?both

17:03 * blackboxsw looks to see if there was a better doc hrm

17:03 <stanguturi> blackboxsw: Oh. But in the case of netplan, why does cloud-init remembers?

17:04 <akik> blackboxsw: does it mean that cloud-init stays running, waiting for new network interfaces to appear?

17:04 <blackboxsw> akik: right, it would mean that you wouldn't have to reboot cloud-init if devices get added at a later time (post-boot)

17:05 <blackboxsw> cloud-init would listen to some sort of event channel and react, re-write, and apply network config to add new devices

17:06 <akik> would it do the same thing as you could do with ansible or puppet? sorry i'm trying to understand why you would do it with cloud-init

17:08 <blackboxsw> akik: you would try to do it with cloud-init if you didn't want to rely on additional configuration management solutions if the only thing you needed was network config to reflect reality (not full system configuration and system automation)

17:09 * blackboxsw has more puppet/chef background than ansible.

17:09 <blackboxsw> cloud-init does currently detect and write network configuration based on what the user/cloud-metadata tell us is the proper config for the instance

17:10 <akik> i only thought of cloud-init to do the initial configuration

17:10 <blackboxsw> so it would follow that if the metadata could dynamically tell the instance that network config has changed, cloud-init should probably try to react to that to fix the config to match the updated network configuration

17:11 <blackboxsw> akik: correct. cloud-init current only handle initial boot config and leaves the rest up whatever mechanism someone uses to update detailed config after that boot

17:11 <akik> ok thanks

17:12 <blackboxsw> akik: and we'd make that feature configurable (handle hotplug:True/False) so if users have other services handling hotplug cloud-init wouldn't collide

17:13 <blackboxsw> ok I think we're hitting the end of office hours. please feel free to continue discussion, we all poke around here throughout the day as our primary means of communication

17:13 <blackboxsw> thanks robjo akik stanguturi and mgerdts for the lively discussion

17:13 <blackboxsw> stanguturi: I'll dig up more info on that bug today

17:13 <mgerdts> thank you

17:13 <blackboxsw> as always notes will be here

17:14 <blackboxsw> #link https://cloud-init.github.io/

17:14 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)