Cloud-init bi-weekly status

Posted on Tue 29 May 2018 in status-meeting-minutes • 10 min read

Meeting information

Meeting summary

LINK: https://cloud-init.github.io/

Previous Actions

The discussion about "Previous Actions" started at 16:09.

Recent Changes

The discussion about "Recent Changes" started at 16:13.

In-progress Development

The discussion about "In-progress Development" started at 16:19.

Office Hours (next ~30 mins)

The discussion about "Office Hours (next ~30 mins)" started at 16:34.

Vote results

Done items

  • (none)

People present (lines said)

  • blackboxsw (118)
  • dmbaturin (20)
  • powersj (6)
  • dpb1 (5)
  • robjo (5)
  • ubot5 (4)
  • smoser (4)
  • meetingology (3)
  • rharper (2)

Full Log

16:05 <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting

16:05 <meetingology> Meeting started Tue May 29 16:05:51 2018 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:05 <meetingology>

16:05 <meetingology> Available commands: action commands idea info link nick

16:06 <dpb1> dmbaturin: there will be a time for open questions in this meeting in just a few minutes. :)

16:06 <dpb1> so you will have the right people around

16:06 <blackboxsw> welcome folks to another cloud-init community status meeting, today's meeting delayed by one day due to US holiday. Next meeting will be June 11th. same time

16:07 <blackboxsw> I've added an actions topic to this meeting so we can wrap up or carry over any actions discussed last time

16:08 <blackboxsw> the topics will be Previous Actions, Recent Changes, In-progress Development, and Office Hours

16:08 <dmbaturin> Oh, cool.

16:09 <blackboxsw> As always notes will be posted to the following site

16:09 <blackboxsw> #link https://cloud-init.github.io/

16:09 <blackboxsw> welcome dmbaturin good timing. :)

16:09 <blackboxsw> #topic Previous Actions

16:09 <dmbaturin> Yeah, I'm just in time it seems. ;)

16:09 <blackboxsw> 2 weeks ago we had a couple of followup items that needed some extra review:

16:10 <blackboxsw> * ACTION: blackboxsw review distro dection and empty modules list

16:10 <blackboxsw> * ACTION: robjo review existing chrony support in master per rharper's work

16:10 <blackboxsw> * ACTION: blackboxsw carryover network hotplug vs network maintenance on reboot-only

16:10 <blackboxsw> we did get through robjo's branches on distro detection and landed them\

16:11 <blackboxsw> and I know our team also discussed a potential approach to network hotplug vs network maintenance to better enable SmartOs folks who want to handle network config across reboots only

16:12 <blackboxsw> I think we decided we needed to draw up a quick shared document on a proposal which would allow for maintenance on reboots only vs true hotplug.

16:12 <blackboxsw> I'll carry over that action to write up a doc on this and send it to list by the next meeting

16:13 * blackboxsw blackboxsw write up short doc/branch on hotplug versus network maintenance on reboot for comment

16:13 <blackboxsw> and I believe robjo from SuSE was able to get through rharper's chrony support branch with a couple comments too

16:13 <blackboxsw> so no other actions from last meeting

16:13 <blackboxsw> #topic Recent Changes

16:14 <blackboxsw> this following content landed in cloud init tip over the last two weeks

16:14 <blackboxsw> - Do not use the systemd_prefix macro, not available in this environment

16:14 <blackboxsw> [Robert Schweikert]

16:14 <blackboxsw> - doc: Add config info to ec2, openstack and cloudstack datasource docs

16:14 <blackboxsw> [Chad Smith]

16:14 <blackboxsw> - Enable SmartOS network metadata to work with netplan via per-subnet

16:14 <blackboxsw> routes [Dan McDonald] (LP: #1763512)

16:14 <blackboxsw> - openstack: Allow discovery in init-local using dhclient in a sandbox.

16:14 <ubot5> Launchpad bug 1763512 in cloud-init "DataSourceSmartOS ignores sdc:routes" [Medium,Fix committed] https://launchpad.net/bugs/1763512

16:14 <powersj> lol!

16:14 <powersj> welcome back

16:14 <blackboxsw> heh looks like I got kicked for the paste :)

16:14 <powersj> blackboxsw: your last message was - openstack: Allow discovery in init-local using dhclient in a sandbox.

16:15 <blackboxsw> - tests: Avoid using https in httpretty, improve HttPretty test case.

16:15 <blackboxsw> 10:14 (LP: #1771659)

16:15 <blackboxsw> 10:14 - yaml_load/schema: Add invalid line and column nums to error message

16:15 <blackboxsw> 10:14 [Chad Smith]

16:15 <blackboxsw> 10:14 - Azure: Ignore NTFS mount errors when checking ephemeral drive

16:15 <blackboxsw> 10:14 [Paul Meyer]

16:15 <ubot5> Launchpad bug 1771659 in cloud-init "unittests fail in OpenSuSE 42.3 with httpretty issues" [Medium,Fix committed] https://launchpad.net/bugs/1771659

16:15 <blackboxsw> - packages/brpm: Get proper dependencies for cmdline distro.

16:15 <blackboxsw> 10:14 - packages: Make rpm spec files patch in package version like in debs.

16:15 <blackboxsw> 10:14 - tools/run-container: replace tools/run-centos with more generic.

16:15 <blackboxsw> 10:14 - Update version.version_string to contain packaged version. (LP: #1770712)

16:15 <blackboxsw> 10:14 - cc_mounts: Do not add devices to fstab that are already present.

16:15 <blackboxsw> 10:14 [Lars Kellogg-Stedman]

16:15 <blackboxsw> 10:14 - ds-identify: ensure that we have certain tokens in PATH. (LP: #1771382)

16:15 <blackboxsw> 10:14 - tests: enable Ubuntu Cosmic in integration tests [Joshua Powers]

16:15 <blackboxsw> 10:14 - read_file_or_url: move to url_helper, fix bug in its FileResponse.

16:15 <blackboxsw> 10:14 - cloud_tests: help pylint [Ryan Harper]

16:15 <blackboxsw> 10:14 - flake8: fix flake8 errors in previous commit.

16:15 <ubot5> Launchpad bug 1770712 in cloud-init "It would be nice if cloud-init provides full version in logs" [Medium,Fix committed] https://launchpad.net/bugs/1770712

16:15 <blackboxsw> 10:14 - typos: Fix spelling mistakes in cc_mounts.py log messages [Stephen Ford]

16:15 <blackboxsw> 10:14 - tests: restructure SSH and initial connections [Joshua Powers]

16:15 <ubot5> Launchpad bug 1771382 in cloud-init "ds-identify: fails to recognize NoCloud datasource on boot cause it does not have /sbin in $PATH and thus does not find blkid" [Low,Fix committed] https://launchpad.net/bugs/1771382

16:15 <blackboxsw> 10:14 - ds-identify: recognize container-other as a container, test SmartOS.

16:15 <blackboxsw> ok hopefully we ended on ds-identify

16:16 <dmbaturin> Yes, we did.

16:16 <blackboxsw> excellent. sorry for the paste, I'll send this out to cloud-init@lists.canonical.com a day before the next meeting so we don't have to IRC flood here

16:17 <blackboxsw> make that cloud-init@lists.launchpad.net

16:17 <blackboxsw> also we finished our SRU (stable release update) of cloud-init 18.2.27 to Bionic.

16:18 <blackboxsw> Ubuntu Cosmic currently reflects near tip of master 18.2.59

16:18 <blackboxsw> ok that's all for Recent Changes

16:18 <blackboxsw> anything I'm missing powersj ?

16:19 <powersj> I think you are good

16:19 <blackboxsw> #topic In-progress Development

16:19 <blackboxsw> We track upstreams progress publicly in trello

16:19 <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:20 <blackboxsw> any blue labeled cards are cloud-init core work

16:21 <blackboxsw> we have been fixing a couple of bugs raised by our CI infrastructure on newer series of Ubuntu . currently a minor issue with salt minion on Bionic or later, and a couple of unit and integration test race conditions

16:23 <blackboxsw> big ticket items for cloud-init in the nearterm are metadata standardization across clouds, so cloud-init scripts/cloud-config template can source these cloud-provided values

16:24 <dmbaturin> Metadata standardization is something I really would like to see, if you need more hands for that, let me know.

16:24 <blackboxsw> the standardization of this instance-data will allow folks to script against any standard values provided to cloud-init in the same way on any cloud. Think hostname, fqdn, ip addrs, region name etc.

16:24 <dmbaturin> SSH keys too!

16:24 <blackboxsw> definitely dmbaturin I'll point you at a couple branches and what we're thinking

16:25 <blackboxsw> this conent will show up in /run/cloud-init/instance-data.json

16:25 <blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/datasources.html?highlight=instance-data#instance-data

16:26 <blackboxsw> and will also be referenced via jinja template variables

16:26 <blackboxsw> and a cloud-init query CLI

16:28 <dmbaturin> The data will be updated whenever a change in the environment is made?

16:28 <blackboxsw> Also powersj will be working toward a common library for cloud-testing in the weeks to come which cloud-init integration tests will leverage to drive lxd, ec2, openstack azure etc for a cloud testing

16:29 <dmbaturin> Also, will it be possible to stop cloud-init from doing anything but writing that data and starting an external script to process it?

16:30 <blackboxsw> dmbaturin: some of that functionality will be handled in the hotplug work we are starting on. There will be operations that can be triggered by either a hotplug monitor on metadata or by cloud-init's CLI to say query from cache (the instance-data.json file) versus query fresh/update

16:30 <dmbaturin> I see.

16:33 <blackboxsw> dmbaturin: cloud-inits init-local or init-network stage is what calls "get_data" on the give datasources to collect and write that data to file. Spawning a script is generally done through runcmd which happens in cloud-init's 'final' stage. Trying to decouple them (and skipping the modules:config stage) is possible by altering /etc/cloud/cloud.cfg in a custom image to specify no modules in a given stage. Though it's not

16:33 <blackboxsw> really recommeded as most of the modules only do a quick sanity check to see if they are specifically enabled before trying to do any realy work

16:33 <blackboxsw> we try to keep boot time as fast as possible and cut out the fat where we can

16:33 <blackboxsw> if that's the concern you had

16:34 <blackboxsw> ok I think that's it for In-progress development we can move to office hours for all addtional discussion

16:34 <blackboxsw> #topic Office Hours (next ~30 mins)

16:35 <dmbaturin> No, boot time is not the primary concern here, my concern is how to ensure no module is trying to treat our system as if it was a normal Debian (which either doesn't work or can potentially get the system into an inconsistent state).

16:36 <dmbaturin> I guess if we are having a real meeting, it may be a good idea to formally introduce myself and the project. :)

16:36 <blackboxsw> All topics of interest to cloud-init development can be brought up and discussed here. If there are merge proposal that need attention, bugs that need work just bring them up here we should have a few sets of eyes on this channel to discuss and comment

16:36 <blackboxsw> sounds good dmbaturin introduce away :)

16:37 <blackboxsw> Chad smith, Canonical, one of the maintainers of cloud-init. We have a few others here (some on vacation). powersj rharper smoser dpb1 all canonical as well.

16:38 <dpb1> idk who blackboxsw is

16:38 <dpb1> he might be crazy

16:38 <blackboxsw> frequently we have other distribution developers and cloud devs here too (SuSE, RedHat, Microsoft Azure, SmartOS, VMWare )

16:38 <blackboxsw> heh, I'm just a bot

16:38 <dmbaturin> So, I'm one of the maintainers of the VyOS project (http://vyos.io). It's a distro for routers and firewalls whose primary goal is to be just like hardware routers, but not tied to any hardware, which includes a single config file and unified CLI with a commit/rollback model, versioning, and cross-checks (e.g. if you try to reference a non-existent NIC in DHCP configuration, commit fails).

16:39 <dpb1> nice to meet you dmbaturin

16:40 <blackboxsw> ahh makes sense. So debian-based os kindof, which is why you'd want to lock down what modules run.

16:40 <dmbaturin> We support all major virtualization platforms now in the sense of including all required drivers and utilities, but autoconfiguration on cloud platforms is only supported for EC2 via a custom script, so we are looking to ways to support more clouds, ideally without doing the work that is already done, or at least contributing those general things into something where more people can benegit from it, not just us.

16:41 <blackboxsw> alos dmbaturin each config module claims what distro is supported in a distro property, so you could vet what modules you want to run, and only add VyOS to the list of compatible distros. Config modules all live in source at cloudinit/config/cc_*py.

16:41 <blackboxsw> but we can discuss that confuig module support (or not) once you dig in to look at supporting VyOS

16:42 <dmbaturin> Yes, I'm thinking how exactly it should be done.

16:42 <blackboxsw> dmbaturin: cloud-init's a pretty good choice for getting that cloud-support breadth for free

16:43 * robjo sorry I'm late

16:43 <blackboxsw> robjo: sorry for the late change from yesterday's normal meet time

16:43 <dmbaturin> The least intrusive option would be to indeed improve the instance data format, so that we can simply pass it to our own script, which is why I'm all for contributing to it.

16:43 <robjo> blackboxsw: noLnxDistro branch has not yet been merged

16:44 <blackboxsw> bah robjo ahh you're right

16:44 <blackboxsw> ok looks like you handled all review comments. I'll get it landed today

16:44 <smoser> powersj: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/347060

16:44 * blackboxsw blackboxsw land https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/336794 today

16:45 <smoser> you moved that to 'approved' i guess ?

16:45 <smoser> which meant the bot didnt comment (sorry blackboxsw ... interupted)

16:45 <dmbaturin> blackboxsw: Could you point me to the branches were the work on instance data is going on?

16:45 <blackboxsw> dmbaturin: here's one stale one I need to get back to this week. for enabling the template reference of instance-data.json content

16:46 <powersj> smoser: ah sorry you are right

16:47 <smoser> powersj: i'm going to land it anyway

16:47 <robjo> also I think emptyStageOK branch should be ready to go

16:47 <powersj> ok

16:47 <blackboxsw> hrm digging on the metadata branch.

16:48 <blackboxsw> dmbaturin: the trello card I'll be tying branches to is this one

16:48 <blackboxsw> #link https://trello.com/c/5n5B8x23/802-cloud-init-query-standardized-json-information

16:48 <robjo> and https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 should be on smoser plate

16:49 <robjo> or anyone else who wants to pick it up and get it merged, please

16:49 <blackboxsw> #link https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/335290

16:50 <blackboxsw> ^ dmbaturin initial template handling thoughts.... I have to create at least 2 branches to standardize datasource class apis to make the metadata content easier to generalize and I can add your launchpad username to the review as I put them up

16:50 <rharper> smoser: re: hostname, yes, that's right; we probably could update the set_hostname docs to mention that detail w.r.t early hostname setting

16:51 <rharper> blackboxsw: sorry to interrupt

16:51 <blackboxsw> dmbaturin: what's your launchpad user name? (mine's chad.smith)

16:51 <dmbaturin> blackboxsw: dmbaturin

16:51 <blackboxsw> heh.

16:51 <blackboxsw> thx

16:51 <dmbaturin> I'm too predictable. ;)

16:51 <blackboxsw> yeah, I lost a bet on blackboxsw :)

16:51 <blackboxsw> robjo: okay adding that branch too for review/landing

16:51 * blackboxsw land https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904

16:52 * blackboxsw land https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/336794

16:53 <blackboxsw> alright, any other topics or discussions?

16:53 <dmbaturin> Trello is integrated with launchpad?

16:55 <blackboxsw> dmbaturin: nope, just easy to use for our agile workflow. And simple to cut-paste links, assign people, drag to different lanes as the work progresses

16:55 <blackboxsw> has a lot of github integraiton if you get the right plugins

16:56 <blackboxsw> we have some minimal tooling that can talk to lauchpad and inject cards, but that's hand-written, not part of trello product.

16:59 <dmbaturin> I mean, if you add my username there, will I get any notifications about card changes.

16:59 <blackboxsw> dmbaturin: I get emails from all trello card moves,changes. let's see

16:59 <blackboxsw> I can subscribe you to the card (you want the standardized json stuff?)

17:00 <dmbaturin> Yes.

17:02 <blackboxsw> hrm can't find your user

17:02 <blackboxsw> ahh

17:03 <blackboxsw> I think I invited you

17:03 <blackboxsw> and added your user to the card so you can watch it progress

17:05 <blackboxsw> ok I think that about wraps up our meeting for today

17:05 <blackboxsw> any parting shots?

17:05 <blackboxsw> I'll post these notes to our github project page

17:05 <blackboxsw> #link https://cloud-init.github.io/

17:05 <blackboxsw> thanks again all

17:05 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)