Cloud-init bi-weekly status

Posted on Mon 11 December 2017 in status-meeting-minutes • 13 min read

Meeting information

Meeting summary

Recent Changes

The discussion about "Recent Changes" started at 16:07.

In-progress Development

The discussion about "In-progress Development" started at 16:15.

Office Hours (for next 30 mins)

The discussion about "Office Hours (for next 30 mins)" started at 16:38.

Vote results

Action items, by person

  • blackboxsw
  • blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns

Done items

  • (none)

People present (lines said)

  • blackboxsw (83)
  • smoser (54)
  • ajorg (38)
  • robjo (32)
  • ubot5 (7)
  • meetingology (4)
  • powersj (2)
  • dpb1 (2)
  • rharper (2)

Full Log

16:05 <blackboxsw> #startmeeting Cloud-init 'bi-weekly' status meeting

16:05 <meetingology> Meeting started Mon Dec 11 16:05:16 2017 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:05 <meetingology>

16:05 <meetingology> Available commands: action commands idea info link nick

16:05 <smoser> thanks for hosting blackboxsw

16:05 <blackboxsw> no problemo.

16:05 <blackboxsw> happy holidays folks and thanks for joining.

16:07 <blackboxsw> #topic Recent Changes

16:07 <blackboxsw> As mentioned @ our 17.1 release, we're promising more frequent cloud-init releases.

16:08 <blackboxsw> smoser has mailed the list informing cloud-init interested parties that we are targeting a 17.2 release for Thursday this week

16:08 <blackboxsw> It's been a few weeks since we've hosted the meeting (I think we missed last meeting), so I'll post some of the development that has landed in trunk

16:08 <smoser> #link https://lists.launchpad.net/cloud-init/msg00114.html

16:09 <blackboxsw> * All integration tests now function with the nocloud-kvm backend

16:09 <blackboxsw> * Fix apport for cloud-name options (LP: #1722564)

16:09 <blackboxsw> * Improve warning message when templates aren't found (Robert Schweikert) (LP: #1730135)

16:09 <blackboxsw> * Perform null checks for enabled/disabled Red Hat repos (Dave Mulford)

16:09 <blackboxsw> * Fix openSUSE and SLES setup of /etc/hosts (Robert Schweikert) (LP: #1731022)

16:09 <blackboxsw> * Catch UrlError when #include'ing URLs (Andrew Jorgensen)

16:09 <ubot5> Launchpad bug 1722564 in Apport "apport question will not accept multi-character responses" [Undecided,Confirmed] https://launchpad.net/bugs/1722564

16:09 <ubot5> Launchpad bug 1730135 in openstack-dev-sandbox ""Too much rain in Sydney"" [Undecided,New] https://launchpad.net/bugs/1730135

16:09 <ubot5> Launchpad bug 1731022 in cloud-init "host template expansion does not work on SUSE distros" [High,Fix committed] https://launchpad.net/bugs/1731022

16:09 <smoser> ajorg replied with a request for https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/329657

16:09 <smoser> that fell on deaf ears

16:09 <blackboxsw> * Released stable release update (SRU) of 17.1-27-geb292c18 (LP: #1721847)

16:09 <blackboxsw> * Cleanup dhclient background process after EC2 network discovery.

16:09 <blackboxsw> * ntp: fix configuration template rendering for openSUSE and SLES (Robert Schweikert) LP: #1726572

16:09 <blackboxsw> * fix manually running cloud-init after upgrade (LP: #1732917)

16:09 <ubot5> Launchpad bug 1721847 in cloud-init (Ubuntu Artful) "sru cloud-init 2017-10-06 (17.1-18-gd4f70470-0ubuntu1) updated to (17.1-27-geb292c18)" [Medium,Fix released] https://launchpad.net/bugs/1721847

16:09 <ubot5> Launchpad bug 1726572 in cloud-init "ntp config handling inconsistent for SLES and openSUSE" [Medium,Fix committed] https://launchpad.net/bugs/1726572

16:09 <ubot5> Launchpad bug 1732917 in cloud-init "17.1 update breaks EC2 nodes" [High,Fix committed] https://launchpad.net/bugs/1732917

16:09 <ajorg> truth

16:09 <smoser> ajorg: i will review shortly

16:09 <blackboxsw> * Queued upstream for merge into Bionic

16:09 <blackboxsw> * Queued 17.1.46 SRU for Xenial, Zesty, and Artful

16:09 <blackboxsw> * Fix EC2 race on sandboxed dhclient's pidfile during tempdir teardown (LP: #1735331)

16:09 <blackboxsw> * Enable Bionic in Integration Tests

16:09 <blackboxsw> * Create LXD and KVM Integration Tests in Jenkins

16:09 <ubot5> Launchpad bug 1735331 in cloud-init "ec2: zesty tempfile sandbox dhclient.pid file can't be created" [High,Fix committed] https://launchpad.net/bugs/1735331

16:10 <blackboxsw> As of end of last week, we are trying to blitz the review queue and dust off anything that has been sitting too long

16:12 <blackboxsw> So a couple fixes went into Amazon's initial network setup, IPv6 support is live for Ubuntu series Xenial, Zesty, Artful and Bionic

16:13 <ajorg> cool

16:14 <blackboxsw> heh I blew that last topic. it should have been #topic Recent Changes.

16:14 <blackboxsw> anyway I'll fix it in the logs when I publish

16:15 <blackboxsw> As always , for historical docs from this meeting check this link

16:15 <blackboxsw> #link http://cloud-init.github.io

16:15 <blackboxsw> #topic In-progress Development

16:15 <blackboxsw> So we have an active queue that is pretty healthy still

16:15 <blackboxsw> #link http://bit.ly/ci-reviews

16:16 <blackboxsw> smoser: rharper are we still trying to get through that queue as best we can for 17.2 or when do we think the window closes there?

16:16 <smoser> i think we can spend some more time on queue today.

16:16 <smoser> but that is about it really.

16:16 <blackboxsw> yeah, want some settle 'bake' time before the 17.2 cut on Thursday

16:17 <blackboxsw> We saw a couple Azure branches come in late last week.... Are there any branches folks are really excited about landing this week (today tomorrow?)

16:18 <blackboxsw> I had hoped to get through a couple of Robert's as they don't seem very contentious.

16:19 <smoser> the reporter bit seems pretty reasonable

16:19 <smoser> other than its not actually used anywhere in the mp

16:19 <smoser> ie, its non-contentious to add a reporter, but adding code that is not used is of not a lot of use :)

16:19 <blackboxsw> true

16:20 <ajorg> which mp is being discussed?

16:20 <smoser> (https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989)

16:20 <blackboxsw> #link https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/334989

16:21 <ajorg> thanks

16:23 <blackboxsw> With the upcoming holidays I expect things will be pretty slow after mid-next week, so we won't likely be landing a lot before the first of the new year.

16:25 <robjo> If it's slow for you more time to review open merge proposals ;)

16:25 <blackboxsw> This week we are also trying to get an SRU into ubuntu xenial, zesty and artful for some VMware/OVF datasource fixes for ds-identify and for pre-cusomization marker files courtesty (smoser & maitriyee)

16:26 <blackboxsw> courtesy rather

16:26 <smoser> ajorg: you could ping matthew on https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897

16:27 <ajorg> yup

16:27 <blackboxsw> and I know powersj is working on EC2 integration test support for cloud-init

16:27 <powersj> yep!

16:27 <powersj> Hoping to have an initial MP up this week

16:27 <blackboxsw> it's gonna be excellent to automatically test these releases

16:28 <blackboxsw> powersj: rharper smoser anything else in progress?

16:28 <rharper> nothing here

16:28 <ajorg> oh very nice.

16:28 <dpb1> powersj: \o/

16:28 <smoser> just the things that are in teh review queue. i put up one this morning for tmp file leakage

16:28 <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/335034

16:29 <smoser> i think the yeazelm mp probably is just missing somethign simple bug haven't spent any time on it.

16:29 <blackboxsw> ahh also we landed the initial /run/cloud-init/instance-data.json which we had talked about with larsks. It captures all metadata and userdata and some standardized properties which could help people script instance data.

16:29 <smoser> yeah, that is neat.

16:30 <blackboxsw> Yeah, we have yet to write up docs on using it (and we have an inprogress branch to allow using jinja templates in #cloud-config modules). But I don't expect this to land by 17.2

16:31 <ajorg> did we? that's great!

16:32 <blackboxsw> yeah only basic standardized properties currently. + 'local-hostname': self.get_hostname(),

16:32 <blackboxsw> 592 + 'instance-id': self.get_instance_id(),

16:32 <blackboxsw> 593 + 'cloud-name': self.cloud_name,

16:32 <blackboxsw> 594 + 'region': self.region,

16:32 <blackboxsw> 595 + 'availability-zone': self.availability_zone}}

16:32 <robjo> is that basically a re-implementation of python-ec2metadata? https://github.com/SUSE/Enceladus/tree/master/ec2utils/ec2metadata

16:32 <blackboxsw> but it's a first pass. We expect to add more

16:33 <blackboxsw> robjo: kindof, though generalized for all datasources

16:34 <rharper> robjo: long term, it's expected to be more than just ec2; rather a common baseline of instance metadata independent the actual cloud, but , IIRC, having a cloud-specific area (or at least access to the raw data)

16:34 <blackboxsw> it leaves a json-foramatted file containing any vendor data and user-data plus generalized/standardized fields extracted from that content which can be expected on all clouds

16:35 <robjo> problem with "all" clouds is that Azure is very different

16:35 <blackboxsw> since each datasource has that data already, it's essentially just formating it in a consumable file that others could levereage

16:35 <robjo> although one can argue that a "name" is an id it still looks weird when 'instance-id' and is a name

16:36 <blackboxsw> agreed robjo, some datasources may not provide different/less content.

16:36 <smoser> ajorg: http://paste.ubuntu.com/26164503/

16:36 <ajorg> Is that one of the Azure differences? name vs. instance-id?

16:36 <smoser> thats a demo of instance-data.json

16:36 <robjo> yes, azure has names no numbers

16:37 <robjo> it's more about "user expectations" as a "name" is an "identifier"

16:37 <blackboxsw> I'll check that azure run now. I think I linked it to the branch originally

16:37 <blackboxsw> ok I think it's probably a good to transition to open office hours now for the next 30 mins

16:38 <smoser> what is 'name' versus 'instance-id' comment ?

16:38 <smoser> ajorg: ?

16:38 <blackboxsw> #topic Office Hours (for next 30 mins)

16:38 <blackboxsw> feel free to continue the discussion now

16:38 <robjo> I'd just caution of making the assumption that we can stick the information from that data sources straight into another format and then call it "generic instance information"

16:38 <ajorg> smoser: re robjo's comment about "Azure is very different"

16:39 <smoser> oh. yes. ok.

16:39 <blackboxsw> azure instance-data

16:39 <blackboxsw> #link http://pastebin.ubuntu.com/26075842/

16:39 <dpb1> smoser: standup

16:39 <smoser> yeah, they do have a 'id'

16:40 <blackboxsw> #link http://paste.ubuntu.com/26164503/

16:40 <ajorg> from DMI?

16:40 <robjo> Which is useless in any any command

16:40 <smoser> from the cd i think.

16:41 <ajorg> robjo: ah, so the ID is unique (is it?) but can't be used to call any Azure APIs?

16:41 <robjo> in EC2 the instance-id is useful to me if I want to run "aws" commands, but the instance ID shown in the pastebin is useless for any "az" command

16:41 <robjo> ajorg: correct

16:41 <robjo> in the "az" tools everything is a name

16:42 <robjo> and thus to make the data cloud-init produces useful the -id should be the name of the VM

16:42 <robjo> then I can parse that information and use it if I need to deal with the API

16:43 <smoser> hm.

16:43 <robjo> but providing that ID as its is basically just sticking information into the json to "fill a field" which is somewhat counter to the point I'd say

16:43 <ajorg> robjo: there's a uniqueness constraint on the name too? but per-account or at-a-time or what?

16:43 <smoser> i odnt knwo. although it is insteresting thought.

16:43 <smoser> the issue is 'instance-id' is supposed to be an instance id

16:43 <smoser> not a user provided name that can be provided mutliple times in a row.

16:44 <smoser> i realize name is per-group unique, but if i

16:44 <smoser> a.) launch

16:44 <smoser> a.) launch 'foobar'

16:44 <robjo> There is a uniqueness constraint in that one cannot run a VM with the same "name" in the same resource group

16:44 <ajorg> so the question is if that ID provides global uniqueness, or if it provides a reference to the instance to be used via APIs

16:44 <smoser> b.) create capture

16:44 <smoser> c.) delete foobar

16:44 <smoser> d.) launch foobar

16:44 <smoser> then 'd' wont look new

16:45 <robjo> yes, it will it just takes a long time in Azure until the backend reaches "eventual" consistency and knows "foorbar" has been deleted previously

16:45 <ajorg> It seems clear enough that cloud-init is looking for a unique ID

16:45 <ajorg> But a user might want either, and probably an ID for API use.

16:47 <robjo> Well if we provide a format of the data that is exposed to the user via documentation and expected to be used by the user than at that point, IMHO, user needs have higher priority than what cloud-init is looking for

16:47 <ajorg> To decide which APIs to use, a script has to first look at which cloud it's on, so it has a chance to decide which value to use.

16:47 <robjo> that cloud-init uses the id to make decisions about "pre-once", "per-always" is a different topic

16:49 <robjo> Well that then kind of defeats the "generic instance information" claim, IMHO

16:49 <robjo> you are basically saying 1.) look for the framework and then decide if on that framework the "generic instance information" is useful or not

16:50 <robjo> 2.) If you happen to be on a platform where the "generic instance information" is not useful, go and collect your own

16:50 <robjo> From a user perspective that is not very nice, IMHO

16:51 <ajorg> oh, I was presuming we'd also include the Azure name, not that we'd include only a useless instance-id in that case.

16:51 <ajorg> clouds that don't have a name, wouldn't include a value for it.

16:52 <robjo> the pastebin only has the ID

16:52 <ajorg> right, I'm saying we should add the name

16:53 <robjo> This is why I am pointing out that "generic instance information" is not necessarily so easy to come by

16:54 <blackboxsw> robjo: ultimately, I'd like the generalized content surfaced in instance-data.json to be something that external user's could get value from and script against. This first pass was a stripped down approach to some of that content.

16:54 <smoser> we could add 'name' and have it be none yes.

16:55 <smoser> the not-yet-written doc will state that consumers should not be confused by new field names.

16:55 <blackboxsw> There are some fixes that need to be proposed to all datasources to better standardize on things like public vs private addresses, external hostnames etc. Those I expect will come in subsequent passes.

16:55 <robjo> it might be worth considering the concept of "equivalent instance information" where the entries in the json files get names/keys that are generic across all cloud frameworks and provide the euivalent information/usefulness to the user

16:55 <smoser> but inside the 'v1', then content of a key will not change.

16:55 <ajorg> robjo: that's a fair point, imho

16:55 <smoser> but 'instance-id' is in fact 'instance-id'. not 'name'.

16:55 <blackboxsw> robjo: I think that is the intent of those 'v1' standardized fields.

16:56 <smoser> note that lxd shares the same generic problem in this regard as azure. it uses user-provided name for instance-id. but does not provide an actual instance id of any sort.

16:56 <blackboxsw> right per name/instance-id discussion, they feel separate, and I think there is value in adding a separate 'name' as smoser mentioned

16:58 <robjo> Lets look at it from an API perspective, if I were to use the .json file wouldn't it be nice if I could just say json.load().get{'instance_api_id')

16:58 <robjo> for EC2 that returns the instance-id, for Azure it gives me the name

16:59 <robjo> part of the idea of cloud-init is to keep the ugly details of the cloud framework away from the user

16:59 <ajorg> if we were talking about the value of "region" we'd certainly want to yield the value that's useful for API calls.

17:00 <robjo> so why would the .json data then retrieve from that idea and make the user know if I am in EC2 I need to use instance-id and if I am in Azure I need to use instance-name?

17:01 <robjo> ajorg: agreed

17:01 <smoser> it seems somewhat non-sense that azure gives an instance a unique id, but cannot take that in as an identifier to the instance.

17:02 <robjo> AWS, GCE, and Azure all have the concept of "region" , not certain how IBM is handling that part in their setup but that may not be of interest to us at this point

17:02 <smoser> your point is good though. but instance-id i really think needs to be a unique identifier (as much as possible) for this instance

17:02 <ajorg> It sounded like smoser's 'v1' comment was meant to imply we could have a 'v2' that yields data differently than 'v1'.

17:02 <smoser> softlayer has "datacenters"

17:03 <robjo> smoser: I agree, but that's the way it is

17:03 <smoser> at some point, ajorg we will of course realize that we're all idiots

17:03 <smoser> and wonder What were we thinking!

17:03 <smoser> and have a 'v2'

17:04 <blackboxsw> `<-- it takes some of us longer than others to realize that

17:05 <ajorg> smoser: you're not convinced that today is that day?

17:05 <smoser> i try to keep acknowledgement of that fact to be more than a few days later

17:06 <ajorg> good to let it sink in first :-)

17:06 <smoser> (compared to when i notice it, to allow for additional occurences)

17:06 <ajorg> I'm not going to say it has to be changed, but I do think at the very least the azure name should be available.

17:07 <blackboxsw> I think this discussion definitely sheds light on the fact that we should continue to bring these standardized instance-data discussions to this meeting for a quick feedback loop from you guys as it evolves :)

17:07 <ajorg> :)

17:08 <blackboxsw> #action blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns

17:08 * meetingology blackboxsw bring up any updates in instance-data.json fields for discussion about common use-cases/patterns

17:08 <ajorg> and it doesn't seem harmful to have the name only if the cloud provides one, just as if the cloud doesn't have a concept of an availability zone we'll skip that too.

17:09 <blackboxsw> +1

17:10 <blackboxsw> well i think this about wraps up our meeting for today

17:10 <blackboxsw> any other topics for today?

17:11 <ajorg> I pinged Matt Yeazel, but he didn't respond yet.

17:11 <smoser> i think 'api-id' would lmake sense as a name.

17:11 <ajorg> so nothing more from my end

17:11 <ajorg> smoser: or 'api-instance-id'

17:12 <smoser> that just seems confusing.

17:12 <smoser> hm..

17:12 <smoser> i see why you want the 'instance' portion there, but the thing i dont like is that implies that this is 'per instance'

17:12 <ajorg> well, it's an API instance identifier.

17:13 <smoser> which in fact it is not.

17:13 <smoser> hm.

17:13 <ajorg> Ah, okay, that's true, but if the cloud doesn't have a unique way to identify the instance to the API...

17:13 <smoser> yeah

17:14 <ajorg> someone should check that assumption... how do you refer to terminated instances? or how are they identified in logs?

17:16 <ajorg> smoser: I just worry that someone's going to say "but in my API an API ID is this other thing"

17:17 <blackboxsw> yeah before surfacing something like that we'd need to vet it

17:17 <ajorg> In general I think there are enough differences between clouds that it's probably a losing battle to try to come up with something that's one-size-fits-all.

17:17 <ajorg> The goal was to make the information available more readily than by calling out to metadata services, right?

17:18 <ajorg> It's much harder to implement meta-data pulling for every cloud than to implement some logic that pulls the right value out of a JSON object, so it's still a big improvement even if it can't provide a unified view.

17:21 <ajorg> anywho, I should go do other things.

17:21 <blackboxsw> ajorg: yes that is the primary goal: more easily access cloud-provided metadata

17:21 <blackboxsw> if there is low-hanging fruit we can standardize I'm +1 on the concept

17:22 <blackboxsw> that's where the standard 'v1' key came from

17:22 <blackboxsw> but yeah I also don't think cloud-init needs to boil the ocean and standardize all fields

17:22 <blackboxsw> we'll capture what low-hanging fruit we can

17:22 <blackboxsw> and it'll take time

17:22 <blackboxsw> ok. Thanks for the great discusssions/suggestions ajorg and robjo. keep 'em coming

17:22 <blackboxsw> think I'll end meeting now

17:22 <blackboxsw> until next time...

17:22 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)