Cloud-init bi-weekly status

Posted on Mon 08 January 2018 in status-meeting-minutes • 11 min read

Meeting information

Meeting summary

Recent changes

The discussion about "Recent changes" started at 16:04.

In-progress Development

The discussion about "In-progress Development" started at 16:22.

Office Hours (next 30 minutes)

The discussion about "Office Hours (next 30 minutes)" started at 16:46.

Vote results

Done items

  • (none)

People present (lines said)

  • blackboxsw (99)
  • ajorg (45)
  • smoser (19)
  • rharper (9)
  • robjo (5)
  • powersj (4)
  • meetingology (3)
  • smoser1 (2)
  • ubot5 (1)

Full Log

16:03 <blackboxsw> #startmeeting Cloud-inin bi-weekly status meeting

16:03 <meetingology> Meeting started Mon Jan 8 16:03:53 2018 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:03 <meetingology>

16:03 <meetingology> Available commands: action commands idea info link nick

16:04 <blackboxsw> Happy 2018 cloud-initers! Thanks ajorg for helping kick us off.

16:04 <blackboxsw> Welcome back from break hope the holidays were good for folks.

16:09 <blackboxsw> #topic Recent changes

16:04 <blackboxsw> It

16:05 <blackboxsw> It's been a while since we've held the meeting due to holidays and vacation time. So, not a ton of content to report for the last bit. Digging up those details now

16:06 <blackboxsw> Testing of 17.2 on EC2, Azure, and GCE and release to Ubuntu Bionic

16:06 <blackboxsw> Complete 17.1.46 SRU to Ubuntu Xenial, Zesty, and Artful

16:06 <blackboxsw> Fix documentation around 'init' mode for modules subcommand (LP: #1736600)

16:06 <blackboxsw> Tooling to merge community authored branches into master

16:06 <ubot5> Launchpad bug 1736600 in cloud-init "CLI: cloud-init modules -h documents unsupported --mode init" [Low,Fix committed] https://launchpad.net/bugs/1736600

16:07 <blackboxsw> So the canonical side of the team worked a bit on getting the latest SRU updates 17.1.46 into Xenial, Zesty and artful. The testing and verification of that release took a bit of time, but we are getting better(faster)

16:07 <blackboxsw> I think this last SRU only took us 2 weeks instead of 4 weeks. so that frees up more time on upstream reviews and increasing cloud-init's velocity

16:07 <ajorg> great

16:08 <blackboxsw> we also added team tools for streamlining community authored branches. so that we stop slowing folks down :/

16:08 <blackboxsw> then the only problem is the reviewer :)

16:10 <blackboxsw> Also 17.2 release was 'cut' prior to Christmas break, this opened master up for more changes to land. so we've pulled in good fixes for VMWare NoCloud and SLES

16:11 <blackboxsw> digging up the changests now.

16:11 <blackboxsw> Also, keep in touch with our active development and the "done" lane on trello. It's out bookkeeper for anything we are working and Done represents anything landed

16:11 <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:13 <blackboxsw> so high-level content that landed between 17.1.46 and 17.2:

16:14 <blackboxsw> * CLI added the clean and status subcommands

16:14 <blackboxsw> * Support for identifying OVF datasource provided by VMware

16:14 <blackboxsw> * NoCloudKVM tests now run in continuous integration

16:14 <blackboxsw> * Formalize DataSource get_data and related properties

16:14 <blackboxsw> * Remove prettytable dependency and introduce simpletable

16:14 <blackboxsw> * VMWare pre and post-customization script support

16:15 <blackboxsw> Thanks ajorg I think you were the author of note on simpletable stuff, it's nice to drop dependencies where we can to increase speed of cloud-init

16:15 <ajorg> it was done selfishly

16:15 <ajorg> we dislike taking on new dependencies :-)

16:17 <blackboxsw> and thanks to robjo(suse) maitree(vmware) too and dojordan and Ryan McCabe(redhat) for recent branches too

16:17 <blackboxsw> :)

16:18 <blackboxsw> Post our 17.2 release we've started work on improved integration..... I think we just got powersj's ec2 integration tests landed right johs?

16:18 <blackboxsw> josh even

16:18 <powersj> \o/ yep!

16:18 <ajorg> nice

16:19 <blackboxsw> sweet, so an extra security blanked for us when we have significant changesets landed in master to ensure ec2 is happy.

16:19 <blackboxsw> powersj: what are out plans for continuous integration frequency

16:19 <blackboxsw> with ec2 specifically

16:19 <ajorg> Can those integration tests be run by others with EC2 accounts?

16:19 <blackboxsw> ajorg: yes they can

16:19 <powersj> I am working on the jenkins jobs this week and hope to have a weekly run as well as a manual run for backport testing

16:19 <blackboxsw> I'll get the cmdline

16:19 <ajorg> thanks!

16:20 <blackboxsw> tox -e citests -m tests.cloud_tests run --os-name=artful --platform=ec2 --preserve-data --data-dir=../results --verbose

16:20 <blackboxsw> or something like that

16:20 <ajorg> got it

16:20 <ajorg> thanks!

16:20 <blackboxsw> powersj: documented it too I think

16:20 <blackboxsw> getting link

16:20 <powersj> https://cloudinit.readthedocs.io/en/latest/topics/tests.html#ec2

16:20 <blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/tests.html#ec2

16:20 <blackboxsw> :)

16:21 <blackboxsw> excellent work Josh

16:21 <powersj> thanks for all the reviews :)

16:21 <blackboxsw> anything else I'm missing about landed work? rharper powersj smoser1 ?

16:22 <blackboxsw> otherwise next topic

16:22 <rharper> blackboxsw: nothing from me

16:22 <blackboxsw> #topic In-progress Development

16:23 <blackboxsw> So we've got a fairly healthy review queue that we need to get through as we get the year started....

16:24 <blackboxsw> we also have a few things we are in flight currently:

16:24 <blackboxsw> - continuous integration improvements per powersj

16:24 <blackboxsw> - dropping dependence on ifup ifdown utils where possible as that's not supported (or installed in some cases) in systemd world

16:24 <smoser1> blackboxsw: wow. sorry, missing.

16:25 <blackboxsw> who is that smoser1 guy anyway

16:25 <smoser1> yeah, i didnt see anything missing sorry.

16:25 <smoser> wonder how that happened.

16:25 <blackboxsw> welcome ;)

16:25 <blackboxsw> - netplan improvements per rharper and jinja template support for all cloud-config modules

16:26 <blackboxsw> - and softlayer support per smoser

16:27 <blackboxsw> know the Azure guys are also posting a couple branches on getting a pre-provisioning setup going for thier datasource which looks pretty exciting

16:27 <blackboxsw> I can't think of anything else off the top of my head.

16:28 <robjo> chrony support

16:28 <ajorg> we're only talking feature work in this topic?

16:29 <blackboxsw> any in progress development to highlight is fair game. bug work. refactoring, feature etc

16:29 <blackboxsw> +10 robjo and again thanks for working with us getting all those branches up and (hopefully soon) landed

16:29 <ajorg> what does "jinja template support for all cloud-config modules" mean?

16:31 <ajorg> I'd guess most modules don't need templating?

16:31 <blackboxsw> ajorg: two things. 1. since we have now landed /run/cloud-instance/instance-data.json to store metadata/userdata it'd be that #cloud-config can new be specified with ## template:jinja header and could leverage anything jinjia has to offer plus sourcing any of the instance-data.json metadata fields

16:33 <ajorg> Ah, right. Is that not being done above the module level?

16:33 <blackboxsw> so if people have repetitive or template-driven content in the runcmd or write_files portion or their #cloud-config they'd be able to leverage jinja templates etc

16:33 <smoser> ajorg: yes, above the module level.

16:33 <blackboxsw> ajorg: not anywhere in cloud-config currently

16:33 <blackboxsw> one sec I misunderstood the question

16:33 <blackboxsw> smoser: can you clarify what you mean?

16:33 <ajorg> I mean, shouldn't #cloud-config template expansion happen before the module sees the config?

16:34 <smoser> blackboxsw: we could/should also allow other part types to be rendered

16:34 <smoser> ttps://trello.com/c/xyqxyOxg

16:35 <smoser> er... bad url. in 2 ways

16:35 <ajorg> The the part handler would be the one to do that expansion.

16:35 <smoser> https://trello.com/c/AYaCdQyT

16:35 <blackboxsw> ahh ok, right that makes sense. I think the cut I made was limited in focus to cloud-config modules and custom scripts supporting the ## template:jinja header.. but nothing would preclude handling other parts

16:36 <blackboxsw> so the link to my WIP branch was

16:36 <blackboxsw> #link https://trello.com/c/xyqxyOxg

16:36 <blackboxsw> and the general feature per smoser

16:36 <blackboxsw> #link https://trello.com/c/AYaCdQyT/21-cloud-init-query-standardized-json-information

16:36 <ajorg> Is there a design doc of some kind of this?

16:37 <blackboxsw> not yet.. but we probably should have a spec as it'd be a good template for the docs we'll need to write

16:38 <blackboxsw> scott captured most of the use cases we'd be going for in that last trello link above

16:38 <ajorg> Small example of where some clarity is needed: if Jinja is interpreting {foo} in a user-script, what will it do when it sees a shell variable ${foo}

16:38 <ajorg> ?

16:39 <smoser> you declare that the content is a jinja template

16:39 <smoser> if you provide it something that is not renderable as a jinja template

16:39 <smoser> then it will fail

16:39 <smoser> it requires input to explicitly say "this is jinja". it does not just attempt to render anything.

16:39 <smoser> (unless explicitly told to)

16:39 <blackboxsw> some brief working examples are in the description of the branch @ https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/334030

16:40 <ajorg> Sure. But as a content author, I need to know if Jinja is going to try to render ${foo} or not.

16:40 <smoser> then as a content author you can read jinja docs :)

16:40 <blackboxsw> jinja would try to render {{ foo }}

16:40 <ajorg> :-

16:41 <smoser> ajorg: we'll document a simple case, and we can even document "for shell, you'll have to be aware that ...."

16:41 <smoser> but we're not going to document all of jinja

16:41 <ajorg> I see.

16:42 <ajorg> My understanding was that Jinja was highly customizable in what it interpreted and how, so that it's important to document how you've configured it to work.

16:42 <blackboxsw> and since to burden is on the #cloud-config or script writer to provide the header ## template: jinja\n#cloud-config\n they should understand what they are doing

16:42 <blackboxsw> we won't implicitly run the #cloud-config through jinja

16:43 <ajorg> I get that, no problem, what I'm saying is that Jinja is an engine that you configure to do something, not a markup that always does the same thing for everyone.

16:43 <ajorg> Am I making any sense?

16:44 <blackboxsw> understood (though I thought it was fairly constrained it it's application and functionality). We'll make sure that the mechanism by which jinja operates is well documented and confined as best we can... for our own sanity we don't want that template engine to be too flexible... too many tough support use cases

16:45 <blackboxsw> ok anything else for "In progress development" otherwise we can move to Office hours for 30 mins

16:46 <blackboxsw> #topic Office Hours (next 30 minutes)

16:47 <blackboxsw> robjo: you've got quite a few branches of goodness up for us to review. Any prioritization on those branches or just take them as we can?

16:47 <rharper> I don't think there are issues w.r.t jinja and shell; they use different variable escape methods, jinja uses {{ variable/expression }}; and it doesn't consume $ AFAIK, ajorg do you know differently ?

16:48 <blackboxsw> #link https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992

16:48 <blackboxsw> I'm guessing is top of the list

16:48 <ajorg> I saw {instance_id} at https://trello.com/c/AYaCdQyT/21-cloud-init-query-standardized-json-information so I assumed it was being customized to look for { instead of {{

16:48 <robjo> blackboxsw: The chrony support should probably be the last as it will take longer over all and more back and forth

16:48 <ajorg> (for one thing)

16:49 <ajorg> rharper: also, there's the whole question of the "extends" feature

16:50 <ajorg> We integrated Jinja into an internal tool a few years back and we spent a very long time making sure the loaders did the right thing.

16:50 <blackboxsw> ajorg: I thought I read somewhere that you couldn't exend jinja for custom functions. maybe I was mistaken

16:50 <robjo> I am also not certain that the "re-write everything" on the first go around for chrony is really what we want to do initially

16:50 <ajorg> blackboxsw: I don't think I'm referring to custom functions

16:51 <robjo> That's probably where we ant to end up, but I am not certain that a "step function" approach is in order

16:51 <rharper> ajorg: hrm, I've always seen {{ variable }} or {% expression %}; so maybe blackboxsw can just update the templates;

16:51 <rharper> the examples in the cards

16:52 <ajorg> rharper: sure, that would have helped in this case.

16:52 <robjo> If we do go down the route of the step function I'll need more gudance then in rharper's comments

16:52 <ajorg> blackboxsw: I was referring to the ability of one template to extend another.

16:53 <ajorg> blackboxsw: and the question of where does the engine look when it's asked to extend another template. It can be tricky.

16:54 <blackboxsw> yeah I honestly hadn't gotten past step one of handling the template markup within an existing single template. so this may need a bit of thought/work

16:55 <ajorg> Personally, I'd be a lot happier with limiting things to Python format() templates, even though it means you can't have loops, but I won't get in the way as long as we're cognizant of the problems we can run into by accepting the full power of an advanced engine like Jinja.

16:56 <smoser> i'm not opposed to allowing ## template: python-format

16:56 <ajorg> heh

16:56 <smoser> honestly.

16:56 <smoser> you can pick a differnt name if you dont like that one.

16:57 <smoser> but we already use jinja, so it makes sense to support jinja

16:57 * smoser has to run. sorry.

16:57 <rharper> I do feel that supplying the template means the user is opting in; and specifically if we've got a good way to provide dry-run based on a instance.json and a script; that certainly can help folks work out the kinks in the template of their choosing

16:57 <ajorg> I'm really not opposed so much as wary of the extensive power of the thing

16:58 <rharper> ajorg: that's a fair warning; given you've experience here; help drawing the line is most welcome

16:58 <ajorg> I'm trying to think of a way to read in /etc/shadow using Jinja, you know?

16:58 <rharper> well, cloud-init is root anyhow; so, what's the deal with that ?

16:59 <blackboxsw> ajorg: heh, right though you can read that with your runcmd section in #cloud-config :)

16:59 <ajorg> If I can come up with a way to do it that doesn't make it look obvious that I'm doing it, and then post that as something others can copy, or use with #include <url> then I win.

16:59 <rharper> I don't think jinja makes that any more troublesome

17:00 <rharper> folks already wget | bash with shell they don't understand either

17:00 <ajorg> I suspect Jijna makes it more opaque.

17:01 <ajorg> The answer to "what file does Jinja read when I use {% extends foo %}" is a very lengthy "it depends"

17:02 <ajorg> anyway, I've said my piece

17:03 * ajorg is a bit of a template naysayer.

17:05 <blackboxsw> +1, there's one in every group. We'll try to keep that in mind as this feature evolves

17:05 <blackboxsw> :)

17:06 <ajorg> nice

17:06 <ajorg> :-)

17:06 <blackboxsw> any pet bugs, new features or burning reviews that need mention?

17:07 <blackboxsw> ajorg: we could do something simple like disable the extends option via policies

17:07 <blackboxsw> it looks like

17:08 <blackboxsw> #link http://jinja.pocoo.org/docs/2.10/api/#policies

17:08 <blackboxsw> or maybe I'm misunderstanding the issue I'll read up more on it

17:08 <ajorg> thanks

17:09 <ajorg> It looked like https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/334074 was blocking https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897 but shouldn't be anymore.

17:09 <ajorg> I'll remind Matt to try it again now.

17:13 <blackboxsw> thanks good dela

17:13 <blackboxsw> dela

17:13 <blackboxsw> deal

17:13 <blackboxsw> geez

17:14 <blackboxsw> on that note. I think it's time for coffee

17:14 <blackboxsw> and time to end the meeting

17:14 <blackboxsw> Happy New Year again folks. Good to be back in the office.

17:15 <blackboxsw> thanks again for the chat, until next time..

17:15 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)