Cloud-init bi-weekly status

Posted on Mon 02 October 2017 in status-meeting-minutes • 10 min read

Meeting information

Meeting summary

Recent Changes / Highlights

The discussion about "Recent Changes / Highlights" started at 16:06.

In Progress Development / Highlights

The discussion about "In Progress Development / Highlights" started at 16:09.

Open Discussion / Office Hours

The discussion about "Open Discussion / Office Hours" started at 16:17.

Vote results

Done items

  • (none)

People present (lines said)

  • smoser (88)
  • ajorg (51)
  • rharper (28)
  • blackboxsw (25)
  • robjo (4)
  • ubot5 (3)
  • meetingology (3)

Full Log

16:05 <smoser> #startmeeting cloud-init

16:05 <meetingology> Meeting started Mon Oct 2 16:05:47 2017 UTC. The chair is smoser. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:05 <meetingology>

16:05 <meetingology> Available commands: action commands idea info link nick

16:06 <smoser> #topic Recent Changes / Highlights

16:06 <smoser> hm..

16:06 <smoser> 17.1 released https://lists.launchpad.net/cloud-init/msg00106.html

16:06 <smoser> thats the biggest thing and only thing i have for this topic

16:06 <blackboxsw> #link https://lists.launchpad.net/cloud-init/msg00106.html

16:06 <smoser> horay for a release, thanks to those who contributed.

16:07 <rharper> \o/

16:07 <smoser> 17.2 is set for 2017-12-14

16:08 <smoser> https://launchpad.net/cloud-init/+milestone/17.2

16:08 <ajorg> yay!

16:08 <smoser> feel free to target bugs to that release.

16:08 <rharper> #link https://launchpad.net/cloud-init/+milestone/17.2

16:08 <rharper> #info please target bugs to the next release

16:09 <smoser> does that do anyting ?

16:09 <rharper> in the meeting summary, it does

16:09 <blackboxsw> smoser: the links showed up in meeting minutes last time

16:09 <smoser> k. i always expected the bot to tell me that in a pm

16:09 <blackboxsw> meetingology didn't echo though

16:09 <rharper> well, smoser may have to do those

16:09 <smoser> #topic In Progress Development / Highlights

16:09 <blackboxsw> same. but something is up that needs attention.

16:10 <blackboxsw> I'll properly handle it when publishing

16:10 <rharper> cool

16:11 <smoser> Merge Proposals

16:11 <smoser> #link http://bit.ly/ci-reviews

16:11 <smoser> there are some there for sure. i know that robjo has some he's interested in, and i think ajorg's instance-identity deserves a look

16:12 <smoser> as well as simpletable for 17.2

16:12 <ajorg> I'd be grateful for both, yes.

16:12 <smoser> i am looking at the 'networkd' one which is a blocker as currently Ubuntu does not work properly on azure

16:12 <blackboxsw> we should have more bandwidth this week

16:12 <smoser> and all sysstemd-networkd systems will not work properly on CloudStack.

16:12 <smoser> anything else there?

16:13 <robjo> The task limit increase has been tested and solves our problem, thus merging would be great

16:13 <smoser> robjo, yeah. athat does seem unlikely to cause issue

16:13 <smoser> Trello Board

16:13 <smoser> # https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:13 <rharper> #link ?

16:13 <smoser> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:14 <robjo> https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+ref/taskLimit that's the branch for the task limit

16:15 <robjo> Sorry, I can never find the link for the open merge proposals :( problem exists between the keyboard and the chair

16:16 <blackboxsw> will get eyes on that today robjo as your addZypper is about wrapped

16:16 <smoser> Bugs

16:16 <smoser> #link https://bugs.launchpad.net/cloud-init

16:16 <smoser> just mostly pointers... above.

16:16 <smoser> for Ubuntu in 16.04, there is one SRU in progress that blackboxsw and i need to verify today

16:16 <smoser> and then we will be soon looking at doing an SRU of 17.1-ish

16:17 <robjo> Thanks, still in europe, my day is about done, will pick it up tomorrow

16:17 <smoser> robjo, link is in topic

16:17 <smoser> http://bit.ly/ci-reviews

16:17 <smoser> ok. so open discussion i guess. no w?

16:17 <smoser> #topic Open Discussion / Office Hours

16:17 <smoser> we'll hang around and pay attention to pings and discussion fro the next 30 minutes or so.

16:18 <smoser> and after that, feel free to ping.

16:19 <ajorg> I'm writing a unit test for a bugfix I'm preparing to submit. It works, but it's slow because readurl retries and waits a second each time

16:20 <ajorg> I'm a bit of a mock / patch newb. Anyone can help me see how to patch that argument?

16:20 <smoser> you can feel free to mock readurl

16:20 <smoser> and set its return_value and look at calls

16:21 <smoser> then you dont have to httppretty.

16:21 <ajorg> The only thing I want to change about it is the sleep though. Is there a straightforward way to just change one arg?

16:21 <blackboxsw> tests/unittests/test_handler/test_handler_chef.py is an example and tests/unittests/test_datasource/test_maas.py

16:21 <blackboxsw> I think

16:21 <ajorg> Hmm, I could do that, yeah. Would be sufficient for this.

16:22 <ajorg> I'd still like to know if there's an easy way to just change one argument to something that gets called somewhere else.

16:22 <smoser> you could also mock the time.sleep from url_helper

16:22 <ajorg> hahaha, yes. that's what I need to do.

16:22 <rharper> ajorg: if you want to mock the sleep you can decorate the test_ method with @mock.patch(time.sleep)

16:22 <smoser> we're suggesting other ways generally because there isn't :) at least that i know of.

16:23 <ajorg> that's cool, patching time.sleep will do nicely

16:24 <ajorg> is there an integration test that looks at what lands on the console?

16:24 <ajorg> (going back to my question from before the meeting)

16:25 <smoser> no. there coudl be on the nocloud-kvm backend

16:25 <smoser> but i think there is no console access currently on lxd

16:25 <smoser> and i think that we do not collect console access on nocloud-kvm

16:25 <smoser> but as you suggest we should for sure

16:25 <ajorg> k

16:26 <ajorg> I wondered if systemd might be swallowing my output, or maybe python (boothook in this case is a python script) is making a strange choice when it sets up logging.

16:27 <rharper> certainly possible; I know there were issues with cloud-init starting before say rsyslog on non-systemd boots

16:28 <rharper> and the python logging has changed w.r.t the default configuration; cloud-init main sort of expects this transition as it starts up in init-local and it has not yet read the cloud-config for logging configuration yet, so it reads that and then does some replay

16:28 <ajorg> interesting

16:29 <smoser> ajorg, you're writing to stdout/err with logging from a boothook ?

16:29 <ajorg> at least I would have thought cloud-init-output.log would contain my logs though, since it's more a redirect of stderr and stdout, right?

16:29 <ajorg> yeah

16:29 <rharper> correct

16:29 <rharper> this is from a bootcmd ?

16:29 <ajorg> no, #cloud-boothook

16:29 * rharper hasn't used boothook

16:29 <smoser> hm..

16:30 <ajorg> boothooks are super useful

16:32 <rharper> smoser: when do boot_hooks run ? local ? net ?

16:33 <rharper> I'm not sure yet where stdout/err for boot hooks occur, but you can dump each from the units via: journalctl -u cloud-init-local.service (or cloud-init.service)

16:33 <blackboxsw> "This is the earliest hook available. Note, that there is no mechanism provided for running only once"

16:34 <blackboxsw> I'm checking the code now (local/pre-local maybe)?

16:34 <rharper> yeah, saw that; it runs in-image scripts programs

16:34 <ajorg> I did that too, and don't see anything.

16:34 <ajorg> (journalctl)

16:34 <rharper> you should see some output

16:34 <rharper> but not seeing your hook in there ?

16:34 <ajorg> I can tell by other means that the hook ran

16:34 <ajorg> just don't see that it printed anything

16:35 <ajorg> my prime suspect if systemd can't do something bad here is that python makes some decision about not actually logging.

16:35 <ajorg> and I need to be more explicit that it should log

16:36 <rharper> so, the boot_hook will sub out each part, I see no capture on the stderr/stdout ; so I would expect those to go to whatever is currently capturing those;

16:37 <rharper> s/sub/subprocess

16:38 <ajorg> yup, that's what I expect too, and what I recall seeing on Amazon Linux

16:38 <rharper> since it's using cloud-init's util.subp, there should be a debug level message saying 'Running command %s with allowed return codes %s' which maches a path to the boothook

16:38 <rharper> I would expect to see that in the cloud-init.log

16:39 <smoser> http://paste.ubuntu.com/25661692/

16:39 <ajorg> yup, I do see that

16:39 <rharper> the output of script, I think should go to cloud-init-output.log

16:39 <smoser> i'm testing ^ on serverstack now.

16:39 <smoser> i verified the commands run on lxd but not look at console there.

16:39 <ajorg> thanks

16:40 <smoser> actually.. yeah, BOOTHOOK does run, but doesnt seem to have stdout tied to same place as bootcmd

16:41 <ajorg> oh?

16:41 <ajorg> I'm slightly surprised it's not just me

16:42 <smoser> http://paste.ubuntu.com/25661714/

16:42 <rharper> it really should be in the journal/console if stdout is not redirected to a file, cloud-init-local service

16:43 <smoser> that reproduces in lxc

16:44 <rharper> smoser: in your instance on serverstack, do you see your BOOTHOOK in journalctl -b -u cloud-init-local.service ?

16:44 <smoser> ajorg, please go ahead and open a bug.

16:44 <ajorg> cool (not cool :-P)

16:44 <ajorg> will do

16:44 <smoser> # journalctl -b -u cloud-init.service | grep BOOT

16:44 <smoser> Oct 02 16:37:41 a5 cloud-init[121]: BOOTCMD: Mon, 02 Oct 2017 16:37:41 +0000: a5

16:44 <smoser> ^ that is from lxd

16:45 <rharper> and what about the welcome message from cloud-init ?

16:45 <smoser> i think not.

16:45 <rharper> bootcmd does: util.subp(cmd, env=env, capture=False) ; where boot_hook does: util.subp([filepath], env=env)

16:46 <smoser> yeah, so boot_hook definitely swallowed. and should not be.

16:47 <rharper> yeah

16:49 <ajorg> https://bugs.launchpad.net/cloud-init/+bug/1720841

16:49 <ubot5> Ubuntu bug 1720841 in cloud-init "Output from boothook is not logged" [Undecided,New]

16:50 <ajorg> Meanwhile I need to setup more explicit logging from that boothook anyway, so I'm okay.

16:51 <blackboxsw> #link https://bugs.launchpad.net/cloud-init/+bug/1720841

16:51 <ubot5> Ubuntu bug 1720841 in cloud-init "Output from boothook is not logged" [Medium,Confirmed]

16:55 <ajorg> Here's another one, which I've confirmed with a unit test and proposed a patch: https://bugs.launchpad.net/cloud-init/+bug/1720844

16:55 <ubot5> Ubuntu bug 1720844 in cloud-init "UrlError from #include aborts stage" [Undecided,New]

16:55 <ajorg> (what is this #link thing you're doing there?)

16:55 <smoser> well, the bot is supposed to care and do somethign with it.

16:56 <smoser> but /me thinks it doesnt work right :)

16:56 <blackboxsw> ajorg: since meetingology is still active, the theory is it is keeping track of links during this meeting

16:56 <smoser> (during the meeting)

16:56 <ajorg> ah, okay

16:57 <ajorg> I think simpletable is completely ready to merge, btw. Any objections? https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/330525

16:57 <blackboxsw> The theory is it writes the meeting notes out for us so we can publish to https://cloud-init.github.io

16:57 <ajorg> #link https://code.launchpad.net/~ajorgens/cloud-init/+git/cloud-init/+merge/330525 ;-)

16:57 <blackboxsw> geg

16:57 <blackboxsw> heh

16:58 <blackboxsw> ajorg: I think we were good on that changeset, and we only wanted to wait post 17.1 cut to avoid potential regression

16:59 <ajorg> great

16:59 <blackboxsw> I'm +1 on that will give it a spin today and then I think we can land it

17:01 <smoser> blackboxsw, ajorg my thoughts on the simpletable...

17:01 <ajorg> here it comes...

17:01 <ajorg> :-P

17:01 <blackboxsw> :)

17:02 <smoser> i would like to have a machine friendly output available

17:02 <smoser> human friendly is good, but machine friendly solves the actual goal of writing the stuff.

17:02 <ajorg> So I modified it to display exactly as prettytable did

17:02 <smoser> oh really.

17:02 <smoser> wow

17:02 <ajorg> which is admittedly not machine friendly

17:02 <ajorg> not especially

17:02 <smoser> then i guess i can't object at all.

17:02 <ajorg> but it's at least as good as prettytable

17:03 <smoser> in that its backwards compat

17:03 <smoser> right.

17:03 <ajorg> ^ this was my goal, to get you to not object

17:03 <smoser> i still think we should probaly additionally write some machine friendly json

17:03 <ajorg> agreed, that would be better

17:03 <smoser> i tihnk having something human friendly is good though

17:03 <smoser> as i know I look at that output

17:04 <ajorg> it meets my goal of not depending on prettytable

17:04 <smoser> and parsing json would be less nice

17:04 <smoser> yeah

17:05 <smoser> so based on your assertion that it outputs the same as pretty table, i have no objections. only future hopes.

17:05 <smoser> and i do agree dropping pretty table is nice

17:05 <ajorg> winning

17:06 <blackboxsw> smoser: yeah I was wondering how we generally expect people/machines to parse cloud-init-output.log. Right now it's kindof hard to do machine parsing of cloud-init-output.log.

17:07 <smoser> blackboxsw, yeah. cloud-init-output.log is not intended to be machine friendly.

17:07 <blackboxsw> do we know already of non-human consumers of the formatting in cloud-init.output.log?

17:07 <smoser> but the console actually has value in having machien friendly things on it.

17:08 <smoser> my feeling is if you have access to /var/log/cloud-init.log, then you could very easily have written whatever you wanted to another file that was purely machine friendly.

17:08 <blackboxsw> was wondering where we would intend to dump machine-friendly json

17:08 <blackboxsw> yeah

17:08 <smoser> but the console (/dev/ttyS0) boot log is different

17:08 <smoser> in that it can give you a couple things you'd not find easy access to

17:09 <smoser> a.) ssh public keys for the system (providing out of band communication of this data)

17:09 <smoser> b.) network configuration info: providn useful bits of data on how you might get to this system

17:10 <smoser> systemd makes it "fun" to get that data to the console in a safe way

17:12 <ajorg> ugh yes. systemd like to stomp all over your consoles.

17:19 <blackboxsw> ok so did we want to iterate on simpletable to dump json, or just look for cloud-init to write supplementary json files under /run/cloud-init

17:20 <blackboxsw> .. after we land ajorg's branch

17:21 <ajorg> I think it's a good idea to dump json fragments to console for some things, but I suspect smoser will still not want to break anyone who might be using the tables, so probably best to land my branch as-is

17:23 <smoser> yeah, i agree with ajorg. for now, we can just take a replacement that drops prettytable

17:23 <smoser> there are improvments to be done there, but this is a simple win

17:29 <smoser> we'll go ahead and end meeting here.

17:29 <smoser> i'll still be around. thanks all!

17:29 <smoser> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)