Cloud-init bi-weekly status

Posted on Mon 22 January 2018 in status-meeting-minutes • 8 min read

Meeting information

Meeting summary

Recent changes

The discussion about "Recent changes" started at 16:10.

In-progress Development

The discussion about "In-progress Development" started at 16:24.

Office Hours (next 30 minutes)

The discussion about "Office Hours (next 30 minutes)" started at 16:45.

Vote results

Done items

  • (none)

People present (lines said)

  • blackboxsw (83)
  • smoser (23)
  • ajorg (16)
  • powersj (10)
  • ubot5 (6)
  • meetingology (3)
  • rharper (1)

Full Log

16:08 <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting

16:08 <meetingology> Meeting started Mon Jan 22 16:08:22 2018 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.

16:08 <meetingology>

16:08 <meetingology> Available commands: action commands idea info link nick

16:08 <blackboxsw> certianly ajorg :) (on office hours)

16:09 <blackboxsw> Welcome to another episode of cloud-init bi-weekly status. We'll chat about about cloud-init updates and in progress work, and we'l drop into office hours for ongoing discussions/bug work etc.

16:10 <blackboxsw> #topic Recent changes

16:11 <blackboxsw> Just walking through git-log for what we have committed in the last couple of weeks, here's the brief summary

16:12 <blackboxsw> thx smoser

16:12 <blackboxsw> - shorten the message in the exception per powersj feedback

16:12 <blackboxsw> - Use the same botocore session so the patched changes stick.

16:12 <blackboxsw> - fix bad use of %

16:12 <blackboxsw> - Fix console_log, improve comments and raise PlatformError on.

16:12 <blackboxsw> - tests: Fix EC2 Platform to return console output as bytes.

16:12 <blackboxsw> - tests: remove zesty as supported OS to test [Joshua Powers]

16:12 <blackboxsw> - Do not log warning on config files that represent None. (LP: #1742479)

16:12 <ubot5> Launchpad bug 1742479 in cloud-init (Ubuntu) "setting manual_cache_clean causes warning" [Medium,Fix released] https://launchpad.net/bugs/1742479

16:12 <blackboxsw> - tests: Use git hash pip dependency format for pylxd.

16:12 <blackboxsw> - tests: add integration requirements text file [Joshua Powers]

16:12 <blackboxsw> - MAAS: add check_instance_id based off oauth tokens. (LP: #1712680)

16:12 <blackboxsw> - tests: update apt sources list test [Joshua Powers]

16:12 <blackboxsw> - tests: clean up image properties [Joshua Powers]

16:12 <blackboxsw> - tests: rename test ssh keys to avoid appearance of leaking private keys.

16:12 <ubot5> Launchpad bug 1712680 in maas-images "cloud-init re-generates network config every reboot overwriting manual admin changes on CentOS." [Undecided,New] https://launchpad.net/bugs/1712680

16:12 <blackboxsw> [Joshua Powers]

16:12 <blackboxsw> - tests: Enable AWS EC2 Integration Testing [Joshua Powers]

16:12 <blackboxsw> - cli: cloud-init clean handles symlinks (LP: #1741093)

16:12 <ubot5> Launchpad bug 1741093 in cloud-init "cloud-init clean traceback on instance dir symlink" [Low,Fix committed] https://launchpad.net/bugs/1741093

16:13 <ajorg> What's being patched in botocore?

16:13 <blackboxsw> So a number of changes went into integration test related work, separating out requirements files.

16:14 <blackboxsw> MAASDatasource now also has smarted cache handling based on oauth token renewal from the maas server

16:14 <blackboxsw> so botocore is used by integration tests only as a mechanism to talk to the instance under test... looking back at the specifics here

16:14 <blackboxsw> it might have just been shuffling out how and where we define the dependency

16:14 <smoser> blackboxsw: (my 'paste' to you was bad... http://paste.ubuntu.com/26438113/ is better, showing only those on master, not my local branch that was currently checked out )

16:15 <blackboxsw> heh, oopsie daisy let's paste again inline then

16:15 <blackboxsw> - tests: remove zesty as supported OS to test [Joshua Powers]

16:15 <blackboxsw> - Do not log warning on config files that represent None. (LP: #1742479)

16:15 <blackboxsw> - tests: Use git hash pip dependency format for pylxd.

16:15 <blackboxsw> - tests: add integration requirements text file [Joshua Powers]

16:15 <blackboxsw> - MAAS: add check_instance_id based off oauth tokens. (LP: #1712680)

16:15 <blackboxsw> - tests: update apt sources list test [Joshua Powers]

16:15 <blackboxsw> - tests: clean up image properties [Joshua Powers]

16:15 <blackboxsw> - tests: rename test ssh keys to avoid appearance of leaking private keys.

16:15 <blackboxsw> [Joshua Powers]

16:15 <blackboxsw> - tests: Enable AWS EC2 Integration Testing [Joshua Powers]

16:15 <blackboxsw> - cli: cloud-init clean handles symlinks (LP: #1741093)

16:15 <ubot5> Launchpad bug 1742479 in cloud-init (Ubuntu) "setting manual_cache_clean causes warning" [Medium,Fix released] https://launchpad.net/bugs/1742479

16:15 <ubot5> Launchpad bug 1712680 in maas-images "cloud-init re-generates network config every reboot overwriting manual admin changes on CentOS." [Undecided,New] https://launchpad.net/bugs/1712680

16:15 <ubot5> Launchpad bug 1741093 in cloud-init "cloud-init clean traceback on instance dir symlink" [Low,Fix committed] https://launchpad.net/bugs/1741093

16:15 <blackboxsw> ok the real deal, that looks better

16:16 <blackboxsw> ahh ajorg that interim commit message on botocore was about integration tests caching the session information during testing so we don't recreate that session with every ssh connection to the instance

16:16 <blackboxsw> just a little time savings per review comments on powersj branch I believe

16:17 <ajorg> okay, so nothing that needs to get upstreamed to botocore?

16:17 <blackboxsw> I don't think so, powersj smoser I have vague recollection of someone filing an upstream botocore issue. did we have to do that for something else though?

16:18 <powersj> https://github.com/boto/botocore/issues/1351

16:18 <powersj> that was the issue smoser put in ^

16:18 <blackboxsw> nice recall powersj thanks.

16:18 <blackboxsw> #link https://github.com/boto/botocore/issues/1351

16:20 <smoser> ajorg: you can read that bug. imo they have a data loss error, but not one that they can easily fix without causing failures in places that previously ran fine.

16:20 <ajorg> I'll ask them to re-open it.

16:21 <ajorg> At the very least they should answer your last.

16:21 <smoser> thanks.

16:22 <blackboxsw> Generally anything significant that we have landed (and any inprogress work) should be available at the following link.

16:22 <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin

16:23 <blackboxsw> anything else we should note over the last couple weeks?

16:23 <blackboxsw> otherwise I'll switch to ongoing work topic

16:24 <blackboxsw> #topic In-progress Development

16:25 <blackboxsw> As you may have seen last week, we've gotten through a few passes and discussions around dojordan's branch to define pre-provisioning

16:25 <blackboxsw> #link https://code.launchpad.net/~dojordan/cloud-init/+git/cloud-init/+merge/334341

16:26 <blackboxsw> some of that discussion resulted in a new context manager: EphemeralDHCPv4 to support a sandboxed dhclient request on an instance.

16:27 <blackboxsw> this context manager affects Ec2 datasource a bit as it encapsulates all of the dhcp request ->` EphemeralIPV4Network calls that Ec2 was doing

16:28 <blackboxsw> there may be a couple other datasources that follow suit with this type of sandboxed dhcp request in weeks to come

16:28 <ajorg> glad it turned out to be generally useful rather than only specifically to ec2

16:28 <blackboxsw> absolutely

16:30 <blackboxsw> Some other in-progress bits look like we might try focusing a bit more on chrony support and gettting robjo's branches some more eyes.

16:31 <blackboxsw> and some work on Ubuntu snappy support per the snappy and snap config modules.

16:31 <smoser> dojordan: i just put one comment on your mp. /me thanks dojordan again for his patience.

16:32 <blackboxsw> rharper: smoser powersj anything more in the immediate pipeline that I'm missing/

16:32 <blackboxsw> ?

16:32 <smoser> blackboxsw: we should get the EphemeralDHCP thingy into the digital ocean datasource also.

16:32 <rharper> blackboxsw: a reply to the network discussion on the list from the azure folks and robjo

16:32 <ajorg> I took another look at https://code.launchpad.net/~yeazelm/cloud-init/+git/cloud-init/+merge/331897 and saw that origin/master seems to be failing some of the integration tests too.

16:32 <ajorg> (at least for me, locally, on a 16.04 instance)

16:32 <blackboxsw> ahhh right forgot about all your work there rharper, thanks!

16:33 <smoser> ajorg: https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci-nightly/

16:33 <smoser> that is nigytly run of trunk

16:33 <blackboxsw> #link https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci-nightly/

16:34 <ajorg> I'll try blackholing IMDS on my instance. Could be that's interfering with something.

16:35 <smoser> it is red, but 218 (green) and 219 (red) used the same git has on trunk (5cc0b19b8).

16:35 <ajorg> I'll follow up during office hours

16:36 <smoser> can you give me example of your failures ? we had "disk full" errors recently on our jenkins, so that might be the cause of the issue for 291.

16:36 <smoser> s/291/219/

16:36 <blackboxsw> I don't remember seeing that traceback recently. w/ warning messages present in cloud-init

16:36 <smoser> powersj: ? can you explain lxc timeout failure at

16:36 <smoser> https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-ci-nightly/219/consoleFull

16:37 <powersj> smoser: we discovered that our qemu-migration test was installing lxd from the archive and causing conflicts with the snap installed lxd

16:37 <powersj> I have a message to christian to prevent it, and I have already cleaned it up

16:37 <powersj> so new runs should pass

16:37 <ajorg> 2018-01-22 16:19:03,550 - tests.cloud_tests - WARNING - test case: modules/ssh_import_id failed TestSshImportId.test_no_stages_errors with: AssertionError: 1 != 0 : errors ['(\'ssh-import-id\', ProcessExecutionError("Unexpected error while running command.\nCommand: [\'sudo\', \'-Hu\', \'ubuntu\', \'ssh-import-id\', \'gh:powersj\', \'lp:smoser\']\nExit code: 1\nReason: -\nStdout: -\nStderr: -",))'] were encountered in stage m

16:38 <smoser> hm.. well, that will hit launchpad.net over https

16:38 <smoser> cloud-init-output.log probaly has more info (should be collected)

16:38 <powersj> the actual error is: File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-ci-nightly/tests/cloud_tests/platforms/instances.py", line 142, in _wait_for_system

16:38 <powersj> raise OSError('timeout: after {}s system not started'.format(time))

16:38 <powersj> it is because when the qemu tests installed lxd it didn't initialize lxd networking

16:38 <powersj> so no IP is received

16:39 <smoser> ajorg: would you have had outbound access to launchpad https ? if not, then that'd be expected failure.

16:40 <smoser> oh, and i guess 'gh:powersj' (github)

16:40 <ajorg> smoser: I'll check some things, but in short yes. Maybe lxc is being weird?

16:40 <smoser> i dont like our user names in that test though...

16:40 <powersj> smoser: we could use the bot instead

16:42 <ajorg> smoser: it's a public ec2 instance with no special outbound rules, and I can connect to public https sites from a normal session.

16:44 <blackboxsw> hrm, ok let's chat about what we can do to anonymize or drop that type of test data if we can

16:44 <blackboxsw> probably time to kick over to office hours

16:45 <blackboxsw> #topic Office Hours (next 30 minutes)

16:45 <smoser> powersj: well, i think i'd prefer some public key that we state "no one has the private key for this."

16:46 <smoser> obviously we could lie about that, but one would expect that you and I would gain access to the system using our public keys.

16:46 <smoser> it doens't make me feel a lot better that a bot could/can.

16:47 <ajorg> Is there a way to limit integration testing to a specific test?

16:47 <blackboxsw> Feel free to bring up any topic/bugs/branches/features you'd like discussion on. We can also continue our discussion on the ssh key imports in teting

16:47 <ajorg> (takes a long time to run the full suite)

16:48 <blackboxsw> ajorg: yes

16:48 <blackboxsw> (reverse-i-search)`cloud_t': python3 -m tests.cloud_tests run --os-name=artful --platform=nocloud-kvm --preserve-data --data-dir=../results --verbose -t modules/locale -t modules/set_password

16:48 <ajorg> thanks, that should help

16:48 <blackboxsw> ajorg: you can specify the test names (like modules/set_password) and modules/locale in this test

16:48 <blackboxsw> yeah those are short ones I frequently test with

16:49 <smoser> http://paste.ubuntu.com/26438334/

16:49 <smoser> that is what i use. and yeah... we've discussed that integration test could be easier to run :)

16:49 <blackboxsw> #link http://paste.ubuntu.com/26438334/

16:49 <blackboxsw> nice 1

16:51 <blackboxsw> smoser: to have a public key we know nobody has a private key for would that mean we'd need a separate github account (or maybe just an additional key associated w/ our bot account in gh

16:51 * blackboxsw checks github for authorizing multiple keys.

16:51 <blackboxsw> hrm, that wouldn't work as we need gh:ubuntu-server-bot (one key) n/m

16:52 <ajorg> I've got meetings most of today, so I'll have to follow up later. thanks everyone!

16:52 <blackboxsw> thanks ajorg

16:59 <blackboxsw> so, bot account for the time being is better than powersj owning the testing world ;)

16:59 <blackboxsw> but I'm not too concerned about it as this are supposed to be throw away instances

16:59 <blackboxsw> but I'm not too concerned about it as there instances under test are supposed to be throw away instances

17:00 <blackboxsw> these instances.... anyway

17:04 <smoser> blackboxsw: right. it would require users on both those services .

17:14 <blackboxsw> alrighty. think we're at the close of office hours. Last call?

17:16 <blackboxsw> Thanks for your time and contributions to cloud-init folks!

17:16 <blackboxsw> #endmeeting

Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/meetingology)