| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-02 | |||
| 13:58:35 | mriedem | so the live migration job is blowing up on the runs that use ceph | |
| 13:58:39 | mriedem | the config is started here: | |
| 13:58:40 | mriedem | http://logs.openstack.org/48/461448/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/a3f4426/console.html#_2017-05-01_22_14_21_133356 | |
| 14:00:45 | mriedem | i wonder if the change to systemd busted something with the ceph plugin | |
| 14:01:03 | mriedem | jbernard: have you noticed anything with the ceph job since yesterday? ^ | |
| 14:01:45 | efried | mriedem The systemd thing hasn't merged yet, has it? | |
| 14:02:01 | mriedem | https://review.openstack.org/#/c/460062/ | |
| 14:02:01 | mriedem | yes | |
| 14:02:03 | mriedem | yesterday | |
| 14:02:11 | sdague | the gate side did | |
| 14:02:23 | sdague | I think the live migration job is doing more cleverness on stop process | |
| 14:02:36 | sdague | which is causing challenges | |
| 14:02:41 | mriedem | sdague: how do i disable systemd as a test? | |
| 14:02:43 | mriedem | USE_SCREEN=True? | |
| 14:04:03 | sdague | mriedem: well, that's going to probably not work in the gate for other reasons | |
| 14:04:10 | sdague | https://github.com/openstack/nova/blob/d6d5c6be0cc0738bb7d67ca8391e5cc4f0a419f2/nova/tests/live_migration/hooks/ceph.sh#L70 | |
| 14:04:19 | sdague | yeh, this stuff is just really weird | |
| 14:05:58 | jbernard | mriedem: lemme check | |
| 14:06:33 | jbernard | mriedem: something merged somewhere last week that broke all of the things | |
| 14:06:52 | sdague | I did not realize how completely hacktastic this all is | |
| 14:07:02 | sdague | mriedem: is it only dying before the ceph reconfigure? | |
| 14:07:09 | sdague | or after I should say | |
| 14:07:12 | mriedem | sdague: after | |
| 14:07:21 | mriedem | non-shared local block live migration is passing | |
| 14:07:30 | sdague | yeh, so the reconfigure is never done, I can hack here for a few on it | |
| 14:07:56 | jbernard | let me know if/how i can help | |
| 14:16:04 | mriedem | sdague: i'm going to skip the ceph setup for now since we're blocked ^ | |
| 14:16:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Disable ceph tests in the live migration job https://review.openstack.org/461801 | |
| 14:18:54 | openstackgerrit | Sean Dague proposed openstack/nova master: Use systemctl to restart services https://review.openstack.org/461803 | |
| 14:19:13 | sdague | mriedem: I think that's the fix, or at least it's close to it | |
| 14:19:54 | jbernard | mriedem: so i need to update our plugin to support systemd, as a second step | |
| 14:20:19 | mriedem | jbernard: i reckon | |
| 14:20:30 | sdague | jbernard: what do you need to update? | |
| 14:20:37 | cdent | jaypipes, edleafe : my interpretation of jay's code is different from edleafe's interpretation. If Ed's is correct than his question of "why not do just the one query?" is a very good one. | |
| 14:20:38 | sdague | most things just worked | |
| 14:20:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use systemctl to restart services https://review.openstack.org/461803 | |
| 14:20:54 | jbernard | sdague: i suposed whatever is causing https://review.openstack.org/#/c/461801/ to be an issue | |
| 14:21:03 | sdague | the reason that this one doesn't is because it wasn't using any devstack functions for run_process / stop_process | |
| 14:21:07 | jaypipes | cdent: working on responding to your review comments now. | |
| 14:21:30 | cdent | jaypipes: cool thanks, you might wanna refresh if you're not seeing ed's too | |
| 14:21:33 | sdague | jbernard: right, that wasn't a plugin issue, it's just the live reconfigure invented it's own way of killing / starting processes | |
| 14:21:37 | sdague | that was no long valid | |
| 14:21:40 | jaypipes | nope, got ed's :) | |
| 14:21:48 | cdent | rad | |
| 14:22:27 | jbernard | sdague: ok, so what is needed to unbreak live migration? | |
| 14:23:14 | sdague | <sdague> mriedem: I think that's the fix, or at least it's close to it | |
| 14:23:14 | sdague | jbernard: I think https://review.openstack.org/461803 | |
| 14:24:13 | jbernard | nice | |
| 14:25:44 | openstackgerrit | Mathieu Gagné proposed openstack/nova master: Add ability to signal and perform online volume size change https://review.openstack.org/454322 | |
| 14:26:18 | cdent | jaypipes, edleafe: a re-read suggests my interpretation is wrong, so bigger "yeah, what about" to ed's question | |
| 14:26:39 | edleafe | cdent: you should know I'm always right :) | |
| 14:27:19 | cdent | edleafe: I'll hold you to that | |
| 14:29:26 | edleafe | cdent: I never make a mistake. I thought I did once, but I was mistaken. | |
| 14:30:12 | cdent | I'll make mistakes for you, shall I? | |
| 14:36:26 | zaneb | mriedem: o/ | |
| 14:36:48 | mriedem | zaneb: hi, so i read through all of https://governance.openstack.org/tc/resolutions/20170317-cloud-applications-mission.html again per your email on getting nova dev attendance at the session https://www.openstack.org/summit/boston-2017/summit-schedule/events/18951/cloud-aware-application-support | |
| 14:37:05 | openstackgerrit | Sean Dague proposed openstack/nova master: remove unused functions https://review.openstack.org/461813 | |
| 14:37:08 | mriedem | zaneb: and am i correct in saying that for nova it really just boils down to talking about this again? https://review.openstack.org/#/c/222293/ | |
| 14:37:44 | mriedem | b/c of the footnote that says "and provide a secure method to provision, rotate, and deliver credentials to the application." | |
| 14:37:49 | zaneb | mriedem: that's one thing that we'll want to discuss, but there are a couple of others that touch on Nova | |
| 14:39:02 | zaneb | mriedem: https://review.openstack.org/#/c/427872/ and https://review.openstack.org/#/c/438134/ are others | |
| 14:39:05 | mriedem | zaneb: so before we talk about that again, | |
| 14:39:13 | mriedem | this has come up several times in the past https://etherpad.openstack.org/p/newton-instance-users | |
| 14:39:19 | mriedem | ^ is the most recent that i remember, from austin, | |
| 14:39:37 | mriedem | but there is nothing in there about what was decided, or next steps | |
| 14:39:54 | jaypipes | edleafe, cdent: k, responded. | |
| 14:40:16 | cdent | thanks jay, looking. Also came up with yet another concern, which I'll inject while reading your repsonses | |
| 14:40:22 | jaypipes | kk | |
| 14:40:33 | mriedem | johnthetubaguy: do you know if you're going to be at the summit? | |
| 14:43:38 | johnthetubaguy | mriedem: I will be there | |
| 14:43:54 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Debug race in delete-while-booting test https://review.openstack.org/461513 | |
| 14:46:50 | mriedem | and we have lost zane | |
| 14:48:25 | jaypipes | mriedem: ? | |
| 14:49:15 | mriedem | jaypipes: see above | |
| 14:50:27 | mriedem | the john hopkins crew should be attending https://www.openstack.org/summit/boston-2017/summit-schedule/events/18951/cloud-aware-application-support | |
| 14:50:38 | mriedem | since it's all about rbac and auth and secrets | |
| 14:53:57 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Added PCI NUMA policies https://review.openstack.org/390520 | |
| 14:55:47 | openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP]Transform instance.exists notification https://review.openstack.org/403660 | |
| 14:57:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: re-Allow adding computes with no ComputeNodes to aggregates https://review.openstack.org/460616 | |
| 14:58:21 | cdent | jaypipes: responded, I hope I'm making a modicum of sense | |
| 15:00:05 | johnthetubaguy | mriedem: zaneb: my memory of the instance-users stuff was the idea to build a prototype that didn't need any Nova integration, so it was clear what bits would be needed in Nova to make the user experience (or security) better | |
| 15:00:34 | johnthetubaguy | mriedem: I don't 100% remember who promised to build the prototype, but I don't remember hearing about what happened | |
| 15:00:50 | zaneb | yeah, so the conclusion of the session was to hack it in as wsgi middleware | |
| 15:01:04 | zaneb | johnthetubaguy: it was kfox1111 | |
| 15:01:22 | johnthetubaguy | zaneb: that sounds right | |
| 15:01:45 | zaneb | IIUC he went away and decided that wouldn't work, because of (IIRC) key rotation | |
| 15:02:29 | johnthetubaguy | zaneb: but surely that means we need a prototype that supports key rotation? | |
| 15:02:30 | zaneb | then he did some further updates to the spec, but eventually momentum died | |
| 15:04:10 | openstackgerrit | Sergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 15:04:21 | zaneb | johnthetubaguy: you're not going to get that from wsgi middleware though, because that only gets to run once, when you create the server | |
| 15:05:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Replace reserved_host_disk_mb with reserved_host_disk_gb https://review.openstack.org/460661 | |
| 15:05:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Reword 'reserved_host_memory_mb' help text https://review.openstack.org/461054 | |
| 15:05:48 | sdague | zaneb: wsgi middleware can have code run on every request | |
| 15:07:36 | cdent | sdague: I would guess an issue with that is predictability of rotation, but for a prototype may not be a concern | |
| 15:07:38 | mriedem | what does the wsgi middleware need to do? fetch a token from keystone and make it available to the guest? | |
| 15:07:44 | dansmith | sdague: right but they're talking about jamming creds into the request to boot, so you'd never have a change to update those | |
| 15:07:52 | jaypipes | cdent: responded. | |
| 15:07:54 | zaneb | sdague: the middleware in question was to inject the keys into the metadata at boot time. there's no way to get that to update the metadata periodically even when there are no API requests. (is metadata even mutable?) | |
| 15:07:57 | cdent | thanks jaypipes | |
| 15:08:01 | jaypipes | cdent: and yes, you're totally making sense. | |