| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-20 | |||
| 10:33:34 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037 | |
| 10:33:54 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | |
| 10:58:28 | johnthetubaguy | lyarwood: I am not sure about stashing things for disconnect_volume | |
| 10:59:04 | johnthetubaguy | lyarwood: it seems like we could easily screw things up if we do that, if thats lots of load on Cinder, we can optimise that later. | |
| 10:59:12 | johnthetubaguy | lyarwood: what do you think? | |
| 10:59:51 | openstackgerrit | Roman Podoliaka proposed openstack/nova master: nova-status: don't coerce version numbers to floats for comparison https://review.openstack.org/447449 | |
| 11:00:00 | lyarwood | johnthetubaguy: well we currently do that right, so not stashing it would be something new and untested | |
| 11:00:18 | johnthetubaguy | lyarwood: its not currently something cinder doesn't have though | |
| 11:00:32 | lyarwood | johnthetubaguy: I'm just worried about the backend c-vol config changing and getting back connection_info for the disconnect that results on os-bick failing | |
| 11:00:32 | johnthetubaguy | oops, I mean its something cinder couldn't give us right now | |
| 11:00:47 | johnthetubaguy | lyarwood: honestly, I am worrying about the reverse | |
| 11:01:04 | johnthetubaguy | lyarwood: stuff gets updated, and we don't find out, and disconnect fails | |
| 11:01:33 | lyarwood | johnthetubaguy: yeah that's something that could happen today and tbh I can't think of an example | |
| 11:01:48 | lyarwood | johnthetubaguy: where we've seen it in the wild that is | |
| 11:02:26 | johnthetubaguy | lyarwood: it just feels like we should keep the API clean here, and we loop back and fix up things after that | |
| 11:03:04 | lyarwood | johnthetubaguy: yeah indeed, we are not dropping connection_info from the db / object models right? | |
| 11:03:13 | lyarwood | johnthetubaguy: so we can always start stashing this again I guess | |
| 11:03:33 | johnthetubaguy | lyarwood: well, we can't drop that will all existing volumes are transitioned, which is a big TODO right now | |
| 11:04:44 | johnthetubaguy | lyarwood: if we start saving that, I just feel we need an open conversation with os-brick and cinder folks on how it should fit in the flow, maybe immutable stuff on their side per attachment is a better way forward | |
| 11:05:47 | lyarwood | johnthetubaguy: yeah that's a fair point, they can and maybe should look after this for us | |
| 11:05:51 | johnthetubaguy | lyarwood: this probably speaks to that improved swap volume work actually, this is a zero copy swap volume use case | |
| 11:06:05 | johnthetubaguy | (ish) | |
| 11:09:45 | lyarwood | johnthetubaguy: unless the instance is down I can't think of a situation where nova would leave swap volume up to cinder entirely, IMHO there would always be a need for nova to rebase and ensure everything is in sync. | |
| 11:15:43 | openstackgerrit | Pooja Jadhav proposed openstack/nova master: Deny volume swapping with non-bootable volume https://review.openstack.org/407346 | |
| 11:22:32 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add spec to use cinder's new attachment API https://review.openstack.org/373203 | |
| 11:24:30 | johnthetubaguy | lyarwood: I was thinking about cinder wanting to rotate a connection password, +1 it always has to involve Nova | |
| 11:35:20 | mhenkel | sean-k-mooney: would you have some time helping me to figure out why this failed: http://logs.openstack.org/83/441183/10/check/gate-tempest-dsvm-py35-ubuntu-xenial/e42be9f/ ? | |
| 11:35:54 | mhenkel | I walked through all logs but couldn't spot any error related to the commit | |
| 11:40:35 | openstackgerrit | Roman Podoliaka proposed openstack/nova master: neutron: re-raise 401 for a non-admin client https://review.openstack.org/422696 | |
| 12:21:10 | Jeffrey4l_ | is there any consumer for nova versioned notifications? https://docs.openstack.org/developer/nova/notifications.html#versioned-notifications | |
| 12:21:32 | Jeffrey4l_ | seems ceilometer can not handle this. should i disable this when deploying? | |
| 12:26:13 | openstackgerrit | Sean Dague proposed openstack/nova master: remove log translation from nova.api.metadata https://review.openstack.org/447466 | |
| 12:53:25 | Chason | Hi guys, since openstack server create does not support the options --swap and --ephemeral, are there some other options alternatives? | |
| 13:27:31 | efried | Chason You specify those things via the flavor | |
| 13:29:19 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/446674 | |
| 13:29:35 | efried | Chason Take a look at `openstack flavor create` | |
| 13:31:22 | Chason | efried, you mean I should create flavor with these options, then to create the instance? | |
| 13:31:31 | efried | Chason Correct. | |
| 13:31:45 | Chason | OK, thanks! | |
| 13:31:47 | efried | The --flavor argument to `openstack server create` is mandatory anyway, I believe. | |
| 13:31:53 | efried | YW, good luck. | |
| 13:31:54 | johnthetubaguy | Chason efried is correct its normally set by the flavor, but do take a look at block_device_mapping_v2 in here: https://developer.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server | |
| 13:31:55 | Chason | Yeah, I see. | |
| 13:35:44 | openstackgerrit | Pushkar Umaranikar proposed openstack/nova master: Placement api: add tox check for webob wsgify decorator https://review.openstack.org/436671 | |
| 13:37:06 | openstackgerrit | Maciej Szankin proposed openstack/nova master: Stop using mox stubs in nova/tests/unit/cells/fakes.py https://review.openstack.org/342043 | |
| 13:42:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - add some functional tests for placement https://review.openstack.org/446123 | |
| 13:45:18 | edleafe- | Scheduler subteam meeting in 15 minutes in #openstack-meeting-alt | |
| 13:45:35 | openstackgerrit | Claudiu Belu proposed openstack/nova master: POC: tests: autospecs all the mock.patch usages https://review.openstack.org/447505 | |
| 13:47:53 | EmilienM | jaypipes: hey, good morning. We're currently investigating a TripleO CI issue where fail to deploy the overcloud (nodes managed by ironic) since March 16th, so we're looking at commits in Nova and we found https://review.openstack.org/#/c/441544/ that might be related. If you want to see bug report: https://bugs.launchpad.net/tripleo/+bug/1674236 - thanks for looking | |
| 13:47:53 | openstack | Launchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged] | |
| 13:50:18 | openstackgerrit | Claudiu Belu proposed openstack/nova master: POC: tests: autospecs all the mock.patch usages https://review.openstack.org/447505 | |
| 13:52:37 | jaypipes | EmilienM: looking... | |
| 13:52:58 | EmilienM | some logs: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-conductor.txt.gz#_2017-03-19_07_21_12_709 | |
| 13:53:43 | EmilienM | and http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-scheduler.txt.gz#_2017-03-19_07_19_34_158 | |
| 13:53:44 | sdague | jaypipes: I'm confused by your reviews on https://review.openstack.org/#/c/427901/ - as it's got a -1 with a change requested, then a +2 | |
| 13:54:32 | mriedem | sdague: because they talked about it in irc and didn't update the comments in the review | |
| 13:54:49 | jaypipes | sdague: yes, what mriedem said. | |
| 13:55:14 | mriedem | EmilienM: the compute logs are probably what is most interesting since that's where the resource tracker logs are | |
| 13:55:17 | mriedem | which feeds the scheduler | |
| 13:55:47 | dansmith | compute is feeding placement correctly | |
| 13:55:59 | dansmith | scheduler just never finds any compute nodes, AFAICT | |
| 13:56:10 | jroll | O_o | |
| 13:58:18 | jroll | oh, we don't take the old path if get_inventory() is implemented: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L754 | |
| 13:58:28 | mriedem | correct | |
| 13:58:32 | jroll | I bet we aren't registering the "capabilities" | |
| 13:58:33 | mriedem | i see this over and over in the compute logs | |
| 13:58:33 | mriedem | 2017-03-19 07:01:23.652 16870 ERROR nova.compute.manager [req-cf2991f8-9fe2-45db-95e2-8cfc8cee2edd - - - - -] No compute node record for host tripleo-centos-7-tripleo-test-cloud-rh1-7948745 | |
| 13:58:59 | dansmith | ah | |
| 13:59:12 | jroll | it wouldn't explain starting with 0 nodes, though | |
| 13:59:29 | dansmith | jroll: if we never create a computenode record though... | |
| 13:59:37 | dansmith | shouldn't this have broken our ironic job too? | |
| 13:59:51 | jroll | dansmith: right, I thought we did that either way | |
| 13:59:56 | mriedem | i thought the compute node record was created before this | |
| 14:00:05 | dansmith | ah yeah, you're right | |
| 14:00:15 | dansmith | L745 | |
| 14:00:18 | jroll | yeah | |
| 14:00:23 | dansmith | we've already got a CN there | |
| 14:00:50 | mriedem | yeah create it here if it doesn't exist https://github.com/openstack/nova/blob/66c3f0860b6d96500539b5e94856f9594d363da7/nova/compute/resource_tracker.py#L593 | |
| 14:00:51 | jaypipes | mriedem: it looks like it's taking approx 30 minutes for the Ironic nodes to register with Nova.. | |
| 14:01:02 | mriedem | update it here using get_inventory https://github.com/openstack/nova/blob/66c3f0860b6d96500539b5e94856f9594d363da7/nova/compute/resource_tracker.py#L642 | |
| 14:02:15 | EmilienM | CN = compute node? | |
| 14:02:19 | mriedem | yes | |
| 14:02:48 | edleafe | Scheduler meeting now running in #openstack-meeting-alt | |
| 14:03:16 | EmilienM | iiuc, you have one job that test Ironic: gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv - am I right? | |
| 14:03:33 | jroll | +1 to what jaypipes said, I don't see the resource tracker even try to run until 07:10, that's odd | |
| 14:03:47 | jroll | EmilienM: yes, and then the entire ironic gate. we'd know by now if it broke ironic :) | |
| 14:04:15 | EmilienM | right, and I've looked at commits in Ironic since March 16th and I haven't found anything suspicious | |
| 14:04:24 | mriedem | one of the compute node records is created here http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-03-19_07_10_44_256 | |
| 14:04:41 | jroll | EmilienM: I must ask, this is repeatable, right? | |
| 14:04:53 | EmilienM | jroll: yes, we have it for 3 days | |
| 14:05:00 | EmilienM | but nobody looked at it this weekend | |
| 14:05:02 | jroll | cool, the timing thing seemed weird | |
| 14:05:04 | EmilienM | (I was offline) | |
| 14:05:17 | jroll | I guess you're trying to boot an instance after it's created anyway | |
| 14:05:21 | jaypipes | mriedem: yes, and looking at the placement log, I see 3 PUTs for creating inventory successfully. | |
| 14:05:37 | jroll | EmilienM: what does your flavor look like? | |
| 14:05:44 | EmilienM | I can try a revert and Depends-On tripleo patch | |
| 14:05:50 | EmilienM | so I can tell if this patch broke us or not | |
| 14:06:01 | EmilienM | jroll: let me show you that | |
| 14:06:04 | EmilienM | jroll: gimme 2 min | |