| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-25 | |||
| 16:39:28 | mriedem | which is why cli users would probably do hypervisor-list and then hypervisor-show on specific ones | |
| 16:39:42 | cfriesen | mriedem: I'm envisioning something like "nova list" where you could specify which fields you wanted. | |
| 16:41:44 | openstackgerrit | melanie witt proposed openstack/nova master: Save updated libvirt domain XML after swapping volume https://review.openstack.org/498983 | |
| 16:47:07 | arxcruz | dansmith: hi, I'm having a problem with a temprest test test_volume_boot_pattern, failing with Block Device Mapping is Invalid: failed to get volume 800c3fc8-aa91-45ac-a27c-a15158582a3e | |
| 16:47:14 | arxcruz | the problem seems to be in nova | |
| 16:47:42 | dansmith | arxcruz: can you link to a CI run or something? | |
| 16:48:05 | arxcruz | dansmith: 2017-10-25 12:14:49.944 99098 INFO nova.api.openstack.wsgi [req-3b691589-73af-4fda-aee3-c5f962e4c8b7 - - - - -] HTTP exception thrown: Block Device Mapping is Invalid: failed to get volume 800c3fc8-aa91-45ac-a27c-a15158582a3e. | |
| 16:48:07 | arxcruz | this is from nova | |
| 16:48:11 | arxcruz | dansmith: let me check | |
| 16:49:04 | dansmith | arxcruz: that message likely means nova couldn't contact cinder or cinder complained | |
| 16:49:16 | arxcruz | dansmith: https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/undercloud/home/jenkins/tempest_output.log.txt.gz | |
| 16:49:22 | arxcruz | dansmith: there's nothing in cinder log | |
| 16:49:33 | arxcruz | dansmith: btw, this is tripleo | |
| 16:49:44 | arxcruz | dansmith: also, I have a env up on rdo-cloud if you want to take a look | |
| 16:51:43 | dansmith | arxcruz: okay, tripleo's log layout is hard for me to navigate, but I'm looking | |
| 16:51:56 | arxcruz | yeah... :/ | |
| 16:52:09 | dansmith | arxcruz: "nothing in cinder log" meaning no big stack traces or something, right? have you tracked the request nova is making to cinder to see that it's not showing up at all? | |
| 16:52:28 | arxcruz | dansmith: yes, I track down, there's no request in cinder | |
| 16:52:45 | dansmith | arxcruz: the timestamps of the failures in tempest are *after* the latest log message in nova-api | |
| 16:52:54 | arxcruz | it's pretty instantly the failure when I was debuging tempest, when it calls the create_server function, it returns pretty faster | |
| 16:52:56 | dansmith | arxcruz: which seems unpossible | |
| 16:53:19 | dansmith | arxcruz: also, the log you link to is for the undercloud.. is there cinder in the undercloud? | |
| 16:53:22 | arxcruz | dansmith: you're talking about the line I pasted ? | |
| 16:53:34 | arxcruz | dansmith: no, the cinder is in overcloud | |
| 16:53:36 | arxcruz | just a sec | |
| 16:54:00 | arxcruz | dansmith: https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/subnode-2/var/log/ | |
| 16:54:04 | arxcruz | logs for cinder and nova | |
| 16:54:57 | dansmith | arxcruz: is this 100% fail? | |
| 16:55:01 | arxcruz | dansmith: yes | |
| 16:55:11 | mriedem | arxcruz: i think i have a tempest patch for this already... | |
| 16:55:12 | mriedem | sec | |
| 16:55:54 | arxcruz | mriedem: cool :D | |
| 16:55:58 | mriedem | arxcruz: https://review.openstack.org/#/c/510951/ ? | |
| 16:56:02 | mriedem | that merged already though | |
| 16:56:22 | mriedem | you need to provide more details on where this actually fails, like trace the request as dan said | |
| 16:56:31 | mriedem | is it on attach, detach, other? | |
| 16:57:39 | mriedem | although that tempest patch wouldn't fix test_volume_boot_pattern which is a scenario test | |
| 16:59:28 | mriedem | looks like req-733a784c-6fed-48c5-a62b-3876edb56d57 is the request id | |
| 16:59:51 | sdague | mriedem: if it's being used to choose and understand, I sort of assumed it would be in GET /flavors | |
| 16:59:56 | sdague | but honestly, I can go either way | |
| 17:00:07 | sdague | maybe just ask the folks that wanted it | |
| 17:01:19 | arxcruz | mriedem: it's on test_volume_boot_pattern test, from my debug, it create the volume successfully, then when it tries to create the instance from the volume, it fails | |
| 17:01:46 | arxcruz | mriedem: here's the track from tempest https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-10-23_17_30_43 | |
| 17:03:15 | mriedem | arxcruz: why don't i see the POST /servers request id in the nova-api undercloud logs? https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/undercloud/var/log/nova/nova-api.log.txt.gz | |
| 17:03:20 | mriedem | or would it be in the overcloud? | |
| 17:03:54 | arxcruz | in overcloud | |
| 17:04:19 | arxcruz | mriedem: https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/subnode-2/var/log/nova/nova-api.log.txt.gz | |
| 17:04:23 | arxcruz | this is the overcloud nova log | |
| 17:05:00 | arxcruz | https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/subnode-2/var/log/nova/nova-api.log.txt.gz#_2017-10-23_17_30_30_020 | |
| 17:05:04 | arxcruz | the failure | |
| 17:05:40 | mriedem | huh, not helpful at all | |
| 17:06:56 | arxcruz | yeah, that's all that I can see... | |
| 17:07:04 | arxcruz | there's no call to cinder, just trow this error | |
| 17:07:33 | mriedem | in the dsvm ci logs we trace the requests to cinder via python-cinderclient | |
| 17:07:44 | mriedem | that would be helpful here in debugging failures in triple0 logs | |
| 17:10:42 | BrazierCustoms | hello. I am looking for a way to import an existing LXD container into glance for use as a base image. is this possible? | |
| 17:10:55 | BrazierCustoms | for nova-lxd | |
| 17:11:50 | mriedem | arxcruz: looks like it's getting the volume fine from the cinder-api side https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/0bb0995/subnode-2/var/log/cinder/cinder-api.log.txt.gz#_2017-10-23_17_30_30_474 | |
| 17:11:54 | mriedem | around the time of the nova-api failure | |
| 17:11:57 | BrazierCustoms | I have tried searching and I have tried a couple things I thought would have worked but unsuccessful. | |
| 17:12:04 | mriedem | BrazierCustoms: this wouldn't be the channel for nova-lxd | |
| 17:12:10 | mriedem | as it's not an in-tree driver | |
| 17:12:19 | BrazierCustoms | i couldnt fint that either :( | |
| 17:12:30 | mriedem | it's in the nova-lxd repo readme i think | |
| 17:12:39 | mriedem | https://github.com/openstack/nova-lxd#support-and-discussions | |
| 17:12:54 | BrazierCustoms | thank you mriedem | |
| 17:12:59 | mriedem | yw | |
| 17:13:25 | mriedem | arxcruz: looking at the nova code that returns that exception, unfortunately we return it for just about any kind of response from cinder | |
| 17:13:59 | arxcruz | mriedem: yeah, i tried to adventure in the wild nova code and notice that | |
| 17:14:13 | mriedem | arxcruz: are the computes running queens code? | |
| 17:14:20 | arxcruz | mriedem: yes | |
| 17:14:59 | mriedem | arxcruz: is this repeatable? | |
| 17:15:03 | mriedem | we could put up a debug patch | |
| 17:15:30 | mriedem | we're either failing on a 400 when reserving the volume, or a 404, but it doesn't seem like it should be a 404 | |
| 17:16:19 | mriedem | i don't see any POST action to reserve the volume in the cinder-api logs though | |
| 17:17:18 | arxcruz | mriedem: yes, it's repeatable, and I have an environment up | |
| 17:17:21 | arxcruz | if you want to take a look | |
| 17:17:29 | arxcruz | on rdo cloud | |
| 17:18:54 | mriedem | arxcruz: can't right now, but i'd just dump a LOG.exception() here before raising https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1354 | |
| 17:19:03 | mriedem | and run the recreate with that | |
| 17:19:54 | arxcruz | mriedem: sure, I'll do that, but not right now as well, but I'll update you guys | |
| 17:35:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: test: Store the OutputStreamCapture fixture https://review.openstack.org/515146 | |
| 17:35:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-status: Migrate to cliff https://review.openstack.org/515147 | |
| 17:35:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename 'policy_check' -> 'policy' https://review.openstack.org/515148 | |
| 17:35:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-policy: Migrate to cliff https://review.openstack.org/515149 | |
| 17:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163 | |
| 17:52:53 | openstackgerrit | Eric Fried proposed openstack/nova master: Process granular resources/traits in extra_specs https://review.openstack.org/515151 | |
| 17:53:15 | efried | jaypipes cdent ^ | |
| 17:53:30 | efried | Doesn't actually do anything with it yet; just parses it. | |
| 17:54:10 | cdent | efried: cool, will look, probably tomorrow morning | |
| 17:57:38 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse granular resources/traits from extra_specs https://review.openstack.org/515151 | |
| 17:57:57 | efried | That's a cool review number :) | |
| 17:58:41 | cdent | yes, yes it is | |
| 18:10:13 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] manage cache headers for allocations https://review.openstack.org/515156 | |
| 18:56:26 | mriedem | sdague: can you think of anything obvious where https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/fakes.py#L66 would ignore the version in the request object? | |
| 18:57:00 | mriedem | i'm having a hell of a time with making this work https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/compute/test_flavor_manage.py#L126 | |
| 18:57:28 | mriedem | even though i've passed version='2.54' to https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/compute/test_flavor_manage.py#L343 | |
| 18:58:29 | mriedem | i get a 400 back and when i dump the headers in the response, the version is always 2.1 | |
| 18:59:50 | openstackgerrit | Merged openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351 | |
| 18:59:56 | openstackgerrit | Merged openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354 | |