| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-01 | |||
| 14:47:01 | bauzas | mriedem: mmm, just saw https://bugs.launchpad.net/nova/+bug/1660878 | |
| 14:47:01 | openstack | Launchpad bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,Confirmed] | |
| 14:47:17 | mriedem | bauzas: yes it's in the ML too | |
| 14:47:21 | mriedem | i have a debug patch up for it | |
| 14:47:27 | mriedem | https://review.openstack.org/#/c/427544/ | |
| 14:48:23 | panda | mriedem: we think yes, but we have problem merging the change since puppet syntax gate is failing due to a new deprecation warning. | |
| 14:48:23 | bauzas | mriedem: not sure I saw this thread, looking | |
| 14:49:14 | mriedem | bauzas: it's the papercuts thread | |
| 14:49:35 | mriedem | bauzas: the thread doesn't matter, the bug has the details https://bugs.launchpad.net/nova/+bug/1660878 | |
| 14:49:35 | openstack | Launchpad bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,Confirmed] | |
| 14:49:40 | bauzas | oh ok | |
| 14:49:58 | bauzas | not [nova] tagged :( | |
| 14:50:04 | bauzas | I saw it, but passed it | |
| 14:50:09 | bauzas | s/passed/skipped | |
| 14:50:11 | sfinucan | bauzas: Just spotted your review on https://review.openstack.org/#/c/392643/. Thank you :) | |
| 14:50:19 | bauzas | I should hire an assistant | |
| 14:50:21 | mriedem | bauzas: melwitt: dansmith: looks like i got a hit on my debug patch http://logs.openstack.org/44/427544/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/f50a5ee/logs/screen-n-cond.txt#_2017-02-01_04_49_11_702 | |
| 14:50:46 | bauzas | like the one our French Presidency candidate is hiring | |
| 14:53:08 | jaypipes | "Harry Harrington" is officially the best-named new OpenStack core nominee. | |
| 14:54:53 | mriedem | dansmith: so i've got a recreate on that failure, we know the build request is gone before we create the instance in conductor, now i'm just not sure what i do with it - we don't put that in cell0 since if the build request is deleted before the instance is created, we just treat it like the instance was deleted, i.e. it wouldn't be in ERROR state in cell0 | |
| 14:55:11 | mriedem | i think i can just return | |
| 14:55:27 | dansmith | mriedem: ah I misunderstood your assertion last night then | |
| 14:55:43 | dansmith | mriedem: yeah, if build request is gone before we create the point is we're supposed to just bail quietly.. we do that in at least one place now | |
| 14:55:53 | mriedem | yup here https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@936 | |
| 14:55:58 | mriedem | which for this bug is too late | |
| 14:56:33 | dansmith | mriedem: ah, yep, indeed | |
| 14:56:57 | mriedem | dansmith: also, | |
| 14:57:02 | mriedem | unrelated, but this return should be a continue right? https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@972 | |
| 14:57:04 | mriedem | as we're in a for loop | |
| 14:57:58 | dansmith | mriedem: that's not in a for loop | |
| 14:58:13 | mriedem | sure it is https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@875 | |
| 14:58:29 | mriedem | it's like a 200LOC method so it's easy to miss | |
| 14:58:41 | dansmith | ooh, well, let me check | |
| 14:59:17 | mriedem | ok we can fix separately, i'll clean up my patch for the actual first fix | |
| 14:59:21 | dansmith | yeah, since we've got multiple build requests I guess so, if someone were to delete one of the multi-instances in the middle | |
| 14:59:22 | dansmith | aye | |
| 14:59:28 | mriedem | yar! | |
| 14:59:29 | mriedem | matey | |
| 15:01:22 | sfinucan | johnthetubaguy: One question on https://review.openstack.org/#/c/421760/10/api-ref/source/servers.inc | |
| 15:01:50 | mriedem | i'll even put on green grass and high tides since this should take 10 minutes | |
| 15:01:54 | dansmith | mriedem: so, I'm not sure why you think this delete-before-create thing is related to the message in your bug about rebuild | |
| 15:02:10 | sfinucan | In that file, the parameter is recorded as 'accessIPv4', but in the release note [1] it's recorded as 'access_ip_v4' [1] https://review.openstack.org/#/c/421760/10/releasenotes/notes/add-whitelist-for-server-list-filter-sort-params-2ae766d03ba895e5.yaml | |
| 15:02:24 | mriedem | dansmith: the test creates the server, then before it's active it deletes it, and then tries to rebuild it, expecting a 404, but gets a 409 | |
| 15:02:25 | sfinucan | I assume the former is automagically transformed to the latter | |
| 15:02:28 | sfinucan | ? | |
| 15:02:34 | mriedem | dansmith: it's a negative test, expecting the deleted instance to be deleted, | |
| 15:02:54 | mriedem | but when we hit the 409, the build request is gone but conductor created the instance (from the stale build request) | |
| 15:03:11 | dansmith | mriedem: ah, okay cool | |
| 15:03:17 | mriedem | dansmith: it's not just rebuild, there is reboot and some other negative tests that have the same pattern | |
| 15:03:23 | dansmith | ack | |
| 15:03:42 | mriedem | i'm just happy it's not more complicated than this :) | |
| 15:03:52 | dansmith | def totes | |
| 15:10:12 | bauzas | mriedem: I have now gained more b/w, which change should I bite now ? | |
| 15:11:08 | bauzas | looking at https://etherpad.openstack.org/p/nova-ocata-rc1-todos | |
| 15:12:51 | mriedem | bauzas: https://review.openstack.org/#/c/427367/ | |
| 15:13:51 | mriedem | bauzas: and https://review.openstack.org/#/c/421760/ | |
| 15:13:52 | bauzas | ack | |
| 15:14:00 | mriedem | and https://review.openstack.org/#/c/426810/ | |
| 15:14:09 | mriedem | basically, just read the etherpad and pick :) | |
| 15:15:31 | mriedem | sdague: you may want to chime in on the postgres job thread in the dev list | |
| 15:15:38 | mriedem | mordred: ^ | |
| 15:15:53 | mordred | uhoh | |
| 15:16:03 | mriedem | the support matrix that won't die :) | |
| 15:18:34 | vsaienk0 | nova team, since today morning/yesterday evening ironic multinode job is failing due to https://bugs.launchpad.net/nova/+bug/1661014 | |
| 15:18:34 | openstack | Launchpad bug 1661014 in OpenStack Compute (nova) "Multinode job fails with "Compute host X not found"" [Undecided,New] | |
| 15:18:43 | sdague | mriedem: sure | |
| 15:19:16 | vsaienk0 | is it known problem? | |
| 15:19:58 | mriedem | vsaienk0: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | |
| 15:20:13 | mriedem | vsaienk0: https://bugs.launchpad.net/nova/+bug/1660973 is a duplicate | |
| 15:20:13 | openstack | Launchpad bug 1660973 in networking-bgpvpn ""No hosts found to map to cell, exiting" / PlacementNotConfigured exception" [Critical,In progress] - Assigned to Thomas Morin (tmmorin-orange) | |
| 15:20:49 | dansmith | mriedem: you sure that's the same? | |
| 15:20:57 | mriedem | it's not | |
| 15:20:58 | mriedem | nvm | |
| 15:21:02 | mriedem | http://logs.openstack.org/75/427675/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial-nv/3ff2401/logs/screen-n-cpu.txt.gz?level=TRACE is clean | |
| 15:21:11 | dansmith | yeah | |
| 15:21:22 | dansmith | n-cond too, an that's the important one for the mapping bug | |
| 15:21:53 | vsaienk0 | mriedem: on ironic multinode job placement api is enabled | |
| 15:22:04 | mriedem | vsaienk0: then it's something else | |
| 15:22:06 | dansmith | vsaienk0: is that test trying to target specific hosts with a boot? | |
| 15:22:53 | mordred | mriedem: done | |
| 15:23:03 | mordred | mriedem: I believe I said the same thing as sdague - exept worse and with more words | |
| 15:23:34 | mriedem | mordred: that's why i pinged you :) | |
| 15:24:54 | vsaienk0 | dansmith: yes, test_schedule_to_all_nodes is failing | |
| 15:25:21 | dansmith | vsaienk0: I see that from the logs, I'm asking what the test is doing, but I'll go look for myself | |
| 15:25:52 | jaypipes | sileht: thx for your fix on https://review.openstack.org/#/c/427667/. zzzeek, if you can give your attention to this patch, that would be appreciated. I will leave it to mriedem, sdague or bauzas to +W. | |
| 15:26:24 | vsaienk0 | dansmith: that is nova test, I don't know what exactly it is doing | |
| 15:26:36 | dansmith | vsaienk0: okay, yeah, so it's trying to specifically boot an instance targeted at a node | |
| 15:26:52 | jaypipes | sileht: for the record, the reason that uuid was the grouping expression there was that the original query I was using only returned the UUIDs of the resource providers :) | |
| 15:27:10 | dansmith | vsaienk0: so it may be similar to the issue mriedem pointed above, but manifesting differently, give me a sec to look at the logs | |
| 15:27:22 | jaypipes | rpodolyaka: thx for the heads up on https://review.openstack.org/#/c/427667/ :) | |
| 15:27:28 | bauzas | sileht: oh, nice catch, thanks for fixing that | |
| 15:28:11 | dansmith | vsaienk0: is this failing all the time now or just occasionally | |
| 15:28:30 | vsaienk0 | dansmith: all the time | |
| 15:28:45 | dansmith | vsaienk0: okay, let me put up a hacky devstack patch to try something | |
| 15:30:11 | vsaienk0 | dansmith thanks! | |
| 15:30:28 | bauzas | sileht: jaypipes: and FWIW, rpodolyaka is right, I did the GROUP BY because I wanted the HAVING clause | |
| 15:30:47 | bauzas | sileht: jaypipes: but that should work by the PK too | |
| 15:30:54 | rpodolyaka | jaypipes: np! | |
| 15:31:43 | dansmith | vsaienk0: can you do a canary change against ironic, depending on this? https://review.openstack.org/427772 | |
| 15:32:51 | vsaienk0 | dansmith: done https://review.openstack.org/#/c/427675/ waiting for results, thanks! | |