| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-01 | |||
| 14:54:53 | mriedem | dansmith: so i've got a recreate on that failure, we know the build request is gone before we create the instance in conductor, now i'm just not sure what i do with it - we don't put that in cell0 since if the build request is deleted before the instance is created, we just treat it like the instance was deleted, i.e. it wouldn't be in ERROR state in cell0 | |
| 14:55:11 | mriedem | i think i can just return | |
| 14:55:27 | dansmith | mriedem: ah I misunderstood your assertion last night then | |
| 14:55:43 | dansmith | mriedem: yeah, if build request is gone before we create the point is we're supposed to just bail quietly.. we do that in at least one place now | |
| 14:55:53 | mriedem | yup here https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@936 | |
| 14:55:58 | mriedem | which for this bug is too late | |
| 14:56:33 | dansmith | mriedem: ah, yep, indeed | |
| 14:56:57 | mriedem | dansmith: also, | |
| 14:57:02 | mriedem | unrelated, but this return should be a continue right? https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@972 | |
| 14:57:04 | mriedem | as we're in a for loop | |
| 14:57:58 | dansmith | mriedem: that's not in a for loop | |
| 14:58:13 | mriedem | sure it is https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@875 | |
| 14:58:29 | mriedem | it's like a 200LOC method so it's easy to miss | |
| 14:58:41 | dansmith | ooh, well, let me check | |
| 14:59:17 | mriedem | ok we can fix separately, i'll clean up my patch for the actual first fix | |
| 14:59:21 | dansmith | yeah, since we've got multiple build requests I guess so, if someone were to delete one of the multi-instances in the middle | |
| 14:59:22 | dansmith | aye | |
| 14:59:28 | mriedem | yar! | |
| 14:59:29 | mriedem | matey | |
| 15:01:22 | sfinucan | johnthetubaguy: One question on https://review.openstack.org/#/c/421760/10/api-ref/source/servers.inc | |
| 15:01:50 | mriedem | i'll even put on green grass and high tides since this should take 10 minutes | |
| 15:01:54 | dansmith | mriedem: so, I'm not sure why you think this delete-before-create thing is related to the message in your bug about rebuild | |
| 15:02:10 | sfinucan | In that file, the parameter is recorded as 'accessIPv4', but in the release note [1] it's recorded as 'access_ip_v4' [1] https://review.openstack.org/#/c/421760/10/releasenotes/notes/add-whitelist-for-server-list-filter-sort-params-2ae766d03ba895e5.yaml | |
| 15:02:24 | mriedem | dansmith: the test creates the server, then before it's active it deletes it, and then tries to rebuild it, expecting a 404, but gets a 409 | |
| 15:02:25 | sfinucan | I assume the former is automagically transformed to the latter | |
| 15:02:28 | sfinucan | ? | |
| 15:02:34 | mriedem | dansmith: it's a negative test, expecting the deleted instance to be deleted, | |
| 15:02:54 | mriedem | but when we hit the 409, the build request is gone but conductor created the instance (from the stale build request) | |
| 15:03:11 | dansmith | mriedem: ah, okay cool | |
| 15:03:17 | mriedem | dansmith: it's not just rebuild, there is reboot and some other negative tests that have the same pattern | |
| 15:03:23 | dansmith | ack | |
| 15:03:42 | mriedem | i'm just happy it's not more complicated than this :) | |
| 15:03:52 | dansmith | def totes | |
| 15:10:12 | bauzas | mriedem: I have now gained more b/w, which change should I bite now ? | |
| 15:11:08 | bauzas | looking at https://etherpad.openstack.org/p/nova-ocata-rc1-todos | |
| 15:12:51 | mriedem | bauzas: https://review.openstack.org/#/c/427367/ | |
| 15:13:51 | mriedem | bauzas: and https://review.openstack.org/#/c/421760/ | |
| 15:13:52 | bauzas | ack | |
| 15:14:00 | mriedem | and https://review.openstack.org/#/c/426810/ | |
| 15:14:09 | mriedem | basically, just read the etherpad and pick :) | |
| 15:15:31 | mriedem | sdague: you may want to chime in on the postgres job thread in the dev list | |
| 15:15:38 | mriedem | mordred: ^ | |
| 15:15:53 | mordred | uhoh | |
| 15:16:03 | mriedem | the support matrix that won't die :) | |
| 15:18:34 | vsaienk0 | nova team, since today morning/yesterday evening ironic multinode job is failing due to https://bugs.launchpad.net/nova/+bug/1661014 | |
| 15:18:34 | openstack | Launchpad bug 1661014 in OpenStack Compute (nova) "Multinode job fails with "Compute host X not found"" [Undecided,New] | |
| 15:18:43 | sdague | mriedem: sure | |
| 15:19:16 | vsaienk0 | is it known problem? | |
| 15:19:58 | mriedem | vsaienk0: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | |
| 15:20:13 | mriedem | vsaienk0: https://bugs.launchpad.net/nova/+bug/1660973 is a duplicate | |
| 15:20:13 | openstack | Launchpad bug 1660973 in networking-bgpvpn ""No hosts found to map to cell, exiting" / PlacementNotConfigured exception" [Critical,In progress] - Assigned to Thomas Morin (tmmorin-orange) | |
| 15:20:49 | dansmith | mriedem: you sure that's the same? | |
| 15:20:57 | mriedem | it's not | |
| 15:20:58 | mriedem | nvm | |
| 15:21:02 | mriedem | http://logs.openstack.org/75/427675/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial-nv/3ff2401/logs/screen-n-cpu.txt.gz?level=TRACE is clean | |
| 15:21:11 | dansmith | yeah | |
| 15:21:22 | dansmith | n-cond too, an that's the important one for the mapping bug | |
| 15:21:53 | vsaienk0 | mriedem: on ironic multinode job placement api is enabled | |
| 15:22:04 | mriedem | vsaienk0: then it's something else | |
| 15:22:06 | dansmith | vsaienk0: is that test trying to target specific hosts with a boot? | |
| 15:22:53 | mordred | mriedem: done | |
| 15:23:03 | mordred | mriedem: I believe I said the same thing as sdague - exept worse and with more words | |
| 15:23:34 | mriedem | mordred: that's why i pinged you :) | |
| 15:24:54 | vsaienk0 | dansmith: yes, test_schedule_to_all_nodes is failing | |
| 15:25:21 | dansmith | vsaienk0: I see that from the logs, I'm asking what the test is doing, but I'll go look for myself | |
| 15:25:52 | jaypipes | sileht: thx for your fix on https://review.openstack.org/#/c/427667/. zzzeek, if you can give your attention to this patch, that would be appreciated. I will leave it to mriedem, sdague or bauzas to +W. | |
| 15:26:24 | vsaienk0 | dansmith: that is nova test, I don't know what exactly it is doing | |
| 15:26:36 | dansmith | vsaienk0: okay, yeah, so it's trying to specifically boot an instance targeted at a node | |
| 15:26:52 | jaypipes | sileht: for the record, the reason that uuid was the grouping expression there was that the original query I was using only returned the UUIDs of the resource providers :) | |
| 15:27:10 | dansmith | vsaienk0: so it may be similar to the issue mriedem pointed above, but manifesting differently, give me a sec to look at the logs | |
| 15:27:22 | jaypipes | rpodolyaka: thx for the heads up on https://review.openstack.org/#/c/427667/ :) | |
| 15:27:28 | bauzas | sileht: oh, nice catch, thanks for fixing that | |
| 15:28:11 | dansmith | vsaienk0: is this failing all the time now or just occasionally | |
| 15:28:30 | vsaienk0 | dansmith: all the time | |
| 15:28:45 | dansmith | vsaienk0: okay, let me put up a hacky devstack patch to try something | |
| 15:30:11 | vsaienk0 | dansmith thanks! | |
| 15:30:28 | bauzas | sileht: jaypipes: and FWIW, rpodolyaka is right, I did the GROUP BY because I wanted the HAVING clause | |
| 15:30:47 | bauzas | sileht: jaypipes: but that should work by the PK too | |
| 15:30:54 | rpodolyaka | jaypipes: np! | |
| 15:31:43 | dansmith | vsaienk0: can you do a canary change against ironic, depending on this? https://review.openstack.org/427772 | |
| 15:32:51 | vsaienk0 | dansmith: done https://review.openstack.org/#/c/427675/ waiting for results, thanks! | |
| 15:33:25 | dansmith | vsaienk0: okay, this is the thing I was pointing to yesterday, which I suspect you're hitting. if this fixes it, we'll need to put it in a proper place of course | |
| 15:34:25 | openstackgerrit | Christopher Martin proposed openstack/nova: Stop using mox in test_compute_cells.py https://review.openstack.org/389750 | |
| 15:34:54 | dansmith | vsaienk0: oh you guys are a devstack plugin, well that's why I couldn't find stuff... going forward we can just put this in your lib/ironic thing, so .. easier | |
| 15:36:35 | sfinucan | bauzas: Think there's a very minor logging mistake in https://review.openstack.org/#/c/427200, though otherwise LGTM | |
| 15:37:10 | bauzas | sfinucan: you can hurt me with a -1, I won't hit you :p | |
| 15:37:41 | sfinucan | bauzas: I don't usually -1 for things I'm not 100% sure are an issue, but you asked for it ;) | |
| 15:37:44 | bauzas | and I even prefer or I could miss the comments :) | |
| 15:38:11 | bauzas | sfinucan: well, I'm French so arguing is like my 2nd nature | |
| 15:39:01 | openstackgerrit | Matt Riedemann proposed openstack/nova: Ensure build request exists before creating instance https://review.openstack.org/427775 | |
| 15:39:02 | mriedem | dansmith: melwitt: bauzas: ^ i'm assuming some sort of transactional lock would be best there, but not sure how to do it | |
| 15:40:18 | dansmith | mriedem: you can't, they're in separate databases | |
| 15:40:25 | mriedem | thought about that | |
| 15:40:26 | mriedem | yeah | |
| 15:40:28 | dansmith | mriedem: you should create the instance, then try to delete the buildreq, and if that fails, delete the instance | |
| 15:40:31 | dansmith | like we do below | |
| 15:42:04 | mriedem | we have a window either way | |
| 15:42:16 | mriedem | i'd rather not create the instance in the first place | |
| 15:42:44 | mriedem | if the build request doesn't exist before we create the instance, this catches it, if the build request doesn't exist after we create the instance, the existing check at the bottom handles it | |
| 15:43:20 | dansmith | mriedem: there's no window this way, | |