| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-10 | |||
| 14:26:18 | stephenfin | mdbooth: I've left comments against the bug. If this is resolved in new version of systemd, we should probably just wait on that rather than unnecessarily seeking root privileges | |
| 14:26:32 | mdbooth | stephenfin: Yeah, that's where I was going to go with that. | |
| 14:26:52 | stephenfin | mdbooth: Cool. Thanks for the input :) | |
| 14:26:57 | mdbooth | It seems we're adding code to Nova to replicate a function of systemd. | |
| 14:27:36 | mriedem | QG: no, tags are free-form defined by the user | |
| 14:29:49 | QG | mriedem: Ok thanks | |
| 14:30:06 | gibi | mriedem: I have bad news about https://bugs.launchpad.net/nova/+bug/1770434 | |
| 14:30:06 | openstack | Launchpad bug 1770434 in OpenStack Compute (nova) "affinity filter not work on master branch" [High,Confirmed] | |
| 14:30:21 | gibi | mriedem: it is really broken | |
| 14:31:00 | gibi | mriedem: I had to change the existing tempest test to reproduce it but it is now failing http://logs.openstack.org/49/581349/1/check/tempest-multinode-full/0ac326a/testr_results.html.gz | |
| 14:33:17 | gibi | mriedem: so far I only found that querying the instances that are members of the group returns an empty list https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/objects/instance_group.py#L422 | |
| 14:38:52 | openstackgerrit | Merged openstack/osc-placement master: Fix the 1.6 release note format https://review.openstack.org/579042 | |
| 14:39:17 | openstackgerrit | Merged openstack/nova master: make incomplete_consumer_project_id a valid UUID https://review.openstack.org/580358 | |
| 14:39:45 | openstackgerrit | Eric Fried proposed openstack/nova master: Update root providers in same tree https://review.openstack.org/580246 | |
| 14:40:51 | openstackgerrit | Merged openstack/osc-placement master: Remove doc/build during tox -e docs https://review.openstack.org/579041 | |
| 14:40:52 | openstackgerrit | Merged openstack/osc-placement master: Fix docstring for delete allocation method https://review.openstack.org/577075 | |
| 14:41:00 | mriedem | gibi: probably want to ask melwitt if that's related to the bug she's already working on | |
| 14:41:15 | gibi | mriedem: to gather the hosts of the members of an instance group we would need to read instances from every cell where are member instances | |
| 14:41:16 | mriedem | https://review.openstack.org/#/c/540258/ | |
| 14:41:48 | gibi | mriedem: thanks, seems related | |
| 14:46:37 | mnaser | thanks to whoeve decided to strip out all the extra crazy quotas out of the api | |
| 14:46:49 | mnaser | things are so much cleaner now. | |
| 14:49:28 | mriedem | mnaser: docs or microversion? | |
| 14:49:50 | mriedem | https://developer.openstack.org/api-ref/compute/#show-a-quota | |
| 14:50:00 | mnaser | microversion i tihnk, nova quota-show shows a lot less things now | |
| 14:50:11 | mriedem | the networking and file injection ones are still there unless you're using nova CLI which defaults to latest available microversion in the server so you won't see those | |
| 14:50:14 | mriedem | yup | |
| 14:50:33 | mnaser | nice on the eyes. | |
| 14:51:07 | mriedem | that's what they said about me in the Mr Minnesota Pageant | |
| 14:52:08 | mnaser | i wouldnt disagree | |
| 14:52:08 | mnaser | ha | |
| 14:52:35 | mriedem | <# | |
| 14:52:37 | mriedem | <3 | |
| 14:54:17 | kashyap | mriedem: I only get hits for "Mr & Ms Natural Minnesota" peasants. /me wonders about the 'natural' word there... | |
| 14:54:27 | kashyap | s/peasant/pageant/ | |
| 14:56:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Use nova.db.api directly https://review.openstack.org/543262 | |
| 14:58:26 | mriedem | in case it's not clear, mr minnesota is not a real thing, nor was i a contestant | |
| 14:58:39 | mriedem | but a guy can dream... | |
| 15:00:18 | gibi | melwitt: Can I help somehow with https://review.openstack.org/#/c/540258 ? | |
| 15:17:24 | kashyap | mriedem: Hehe, I know it's not a _real_ thing! Was just teasing :P | |
| 15:21:53 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Refactor _build_device_metadata https://review.openstack.org/581398 | |
| 15:21:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/queens: Consider hostdev devices when building metadata https://review.openstack.org/581399 | |
| 15:23:31 | mriedem | gibi: until we're using placement for affinity scheduling, i don't think we can ever do https://review.openstack.org/#/c/581349/ in tempest | |
| 15:23:55 | mriedem | it will always be racy - which is why we have the late affinity check in the compute (which is disabled in default devstack b/c we can't upcall to the api db) | |
| 15:24:21 | gibi | mriedem: that test does not intended to show a race condition | |
| 15:24:46 | gibi | mriedem: I can add sleep between the server create requests and it will still fail | |
| 15:25:37 | mriedem | sure, but when the bug is fixed and we're properly pulling group members, it would start to race | |
| 15:26:04 | mriedem | i'd been meaning to write a functional test for https://review.openstack.org/#/c/540258 but need to look at my notes - it's been awhile | |
| 15:26:33 | mriedem | oh i guess i was going to write a new bug for something that came up during review: | |
| 15:26:33 | mriedem | "We need a new bug and test for the multi-cell affinity scenario where two instances are scheduled at the same time in the same affinity group. We need 2 cells with 1 host each, with exactly enough capacity to fit just one instance so that placement will fail the first request and throw it into the other host in the other cell. The late affinity check in the compute won't fail because it can't see the other member in the | |
| 15:26:34 | mriedem | her cell, so it will think it's fine." | |
| 15:27:28 | mriedem | the tricky part there is our functional tests are able to make up-calls from nova-compute | |
| 15:27:34 | mriedem | we don't have a split mq in our tests | |
| 15:27:38 | mriedem | not sure how easy that would be to do | |
| 15:27:40 | mriedem | dansmith: ^ ? | |
| 15:28:33 | mriedem | well, maybe won't need split mq - just the instances being in different hosts in different cell dbs might do it | |
| 15:28:34 | dansmith | ....yeah | |
| 15:28:53 | dansmith | with the workaround flag you can disable the check right? | |
| 15:29:09 | mriedem | yes | |
| 15:29:54 | dansmith | although I guess that doesn't really do much, | |
| 15:29:58 | gibi | mriedem: do I understand correclty that on master server groups only work if the late check using upcall is enabled? | |
| 15:30:00 | dansmith | obviously if you disable that check it's not going to check | |
| 15:30:02 | mriedem | maybe that type of request wouldn't get past scheduling | |
| 15:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Remove irrelevant comment https://review.openstack.org/578821 | |
| 15:30:04 | openstackgerrit | Matthew Booth proposed openstack/nova master: Move static _get_power_off_values to compute_utils https://review.openstack.org/578822 | |
| 15:30:30 | mriedem | gibi: i don't think so... | |
| 15:30:34 | mriedem | tempest has an anti-affinity test | |
| 15:30:50 | dansmith | mriedem: so, scheduler will look at group.members, get the instance.host for each, and affine to those hosts, right? | |
| 15:30:50 | gibi | mriedem: that is the test I changed to show that master is broken | |
| 15:31:02 | mriedem | gibi: test_create_server_with_scheduler_hint_group_anti_affinity | |
| 15:31:03 | dansmith | which is lossy because those instances might be pre-scheduling | |
| 15:31:07 | dansmith | or rather, pre-build | |
| 15:31:24 | openstackgerrit | Chen proposed openstack/nova master: WIP https://review.openstack.org/581403 | |
| 15:32:06 | gibi | mriedem: that tempest test only works because it sends two server create in the same API request | |
| 15:32:17 | mriedem | right, because servers in separate requests is a known race | |
| 15:32:34 | gibi | mriedem: if you wait 10 minutes between the two requests it still doesnt work | |
| 15:32:37 | gibi | mriedem: I tried :) | |
| 15:32:46 | dansmith | gibi: really? why is that? | |
| 15:33:19 | dansmith | the filter is using a potentially stale version of server group, from the beginning of the request, I just noticed | |
| 15:33:22 | mriedem | yeah the HostState.instances dict should be populated per request | |
| 15:33:29 | dansmith | that shouldn't be a problem 10 minutes later though | |
| 15:33:36 | gibi | dansmith: this returns an empty list for the second scheduling https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/objects/instance_group.py#L422 | |
| 15:33:51 | mdbooth | Just working on this patch: https://review.openstack.org/#/c/578846/ I called it out in the commit message: the cut/paste in there is horrible, but avoids a refactor. Am I going to get that landed as is, or do I need to look at refactoring the driver interface? | |
| 15:34:01 | mriedem | gibi: and i think that's the bug melwitt is fixing | |
| 15:34:01 | dansmith | gibi: oh on multi-cell? | |
| 15:34:05 | dansmith | that needs to stripe | |
| 15:34:06 | dansmith | yeah | |
| 15:34:24 | gibi | don't we deploy with multi cell _by default_ ? | |
| 15:34:39 | mdbooth | It goes without saying that I don't mind refactoring :) However, I'd also like to both land and backport the code. | |
| 15:34:55 | mriedem | gibi: in devstack yes | |
| 15:34:58 | dansmith | gibi: yeah, this would hit cell0 if your default is cell0, and always be empty | |
| 15:35:14 | mriedem | it's not really 'multi cell' so much as superconductor mode in devstack and how the services are configured | |
| 15:35:30 | gibi | let me rephrase. I our suggested deployment mode (cell_v2) the server group policies are broken | |
| 15:35:41 | gibi | that is my understanding | |
| 15:35:50 | dansmith | gibi: because your default connection is to cell0, which will never return the list of instances, yeah | |
| 15:36:03 | gibi | dansmith: thanks, that explains what I see | |
| 15:36:13 | dansmith | but striping across cells is what needs to happen, which is that bug afaik | |
| 15:36:48 | gibi | dansmith: yeah. this is why offered my help to melwitt above on the fix | |
| 15:36:59 | dansmith | gibi: are we in violent agreement? :) | |
| 15:37:06 | gibi | dansmith: yes | |