| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 17:43:29 | bauzas | cfriesen: it's a capacity problem | |
| 17:43:37 | cfriesen | bauzas: the user has said "give me as many instances as you can, up to a max of 10" | |
| 17:43:39 | bauzas | and we shouldn't leak the capacity to the user | |
| 17:43:42 | mriedem | how is the min useless? give me at least 2, but 10 if you can | |
| 17:43:55 | bauzas | mriedem: the "if you can" is only quota-wise | |
| 17:44:04 | cfriesen | bauzas: no, it's in the boot request | |
| 17:44:31 | bauzas | cfriesen: but the info we give to the scheduler is "how many the user asked" | |
| 17:44:45 | bauzas | cfriesen: not "how many the user is agreeing to only have" | |
| 17:44:55 | mriedem | also depends on the port quota, don't forget | |
| 17:45:04 | bauzas | yeah | |
| 17:45:05 | cfriesen | bauzas: I totally agree it's all messed up. but the API describes it as the min/max nubmer of servers to be created | |
| 17:45:15 | bauzas | the API docs are wrong | |
| 17:45:28 | cfriesen | the API docs are by definition correct, the implementation is wrong. :) | |
| 17:45:49 | cfriesen | the user wants at least 2 and at most 10 instances. If the quota is 4, we should give them 4 | |
| 17:45:54 | bauzas | that's an interesting PoV | |
| 17:45:56 | cfriesen | If we can only schedule 3, we should give them 3 | |
| 17:46:08 | mriedem | i'm pretty sure we've said in the past that the docs don't define the api | |
| 17:46:11 | dansmith | cfriesen: I think the actual behavior, which people may have built dependencies on, is the canonical thing | |
| 17:46:13 | mriedem | the current behavior defines the api | |
| 17:46:16 | mriedem | right | |
| 17:46:16 | dansmith | right | |
| 17:46:18 | mriedem | right | |
| 17:46:21 | dansmith | right | |
| 17:46:22 | melwitt | right | |
| 17:46:23 | mriedem | right | |
| 17:46:24 | bauzas | right | |
| 17:46:26 | bauzas | right | |
| 17:46:28 | bauzas | right | |
| 17:46:42 | dansmith | (man we're dorks) | |
| 17:46:43 | mriedem | remember the docs of yore for the compute api were written by people that didn't actually code up nova api | |
| 17:46:57 | melwitt | I think we've said in the past that it makes more sense for the --min to be honored if we can but if we change that, we have to do it with a microversion | |
| 17:46:58 | mriedem | and until sdague lifted them in tree they were all sorts of wrong | |
| 17:47:02 | mriedem | and still are in parts | |
| 17:47:25 | bauzas | anyway, time to say goodbye | |
| 17:47:33 | mriedem | adieu | |
| 17:47:36 | cfriesen | have fun | |
| 17:48:05 | bauzas | "adieu" has a very strong meaning of a final goodbye :) | |
| 17:48:13 | bauzas | I just hope to be back in 5 days :) | |
| 17:48:14 | cfriesen | au revoir | |
| 17:49:22 | cfriesen | for what it's worth, it seems odd to say that nobody can trust our published API specification, and instead they have to observe the current behaviour. | |
| 17:50:16 | cfriesen | but I get the argument about people already depending on the current behaviour | |
| 17:54:11 | melwitt | I wonder if it only behaves that way for quota. like, what happens if you do --min 2 --max 10 and there's only compute capacity for 2. I wonder if that would pass scheduling and give back only 2 instances | |
| 17:54:13 | dansmith | cfriesen: I think it's odd to say that we'd jump through hoops which might be very difficult to implement something the way someone guessed it may have worked long ago | |
| 17:54:40 | dansmith | ...just because they put that in a doc | |
| 17:55:04 | mriedem | arvindn05: ok thanks for the clarifications. replies inline for what i'd like to see changed, but it should be trivial | |
| 17:55:36 | mriedem | cfriesen: if the docs are inaccurate, let's clean up the docs | |
| 17:55:46 | mriedem | coincidentally i was just looking at https://bugs.launchpad.net/nova/+bug/1684261 again | |
| 17:55:47 | openstack | Launchpad bug 1684261 in OpenStack Compute (nova) "AggregateImagePropertiesIsolation example doesn't actually indicate how it works" [Low,Confirmed] | |
| 17:56:44 | arvindn05 | mriedem: awesome...will review them and let you know | |
| 17:57:05 | mriedem | would be really nice if someone would take on fixing the doc for that filter | |
| 17:58:48 | arvindn05 | mriedem: sure. Updating the docs should be straitforward...will look into it | |
| 17:59:02 | mriedem | famous, last, words | |
| 17:59:10 | arvindn05 | mriedem: :) | |
| 18:00:01 | cfriesen | there's a truth table for that filter in https://review.openstack.org/#/c/381912/17/specs/rocky/approved/strict_isolation_of_group_of_hosts_for_image.rst | |
| 18:00:20 | cfriesen | might be handy to reference when reworking the docs | |
| 18:00:38 | melwitt | let's put it in the docs! | |
| 18:01:49 | arvindn05 | yes | |
| 18:02:00 | melwitt | mriedem: thanks for the comments on the cells summary, updated it and will send it out | |
| 18:04:30 | openstackgerrit | Chris Dent proposed openstack/nova master: Move placement exceptions into the placement package https://review.openstack.org/549862 | |
| 18:04:31 | openstackgerrit | Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435 | |
| 18:04:31 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 18:12:48 | mnaser | jaypipes: don't want to bother you too much but if you have some free time about the vif plugging on reboot issue | |
| 18:19:13 | jaypipes | mnaser: crap, sorry man. got distracted. looking now while tests are running locally. | |
| 18:19:38 | mnaser | jaypipes: np! i can get you a bit to what point i reached from my debugging | |
| 18:20:18 | jaypipes | sure thing. | |
| 18:20:21 | jaypipes | got for it. | |
| 18:21:13 | mnaser | jaypipes: so https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 which was merged in master and in the process of getting backported changed behavior (after another change) where now, nova expects a vif-network-plugged event even on reboots | |
| 18:21:45 | mnaser | now, it looks like linuxbridge broke because it didn't send that event, so a little work around was added in that patch above to skip that. now, opendaylight is broken because it doesn't send a notification either. | |
| 18:22:07 | mnaser | upon digging on *why* it doesn't send one, it looks like unplug with ovs on linux is a noop in os_vif | |
| 18:22:31 | mnaser | which means that the port is never really unplugged in odl, so when it tries to 'plug' it, neutron never really does anything because the port state never changes, and the server times out booting because vif_plugging_timeout | |
| 18:22:57 | mnaser | noop unplug here = https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:23:13 | jaypipes | mnaser: and this is only on hard reboot, yes? | |
| 18:23:26 | mnaser | jaypipes: correct, but afaik a clean state is a hard reboot too in the nova codebase? | |
| 18:23:42 | mnaser | clean start* | |
| 18:23:54 | mnaser | so first time you start an instance, it'll be okay, but stop and start again will cause it to want a plugged event | |
| 18:23:58 | jaypipes | mnaser: no. a clean start does not issue a call to unplug() in the os-vif API. | |
| 18:24:42 | mnaser | jaypipes: right, but a clean start means the port is created the first time and the vif plugged event comes through fine, but when the instance is stopped, nova calls unplug (but nothing happens) and when it starts again, it waits for vif plugged event (those patches changed that behaviour) | |
| 18:25:11 | mnaser | this was noticed when the tempest start_stop tests failed because the startup would time out | |
| 18:26:02 | jaypipes | hm | |
| 18:26:03 | mnaser | http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/testr_results.html.gz (you can look at nova logs there, but pretty much test_stop_start_server / test_reboot_server_hard / etc are the ones that arent working now) | |
| 18:26:43 | mnaser | and failure is .. "Details: (ServerActionsTestJSON:test_stop_start_server) Server da9cead9-c217-495e-97a8-65a6adacf37c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHUTOFF. Current task state: powering-on." .. nova logs shows it timing out after 5 minutes | |
| 18:28:07 | jaypipes | mnaser: k. just a minute. reading through these patches... | |
| 18:28:13 | mnaser | sure thing | |
| 18:35:31 | jaypipes | mnaser: ok, done. so are you suggesting the fix here is to add VIF_TYPE_OVS to line 5392 here? https://review.openstack.org/#/c/541442/6/nova/virt/libvirt/driver.py | |
| 18:36:18 | mnaser | jaypipes: that would be a fix, or os_vif *actually* unplugging things could be a fix too | |
| 18:36:27 | mnaser | i do feel if that list starts growing, it might start become confusing for users :< | |
| 18:36:43 | jaypipes | mnaser: agreed. | |
| 18:37:12 | jaypipes | mnaser: for os-vif linux bridge, though, I'm not entirely sure what should be done on unplug... | |
| 18:37:50 | mnaser | jaypipes: yeah.. that's beyond me, but for openvswitch, im not sure if there is a link state, or maybe just deleting the actual port if its 'unplugged' | |
| 18:38:05 | mnaser | im sure there's a really good reason why it's not being deleted though, but i don't know why | |
| 18:38:37 | jaypipes | mnaser: I'm also not sure whether that would *ensure* that a vif-unplugged *event* from Neutron would be received... | |
| 18:38:43 | jaypipes | sean-k-mooney: you still around? | |
| 18:39:21 | mnaser | jaypipes: well, nova doesnt care about vif-unplugged afaik, issue is surrounding vif-plugged event | |
| 18:39:50 | mnaser | jaypipes: based on my reading, the nova notifications are a hook to the db model in neutron when an object changes. so if it goes from active => active, nothing is being updated and it never sent anything | |
| 18:40:03 | mriedem | lyarwood: melwitt: probably want to listen to ^ | |
| 18:40:03 | mnaser | now, maybe neutron-openvswitch-agent does things differently to trigger a change, i'm not sure. | |
| 18:40:34 | jaypipes | mnaser: not the neutron DB. the ovsdb... | |
| 18:40:35 | mriedem | related https://review.openstack.org/#/c/550046/ | |
| 18:40:41 | jaypipes | mnaser: and that's only for OVS of course. | |
| 18:40:47 | melwitt | mriedem: thanks | |