Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-14
17:41:01 mriedem arvindn05: not yet, still reading
17:41:17 bauzas the current behaviour is that if you asked for 10 instances but your quota only tells 2 left, then it will just check that
17:41:25 arvindn05 mriedem: great....just let me know when you have questions.
17:41:27 bauzas but honestly, who cares ?
17:41:34 mriedem bauzas: heh "Error handling for multiple create is not as consistent as for single server create, and there is no guarantee that all the servers will be built. This call should generally be avoided in favor of clients doing direct individual server creates."
17:41:39 mriedem so actually,
17:41:49 mriedem concurrent creates is actually a problem for the affinity scenarios
17:41:58 mriedem but not a single multi-create request
17:42:02 mriedem damned if you do, damned if yo udon't
17:42:28 bauzas I'm just saying that the "minimum" value is useless
17:42:46 bauzas you ask for a max of 10, the scheduler will try to find you room for 10, or kick all of them
17:42:53 mriedem maybe we should amend that to say, "unless of course you're creating servers in a server group, then because of race issues we have'nt fixed yet, you should use multi-create"
17:43:04 cfriesen bauzas: and it shouldn't kick all of them if some of them pass
17:43:21 bauzas cfriesen: I'm not sure
17:43:29 bauzas cfriesen: it's a capacity problem
17:43:37 cfriesen bauzas: the user has said "give me as many instances as you can, up to a max of 10"
17:43:39 bauzas and we shouldn't leak the capacity to the user
17:43:42 mriedem how is the min useless? give me at least 2, but 10 if you can
17:43:55 bauzas mriedem: the "if you can" is only quota-wise
17:44:04 cfriesen bauzas: no, it's in the boot request
17:44:31 bauzas cfriesen: but the info we give to the scheduler is "how many the user asked"
17:44:45 bauzas cfriesen: not "how many the user is agreeing to only have"
17:44:55 mriedem also depends on the port quota, don't forget
17:45:04 bauzas yeah
17:45:05 cfriesen bauzas: I totally agree it's all messed up. but the API describes it as the min/max nubmer of servers to be created
17:45:15 bauzas the API docs are wrong
17:45:28 cfriesen the API docs are by definition correct, the implementation is wrong. :)
17:45:49 cfriesen the user wants at least 2 and at most 10 instances. If the quota is 4, we should give them 4
17:45:54 bauzas that's an interesting PoV
17:45:56 cfriesen If we can only schedule 3, we should give them 3
17:46:08 mriedem i'm pretty sure we've said in the past that the docs don't define the api
17:46:11 dansmith cfriesen: I think the actual behavior, which people may have built dependencies on, is the canonical thing
17:46:13 mriedem the current behavior defines the api
17:46:16 mriedem right
17:46:16 dansmith right
17:46:18 mriedem right
17:46:21 dansmith right
17:46:22 melwitt right
17:46:23 mriedem right
17:46:24 bauzas right
17:46:26 bauzas right
17:46:28 bauzas right
17:46:42 dansmith (man we're dorks)
17:46:43 mriedem remember the docs of yore for the compute api were written by people that didn't actually code up nova api
17:46:57 melwitt I think we've said in the past that it makes more sense for the --min to be honored if we can but if we change that, we have to do it with a microversion
17:46:58 mriedem and until sdague lifted them in tree they were all sorts of wrong
17:47:02 mriedem and still are in parts
17:47:25 bauzas anyway, time to say goodbye
17:47:33 mriedem adieu
17:47:36 cfriesen have fun
17:48:05 bauzas "adieu" has a very strong meaning of a final goodbye :)
17:48:13 bauzas I just hope to be back in 5 days :)
17:48:14 cfriesen au revoir
17:49:22 cfriesen for what it's worth, it seems odd to say that nobody can trust our published API specification, and instead they have to observe the current behaviour.
17:50:16 cfriesen but I get the argument about people already depending on the current behaviour
17:54:11 melwitt I wonder if it only behaves that way for quota. like, what happens if you do --min 2 --max 10 and there's only compute capacity for 2. I wonder if that would pass scheduling and give back only 2 instances
17:54:13 dansmith cfriesen: I think it's odd to say that we'd jump through hoops which might be very difficult to implement something the way someone guessed it may have worked long ago
17:54:40 dansmith ...just because they put that in a doc
17:55:04 mriedem arvindn05: ok thanks for the clarifications. replies inline for what i'd like to see changed, but it should be trivial
17:55:36 mriedem cfriesen: if the docs are inaccurate, let's clean up the docs
17:55:46 mriedem coincidentally i was just looking at https://bugs.launchpad.net/nova/+bug/1684261 again
17:55:47 openstack Launchpad bug 1684261 in OpenStack Compute (nova) "AggregateImagePropertiesIsolation example doesn't actually indicate how it works" [Low,Confirmed]
17:56:44 arvindn05 mriedem: awesome...will review them and let you know
17:57:05 mriedem would be really nice if someone would take on fixing the doc for that filter
17:58:48 arvindn05 mriedem: sure. Updating the docs should be straitforward...will look into it
17:59:02 mriedem famous, last, words
17:59:10 arvindn05 mriedem: :)
18:00:01 cfriesen there's a truth table for that filter in https://review.openstack.org/#/c/381912/17/specs/rocky/approved/strict_isolation_of_group_of_hosts_for_image.rst
18:00:20 cfriesen might be handy to reference when reworking the docs
18:00:38 melwitt let's put it in the docs!
18:01:49 arvindn05 yes
18:02:00 melwitt mriedem: thanks for the comments on the cells summary, updated it and will send it out
18:04:30 openstackgerrit Chris Dent proposed openstack/nova master: Move placement exceptions into the placement package https://review.openstack.org/549862
18:04:31 openstackgerrit Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435
18:04:31 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
18:12:48 mnaser jaypipes: don't want to bother you too much but if you have some free time about the vif plugging on reboot issue
18:19:13 jaypipes mnaser: crap, sorry man. got distracted. looking now while tests are running locally.
18:19:38 mnaser jaypipes: np! i can get you a bit to what point i reached from my debugging
18:20:18 jaypipes sure thing.
18:20:21 jaypipes got for it.
18:21:13 mnaser jaypipes: so https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 which was merged in master and in the process of getting backported changed behavior (after another change) where now, nova expects a vif-network-plugged event even on reboots
18:21:45 mnaser now, it looks like linuxbridge broke because it didn't send that event, so a little work around was added in that patch above to skip that. now, opendaylight is broken because it doesn't send a notification either.
18:22:07 mnaser upon digging on *why* it doesn't send one, it looks like unplug with ovs on linux is a noop in os_vif
18:22:31 mnaser which means that the port is never really unplugged in odl, so when it tries to 'plug' it, neutron never really does anything because the port state never changes, and the server times out booting because vif_plugging_timeout
18:22:57 mnaser noop unplug here = https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268
18:23:13 jaypipes mnaser: and this is only on hard reboot, yes?
18:23:26 mnaser jaypipes: correct, but afaik a clean state is a hard reboot too in the nova codebase?
18:23:42 mnaser clean start*
18:23:54 mnaser so first time you start an instance, it'll be okay, but stop and start again will cause it to want a plugged event
18:23:58 jaypipes mnaser: no. a clean start does not issue a call to unplug() in the os-vif API.
18:24:42 mnaser jaypipes: right, but a clean start means the port is created the first time and the vif plugged event comes through fine, but when the instance is stopped, nova calls unplug (but nothing happens) and when it starts again, it waits for vif plugged event (those patches changed that behaviour)
18:25:11 mnaser this was noticed when the tempest start_stop tests failed because the startup would time out
18:26:02 jaypipes hm
18:26:03 mnaser http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/testr_results.html.gz (you can look at nova logs there, but pretty much test_stop_start_server / test_reboot_server_hard / etc are the ones that arent working now)
18:26:43 mnaser and failure is .. "Details: (ServerActionsTestJSON:test_stop_start_server) Server da9cead9-c217-495e-97a8-65a6adacf37c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHUTOFF. Current task state: powering-on." .. nova logs shows it timing out after 5 minutes
18:28:07 jaypipes mnaser: k. just a minute. reading through these patches...
18:28:13 mnaser sure thing
18:35:31 jaypipes mnaser: ok, done. so are you suggesting the fix here is to add VIF_TYPE_OVS to line 5392 here? https://review.openstack.org/#/c/541442/6/nova/virt/libvirt/driver.py
18:36:18 mnaser jaypipes: that would be a fix, or os_vif *actually* unplugging things could be a fix too
18:36:27 mnaser i do feel if that list starts growing, it might start become confusing for users :<

Earlier   Later