Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-30
20:47:28 artom He's +2 except for 1 patch with nits
20:47:31 artom Which I'm trying to fix
20:47:45 mriedem artom: MN
20:47:47 mriedem no coast
20:47:58 jaypipes artom: ok. can try tomorrow morning (there's a bunch of stuff ahead of that right now)
20:48:10 artom jaypipes, heh, it wasn't a request, but thank you!
20:48:16 mriedem we can throw it in the runways queue
20:48:23 mriedem sahid should have already done that anyway
20:48:28 mriedem i'll add it
20:48:45 mriedem artom: east coast rap-wise
20:49:00 jaypipes mriedem: definitely west coast.
20:49:04 artom mriedem, ok, you have an hour or so on me in terms of time left in day, you might still be around to +2 it if I succeed in making those damn tests pass
21:12:54 mriedem dansmith: before Kevin_Zheng goes off and adds some backoff looping call logic here, you might want to make sure we're not missing something on the problem for which he's trying to account https://review.openstack.org/#/c/568542/1/nova/compute/manager.py@6222
21:13:02 mriedem basically a race window during abort of a live migration
21:26:30 mriedem hmm, also, on surya's patch, she already has a release note https://review.openstack.org/#/c/567878/5/releasenotes/notes/migration-tool-to-populate-inst.avz-29fed2fe57a9764d.yaml
21:26:37 mriedem so i'm not sure what to add to that in the backport
21:27:54 dansmith mriedem: right which is why I suggested a new one
21:28:01 dansmith mriedem: commented on that live migration abort thing
21:28:29 dansmith I think proper ordering will make that mostly unracy for our purposes without a polling loop
21:35:09 mriedem dansmith: alright this is what i had added, but it overlaps pretty heavily http://paste.openstack.org/show/722381/
21:37:11 dansmith mriedem: I just wanted something that indicates that they have to run them again, if you think what is there now is enough, then that's fine
21:37:19 dansmith I agree your paste adds nothing to what is there,
21:37:28 dansmith I was just thinking about calling it out specifically
21:37:30 dansmith but it's cool
21:38:09 dansmith a couple days ago I was working an internal bug where someone updated the code, reported the fix didn't work
21:38:22 dansmith and turns out they didn't even restart nova-compute to get the new code
21:38:29 dansmith so maybe I'm overly sensitive at the moment
21:39:45 mriedem alright, i'll remove my extra reno in the backport. i figured that most deployment tools will run nova-manage db sync and the online data migrations during an update as well as a major upgrade, since it shouldn't hurt if there is nothing new, but can hurt if you don't run it and there is
21:40:22 mriedem also, thanks for the comments on kevin's patch, makes sense when i read it
21:40:54 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/571317
21:41:47 dansmith I don't think they do actuall
21:41:50 dansmith fwiw
21:42:48 dansmith at least our process for rolling out updates (not upgrades) is basically just yum
21:43:56 cfriesen yeah, we normally wouldn't re-run online data migrations unless it's specifically highlighted.
21:44:17 cfriesen previous fixes have generally fixed things up on accessing the data, and the online data migrations fixed up ones that hadn't been accessed "live"
21:50:36 mriedem alright, well, meh
21:50:45 mriedem i guess people doing updates can run them if needed,
21:50:53 mriedem otherwise people rolling up to those during FFU can get it
21:52:09 openstackgerrit Merged openstack/nova master: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/567878
21:52:15 openstackgerrit Merged openstack/nova master: Make nova service-list use scatter-gather routine https://review.openstack.org/568271
21:52:25 openstackgerrit Merged openstack/nova master: doc: cleanup API guide about instance faults https://review.openstack.org/560178
21:54:07 melwitt interesting, looking at a recent fail of a VolumesBackupTest in the ceph job, it looks like if you issue a delete request twice for the same volume, the second one can fail with a 400 for having not allowed volume status http://logs.openstack.org/53/570653/1/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/c5d03e7/logs/screen-c-api.txt.gz#_2018-05-30_04_41_07_095
21:54:29 melwitt anyone seen that before? is that usual?
21:56:40 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/571320
21:56:49 melwitt smcginnis ^
21:57:40 mriedem melwitt: this? http://logs.openstack.org/53/570653/1/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/c5d03e7/logs/screen-c-api.txt.gz#_2018-05-30_04_41_07_306
21:58:14 mriedem too bad it didn't tell you what the volume status was when it puked
21:58:22 mriedem must be 'deleting' or something?
21:58:41 melwitt yes. that was a second request to delete the same volume. and yeah, also wish it said, I can only assume it was 'deleting' or the like
21:59:37 mriedem is tempest doing the 2nd delete during a cleanup?
21:59:43 melwitt I don't yet know why the teardown of the test is trying to delete an already deleted volume
21:59:50 melwitt yeah, it's the cleanup step that fails
21:59:51 mriedem i.e. first delete is explicit in the test, but not waiting for the volume to actually be gone
21:59:58 melwitt that's my guess but haven't confirmed yet
22:00:08 mriedem because most of tempest adds an automatic cleanup if you create the resource in one of the base test methods
22:00:17 mriedem so the test likely just needs to wait for the volume to be gone,
22:00:22 mriedem then on cleanup, it will handle the 404 and ignore it
22:00:42 melwitt ah, got it
22:01:34 melwitt hm, none of the tests seem to be doing an explicit delete in the test https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_backup.py
22:04:38 melwitt ohhh, wait, this is a cinder in-tree tempest test from ocata, so different than what I just linked
22:09:58 openstackgerrit Merged openstack/nova master: Expand tests for multiple shared resources case https://review.openstack.org/569637
22:17:25 melwitt ah, indeed we have redundant cleanups because the test is adding a cleanup and the base class is also adding a cleanup during self.create_volume https://github.com/openstack/cinder/blob/stable/ocata/cinder/tests/tempest/api/volume/test_volume_backup.py#L77
22:18:54 eandersson Is numa pinning exposed in any api calls?
22:21:59 openstackgerrit Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325
22:22:06 openstackgerrit Jan Gutter proposed openstack/os-vif master: Add multiqueue field to InstanceInfo https://review.openstack.org/570959
22:22:42 melwitt eandersson: do you mean the flavor extra specs that show what numa pinning was requested? it will show that in the server detail GET call and recently will also show it via the flavors API as of this recent bp https://blueprints.launchpad.net/nova/+spec/add-extra-specs-to-flavor-list
22:24:46 eandersson I think case I actually wanted to see the pinning on the compute itself.
22:25:14 eandersson For each VM
22:26:17 eandersson We wrote an ansible playbook for this, and checked each host using that, but would have been nice to be able to get that kind of data from the api as well.
22:26:22 eandersson (as an admin at least)
22:27:11 melwitt I see. I have limited knowledge about it, but maybe cfriesen might know if he's still around today
22:27:30 eandersson In this particular case we figured out that most of our hosts had the network interrupt rx queue pinned to a specific core
22:27:46 eandersson (well the ones that didn't have multiqueue enabled anyway)
22:29:55 mriedem eandersson: there isn't anything in the server details that say where an instance is pinned
22:30:31 mriedem stephenfin: dansmith: went over the numa aware vswitch spec again, i think it's pretty close, just missed some cleanups and other small details https://review.openstack.org/#/c/541290/
22:31:18 mriedem eandersson: there isn't anything in the os-hypervisors API either
22:31:23 eandersson figured as much, but wanted to double check (as the data is available in the db at least)
22:31:32 mriedem NUMA and CPU pinning stuff will eventually be expressed in the placement API
22:31:38 eandersson Nice
22:31:51 mriedem there used to be an os-pci API added back in juno and removed later in i think pike (or maybe older)
22:31:55 mriedem since it never did anything
22:32:43 mriedem eandersson: btw, was it you or someone else from your company that asked about the TypeAffinityFilter being removed during the nova project update session in vancouver?
22:32:48 mriedem or just totally coincidental?
22:33:14 eandersson One of my guys asked about it, but heard that someone else also asked about it
22:33:29 eandersson as well
22:34:12 mriedem i heard someone named their first born "TypeAffinityFilter" they loved it so much
22:34:17 eandersson haha
22:34:45 eandersson To be fair the version in nova stinks, especially since it's based on uuid
22:35:19 mriedem uuid? the one that was in tree was based on the flavor.id primary key
22:35:26 mriedem which is pretty much the reason it was removed
22:35:29 eandersson Yea
22:35:35 eandersson Our version was always based on name, not id
22:35:49 mriedem because i can create flavorid=foo with id=1, delete it and recreate it with the same flavorid but now it's id=2
22:35:54 mriedem and your affinity stuff is all busted
22:36:01 mriedem ok
22:36:03 mriedem well, fyi http://lists.openstack.org/pipermail/openstack-operators/2018-May/015277.html
22:36:45 mriedem https://review.openstack.org/#/c/569247/ will cause a perf penalty on out of tree filters that need to access any fields on HostState.instances values
22:37:03 mriedem so if people (ops at large) are needing a type affinity filter, one should probably be proposed upstream
22:40:03 eandersson btw unrelated but was a feature to allow rebuild to move the VM to a new compute?
22:40:15 eandersson or even restart (similar to how Amazon does it)

Earlier   Later