Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-30
21:37:30 dansmith but it's cool
21:38:09 dansmith a couple days ago I was working an internal bug where someone updated the code, reported the fix didn't work
21:38:22 dansmith and turns out they didn't even restart nova-compute to get the new code
21:38:29 dansmith so maybe I'm overly sensitive at the moment
21:39:45 mriedem alright, i'll remove my extra reno in the backport. i figured that most deployment tools will run nova-manage db sync and the online data migrations during an update as well as a major upgrade, since it shouldn't hurt if there is nothing new, but can hurt if you don't run it and there is
21:40:22 mriedem also, thanks for the comments on kevin's patch, makes sense when i read it
21:40:54 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/571317
21:41:47 dansmith I don't think they do actuall
21:41:50 dansmith fwiw
21:42:48 dansmith at least our process for rolling out updates (not upgrades) is basically just yum
21:43:56 cfriesen yeah, we normally wouldn't re-run online data migrations unless it's specifically highlighted.
21:44:17 cfriesen previous fixes have generally fixed things up on accessing the data, and the online data migrations fixed up ones that hadn't been accessed "live"
21:50:36 mriedem alright, well, meh
21:50:45 mriedem i guess people doing updates can run them if needed,
21:50:53 mriedem otherwise people rolling up to those during FFU can get it
21:52:09 openstackgerrit Merged openstack/nova master: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/567878
21:52:15 openstackgerrit Merged openstack/nova master: Make nova service-list use scatter-gather routine https://review.openstack.org/568271
21:52:25 openstackgerrit Merged openstack/nova master: doc: cleanup API guide about instance faults https://review.openstack.org/560178
21:54:07 melwitt interesting, looking at a recent fail of a VolumesBackupTest in the ceph job, it looks like if you issue a delete request twice for the same volume, the second one can fail with a 400 for having not allowed volume status http://logs.openstack.org/53/570653/1/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/c5d03e7/logs/screen-c-api.txt.gz#_2018-05-30_04_41_07_095
21:54:29 melwitt anyone seen that before? is that usual?
21:56:40 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/571320
21:56:49 melwitt smcginnis ^
21:57:40 mriedem melwitt: this? http://logs.openstack.org/53/570653/1/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/c5d03e7/logs/screen-c-api.txt.gz#_2018-05-30_04_41_07_306
21:58:14 mriedem too bad it didn't tell you what the volume status was when it puked
21:58:22 mriedem must be 'deleting' or something?
21:58:41 melwitt yes. that was a second request to delete the same volume. and yeah, also wish it said, I can only assume it was 'deleting' or the like
21:59:37 mriedem is tempest doing the 2nd delete during a cleanup?
21:59:43 melwitt I don't yet know why the teardown of the test is trying to delete an already deleted volume
21:59:50 melwitt yeah, it's the cleanup step that fails
21:59:51 mriedem i.e. first delete is explicit in the test, but not waiting for the volume to actually be gone
21:59:58 melwitt that's my guess but haven't confirmed yet
22:00:08 mriedem because most of tempest adds an automatic cleanup if you create the resource in one of the base test methods
22:00:17 mriedem so the test likely just needs to wait for the volume to be gone,
22:00:22 mriedem then on cleanup, it will handle the 404 and ignore it
22:00:42 melwitt ah, got it
22:01:34 melwitt hm, none of the tests seem to be doing an explicit delete in the test https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_backup.py
22:04:38 melwitt ohhh, wait, this is a cinder in-tree tempest test from ocata, so different than what I just linked
22:09:58 openstackgerrit Merged openstack/nova master: Expand tests for multiple shared resources case https://review.openstack.org/569637
22:17:25 melwitt ah, indeed we have redundant cleanups because the test is adding a cleanup and the base class is also adding a cleanup during self.create_volume https://github.com/openstack/cinder/blob/stable/ocata/cinder/tests/tempest/api/volume/test_volume_backup.py#L77
22:18:54 eandersson Is numa pinning exposed in any api calls?
22:21:59 openstackgerrit Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325
22:22:06 openstackgerrit Jan Gutter proposed openstack/os-vif master: Add multiqueue field to InstanceInfo https://review.openstack.org/570959
22:22:42 melwitt eandersson: do you mean the flavor extra specs that show what numa pinning was requested? it will show that in the server detail GET call and recently will also show it via the flavors API as of this recent bp https://blueprints.launchpad.net/nova/+spec/add-extra-specs-to-flavor-list
22:24:46 eandersson I think case I actually wanted to see the pinning on the compute itself.
22:25:14 eandersson For each VM
22:26:17 eandersson We wrote an ansible playbook for this, and checked each host using that, but would have been nice to be able to get that kind of data from the api as well.
22:26:22 eandersson (as an admin at least)
22:27:11 melwitt I see. I have limited knowledge about it, but maybe cfriesen might know if he's still around today
22:27:30 eandersson In this particular case we figured out that most of our hosts had the network interrupt rx queue pinned to a specific core
22:27:46 eandersson (well the ones that didn't have multiqueue enabled anyway)
22:29:55 mriedem eandersson: there isn't anything in the server details that say where an instance is pinned
22:30:31 mriedem stephenfin: dansmith: went over the numa aware vswitch spec again, i think it's pretty close, just missed some cleanups and other small details https://review.openstack.org/#/c/541290/
22:31:18 mriedem eandersson: there isn't anything in the os-hypervisors API either
22:31:23 eandersson figured as much, but wanted to double check (as the data is available in the db at least)
22:31:32 mriedem NUMA and CPU pinning stuff will eventually be expressed in the placement API
22:31:38 eandersson Nice
22:31:51 mriedem there used to be an os-pci API added back in juno and removed later in i think pike (or maybe older)
22:31:55 mriedem since it never did anything
22:32:43 mriedem eandersson: btw, was it you or someone else from your company that asked about the TypeAffinityFilter being removed during the nova project update session in vancouver?
22:32:48 mriedem or just totally coincidental?
22:33:14 eandersson One of my guys asked about it, but heard that someone else also asked about it
22:33:29 eandersson as well
22:34:12 mriedem i heard someone named their first born "TypeAffinityFilter" they loved it so much
22:34:17 eandersson haha
22:34:45 eandersson To be fair the version in nova stinks, especially since it's based on uuid
22:35:19 mriedem uuid? the one that was in tree was based on the flavor.id primary key
22:35:26 mriedem which is pretty much the reason it was removed
22:35:29 eandersson Yea
22:35:35 eandersson Our version was always based on name, not id
22:35:49 mriedem because i can create flavorid=foo with id=1, delete it and recreate it with the same flavorid but now it's id=2
22:35:54 mriedem and your affinity stuff is all busted
22:36:01 mriedem ok
22:36:03 mriedem well, fyi http://lists.openstack.org/pipermail/openstack-operators/2018-May/015277.html
22:36:45 mriedem https://review.openstack.org/#/c/569247/ will cause a perf penalty on out of tree filters that need to access any fields on HostState.instances values
22:37:03 mriedem so if people (ops at large) are needing a type affinity filter, one should probably be proposed upstream
22:40:03 eandersson btw unrelated but was a feature to allow rebuild to move the VM to a new compute?
22:40:15 eandersson or even restart (similar to how Amazon does it)
22:40:24 eandersson ever discussed?
22:41:09 eandersson We are trying to find a strategy to make it easier for our customers when we need to bring a compute down for maintenance
22:42:49 mriedem eandersson: that's the evacuate API
22:43:14 eandersson Yea - I guess it's the same thing.
22:43:22 mriedem that's after nova-compute is down though
22:43:30 mriedem https://docs.openstack.org/nova/latest/admin/evacuate.html
22:44:13 eandersson Our ideal scenario would be that if we want to do maintenance on compute X, we mark it as "going down in 7 days", and anyone that hits rebuild on their VM on that host gets moved to a new compute
22:44:24 eandersson and after 7 days they would just automatically get rebuilt and moved
22:44:48 mriedem i think you could build that on top of existing APIs in an external service
22:45:00 eandersson Yea - that is our general plan of action for now
22:45:03 mriedem you can force a compute service down so new things can't go to it but the existing VMs continue to run
22:45:27 mriedem https://developer.openstack.org/api-ref/compute/#update-forced-down
22:45:46 mriedem so you can force it down to evacuate things off it,
22:45:50 mriedem or disable it so new things can't go to it
22:45:53 eandersson Nice
22:46:24 mriedem rebuild to the same host will work while the service is disabled, it won't auto-migrate it
22:46:45 mriedem non-admins can only move during things like shelve_offload/unshelve and resize
22:47:08 mriedem your external thing could mark the compute as disabled/down, and then after 7 days, trigger a cold migration
22:47:17 mriedem or live if you support that
22:48:13 mriedem actually if you disable the compute, shelve all the instances, unshelve them, you get the same thing you're asking for
22:48:38 mriedem with some additional glance snapshot traffic, but otherwise the same as rebuild
23:20:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330
23:52:06 openstackgerrit karim proposed openstack/nova master: Handle rebuild of instances with image traits https://review.openstack.org/569498

Earlier   Later