Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-24
13:37:19 mdbooth mriedem: Briefly back to the topic of evacuation tests, am I right in thinking we probably won't be able to write a full job which does failure injection? If not, I don't think I don't think I'm blocked on that. If we *can* I'm happy to write it if you point me at the right repo and an example to crib from.
13:37:56 mnaser sean-k-mooney: was rax doing x86 on ppc via software or hardware emulation?
13:38:18 mdbooth 'example' wouldn't have to be that similar, just enough to orient myself. I haven't touched the top-level CI jobs before.
13:38:20 sean-k-mooney mnaser: i belive xen allows cross arch hardware emulation
13:40:17 mdbooth mriedem: The ability to assert success in the non-failure case might still be interesting, I guess. It would essentially be a regression test.
13:40:27 mnaser sean-k-mooney: i am not too familiar with xen but i would assume that wouldn't be hardware assisted virt (i.e. using VT-x in intel, etc)
13:40:36 mnaser but that's something i dunno much about so i could be entirely wrong
13:41:33 sean-k-mooney mnaser: well xen does use vt-x normally if you are running x86 on x86
13:41:40 sean-k-mooney as does kvm
13:42:04 mnaser right, so this would be the same as software virtualization which can be really slow, which i'm trying to avoid having anyone end up with
13:42:22 mnaser and all of this is optional, if you set none of those settings, it won't affect you. but if you run a cloud where you want to make sure all VMs are hardware virt'd, then you can do that
13:47:11 sean-k-mooney mnaser: its not so much that this change was done that i object to its that the semantics of the existing api were changed without a spec.
13:47:49 mnaser sean-k-mooney: but if you don't configure this very optional value, there will zero behavior change
13:48:36 sean-k-mooney mnaser: yes and if you had those configure before an you upgrade then your cloud may now be broken if they do not have the same semantics
13:48:56 mnaser sean-k-mooney: you couldn't possible have them configured because they don't even exist before?
13:49:13 mnaser s/possible/possibly
13:49:25 sean-k-mooney the image properties
13:50:35 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Remove unnecessary default parameter https://review.openstack.org/564451
13:58:48 mriedem mdbooth: what type of failure needs to be injected? like delete the disk before starting the evacuatoin?
13:59:06 mdbooth mriedem: Any failure in spawn on the destination
13:59:08 mriedem can't really do that from tempest, but could if run from a script
13:59:21 mdbooth I think I chose plug_vifs in my functional test
13:59:36 dansmith mriedem: so on your gate job,
14:00:02 dansmith mriedem: just force-downing the service isn't going to be enough I think, since it will still be running, potentially mucking with allocations and state correction, etc right?
14:00:07 mdbooth Just need to assert that if we fail in spawn in the dest, we only cleanup what we're supposed to and don't delete shared data.
14:00:18 mdbooth Whether that's shared instance dir or shared block.
14:00:20 dansmith you have to actually kill the n-cpu process and force-down it to avoid the wait
14:00:54 dansmith (well, not mucking with allocations until it's deleted maybe anymore, but screaming at least)
14:01:06 dansmith but I think it's more legit if you stop it
14:01:25 mriedem dansmith: force down is what we do in functional tests
14:01:30 mriedem that's all the API checks for
14:01:38 dansmith mriedem: I know it will make evac work,
14:02:05 dansmith but the n-cpu for the "down" one will still be running and doing stuff, which isn't how we describe evac as working properly in real life
14:02:13 dansmith we say you have to fence it
14:02:50 mriedem in a multinode job the post-test script runs on the control node which has an n-cpu process on it which we could kill
14:03:06 mriedem we'd have to force the server to that host so it's the source
14:03:37 stephenfin mriedem, melwitt, dansmith: btw, given that sahid immediately replied yesterday, I figured it wasn't necessary to investigate the queue size thing any further myself. Sorry if I picked that up wrong
14:03:46 mriedem stephenfin: it's fine
14:04:11 mriedem after looking at the existing release note and config option help for the rx/tx queue size stuff i think they were good enough to say that it was very limited to certain types of vifs
14:04:27 mriedem and i think what moshe is trying to do is make it work for something else
14:05:37 stephenfin mriedem: aight, good to hear
14:06:06 melwitt .
14:06:35 mriedem mdbooth: btw yes i'm also interested in simply have regression testing for evac in a real env
14:06:40 mriedem *having
14:07:28 mriedem melwitt: that last rocky change is in the gate
14:09:51 melwitt mriedem: I saw, thank you for getting that. the master change didn't land until late last night after I went to sleep
14:13:59 mriedem mdbooth: btw do you still care about this? https://review.openstack.org/#/c/572790/
14:38:15 mriedem dansmith: a couple of thoughts on batch size floor in https://review.openstack.org/#/c/593131/
14:41:48 dansmith mriedem: thanks will look shortly
14:57:22 mriedem and i'm done with that stack \o/
15:02:35 mdbooth mriedem: I do. Came up yesterday, in fact.
15:02:54 mdbooth mriedem: I don't think I had any strong objections there?
15:03:47 mriedem i'm pretty sure you wanted to +1 it
15:04:28 mdbooth Just re-reading my last review. Sounds like I was prevaricating rather than objecting.
15:06:12 mdbooth Ah... the attach_mode thing.
15:13:46 mdbooth mriedem: So there's attach_mode and access_mode
15:14:23 mdbooth It seems like we only *use* access_mode, but that code is checking attach_mode
15:23:34 openstackgerrit Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211
15:23:42 openstackgerrit Claudiu Belu proposed openstack/nova master: WIP: replace spec with autospec https://review.openstack.org/557299
15:29:11 mdbooth mriedem: Who's a friendly cinder person?
15:31:34 mriedem for attachment record stuff it would be jgriffith
15:33:54 mdbooth I've hopefully added the right one.
15:34:24 giblet_off I will be back on Tuesday
15:38:56 mdbooth dansmith: Is your no timeouts messaging thing in Queens, btw? Or backportable to Queens?
15:46:23 mriedem it is not in queens
15:46:27 mriedem and not really backportable
15:46:35 mriedem as that would require a min required version of oslo.messaging
15:46:39 mriedem and we don't bump mins on libs in stable
15:47:21 mdbooth mriedem: ack, thanks.
15:47:58 mdbooth mriedem: Means I can't promise a fix to a customer in our next stable release :/
15:48:16 sean-k-mooney mdbooth: without it could you just set the time out really high to get a similar effect?
15:48:25 mdbooth sean-k-mooney: Yeah.
15:49:53 mdbooth sean-k-mooney: I still need to look way harder at the impact, anyway.
15:50:09 mriedem mdbooth: you can fork whatever you want
15:50:35 mdbooth mriedem: Well I would, but dansmith won't let me.
15:51:06 mdbooth I was just going to rewrite in Go.
15:53:16 mdbooth In all seriousness, though: pretty sure you were joking, but in case not we don't do that. At least not in any incompatible way.
15:54:06 mdbooth If we can't do it, that's the way it is.
16:14:51 openstackgerrit Stephen Finucane proposed openstack/nova master: doc: Note NUMA topology requirements for numa-aware-vswitches https://review.openstack.org/596393
16:16:24 mdbooth melwitt: Do you remember anything about either adding or removing refresh_conn_info from hard reboot?
16:16:54 mdbooth It's vaguely ringing a bell, but I can't find any evidence of it.
16:17:13 melwitt mdbooth: no, I never understood much around that. lyarwood would be the one to ask
16:17:49 melwitt might be a WIP patch from him or something? not sure
16:18:23 openstackgerrit Merged openstack/nova stable/rocky: Correct the release notes related to nova-consoleauth https://review.openstack.org/595890
16:18:32 openstackgerrit Merged openstack/nova master: tests: Move mocking to setUp https://review.openstack.org/595802
16:20:50 mdbooth melwitt: I think the change I'm thinking of is when we added the call to destroy() which calls cleanup() and therefore does other stuff like detaching volumes. We don't seem to refresh_conn_info, though
16:21:21 mdbooth mriedem: Is refresh_conn_info still relevant in the new world order?
16:21:43 melwitt mdbooth: ah yeah, the "redo all the things" change
16:21:50 mdbooth melwitt: Yeah.
16:22:02 melwitt didn't have anything to do with refresh_conn_info tho (as you saw)
16:22:38 mdbooth I'm trying to come up with a recovery recommendation for this pre_live_migration timeout.
16:23:43 mdbooth The bug is: we called cleanup on the source when the dest timed out, even though it continued to run. The result is that the instance still has all volumes connected on the source where it continues to run, but some bdms have conn_info for source and others for dest.
16:24:05 mdbooth So really we want some operation which would cause a refresh_conn_info.
16:24:29 mdbooth Options seem to be cold and live migrations.
16:24:38 mdbooth Wonder if reboot ought to be another.
16:25:09 mriedem mdbooth: refresh_conn_info with the new attachment record flow isn't the same as how things worked with the old flow
16:25:50 smcginnis melwitt: Looks like that last stable/rocky patch merged, and you were right that the hash needs to be updated.
16:26:02 mdbooth mriedem: I'll take a look.
16:26:03 melwitt smcginnis: doing it right now, thanks
16:26:35 mriedem mdbooth: https://review.openstack.org/#/c/584648/20/nova/scheduler/client/report.py@2055 the difference is in the old flow, we'd always create a new connection to the storage backend which gives us a fresh connection_info from cinder,

Earlier   Later