| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-30 | |||
| 09:34:54 | sean-k-mooney | i just grabbed the instance uuid from the html report of the first failure then when to the n-cpu log and searched for a trasback with that uuid | |
| 09:35:44 | kashyap | Yep, thank you for the (non-null) pointer | |
| 09:42:11 | kashyap | sean-k-mooney: How can I reproduce this locally? | |
| 09:42:18 | kashyap | sean-k-mooney: By running one of the Tempest tests? | |
| 09:43:20 | sean-k-mooney | kashyap: tempest will work if you have a 2 node devstack running yes | |
| 09:43:35 | sean-k-mooney | https://review.openstack.org/#/c/567258/10/nova/virt/libvirt/guest.py | |
| 09:43:50 | sean-k-mooney | you removed a null check on domain_xml | |
| 09:43:54 | kashyap | sean-k-mooney: Afraid, need to set up afresh, and I'm doing last-minute stuff before I head out. | |
| 09:43:58 | sean-k-mooney | could that be the issue | |
| 09:44:40 | kashyap | sean-k-mooney: Aah, good pointer; let me look! | |
| 09:45:59 | kashyap | sean-k-mooney: But 'domain_xml' is no longer needed | |
| 09:46:10 | sean-k-mooney | its now destination_xml | |
| 09:46:25 | sean-k-mooney | which you default to None and never check | |
| 09:47:25 | kashyap | Hmm, true. I need to put such a check for 'destination_xml' | |
| 09:48:23 | sean-k-mooney | or you chould do 'params['destination_xml'] = destination_xml or ""' but really if you dont have a dest xml it should proably be the source xml? | |
| 09:48:34 | sean-k-mooney | or just not add it to the params dict | |
| 09:49:04 | kashyap | Hmm, that last option sounds cleaner: if 'destination_xml' is None, then don't add it to the 'params' dict | |
| 09:49:09 | sean-k-mooney | im not sure what the preconditions on migrateToURI3 are | |
| 09:50:11 | sean-k-mooney | in theory you should be able to test this with a functional test locally but i that is non trivail to write correctly | |
| 09:50:52 | kashyap | Yeah | |
| 09:52:24 | kashyap | sean-k-mooney: Meanwhile, here is the API definition: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainMigrateToURI3 | |
| 09:53:35 | kashyap | The Python bindings use the 'destination_xml', and the C equivalent: https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_MIGRATE_PARAM_DEST_XML | |
| 09:57:14 | sean-k-mooney | kashyap: im not seeing anything that explcitly eays the destination_xml is required. so maybe just omitting it for the params is correct | |
| 09:57:32 | kashyap | sean-k-mooney: Yeah, so how about: http://paste.openstack.org/show/726818/ | |
| 09:58:18 | sean-k-mooney | am you are always leaving it out in that case. | |
| 09:59:11 | sean-k-mooney | just do if destination_xml: | |
| 09:59:16 | sean-k-mooney | params['destination_xml'] = destination_xml | |
| 10:00:52 | kashyap | Ah, let me try | |
| 10:03:08 | kashyap | sean-k-mooney: Err, the 'else' thing had the wrong snippet | |
| 10:03:14 | kashyap | Let me re-pase what I have: | |
| 10:04:31 | kashyap | sean-k-mooney: http://paste.openstack.org/show/726820/ | |
| 10:08:54 | kashyap | Thanks for the discussion here! | |
| 10:16:19 | s10 | https://bugs.launchpad.net/nova/+bug/1322921 | |
| 10:16:19 | openstack | Launchpad bug 1322921 in OpenStack Compute (nova) "hypervisor-servers command always search by wildcard as '%hypervisor_hostname%'" [Wishlist,Opinion] | |
| 10:16:35 | s10 | Could this bug be changed to another status? | |
| 10:16:52 | s10 | Because this bug is awful. Execute: nova host-evaucate-live cmp2 | |
| 10:17:17 | s10 | Result: all hosts cmp2* will be live evacuated. | |
| 10:22:50 | kashyap | sean-k-mooney: Okay, I'll go with this: | |
| 10:23:07 | kashyap | params = {} | |
| 10:23:07 | kashyap | params['bandwidth'] = bandwidth | |
| 10:23:07 | kashyap | params['migrate_disks'] = migrate_disks | |
| 10:23:07 | kashyap | params['migrate_uri'] = migrate_uri | |
| 10:23:07 | kashyap | if destination_xml: | |
| 10:23:09 | kashyap | params['destination_xml'] = destination_xml | |
| 10:53:58 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258 | |
| 10:54:34 | kashyap | sean-k-mooney: When you have a minute, please have a gander ^. I've addressed the 'destination_xml' and both your consistency nits | |
| 11:04:18 | sean-k-mooney | kashyap: taking a look now. was on 1:1 call | |
| 11:09:21 | kashyap | sean-k-mooney: Thank you. I normally don't expect instant turn-around :-) | |
| 11:44:23 | openstackgerrit | Brin Zhang proposed openstack/nova stable/pike: Add unshelve instance error info to fault table https://review.openstack.org/583426 | |
| 11:46:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enhance service restart in functional env https://review.openstack.org/512552 | |
| 11:46:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1724172 in the functional test env https://review.openstack.org/512553 | |
| 11:46:39 | openstack | bug 1724172 in OpenStack Compute (nova) "Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor" [Medium,In progress] https://launchpad.net/bugs/1724172 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 11:46:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cleanup evacuated instances not on hypervisor https://review.openstack.org/512623 | |
| 11:47:47 | sean-k-mooney | kashyap: looking at the zuul jobs we are still hitting the same error http://logs.openstack.org/58/567258/11/check/nova-live-migration/93ce085/logs/screen-n-cpu.txt.gz#_Jul_30_11_39_23_991580 | |
| 11:48:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: fixtures: Track attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 11:48:41 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 11:58:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 12:07:08 | kashyap | sean-k-mooney: Hmm, Zul was so quick? | |
| 12:07:50 | kashyap | Guess it's still running, isn't it? | |
| 12:07:56 | sean-k-mooney | kashyap: no but you can get the live logs from zuul.openstack.org | |
| 12:08:22 | sean-k-mooney | put 567258 into the filter | |
| 12:08:56 | sean-k-mooney | you can see the nova-live-migration sub job has completed so you can get to the logs | |
| 12:09:27 | sean-k-mooney | if the job is still running you can connect to a telnet stream of the console but that is less useful | |
| 12:12:24 | kashyap | sean-k-mooney: Got a URL of the live log? | |
| 12:12:32 | kashyap | Sorry for the sluggish responses, /me is shuffling b/n kitchen & IRC | |
| 12:12:55 | kashyap | sean-k-mooney: /me goes to put that ID in the filter | |
| 12:13:04 | kashyap | TIL | |
| 12:13:54 | sean-k-mooney | most people dont know you can do this with the gate but its been like this for years. module some minor refinement with zuul v3 | |
| 12:14:18 | sean-k-mooney | amy well here is a live log of your neutron-multinode subjob http://zuul.openstack.org/stream.html?uuid=6bc29a9c0ffb44339b36ee4b42b2b1b2&logfile=console.log | |
| 12:14:21 | kashyap | Yeah, I see the URL here: http://logs.openstack.org/58/567258/11/check/nova-live-migration/93ce085/ | |
| 12:14:55 | sean-k-mooney | kashyap: ya the migration subjob has finished so it has published everything to the normal log server | |
| 12:18:09 | openstackgerrit | Vieri proposed openstack/osc-placement master: Add image link in README.rst https://review.openstack.org/586839 | |
| 12:18:57 | kashyap | sean-k-mooney: Nod; thanks for the education | |
| 12:19:39 | kashyap | sean-k-mooney: Hmm, indeed, still the same TypeError | |
| 12:19:39 | kashyap | http://logs.openstack.org/58/567258/11/check/nova-live-migration/93ce085/logs/screen-n-cpu.txt.gz#_Jul_30_11_39_24_000563 | |
| 12:20:18 | sean-k-mooney | im currently trying to get a 2 node dev env up and running | |
| 12:21:04 | kashyap | Oh, you mean, you might be able to run the Tempest test locally? | |
| 12:21:05 | sean-k-mooney | when i get it stacked i can try and pull this down and see whats going on. i want to do some testing of the numa aware vswich code locally so i have to do a multinode deploy anyway | |
| 12:21:16 | sean-k-mooney | kashyap: ya i have the hardware to do that | |
| 12:21:30 | sean-k-mooney | just not the devstack cloud deployed | |
| 12:22:10 | kashyap | Most excellent, would be curious | |
| 12:24:07 | kashyap | Hmm, maybe something else is None | |
| 12:25:39 | kashyap | sean-k-mooney: Oh: 'migrate_disks' and 'migrate_uri' are None by default! | |
| 12:25:45 | sean-k-mooney | kashyap: there used to be an if check on mirate_uri also. | |
| 12:25:49 | sean-k-mooney | yes | |
| 12:26:31 | sean-k-mooney | i would be surprised if it was migrate_disk however as it is normally a list | |
| 12:27:08 | sean-k-mooney | the none type failure was expecting a sting so im guessing migrate_uri is more likely | |
| 12:28:41 | sean-k-mooney | we might want to add a debug line to print each of the values. | |
| 12:29:05 | kashyap | sean-k-mooney: I think that's pretty much it, actually. | |
| 12:29:18 | kashyap | To be consistent, I'll do this: | |
| 12:29:21 | kashyap | params = {} | |
| 12:29:22 | kashyap | params['bandwidth'] = bandwidth | |
| 12:29:22 | kashyap | if destination_xml: | |
| 12:29:22 | kashyap | params['destination_xml'] = destination_xml | |
| 12:29:22 | kashyap | if migrate_disks: | |
| 12:29:24 | kashyap | params['migrate_disks'] = migrate_disks | |
| 12:29:26 | kashyap | if migrate_uri: | |
| 12:29:29 | kashyap | params['migrate_uri'] = migrate_uri | |
| 12:29:46 | sean-k-mooney | ya that should work or at least help | |
| 12:30:13 | sean-k-mooney | im going to need to do an os reinstall on one of the nodes so it will be a while before i have a working env. | |