| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 02:34:59 | melwitt | you could do tomorrow morning? | |
| 02:35:12 | sean-k-mooney | i can try this in the morning too. i just need a 2 node vanila devstack install right | |
| 02:35:13 | melwitt | unless you were thinking to fast-approve this tonight | |
| 02:36:26 | mriedem | why would the vif type be unbound? | |
| 02:36:30 | mriedem | shouldn't it be bound to the dest host? | |
| 02:36:35 | mriedem | since we activated it there? | |
| 02:37:13 | sean-k-mooney | mriedem: it is. each host has its own binding now. only one will be in the bound state all the rest will be unbound | |
| 02:38:36 | mriedem | but i think the port in our info cache is not host-aware... | |
| 02:38:39 | mriedem | i need to check | |
| 02:39:56 | mriedem | http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_726935 | |
| 02:40:04 | mriedem | that's where we refresh the info cache in _post_live_migration | |
| 02:40:10 | mriedem | after activating the dest host port binding | |
| 02:40:23 | mriedem | ed": false, "tenant_id": "7dbeedd7076e472091193779ebbcf887", "mtu": 1400}, "id": "1d8de970-331e-46b5-8c7b-574821e891e5", "label": "tempest-LiveMigrationTest-411356071-network"}, "devname": "tap21095ff0-6b", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {}, "address": "fa:16:3e:34:c9:90", "active": false, "type": "unbound", "id": "21095ff0-6bcd-414b-9d6f-b63e03aacb23", "qbg_params": null}] | |
| 02:40:23 | mriedem | [{"profile": {"migrating_to": "ubuntu-xenial-rax-dfw-0001002004"}, "ovs_interfaceid": null, "preserve_on_delete": false, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.10"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"in | |
| 02:40:50 | mriedem | yeah...that's wrong | |
| 02:40:55 | mriedem | it should be bound to the dest host | |
| 02:44:42 | sean-k-mooney | well it was bound shortly before http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_311896 | |
| 02:46:49 | mriedem | yup we hit post-copy callback here and activate the dest host port binding http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_58_561391 | |
| 02:47:34 | mriedem | refresh nw info cache here http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_310738 | |
| 02:48:16 | mriedem | then we get an unplugged vif event from neutron | |
| 02:48:19 | sean-k-mooney | whats happening is liekly that when the ovs neutron agent sees the tap device disapear it is sending an update to notify us the port state has changed on the souce node. | |
| 02:48:19 | mriedem | could be concurrently | |
| 02:48:50 | mriedem | yeah we get the unplugged event and refresh the cache and it's unbound http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_726935 | |
| 02:49:14 | mriedem | post live migrate the dest host port binding is already active http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_974248 | |
| 02:50:01 | mriedem | then we unplug and kablammo | |
| 02:50:36 | mriedem | doesn't help that we route all of these plug/unplug neutron events to the source host only, that's a nova limitation during live migration right now | |
| 02:50:57 | mriedem | and there might be some kind of delay in the state updates or something in the neutron db? | |
| 02:51:09 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix create_all() to replace_all() in comments https://review.openstack.org/586396 | |
| 02:51:16 | mriedem | anyway, i can hack around this a bit i think but kind of sucks | |
| 02:51:53 | sean-k-mooney | mriedem: well there is a delay in the neutron agent sendign the update over the rabbit rpc bus to the neutron-server and then the rest call to nova. | |
| 02:52:37 | mriedem | i just worry the port isn't wired up on the dest or something, but that shouldn't be the case b/c we plug_vifs on the dest host during pre_live_migration now | |
| 02:52:41 | mriedem | it's just inactive until post-copy | |
| 02:52:48 | sean-k-mooney | we could prabably hack in a filter to ignore nay info cache updates where teh vif type is unbound and the port profile containts a migrating_to field | |
| 02:53:11 | mriedem | yeah... | |
| 02:53:49 | mriedem | that would coincide with this http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_310738 | |
| 02:54:04 | sean-k-mooney | mriedem: yes if the pulgin fails in pre_live_migration we bail out early and try another host so at this point the dest networking shoudl be fully set up | |
| 02:55:59 | mriedem | also, if we get the info cache based on what's setup for the dest host, we could have changed vif types, so unplugging on the source could be a different vif type...couldn't it? | |
| 02:56:03 | mriedem | this gets a bit wonky | |
| 02:56:24 | mriedem | we do have an exact copy of the source_vif in the migrate data vifs | |
| 02:56:33 | sean-k-mooney | yes it could have changed. | |
| 02:56:35 | sean-k-mooney | yep | |
| 02:56:49 | sean-k-mooney | the migrate data has everything you need. | |
| 02:57:18 | sean-k-mooney | just look up the vif by the port uuid and unplug or better yet just loop over all the vifs in migrate data instead of instance | |
| 02:57:50 | mriedem | that's kind of what i'm going to do, will hack something up quick and post it then flesh it out more in the morning | |
| 02:57:59 | mriedem | sean-k-mooney: and for the love of toast go to bed | |
| 02:58:11 | bzhao__ | Sorry for a nic break, I have a brief in the neutron log from the link shows. For the failure test instance, seem It works correct in Neutron side. | |
| 02:58:36 | sean-k-mooney | haha its only 4 am. but ya. il be back only in 6-8 hours and ill take a look at it then. nighto/ | |
| 03:01:55 | openstack | Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Confirmed] | |
| 03:01:55 | melwitt | bzhao__: thanks. feel free to add a comment to explain about the neutron side in https://bugs.launchpad.net/neutron/+bug/1783917 see comment #6 | |
| 03:02:39 | bzhao__ | melwitt: Thanks, I will. ;-) | |
| 03:07:32 | mriedem | got a patch, pretty simple, no tests but can be easily added by someone else tonight or in the morning | |
| 03:09:16 | sapd | Hi everyone. I got this error when attach a SR-IOV port to instance http://paste.openstack.org/show/726723/ Please help me | |
| 03:10:35 | mriedem | sapd: read through https://docs.openstack.org/neutron/latest/admin/config-sriov.html and check everything in there | |
| 03:10:54 | melwitt | mriedem: coolness, sounds good | |
| 03:14:12 | sapd | mriedem: yep. I have read it. And follow the guide to config. Everything I setup is correct. Because I already launched an instance using SR-IOV successful. But It did not receive DHCP. So I launched another instance using Openvswitch then add SR-IOV port to the instance. But got above error. | |
| 03:16:57 | openstack | Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Undecided,Expired] | |
| 03:16:57 | melwitt | sapd: looks like the bug has been around for awhile and still not resolved https://bugs.launchpad.net/nova/+bug/1708433 they say you can boot with the port if you pass it during server create, but that attaching port separately is broken | |
| 03:18:17 | melwitt | sapd: what release of nova are you using? | |
| 03:18:32 | sapd | melwitt: I'm using queens version. 17.0.4 | |
| 03:18:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Use source vifs when unplugging on source during post live migrate https://review.openstack.org/586402 | |
| 03:18:35 | mriedem | melwitt: bzhao__: Kevin_Zheng: sean-k-mooney: ^ just needs unit tests | |
| 03:19:03 | melwitt | sapd: okay, I'm going to re-open that bug and mention what version you saw it in. it will need to be worked on | |
| 03:19:13 | Kevin_Zheng | mriedem, got it, just finish reading launchpad report | |
| 03:19:15 | mriedem | ask sahid to look at it | |
| 03:19:19 | mriedem | the sriov bug i mean | |
| 03:19:54 | melwitt | k | |
| 03:20:33 | sean-k-mooney[m] | Melwitt we used ti have an api check at one point to expresly forbid attach sriov port to existing instances. | |
| 03:20:50 | melwitt | hmm, interesting. I wonder what happened to that | |
| 03:21:03 | sapd | melwitt: I'm waiting. | |
| 03:21:12 | melwitt | hah | |
| 03:21:24 | sean-k-mooney[m] | Melwitt im guessing some of artoms changes | |
| 03:23:25 | melwitt | okay, I'll ask him about it | |
| 03:26:04 | openstackgerrit | Merged openstack/os-vif stable/rocky: Add vif_plug_noop to setup.cfg packages https://review.openstack.org/586340 | |
| 03:26:20 | melwitt | hot dog | |
| 03:29:02 | bzhao__ | mriedem: So so quick.... =。= | |
| 03:31:48 | melwitt | I think I'm gonna give up on rechecking the r-3 patches, seems like a pretty high fail rate with the live migration thing | |
| 03:32:10 | melwitt | get the fix sorted in the morning and go from there | |
| 03:48:05 | mriedem | should have tests done pretty soon | |
| 03:56:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use source vifs when unplugging on source during post live migrate https://review.openstack.org/586402 | |
| 03:57:22 | mriedem | alright gang there it is with a test ^ | |
| 04:04:03 | mriedem | and now i'm going to bed | |
| 04:04:04 | mriedem | o/ | |
| 04:04:13 | melwitt | gnite | |
| 04:47:28 | openstackgerrit | Xiaohan Zhang proposed openstack/nova master: compute node local_gb_used include swap disks https://review.openstack.org/585928 | |
| 05:11:45 | vishakha | melwitt : Hi, waiting for your response https://review.openstack.org/#/c/580271/. Thanks | |
| 06:08:33 | openstackgerrit | Vishakha Agarwal proposed openstack/nova master: No change in field 'updated' in server https://review.openstack.org/586446 | |
| 06:25:05 | openstackgerrit | Vishakha Agarwal proposed openstack/nova master: No change in field 'updated' in server https://review.openstack.org/586446 | |
| 06:32:25 | ileixe | Hello again | |
| 06:33:11 | ileixe | Does any body know how to expand APIExtensionBase for pre-processing not for post-processing..? | |
| 06:47:47 | openstackgerrit | Xiaohan Zhang proposed openstack/nova master: compute node local_gb_used include swap disks https://review.openstack.org/585928 | |
| 07:15:26 | openstackgerrit | Chen proposed openstack/nova master: Make nova-manage capable of syncing all cell databases https://review.openstack.org/519275 | |
| 07:33:02 | openstackgerrit | Vishakha Agarwal proposed openstack/nova master: No change in field 'updated' in server https://review.openstack.org/586446 | |
| 07:43:12 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix create_all() to replace_all() in comments https://review.openstack.org/586396 | |
| 08:20:50 | kashyap | Hey folks, I'm hitting a "POST_FAILURE" state for the 'nova-live-migration' CI job; seems like a Zuul problem? | |
| 08:20:55 | kashyap | (For this change: https://review.openstack.org/#/c/567258/) | |
| 08:21:28 | PapaOurs | kashyap: nothing raised by infra AFAIK | |
| 08:21:46 | PapaOurs | kashyap: but maybe you should ask in #openstack-infra ? | |
| 08:21:48 | kashyap | Nod; in the past I've seen channel topic being changed when such errors occurreed. | |
| 08:21:56 | kashyap | PapaOurs: Yep, was just about to check there. | |
| 08:22:31 | kashyap | When I look into the log, it's the SSH failing | |