| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 00:26:47 | dansmith | apparently when we create that the first time, we also put those files in there (TIL) | |
| 00:27:08 | dansmith | but we can't re-create it later, which is the #2 part of the spec problem section | |
| 00:27:10 | dansmith | so, | |
| 00:27:33 | dansmith | you're using the API part, and the config drive part, but not the actual injection thing which is the most smelly bit | |
| 00:27:35 | rm_work | ha, right, which is funny because the #2 "problem" is actually WHY we chose this method | |
| 00:27:45 | dansmith | fine, but whatever | |
| 00:27:55 | rm_work | ok so if #2 was the bad part, and that's just not done anymore... why is the first part being removed? | |
| 00:28:17 | dansmith | #2 is related to the API not the really bad part | |
| 00:28:41 | rm_work | err | |
| 00:28:46 | rm_work | sorry, PART 1 and 2 | |
| 00:29:21 | dansmith | the API part is bad because it takes arbitrary files and then kind keeps track of them, until a rebuild or something and then we lose them | |
| 00:29:25 | rm_work | per "1. The API (personality files) by which people provide this data" and "the #2 part is the really nasty bit, which has been disabled by default, and which we _actually_ want to be rid of" | |
| 00:29:36 | rm_work | hmmm | |
| 00:29:49 | dansmith | the #2 part is the libvirt injection partition thing | |
| 00:29:52 | dansmith | sorry | |
| 00:29:56 | dansmith | eff, | |
| 00:30:04 | rm_work | yeah | |
| 00:30:05 | dansmith | this straightening isn't going well | |
| 00:30:21 | rm_work | so right, #2 part (libvirt) isn't even done anymore | |
| 00:30:25 | rm_work | now it puts things into config-drvie | |
| 00:30:28 | rm_work | which is ... fine? | |
| 00:30:41 | rm_work | it's just that nova then loses track of that data, which you consider bad (but we don't) | |
| 00:31:01 | rm_work | (and it has worked that way for a while?) | |
| 00:31:19 | dansmith | okay, you know, it's after 5pm and I'm getting more frustrated here, so I'm just going to go | |
| 00:31:26 | rm_work | kk | |
| 00:31:36 | rm_work | prolly just discussing at the PTG is best | |
| 00:32:21 | Ileixe | Hello guys | |
| 00:33:37 | Ileixe | Recently I implement custom hooking code for server create api in nova-api by hook api. | |
| 00:33:59 | johnsom | My take away. There was some nasty bit taking files and making some strange partition at boot. We aren't using that and never have. Then there is the bit that takes files, stashes them in the config drive and cloud-init drops them in the guest filesystem. This what we use. However to remove the partition stuff the config drive part got removed too | |
| 00:35:16 | Ileixe | Oh sorry there was converstation in now. Never mind. I ask later | |
| 00:35:39 | rm_work | Ileixe: we are ... wrapped up on that :P | |
| 00:35:40 | rm_work | it's fine | |
| 00:35:41 | rm_work | lol | |
| 00:37:20 | Ileixe | Thanks rm_work :) just simple qeustion. I found hook api was deprecated, and the api was the right thing for my logic, so i wonder what replace hook api | |
| 00:57:41 | melwitt | argh, looks like we have a new gate failure as of today | |
| 00:57:55 | melwitt | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Unsupported%20VIF%20type%20unbound%20convert%20'_nova_to_osvif_vif_unbound'%5C%22%20AND%20tags:screen-n-cpu.txt&from=7d | |
| 00:58:58 | melwitt | unless it's only the numa-aware-vswitches patches that are affected... looking closer | |
| 01:04:45 | melwitt | it's hitting several of the numa-aware-vswitches patches but is hitting other patches as well. started very recently | |
| 01:07:55 | openstackgerrit | Xiaohan Zhang proposed openstack/nova master: compute node local_gb_used include swap disks https://review.openstack.org/585928 | |
| 01:44:17 | mriedem | melwitt: i was noticing those randomly the last couple of weeks | |
| 01:44:21 | mriedem | unless it's major, just recheck | |
| 01:45:03 | melwitt | mriedem: oh, logstash was claiming it started today. and I was wondering if it might be related to https://review.openstack.org/522537 | |
| 01:45:43 | melwitt | I've rechecked the numa patches at least twice because of it so far. maybe it's a coincidence. I'll keep trying to recheck | |
| 01:46:42 | mriedem | hmm, yeah it might be, mostly hitting on the live migration and multinode jobs | |
| 01:46:48 | mriedem | which is where that is turned on | |
| 01:47:00 | mriedem | well that would be...awesome | |
| 01:47:09 | mriedem | can you report a neutron bug? | |
| 01:47:29 | melwitt | that patch landed at 13:00 (my time) which coincides with the logstash start of hits | |
| 01:48:07 | melwitt | mriedem: can do. was just writing it up for nova not realizing it's neutron. will copy it over and open for neutron | |
| 01:48:59 | mriedem | it could be either | |
| 01:49:01 | mriedem | just add both | |
| 01:49:09 | melwitt | oh, right. we can do that | |
| 01:49:23 | mriedem | Kevin_Zheng: fyi, might need to see if zhaobo can investigate this ^ | |
| 01:49:37 | mriedem | mlavalle is already gone for the day | |
| 01:49:54 | Kevin_Zheng | ACK, I will ask him | |
| 01:50:06 | mriedem | melwitt: there would be an easy way to disable it in nova if needed | |
| 01:50:13 | melwitt | k | |
| 01:50:18 | mriedem | and then could be tracked as an rc bug (it will need to be an rc bug anyway) | |
| 01:50:24 | mriedem | rather than revert | |
| 01:51:23 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Fix problem description number in deprecate file injection spec https://review.openstack.org/586385 | |
| 01:51:31 | mriedem | i'm also going to fast approve ^ b/c of the confusion i saw in the backscroll | |
| 01:55:17 | Kevin_Zheng | mriedem, could you provide a error log? | |
| 01:55:44 | dansmith | mriedem: way ahead of you | |
| 01:55:51 | Kevin_Zheng | mriedem, never mind, Igot it | |
| 01:57:17 | melwitt | mriedem: https://bugs.launchpad.net/neutron/+bug/1783917 | |
| 01:57:17 | openstack | Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Undecided,New] | |
| 01:57:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: document user_data length restriction https://review.openstack.org/586388 | |
| 01:57:26 | melwitt | Kevin_Zheng ^ | |
| 01:57:47 | Kevin_Zheng | Thanks | |
| 01:57:57 | mriedem | i'll push up an e-r and nova wip patch and then i have to run i think | |
| 01:58:07 | melwitt | oh, I'm not 100% sure it makes live migration "fail", I meant to change that to "raises" | |
| 01:59:35 | mriedem | e-r query https://review.openstack.org/#/c/586389/ | |
| 01:59:41 | mriedem | it fails | |
| 01:59:55 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Live%20migration%20failed%5C%22%20AND%20message%3A%5C%22Unsupported%20VIF%20type%20unbound%20convert%20'_nova_to_osvif_vif_unbound'%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 02:00:05 | melwitt | although yeah, all the logstash hits containing the message are build failures | |
| 02:00:20 | melwitt | bah *changes it back* | |
| 02:01:22 | melwitt | cool, thanks for adding the e-r query | |
| 02:07:37 | sean-k-mooney | so im going to sleep now but http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_01_083831 looks like its happening because we are calling unplug on the source node after we have activated the binding on the dest | |
| 02:08:19 | melwitt | sean-k-mooney: thanks. so maybe something we need to adjust given the use of the new binding API? I dunno | |
| 02:08:46 | melwitt | I'll add your comment to the bug | |
| 02:09:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Temporarily disable port binding flows for live migration https://review.openstack.org/586391 | |
| 02:09:48 | mriedem | ^ is an option for temporarily disabling this while debugging a fix | |
| 02:10:03 | mriedem | i hope it doesn't have to come to that, but would understand if it's causing a lot of failures | |
| 02:10:41 | sean-k-mooney | melwitt: i can try and reporduce this in the morning. we proably need to stor the original vif type and use that to constuct the vif object and use that or do the unplug on the host. | |
| 02:11:19 | melwitt | mriedem: okay, we'll decide what to do in the morning tomorrow when other people are around | |
| 02:11:48 | openstackgerrit | Merged openstack/nova-specs master: Fix problem description number in deprecate file injection spec https://review.openstack.org/586385 | |
| 02:12:00 | mriedem | yeah the error is from unplugging vifs in _post_live_migration which happens on the source, | |
| 02:12:41 | mriedem | https://github.com/openstack/nova/blob/2afc5fed1f60077e7ff0b9e81b64cff4e4dbabfc/nova/compute/manager.py#L6581 | |
| 02:12:53 | mriedem | right before that, | |
| 02:12:54 | mriedem | https://github.com/openstack/nova/blob/2afc5fed1f60077e7ff0b9e81b64cff4e4dbabfc/nova/compute/manager.py#L6572 | |
| 02:13:01 | mriedem | we activate the port bindings for the dest host | |
| 02:13:47 | melwitt | ah, I see | |
| 02:13:56 | sean-k-mooney | mriedem: yep that will deactivaate all other port bindings for that port meaning it will be in the unbound state on the sorce host | |
| 02:13:59 | melwitt | so just flip that? | |
| 02:14:02 | mriedem | https://github.com/openstack/nova/blob/2afc5fed1f60077e7ff0b9e81b64cff4e4dbabfc/nova/network/neutronv2/api.py#L2534 | |
| 02:14:26 | mriedem | i didn't know we couldn't unplug a deactivated port... | |
| 02:14:32 | melwitt | I wonder how it doesn't fail 100% of the time | |
| 02:14:36 | mriedem | melwitt: race | |
| 02:14:39 | mriedem | apparently | |
| 02:14:40 | melwitt | ah | |