Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-12
15:26:09 sean-k-mooney jangutter: you leave it bound to the host kernel driver at all times
15:26:09 slaweq mriedem: it is this fix which will finally fix neutron pgsql periodic job, right?
15:26:23 jangutter sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround.
15:26:46 jangutter sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver.
15:26:53 sean-k-mooney jangutter: it could yes but it wont today meaning we would need a new libvirt release and need to check for it in the code
15:27:16 jangutter sean-k-mooney: yep, tell me about it, I know all about people wanting to propagate libvirt "fixes".
15:28:01 sean-k-mooney i think this would be somthing that the sriov nic agent could resonalby handel too
15:28:10 sean-k-mooney well maybe not
15:28:15 sean-k-mooney that could be racy
15:29:02 sean-k-mooney it would have to be nova/libvirt to avoid races with other vms booting
15:29:09 adrianc sean-k-mooney: there is no FLR involved during detach/attach, the driver unbinds from the guest, and binds on the host whether or not it triggers an FLR is something the driver decides AFAIK
15:29:39 jangutter adrianc: yep, FLR is mandatory on iommu passthrough I think, but not for driver bind/unbind
15:30:32 sean-k-mooney well it depend on if the vf is bound to say ixgbeVf when its detach or if it stays with vfio-pci
15:30:54 jangutter sean-k-mooney: yeah.
15:31:59 adrianc would have been easier if setting VF mac address via ip link on the VF idx would have synced the mac address on the corresponding VF netdev :)
15:32:00 sean-k-mooney in any case however in macvtap mode we aparent dont update the driver the vf is bound to or trigger a vf reset
15:32:33 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055
15:32:33 sean-k-mooney adrianc: that silly talk why would a hardware/driver implementer ever do a thing like that :P
15:32:34 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740
15:32:39 gibi cdent: ^^
15:32:48 cdent roger
15:33:16 mriedem slaweq: yeah
15:33:22 jangutter sean-k-mooney, adrianc: years ago we had looked at how to propagate MAC config from host to the VF and the conclusion we came to was that at the FLR is the safest way to do so. I don't think there was an async event able to propagate that info from the device to the driver without either a reset or rebind.
15:33:42 slaweq mriedem: thx for taking care of it :)
15:33:47 sean-k-mooney adrianc: speacking of ip link ther eis no way to reset it via ip tools right.
15:34:17 jangutter sean-k-mooney, adrianc: the only way I know to trigger an FLR is via setpci...
15:35:02 sean-k-mooney jangutter: i was hoping there would be a standard sysfs file we could echo too but i guess not
15:35:18 aspiers mriedem: if I understand the code right, the config drive disk mapping is built alongside the other disk mappings prior to instance launch (even though the config drive itself seems to be built late via post_xml_callback). so it's not hotplugged post-launch. Also I think it will either use "ide" or "virtio" bus (depending on whether iso9660 or vfat) and therefore we don't anticipate any problems with
15:35:19 aspiers SEV. Nevertheless we're testing to make sure
15:35:22 jangutter sean-k-mooney: there might be these days, though.
15:35:40 sean-k-mooney jangutter: it would still likely be vendor specific
15:36:15 jangutter sean-k-mooney: very, I believe our driver only recently could propagate MAC on rebind. Previously you _had_ to get an FLR.
15:36:20 sean-k-mooney adrianc: im just getting around to setting up that test env now but i will try a few different things and let you know
15:37:20 sean-k-mooney should i put this in the spec by the way. im inclined to say no as this is too technical/hardware specific
15:37:31 adrianc sean-k-mooney: ack, let me know if you can find a nicer WA
15:38:01 adrianc well i dont think its related to the spec, its a long existing problem IMO and not related to live-migration
15:38:19 sean-k-mooney we need to adress this in the code obviously but i think code commetns/dev docs
15:38:47 sean-k-mooney adrianc: i guess you could consider it a cold migration bug
15:38:48 adrianc there are additional gaps with macVtap like multiple port binding so
15:40:38 adrianc sean-k-mooney: it can happen also without LM being involved, easiest is to boot a VM on node A delete and boot it on node B
15:41:02 adrianc the mac will be reused and you hit the issue
15:41:14 sean-k-mooney adrianc: reusing the same neutorn port so that the mac and vlan remain the same
15:41:49 adrianc adrianc: yes, using the same neutron port
15:42:13 adrianc sean-k-mooney: ^
15:42:22 sean-k-mooney ya ok that is proably ground enought to just treat this as a seperate bug and maybe backport the fix
15:43:53 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to cleanup instance dir during evac with rbd https://review.openstack.org/618478
15:45:41 lyarwood mriedem / mdbooth / melwitt ; ^ thanks again for the reviews, I appreciate just how ugly this is but hopefully this is btw but with downstream customers asking for this to be fixed back in Newton I'm being forced into a corner here tbh.
15:46:02 lyarwood thanks weechat--
15:46:53 sean-k-mooney lyarwood: what wrong with weechat++
15:47:53 lyarwood ^ the above mess, I was trying to edit the line and it decided to send it out, fun times.
16:02:27 mriedem melwitt: did you see my question in your spec for configurable max volumes? https://review.openstack.org/#/c/597306/1/specs/stein/approved/conf-max-attach-volumes.rst@44
16:05:01 melwitt mriedem: I did, sorry. the immediate answer was "not 100% sure, need to check my notes again". it was "fun" tracing it all and I can't remember whether bfv vs attach additional end up using the same code path that leads to the limit of 26. I need to double check it in order to reply and haven't gotten to it yet
16:09:09 mriedem ok, but i'm assuming even if it's not the same code path, bfv would need a check somewhere for the config option to limit the number of volumes attached to an instance during spawn
16:10:21 mriedem it might have to be here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1598
16:10:34 mriedem or a new driver method called from _prep_block_devices
16:10:53 mriedem because you don't want to create or attach more than the limit before you get to driver.spawn just to have it fail and have to roll those all back
16:20:00 melwitt mriedem: yeah, I guess the context is in my WIP patch for the implementation I've put the check at a lower level where all of the code paths end up https://review.openstack.org/616777 there's a default impl in compute/utils that all drivers other than libvirt use, and libvirt has its own override
16:20:36 melwitt I don't have any func test coverage yet to verify it all works right, so I'm in process of writing those
16:20:53 melwitt but if you want to take a quick look to point out anything bad that jumps out, I'd appreciate that
16:27:24 mriedem yeah that won't cut it for bfv
16:27:27 mriedem i'll leave a comment
16:28:03 melwitt thanks
16:38:56 openstackgerrit Merged openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347
16:53:46 openstackgerrit Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370
16:53:47 openstackgerrit Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384
16:56:47 moshele jangutter: hi
16:57:09 jangutter moshele: hi, long time no chat!
16:57:58 moshele jangutter: yes indeed
16:58:30 moshele jangutter: I am not sure I understand you comments
16:58:44 jangutter moshele: sorry, it's a bit late in the year....
16:58:45 moshele jangutter: https://review.openstack.org/#/c/619920/
16:58:47 dansmith mriedem: was your lazy-load patch trying to get rid of the billion "Lock placement_client acquired" messages on startup?
17:00:02 jangutter moshele: the first question is possibly the easiest: on the NIC you have two ports, B and C. I would have thought that it makes sense specifying "connect B and C" to the bridge?
17:02:52 mriedem dansmith: yes
17:02:59 mriedem https://review.openstack.org/#/c/623246/
17:03:01 dansmith mriedem: yeah that's pretty bad
17:03:11 mriedem it now removes the lock as well on jay's request
17:06:08 dansmith mriedem: just noticed while reviewing the logs forthis: https://review.openstack.org/#/c/623284/4
17:06:19 dansmith mriedem: which does now select the right rpc version once computes get started
17:06:23 dansmith which was not happening before
17:07:22 sean-k-mooney adrianc: so yes if i just do "sudo ifconfig enp1s17f5 0" i does not fix the issue either
17:07:39 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensure that allocated PF matches the used PF https://review.openstack.org/623543
17:08:01 mriedem dansmith: which job was that message from?
17:08:10 mriedem because on tempest-full nova-api starts up before there are any computes
17:08:12 dansmith mriedem: tempest-full
17:08:17 sean-k-mooney adrianc: sorry "sudo ifconfig enp1s17f5 down"
17:08:37 dansmith mriedem: right you see it fail once first, and then once computes are started, it finds the version
17:08:58 dansmith mriedem: before that "all cells" patch, you saw the failure on start and it never finds them after that
17:09:03 mriedem ah yeah http://logs.openstack.org/84/623284/4/check/tempest-full/23568c2/controller/logs/screen-n-api.txt.gz#_Dec_11_20_56_12_002867
17:09:15 mriedem http://logs.openstack.org/84/623284/4/check/tempest-full/23568c2/controller/logs/screen-n-api.txt.gz#_Dec_11_21_00_06_274322
17:09:25 dansmith yep
17:09:59 mriedem huh wonder if i made devstack switch upgrade_levels=auto in stein or rocky...
17:10:16 dansmith it's been a while I thought, so I'm surprised this didn't break us
17:10:23 dansmith but we also haven't had many rpc versions recently
17:10:25 mriedem https://review.openstack.org/#/c/550108/
17:10:34 mriedem we went to auto upgrade levels in rocky
17:11:01 dansmith well, we'll definitely want to backport this
17:11:04 mriedem how far back do you think we need this to go?
17:11:07 mriedem pike?
17:11:28 dansmith pike was when you could actually have multiple cells right? so yeah, there

Earlier   Later