Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-04
15:04:44 naichuans efried: right, one pgug has one id, but it would support more than one vgpu type, each of them has different resource amount
15:04:50 bauzas jaypipes: wait? while I haven't replied to your comments, I think I fixed given your opinions :)
15:05:25 bauzas jaypipes: https://review.openstack.org/#/c/557065/5..6/specs/rocky/approved/vgpu-rocky.rst ;)
15:06:24 jaypipes bauzas: ok. I was just expecting you to come back at me with words :)
15:07:04 bauzas jaypipes: heh, sometimes I'm just revisiting my patches if the comment is good for me :)
15:08:22 efried naichuans: I'm having a hard time understanding that. But okay, so the code we're looking at is meant to handle #2. What are we doing to handle #1?
15:08:38 jaypipes bauzas: good to know! :)
15:09:55 bauzas jaypipes: sorry about the confusion :)
15:10:38 jaypipes bauzas: no worries!
15:12:03 bauzas jaypipes: cdent: efried: btw. not sure when the hangout will be ?
15:12:11 bauzas for the upgrade issues ?
15:12:16 efried bauzas: I don't think we discussed it further.
15:12:24 bauzas k
15:12:31 efried I think tomorrow morning (for some U.S. value of "morning")
15:12:31 naichuans efried: Eric, I can't catch your minde... in truth we have only one way to configure vgpu type on OpenStack side, change `CONF.devices.enabled_vgpu_type` by modify nova.conf. So I think #1 and #2 comes from the same reason, nova.conf changed. could you explain more?
15:12:48 bauzas lemme look at my agenda
15:12:56 cdent that's my understanding as well. we were going to figure it out here. hopefully with some input from edleafe
15:13:28 efried naichuans: Okay, enabled_vgpu_types=foo. Create an instance with a vGPU. Change enabled_vgpu_types=bar, restart compute service. Try to create an instance with a vGPU. It should fail. Where's the code that makes it fail?
15:14:14 bauzas efried: FWIW about the discussion we had with naichuans, I think we agreed on saying that if an operator modifies nova.conf for changing the types, then meh, it's their responsibility to make sure there is not existing instances
15:14:20 bauzas efried: we can document it
15:14:46 efried bauzas: Hm, okay. It's certainly possible to detect this and throw an error.
15:15:05 naichuans No, it would not failed, we changed the `reserve` when you restart(after modify conf), the schduling to this rp would be impossile
15:15:52 naichuans impossible
15:17:19 openstackgerrit melanie witt proposed openstack/nova stable/pike: zuul: Move legacy jobs to project https://review.openstack.org/572130
15:17:27 bauzas naichuans: you mean, we *should* do that ?
15:20:19 naichuans efried: Although the customer may want to configure to a new vgpu type, but we stop to support fureture allocation. bauzas: I also think it is a mistake by OpenStack operator. Fix by us, or just let scheduling fail and let operator to mange it, both OK for me, my code now take the first choice.
15:20:45 efried scheduling won't fail, though.
15:20:50 naichuans Yes
15:21:16 naichuans It would looking for another rp on a anther host
15:21:38 efried naichuans: Right now you're looping over the vgpu types. If you restart the compute service with a different configured type, you'll ignore the old one entirely and just start using the new one.
15:21:43 openstackgerrit melanie witt proposed openstack/nova stable/ocata: zuul: Move legacy jobs to project https://review.openstack.org/572132
15:22:14 bauzas efried: naichuans: so we would need to check the existing allocations when init_host() then
15:22:23 bauzas just for that
15:23:30 efried But this also dovetails with my other issue, which is that you need to be detecting and deleting resource providers for other vgpu types than the one (or in future, ones) you're currently managing.
15:25:05 bauzas efried: yup, that'z the problem
15:25:33 efried bauzas: I'm saying with the naming convention being used, you can totally do that.
15:26:42 naichuans efried: currently yes, we detect the configure changes by checking inventory changes. Like bauzas said, we also can communicate with hypervisor to check previous allocated vgpu instances
15:27:26 bauzas naichuans: we could pass the allocations to the virt driver with init_host()
15:27:36 bauzas exactly like we do for spawn() or other virt methods
15:27:47 mnaser melwitt, dansmith: https://bugs.launchpad.net/nova/+bug/1581977
15:27:48 bauzas naichuans: but for the moment, I think we can just document this
15:27:48 openstack Launchpad bug 1581977 in OpenStack Compute (nova) "Invalid input for dns_name when spawning instance with .number at the end" [Undecided,Invalid]
15:29:42 naichuans bauzas: yes, we can do it. could we use it to determin vgpu type current used? Looks difficult
15:29:59 naichuans previously used
15:30:13 bauzas naichuans: I did that for reboot :)
15:30:34 bauzas I'm just looking up the existing instances and check whether the mdev is there or not
15:30:38 bauzas and just recreate it
15:32:07 naichuans bauzas: currently, we defined vgpu type in nova.conf. these types is same with the types in hpyervisor record, so it may easier to compare with hypervisor
15:35:18 naichuans and vgpu type is transparent for placement
15:40:19 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec for volume multiattach enhancements https://review.openstack.org/552078
15:43:32 naichuans bauzas: efried: It is late at night in China now, I'm going to sleep. I will keep close touch with you in the email.
15:44:00 openstackgerrit Dan Smith proposed openstack/nova master: Use oslo.messaging per-call monitoring https://review.openstack.org/566696
15:44:18 efried naichuans: Thanks for helping me understand this stuff. I think with some updates on the comments we might be there now.
15:44:47 naichuans efried: Np, thank you very much for the review :)
15:53:10 openstackgerrit MultipleCrashes proposed openstack/nova master: Retry decorator fix for autoscale delete https://review.openstack.org/570370
17:39:52 karimull Hi ...need your reviews on https://review.openstack.org/#/c/569498/
18:17:59 mriedem karimull: i suggest putting that into the runways review queue https://etherpad.openstack.org/p/nova-runways-rocky
18:43:12 efried karimull: Reviewed.
18:43:57 efried karimull: But I agree with mriedem - this definitely needs eyes from the likes of jaypipes, dansmith, mriedem.
18:51:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix typo in enable_certificate_validation config option help https://review.openstack.org/572185
18:55:17 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081
19:47:53 openstackgerrit Dan Smith proposed openstack/nova master: Change consecutive build failure limit to a weigher https://review.openstack.org/572195
20:06:21 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Remove device from volume attach requests (spec) https://review.openstack.org/452546
20:06:28 mriedem dansmith: finally got around to updating this old spec ^
20:18:40 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081
20:43:57 lyarwood efried: https://bugs.launchpad.net/nova/+bug/1775075 - afternoon, if you have a second could you take a look at this bug and the comment I left on the original change? https://review.openstack.org/#/c/510947/1/nova/context.py@121
20:43:58 openstack Launchpad bug 1775075 in OpenStack Compute (nova) "EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api" [Undecided,New]
20:45:32 efried lyarwood: looking...
20:45:47 lyarwood efried: thanks :)
20:48:21 efried lyarwood: Okay, I think this may come down to what version of ksa exists on pike, and whether os-service-types is available.
20:50:02 lyarwood efried: 3.1.0 iirc
20:50:40 lyarwood efried: adding volumev2 back in nova/context.py is enough in my local env FWIW
20:50:57 lyarwood efried: on the n-api host that is
20:51:19 cdent is tomorrow spec sprint again?
20:51:34 efried lyarwood: I'm fine with that as a resolution; but we should probably bring mordred in on this issue, since he was basically leading me through a lot of the service catalog work.
20:51:43 melwitt cdent: yes
20:51:43 efried cdent: yes, afaik
20:51:49 cdent cool, thanks
20:52:08 melwitt I'll send another email to remind to the dev ML late today
20:52:30 lyarwood efried: ack, I'll push something to stable/queens now
20:52:40 lyarwood efried: thanks :)
20:53:02 efried lyarwood: os-service-types is not in requirements in pike, and I think that's probably the source of the problem.
20:54:26 lyarwood efried: hmm okay, AFAICT the Queens n-api was just stripping the volumev2 endpoints from the request context it was sending to the Pike compute, I'm not sure how os-service-types could help there but I've never really touched this area before.
20:55:37 efried lyarwood: The endpoints themselves should be getting picked up correctly on the queens side. If we're sending actual endpoints across the wire, that should work fine.
20:56:05 efried lyarwood: Because the queens side will have os-service-types, which means it'll find the correct endpoints (whether they're called volumev2 or whatever) for block-storage.
20:57:05 mordred efried: aroo?
20:59:46 efried mordred: See https://bugs.launchpad.net/nova/+bug/1775075 and https://review.openstack.org/#/c/510947/1/nova/context.py@121
20:59:47 openstack Launchpad bug 1775075 in OpenStack Compute (nova) "EndpointNotFound raised by Pike n-cpu when running alongside Queens n-api" [Undecided,New]
21:00:59 efried mordred: What lyarwood is finding is that when he adds 'volumev2' back in on the queens side, it fixes the problem. Which I don't quite understand why doing that on the *queens* side would fix. Because that guy has os-service-types and the right ksa... oh, unless this is one of those ksa bugs we've since fixed, where endpoint lookup isn't happening right.
21:02:10 lyarwood efried: so the Pike side is looking for the cinderv2 endpoint in the catalog stashed in the request context
21:02:28 lyarwood efried: the cinderv3 endpoint is there correctly, but without the cinderv2 endpoint it just fails
21:02:37 mordred efried: probably. when you say "queens side" and "pike side" - what does that mean here?
21:02:51 lyarwood mordred: queens n-api, pike n-cpu
21:02:56 mordred gotcha
21:03:49 mordred yah - I think you'll need the volumev2 in there until everything is on queens - I think efried is correct about there being missing fixes
21:05:11 lyarwood oh I forgot to say that our P n-cpu's have catalog_info=volumev2:cinderv2:internal in nova.conf forcing them to look for cinderv2
21:06:19 efried ah
21:06:31 efried well then yeah
21:06:38 lyarwood yeah my bad, the default is cinderv3 right?
21:06:48 efried so is this a bug in the code, or is this something that should be fixed via conf on site?

Earlier   Later