| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 18:05:47 | efried | gmann: We can help educate the author :P | |
| 18:05:48 | gmann | but ok to wait till Direction:Approved decision | |
| 18:07:34 | gmann | commented on review the same. | |
| 18:20:03 | efried | melwitt: are you now owning nova-audit? (https://review.opendev.org/#/c/693226/) | |
| 18:20:46 | melwitt | efried: I didn't want to but I think the answer is technically yes because dansmith lost interest | |
| 18:21:28 | efried | melwitt: well, I ask because we're at spec freeze, so you need to get a couple cores on board, ahem, today if it's going to happen in ussuri. | |
| 18:22:26 | bauzas | efried: melwitt: FWIW, this is related https://review.opendev.org/#/c/670112/ | |
| 18:22:43 | efried | it is? | |
| 18:22:54 | bauzas | technically, it's just a rename | |
| 18:23:08 | bauzas | but the intent of the spec is to provide a new specific command AFAICR | |
| 18:23:20 | bauzas | this change ^ would just be another subcommand | |
| 18:24:34 | melwitt | efried: yeah, I don't think that's going to happen. operators are interested but the spec didn't attract review from cores thus far and I don't think I could wrangle two that would not be considered part owners by the end of today | |
| 18:25:30 | efried | melwitt: if "tomorrow" would make the difference, I'm fine with that. Or do you just want me to defer? | |
| 18:26:28 | melwitt | bauzas: the intent of the spec is to organize all of the heal commands in one place and make them runnable as a daemon service so that they automatically heal your cloud periodically | |
| 18:27:27 | bauzas | oh missed the last part | |
| 18:27:31 | bauzas | gtk | |
| 18:28:01 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.opendev.org/552924 | |
| 18:28:10 | bauzas | efried: ^ | |
| 18:28:12 | efried | ack | |
| 18:29:18 | bauzas | anyway, bailing out | |
| 18:29:21 | melwitt | efried: I guess yeah if you'll give it till tomorrow, I'll send some email and see if anyone's willing to review. if there's not interest after that, then punt it | |
| 18:29:34 | Sundar | dansmith: If https://review.opendev.org/#/c/673735/37/nova/conductor/manager.py@524 is not the right place to delete ARQs on a reschedule, do you have any suggestion for a better plac? I could do it in the callers. | |
| 18:29:51 | efried | melwitt: ack. I'm adding it (with other open specs) to today's meeting agenda, if you want to drum up interest there. | |
| 18:29:55 | dansmith | melwitt: efried it seems highly unlikely that anything would get implemented in U either way, so I'm not sure it's worth that | |
| 18:30:16 | dansmith | I thought we were supposed to be trying to reduce the number of things we approved that aren't likely to make it, | |
| 18:30:36 | dansmith | but it kinda seems like we're doing the same ol' kind of behavior | |
| 18:31:18 | dansmith | Sundar: do it where it needs to be done, not inside a thing called something else.. so yes, wherever that's called from that is the right place | |
| 18:32:37 | melwitt | dansmith, efried: well, I could implement it quickly/dumbly (I'm imagining just moving the commands and adding a service) but getting review would be another story. worst case it sits there ready to go for V if ppl can't review in time. so, I dunno | |
| 18:33:21 | efried | dansmith: Yes, intend to do a sweep of Definition:Approved blueprints "soon" to decide which of those we can/should defer. | |
| 18:34:06 | efried | "spec freeze" -- no more definition approvals -- is what's happening now. | |
| 18:37:54 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201 | |
| 18:45:32 | efried | johnthetubaguy: Save me looking, did you squash the fup? | |
| 18:45:38 | efried | (abandon if so) | |
| 18:51:31 | dking_desktop | I'm attempting to troubleshoot why I get the "No valid host was found." error when attempting to create a baremetal server, and just found this when I enabled debugging for the nova-scheduler: compute_status_filter request filter added forbidden trait COMPUTE_STATUS_DISABLED | |
| 18:52:07 | dking_desktop | Could that be the reason why I'm not able to find a valid host? How would I troubleshoot this further? | |
| 19:03:49 | efried | dking_desktop: We always add that trait. It's only going to have an effect if the compute host is exposing that trait. You can check with a command like | |
| 19:03:49 | efried | openstack resource provider trait list $host_uuid | |
| 19:04:01 | efried | (I may not have the syntax exactly right -- see the docs) | |
| 19:07:00 | dking_desktop | I'm using Ironic if that helps. I don't see anything for "openstack resource". Is it "openstack service provider list"? | |
| 19:08:11 | dking_desktop | Oh, maybe "openstack baremetal node trait list" | |
| 19:09:45 | dking_desktop | efried: I tried "openstack baremetal node trait list <UUID>", but that gave no results. Is that the problem? | |
| 19:10:14 | efried | dking_desktop: You need to install the osc-placement plugin to get the 'resource provider' subcommands | |
| 19:10:28 | efried | pip install osc-placement (or equivalent for your distro) | |
| 19:10:43 | efried | COMPUTE_STATUS_DISABLED isn't a trait that ironic itself would know about. | |
| 19:12:59 | dking_desktop | Odd. I get "Operation or argument is not supported with version 1.0; requires at least version 1.6" | |
| 19:14:45 | dking_desktop | I wonder what software that refers to. The osc-placement package should be 1.8.0. | |
| 19:16:18 | dking_desktop | python-openstackclient is 4.0.0. Is there another way to check? It's good to know that isn't specifically an Ironic thing. However, my regular VMs work fine. It's only the baremetal nodes causing me trouble. | |
| 19:19:47 | dking_desktop | Oh, I add that to the command line. Okay, I can run that, but no mention of the above trait. | |
| 19:21:38 | dking_desktop | http://paste.openstack.org/show/789544/ | |
| 19:25:13 | dking_desktop | efried: I notice that the above output doesn't show nearly as much information as I see for my compute node. Would the problem be that there's just not any information there about the CPU, etc.? | |
| 19:26:41 | efried | dking_desktop: sorry, yes, you need to specify a microversion for almost every OSC command with placement, as OSC defaults to 1.0 and very little in placement worked at that microversion. You can use an environment variable if you'd rather not have to think about it with every command. | |
| 19:27:34 | efried | dking_desktop: When you say "as I see for my compute node", you mean a libvirt host? | |
| 19:27:58 | efried | is that what compute1.stack1 is? | |
| 19:30:45 | dking_desktop | Correct | |
| 19:30:53 | efried | Next thing to look at is the inventory of your ironic node vs. the flavor you're trying to deploy. | |
| 19:30:53 | efried | If you're seeing COMPUTE_STATUS_DISABLED in play, your control plane is at least at Train, which means your node is supposed to be at least at Stein, by which time we had cut ironic over to reporting single-unit custom resource classes. | |
| 19:31:28 | efried | something like | |
| 19:31:28 | efried | openstack resource provider inventory list (or maybe show) $node_uuid | |
| 19:31:28 | efried | should show you that. | |
| 19:31:38 | dking_desktop | I'm using train. | |
| 19:32:45 | efried | okay, so you might want to make life easier with | |
| 19:32:45 | efried | export OS_PLACEMENT_API_VERSION=1.36 | |
| 19:32:45 | efried | (I think I'm spelling that var name right) | |
| 19:32:45 | efried | Then you won't have to add --os-placement-api-version with every command. | |
| 19:32:50 | dking_desktop | Okay, that shows me the resource class and a few other pieces of info: http://paste.openstack.org/show/789545/ | |
| 19:34:06 | efried | Great. So the flavor you're using should be asking for resources:CUSTOM_BAREMETAL_RESOURCE_CLASS=1 | |
| 19:34:08 | efried | is it? | |
| 19:35:05 | dking_desktop | Yes, it is: http://paste.openstack.org/show/789546/ | |
| 19:35:18 | efried | the fact that your trait list didn't show COMPUTE_STATUS_DISABLED, and your inventory showed reserved=0, is two out of the three things that should make this node eligible for scheduling. | |
| 19:36:17 | dking_desktop | That sounds good. Any idea why I would be getting "No valid host was found." | |
| 19:36:28 | efried | okay, I need to go check whether you need to do something special the ram and disk (set them to zero) but I think those should be ignored. Meanwhile, the last of the three ^ things is to make sure there's no allocation present. | |
| 19:36:45 | efried | openstack resource provider allocation list (or show?) $node_uuid | |
| 19:37:34 | dking_desktop | That's empty. | |
| 19:37:53 | efried | Okay. Easier than me finding that code would be checking your placement logs. | |
| 19:38:25 | efried | Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS | |
| 19:40:27 | efried | Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work. | |
| 19:41:32 | efried | dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 19:42:27 | efried | (not just VCPU, MEMORY_MB and DISK_GB too) | |
| 19:42:33 | dking_desktop | efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/ | |
| 19:42:52 | dking_desktop | Ah, so the VCPUs could be the problem? Let me see if I can update that. | |
| 19:43:58 | efried | Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)? | |
| 19:44:13 | efried | your baremetal node's inventory doesn't have any of those resources. | |
| 19:44:46 | efried | Those are being fed in from your base flavor's disk/ram/vcpus | |
| 19:45:07 | efried | So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query. | |
| 19:46:02 | efried | And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places | |
| 19:46:45 | efried | dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work. | |
| 19:47:38 | dking_desktop | I'm having trouble getting it to accept 0. | |
| 19:48:25 | efried | dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB} | |
| 19:48:28 | efried | ... to zero | |
| 19:49:02 | efried | That will cause the scheduler to ignore the base vcpus/ram/disk values. | |
| 19:49:08 | efried | ...which can be set to whatever. | |
| 19:50:41 | dking_desktop | Ah. Yes, that's what's in the article, so that makes sense. Let me try that. | |
| 19:55:49 | dking_desktop | Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying. | |
| 19:56:30 | efried | Okay, good deal. | |
| 19:59:16 | dking_desktop | I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.' | |
| 19:59:35 | dking_desktop | Do you know where in the logs I might see where that failed? | |
| 20:01:37 | dking_desktop | Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name. | |
| 20:01:58 | efried | dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced. | |
| 20:02:48 | dking_desktop | Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID. | |
| 20:06:48 | dking_desktop | efried: Thank you very much for your help! I wouldn't have been able to make progress today without it. | |