| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-07 | |||
| 20:46:57 | dansmith | maybe we should start enacting a french work-week to avoid overloading the gate | |
| 20:46:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | |
| 20:46:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | |
| 21:01:53 | radez | ok, I'm back now, sry about that | |
| 21:02:00 | radez | dansmith: mriedem: http://paste.openstack.org/show/736843/ | |
| 21:02:25 | radez | this shows the uuids of the bm nodes and the nova instances that seems to be behaving like this | |
| 21:03:20 | radez | and the nova logs where nova seems to assign the already assigned bm node to the new nova instance | |
| 21:03:51 | radez | let me know if there's more logs I can get to help | |
| 21:04:41 | dansmith | radez: so these nodes should all exist in placement with a single inventory item for the resource/node class, | |
| 21:04:51 | dansmith | which means they should never get past the scheduler with two instances on the same node | |
| 21:05:12 | radez | translation was helpful thx :) | |
| 21:05:31 | radez | [instance: a840afbc-1314-44d6-83b3-c20b790b9322] Claim successful on node 3c3eb5ee-a358-421a-b2e9-7fd9a017ba13 | |
| 21:06:34 | radez | hm, well now I think I've given you the logs that show that the associated ironic node and nova instance are properly associated together | |
| 21:07:42 | mriedem | do the flavors have cpu/ram/disk set to anything !0? | |
| 21:07:44 | radez | oh wait, sry I'm getting my self confused... right so does that log line I just pasted here show that nova is trying to | |
| 21:07:46 | dansmith | radez: what log is this? just nova compute? | |
| 21:08:06 | radez | assign ironic node 3c3eb5ee to nova instance a840afbc? | |
| 21:08:34 | radez | yea nova-compute log | |
| 21:08:43 | dansmith | radez: so probably want to look at the scheduler log | |
| 21:09:00 | dansmith | radez: scheduler is what decides which node it is going to tell compute to use for each instance | |
| 21:10:10 | radez | k lemme find a840afbc in the schedule logs | |
| 21:13:17 | radez | http://paste.openstack.org/show/736844/ | |
| 21:13:43 | openstackgerrit | Merged openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | |
| 21:13:46 | radez | that also shows a840afbc trying to select 3c3eb5ee | |
| 21:14:32 | radez | hypervisor list incase that's helpful? http://paste.openstack.org/show/736845/ | |
| 21:14:33 | melwitt | radez: what version of rocky is this? we had a bug around ironic inventory update in the middle of rocky https://review.openstack.org/593678 but as long as you have 18.0.0.0rc2 or later, you should have the fix for that | |
| 21:14:35 | melwitt | er, at the end of rocky | |
| 21:14:42 | dansmith | radez: can you show more of the log so we can see both? | |
| 21:15:07 | radez | yea, that was just a quick grep, lemme grab more of the scheduler log and check my version | |
| 21:16:54 | dansmith | melwitt: that would only affect nodes in cleaning right? | |
| 21:18:11 | melwitt | dansmith: I don't know, tbh | |
| 21:19:14 | melwitt | I remember there were some ironic driver bugs we fixed near the end of rocky and was curious if those can be ruled out based on what version of rocky they're running | |
| 21:19:43 | dansmith | the only way this should be happening, AFAIK, is if we're not exposing one inventory per node, or somehow not claiming in placement | |
| 21:20:15 | melwitt | yeah, this bug was related to the claiming of a node via setting reserved == total | |
| 21:20:36 | melwitt | and one spot where the newer placement api microversion needed to be set was missed | |
| 21:21:00 | melwitt | so I was thinking that might possible cause a node to not be claimed when it was supposed to. but as for if it only comes up during cleaning, that I didn't know | |
| 21:21:20 | melwitt | *possibly | |
| 21:21:57 | mriedem | radez: what are the values for cpu/ram/disk for the flavors being used for these baremetal nodes? | |
| 21:21:58 | melwitt | I had thought the reserved == total thing was claiming in general and not only for cleaning, but could easily be wrong as I don't know _that_ much about the ironic driver | |
| 21:22:11 | radez | openstack-nova-scheduler-18.0.3 | |
| 21:22:20 | radez | ok, got hte version, sry fighting with docker :/ | |
| 21:22:42 | melwitt | thanks radez. so yeah, the thing I wondered can be ruled out | |
| 21:22:44 | dansmith | melwitt: reserved=total is only for making the node look unschedulable while being cleaned right? | |
| 21:22:55 | dansmith | melwitt: reserved=total does't make any sense when the node is actually allocated | |
| 21:23:30 | mriedem | i'm wondering if you're hitting https://bugs.launchpad.net/nova/+bug/1796920 | |
| 21:23:30 | openstack | Launchpad bug 1796920 in OpenStack Compute (nova) queens "Baremetal nodes should not be exposing non-custom-resource-class (vcpu, ram, disk)" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:23:31 | radez | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'40', u'cpus': u'4', u'capabilities': u'boot_option:local' | |
| 21:23:35 | radez | both look liek that | |
| 21:23:40 | mriedem | yeah so check that bug | |
| 21:23:44 | mriedem | or https://review.openstack.org/#/c/609043/ | |
| 21:23:45 | radez | getting the logs... | |
| 21:23:59 | dansmith | mriedem: yeah, that's the kind of thing I was thinking of | |
| 21:24:05 | melwitt | dansmith: I didn't know that. but what you say makes sense, you're probably right | |
| 21:24:06 | dansmith | mriedem: we consume one CUSTOM_FOO, and then schedule the next one based on there looking like there's ram or something | |
| 21:24:22 | mriedem | i'm very much half assing my involvement here, | |
| 21:24:25 | mriedem | but that's something i'd check out | |
| 21:25:45 | dansmith | I have to leave soon | |
| 21:28:24 | radez | here's more of the scheduler log http://paste.openstack.org/show/736846/ | |
| 21:31:23 | dansmith | radez: okay but that's still just one instance, right? | |
| 21:31:40 | dansmith | however, it looks like whatever host it chose got successfully claimed in placement, | |
| 21:31:50 | dansmith | which can't happen if each host is exposing only one inventory item | |
| 21:32:32 | dansmith | radez: can you correlate the "Attempting to claim.." line with the relevant context in the placement log? | |
| 21:32:52 | radez | placement log is the scheduler log? | |
| 21:32:57 | mriedem | no | |
| 21:33:00 | dansmith | nope, a different service | |
| 21:33:01 | mriedem | placement-api log | |
| 21:33:12 | radez | oh, sec lemme find that log | |
| 21:33:16 | dansmith | that should (I think) show you what is being claimed for the instance | |
| 21:33:28 | dansmith | which really should be only one resource, something like CUSTOM_IRONIC_FOO=1 | |
| 21:34:56 | radez | would that coorelate on the insance uuid? | |
| 21:35:24 | dansmith | I'm not sure what the string looks like, so not sure if it dumps the consumer (instance) uuid or not | |
| 21:35:26 | mriedem | if the flavor has vcpu/ram/disk, and the compute node is reporting vcpu/ram/disk inventory - which it will be still before stein, the claim will be on vcpu/ram/disk as well | |
| 21:35:28 | dansmith | so I would go by date | |
| 21:35:51 | dansmith | mriedem: wait, what? | |
| 21:36:04 | dansmith | I thought it only reported the old resources in some compat situation? | |
| 21:36:06 | radez | kk | |
| 21:37:09 | mriedem | dansmith: no, | |
| 21:37:22 | mriedem | we didn't neuter the ironic driver until stein, right around the ptg | |
| 21:37:26 | dansmith | I think I recoiled in horror the last time we talked about this | |
| 21:37:36 | mriedem | that's why we needed https://review.openstack.org/#/c/609043/ | |
| 21:37:49 | dansmith | so maybe he just doesn't have the resource class in his flavor? | |
| 21:39:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | |
| 21:39:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | |
| 21:39:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop old service version check compat from _delete_while_booting https://review.openstack.org/623589 | |
| 21:39:46 | mriedem | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'40', u'cpus': u'4', u'capabilities': u'boot_option:local' | |
| 21:39:54 | mriedem | those are extra specs? | |
| 21:40:03 | dansmith | no, | |
| 21:40:08 | mriedem | then correct the baremetal flavor doesn't have the custom resource class | |
| 21:40:10 | dansmith | node detaisl Ithink | |
| 21:40:52 | mriedem | radez: have you gone through this? https://docs.openstack.org/ironic/rocky/install/configure-nova-flavors.html | |
| 21:41:02 | mriedem | especially: openstack flavor set --property resources:CUSTOM_BAREMETAL_SMALL=1 my-baremetal-flavor | |
| 21:41:15 | dansmith | and/or.. who created your flavor? | |
| 21:41:16 | radez | http://paste.openstack.org/show/736847/ | |
| 21:41:48 | dansmith | o.O | |
| 21:42:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop old service version check compat from _delete_while_booting https://review.openstack.org/623589 | |
| 21:42:48 | radez | jlibosvar created the flavor, it looks like that custom baremetal small is not on it | |
| 21:43:13 | mriedem | CUSTOM_BAREMETAL_SMALL is an example | |
| 21:43:14 | dansmith | radez: it needs to be equal to whatever is set on the node | |
| 21:43:18 | dansmith | right | |
| 21:43:25 | radez | http://paste.openstack.org/show/736848/ | |