| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-07 | |||
| 19:09:46 | mriedem | *ishmael | |
| 19:11:27 | mriedem | dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug | |
| 19:11:32 | mriedem | http://status.openstack.org/elastic-recheck/#1800472 | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568 | |
| 19:45:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981 | |
| 19:48:29 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144 | |
| 19:57:32 | cfriesen | is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately? | |
| 19:58:27 | cfriesen | never mind, found "openstack compute service list" | |
| 20:10:51 | dansmith | mriedem: see comment | |
| 20:10:58 | dansmith | +3 on the code part | |
| 20:18:38 | radez | dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks | |
| 20:19:06 | radez | this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants | |
| 20:19:09 | dansmith | radez: on master or rocky? | |
| 20:19:15 | radez | rocky | |
| 20:19:28 | radez | when we schedule the first node it deploys just fine | |
| 20:19:41 | dansmith | okay just to be clear.. nova with compute_driver=ironic, right? | |
| 20:20:12 | radez | yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me | |
| 20:21:04 | dansmith | just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic | |
| 20:22:14 | radez | sry, you're correct we're using nova to schedule and ironic node | |
| 20:22:23 | dansmith | okay, sorry, continue :) | |
| 20:25:21 | radez | gathering my details here to try and depict this correctly | |
| 20:25:46 | radez | so we deploy the first node and that goes swimmingly. comes active we can connect to it. | |
| 20:25:52 | radez | this is in one tenant | |
| 20:26:40 | radez | in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node | |
| 20:27:01 | radez | lemme gather a few logs that depict this | |
| 20:27:07 | dansmith | are you sure it's selecting the same *node* and not the same nova-compute? | |
| 20:27:35 | dansmith | because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes | |
| 20:28:14 | dansmith | I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant | |
| 20:28:46 | dansmith | also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus | |
| 20:29:06 | radez | I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs | |
| 20:29:35 | radez | fair enough, just laying out the context incase that was helpful | |
| 20:29:47 | dansmith | okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw | |
| 20:29:53 | dansmith | ack, yep, just letting you know | |
| 20:31:25 | dansmith | radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :) | |
| 20:31:44 | dansmith | also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P | |
| 20:32:24 | radez | lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help! | |
| 20:33:17 | mriedem | what's good for the goose is good for the gander | |
| 20:34:00 | dansmith | radez: okay, also keep in mind it's friday afternoon so probably not many hours left in the day (and I'm sure you're ahead of the rest of us in EST) :D | |
| 20:42:01 | dansmith | taco consumption complete | |
| 20:43:01 | mriedem | dansmith: replied on the reno thing - i'm ok with dropping that if it would just cause more concern than it's worth | |
| 20:43:18 | mriedem | that compat was really kind of out the window since pike i think | |
| 20:43:21 | mriedem | when the api was made cells aware | |
| 20:43:59 | dansmith | mriedem: yeah, so I'd just drop it if you're cool with it | |
| 20:44:13 | dansmith | hah | |
| 20:44:23 | mriedem | i'm very cool | |
| 20:44:24 | dansmith | zuul says 135h remaining on my patch I submitted at like 7am this morning | |
| 20:45:04 | mriedem | https://review.openstack.org/#/c/602804/ queued for at least 36 hours, hit a job timeout | |
| 20:46:36 | dansmith | good lord | |
| 20:46:57 | dansmith | maybe we should start enacting a french work-week to avoid overloading the gate | |
| 20:46:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | |
| 20:46:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | |
| 21:01:53 | radez | ok, I'm back now, sry about that | |
| 21:02:00 | radez | dansmith: mriedem: http://paste.openstack.org/show/736843/ | |
| 21:02:25 | radez | this shows the uuids of the bm nodes and the nova instances that seems to be behaving like this | |
| 21:03:20 | radez | and the nova logs where nova seems to assign the already assigned bm node to the new nova instance | |
| 21:03:51 | radez | let me know if there's more logs I can get to help | |
| 21:04:41 | dansmith | radez: so these nodes should all exist in placement with a single inventory item for the resource/node class, | |
| 21:04:51 | dansmith | which means they should never get past the scheduler with two instances on the same node | |
| 21:05:12 | radez | translation was helpful thx :) | |
| 21:05:31 | radez | [instance: a840afbc-1314-44d6-83b3-c20b790b9322] Claim successful on node 3c3eb5ee-a358-421a-b2e9-7fd9a017ba13 | |
| 21:06:34 | radez | hm, well now I think I've given you the logs that show that the associated ironic node and nova instance are properly associated together | |
| 21:07:42 | mriedem | do the flavors have cpu/ram/disk set to anything !0? | |
| 21:07:44 | radez | oh wait, sry I'm getting my self confused... right so does that log line I just pasted here show that nova is trying to | |
| 21:07:46 | dansmith | radez: what log is this? just nova compute? | |
| 21:08:06 | radez | assign ironic node 3c3eb5ee to nova instance a840afbc? | |
| 21:08:34 | radez | yea nova-compute log | |
| 21:08:43 | dansmith | radez: so probably want to look at the scheduler log | |
| 21:09:00 | dansmith | radez: scheduler is what decides which node it is going to tell compute to use for each instance | |
| 21:10:10 | radez | k lemme find a840afbc in the schedule logs | |
| 21:13:17 | radez | http://paste.openstack.org/show/736844/ | |
| 21:13:43 | openstackgerrit | Merged openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | |
| 21:13:46 | radez | that also shows a840afbc trying to select 3c3eb5ee | |
| 21:14:32 | radez | hypervisor list incase that's helpful? http://paste.openstack.org/show/736845/ | |
| 21:14:33 | melwitt | radez: what version of rocky is this? we had a bug around ironic inventory update in the middle of rocky https://review.openstack.org/593678 but as long as you have 18.0.0.0rc2 or later, you should have the fix for that | |
| 21:14:35 | melwitt | er, at the end of rocky | |
| 21:14:42 | dansmith | radez: can you show more of the log so we can see both? | |
| 21:15:07 | radez | yea, that was just a quick grep, lemme grab more of the scheduler log and check my version | |
| 21:16:54 | dansmith | melwitt: that would only affect nodes in cleaning right? | |
| 21:18:11 | melwitt | dansmith: I don't know, tbh | |
| 21:19:14 | melwitt | I remember there were some ironic driver bugs we fixed near the end of rocky and was curious if those can be ruled out based on what version of rocky they're running | |
| 21:19:43 | dansmith | the only way this should be happening, AFAIK, is if we're not exposing one inventory per node, or somehow not claiming in placement | |
| 21:20:15 | melwitt | yeah, this bug was related to the claiming of a node via setting reserved == total | |
| 21:20:36 | melwitt | and one spot where the newer placement api microversion needed to be set was missed | |
| 21:21:00 | melwitt | so I was thinking that might possible cause a node to not be claimed when it was supposed to. but as for if it only comes up during cleaning, that I didn't know | |
| 21:21:20 | melwitt | *possibly | |
| 21:21:57 | mriedem | radez: what are the values for cpu/ram/disk for the flavors being used for these baremetal nodes? | |
| 21:21:58 | melwitt | I had thought the reserved == total thing was claiming in general and not only for cleaning, but could easily be wrong as I don't know _that_ much about the ironic driver | |
| 21:22:11 | radez | openstack-nova-scheduler-18.0.3 | |
| 21:22:20 | radez | ok, got hte version, sry fighting with docker :/ | |
| 21:22:42 | melwitt | thanks radez. so yeah, the thing I wondered can be ruled out | |
| 21:22:44 | dansmith | melwitt: reserved=total is only for making the node look unschedulable while being cleaned right? | |
| 21:22:55 | dansmith | melwitt: reserved=total does't make any sense when the node is actually allocated | |
| 21:23:30 | mriedem | i'm wondering if you're hitting https://bugs.launchpad.net/nova/+bug/1796920 | |
| 21:23:30 | openstack | Launchpad bug 1796920 in OpenStack Compute (nova) queens "Baremetal nodes should not be exposing non-custom-resource-class (vcpu, ram, disk)" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 21:23:31 | radez | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'40', u'cpus': u'4', u'capabilities': u'boot_option:local' | |
| 21:23:35 | radez | both look liek that | |
| 21:23:40 | mriedem | yeah so check that bug | |
| 21:23:44 | mriedem | or https://review.openstack.org/#/c/609043/ | |
| 21:23:45 | radez | getting the logs... | |
| 21:23:59 | dansmith | mriedem: yeah, that's the kind of thing I was thinking of | |