| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-30 | |||
| 19:27:46 | mriedem | how about a public paste? | |
| 19:28:03 | mriedem | paste.openstack.org or gist.github.com | |
| 19:28:21 | tzumainn | whoops, sorry! | |
| 19:29:43 | tzumainn | melwitt, http://paste.openstack.org/show/729177/ | |
| 19:29:51 | melwitt | the error is saying there's the 'id' field missing from the ComputeNode object, which means it wasn't created/obtained from the database (where the 'id' field comes from). but in the code, I see a cn.create() before _setup_pci_tracker is called, so 'id' should be populated | |
| 19:29:59 | mriedem | cdent: vmware ci might be hitting the same scheduling issues? http://207.189.188.190/logs/16/270116/12/check-vote/ext-nova-zuul/7a47690/ | |
| 19:30:04 | mriedem | lots of novalidhost in there | |
| 19:30:11 | mriedem | that's on the live migration for vmware change | |
| 19:30:12 | melwitt | tzumainn: what release is this? | |
| 19:30:17 | tzumainn | melwitt, this is rocky | |
| 19:30:25 | melwitt | ok | |
| 19:31:40 | mriedem | i was going to say https://review.openstack.org/#/c/520024/ but that's not in rocky | |
| 19:33:36 | melwitt | heh, that patch again | |
| 19:34:10 | mriedem | it's in the same code path | |
| 19:36:31 | mriedem | by the time we call _setup_pci_tracker there in that block on L563 we should have an existing instance, either created from the RT or pulled from the DB | |
| 19:36:41 | mriedem | *existing compute node record | |
| 19:36:54 | melwitt | yeah, according to the trace, the ComputeNode object in self.compute_nodes has no 'id' field populated | |
| 19:37:26 | melwitt | so there must be a way we're adding things to self.compute_nodes that are object shells, not gotten from the DB or newly created | |
| 19:41:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API https://review.openstack.org/469675 | |
| 19:49:07 | cdent | thanks mriedem will do some poking and prodding | |
| 19:49:50 | openstackgerrit | Chris Dent proposed openstack/nova master: VMware: Live migration of instances https://review.openstack.org/270116 | |
| 19:52:12 | mriedem | tzumainn: which virt driver? libvirt? ironic? | |
| 19:52:21 | mriedem | and are you doing anything when this happens? | |
| 19:52:35 | melwitt | it's ironic | |
| 19:52:39 | mriedem | like, is this on start of nova-compute or during a periodic task? | |
| 19:52:48 | mriedem | hmmm, is a rebalance happening? | |
| 19:53:10 | tzumainn | mriedem, ah, this is ironic - I've just enrolled four nodes, and am trying to figure out why they don't show up in 'openstack hypervisor list' | |
| 19:53:56 | mriedem | they won't show up in openstack hypervisor list until you've "discovered" them | |
| 19:54:02 | mriedem | see the nova-manage cell_v2 discover_hosts command | |
| 19:54:07 | mriedem | you'll need to discover by service | |
| 19:55:12 | tzumainn | ah, okay! I wasn't aware - I'm following the instructions in http://tripleo.org/install/advanced_deployment/baremetal_overcloud.html which I guess are out of date | |
| 19:56:26 | melwitt | the traceback was unrelated then. still don't see how the condition of no 'id' on a ComputeNode in self.compute_nodes can happen (thought it obviously can happen somehow) | |
| 19:56:43 | mriedem | yeah i don't really know how that's being hit | |
| 19:57:13 | mriedem | tzumainn: no idea re tripleo deployment, they have their own irc channel for that | |
| 19:57:16 | melwitt | remove_node removes things from the dict if orphaned, all of the setting of self.compute_nodes seem to be covered by actual DB gets or creates. weird | |
| 19:57:17 | mriedem | plus like 50 red hat cores | |
| 19:58:39 | tzumainn | mriedem, haha, yep - I started out talking with some ironic folks, and confusion all around has led me here : ) | |
| 19:58:55 | tzumainn | thanks for the information, I really appreciate it! | |
| 19:59:30 | mriedem | maybe related to https://review.openstack.org/#/c/587922/ ? | |
| 20:00:19 | jungleboyj | mriedem: I don't have a separate one. Asked people to mark the topics and then I was going to collect them up. | |
| 20:00:22 | mriedem | remove_node should likely be in the same semaphore as _update_available_resource_for_node | |
| 20:00:52 | mriedem | i guess i already said that https://review.openstack.org/#/c/587922/2/nova/compute/resource_tracker.py | |
| 20:01:21 | mriedem | so self.old_resources will default a ComputeNode object if an entry isn't in the dict... | |
| 20:02:25 | zigo | mriedem: Should I try your patch at https://review.openstack.org/#/c/598176/ and report the result? | |
| 20:02:45 | melwitt | does self.compute_nodes refer to self.old_resources at all? | |
| 20:03:01 | mriedem | they are compared in _resource_change | |
| 20:03:11 | mriedem | to determine if we should call ComputeNode.save() | |
| 20:03:28 | mriedem | zigo: we aren't going to ship that revert i don't think so probably would be a waste of your time | |
| 20:04:13 | zigo | mriedem: If it's only a temporary fix that I can use to validate all of Rocky, that's nice already, then I can still remove the patch... | |
| 20:04:34 | zigo | Hum... | |
| 20:04:39 | zigo | It doesn't apply at all anyway. | |
| 20:05:12 | zigo | mriedem: This wasn't in rocky. | |
| 20:05:16 | mriedem | cdent: melwitt: was also wondering if this somehow is contributing to the allocation ratio bug https://review.openstack.org/#/c/518294/ | |
| 20:05:19 | mriedem | but that was in queens | |
| 20:05:27 | mriedem | zigo: right | |
| 20:05:56 | melwitt | ack | |
| 20:06:44 | zigo | mriedem: Could you see something with the added logs in https://review.openstack.org/#/c/597175/ ? | |
| 20:06:54 | zigo | Both your patches were added there ... | |
| 20:06:58 | zigo | (the ones for logging...) | |
| 20:08:07 | mriedem | i added more debug logs this morning after a recreate in the xen ci, was just about to check those results | |
| 20:08:42 | mriedem | tzumainn: you might want to report a nova bug regardless so we don't lose track of what you hit | |
| 20:08:52 | mriedem | i'm not sure *how* you're hitting it, but that seems to be the theme this week with all bugs | |
| 20:09:00 | mriedem | "how in the hell is this even possible?" | |
| 20:09:12 | tzumainn | mriedem, hahaha - okay, I'll do that, thanks! | |
| 20:09:58 | openstackgerrit | Merged openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 20:10:00 | mriedem | dansmith: is it weird that ComputeNode.save() doesn't check to see if 'id' is changed? | |
| 20:10:01 | mriedem | or not set? | |
| 20:10:31 | dansmith | create or save? | |
| 20:10:32 | mriedem | i guess if it weren't set we'd blow up | |
| 20:10:33 | mriedem | save | |
| 20:10:36 | melwitt | I was thinking save() would just fail if there's no id, because then how could it find the thing to update | |
| 20:10:46 | dansmith | not sure we usually check that it's set on save, but we could | |
| 20:11:01 | dansmith | on create we usually check to avoid re-create | |
| 20:11:03 | mriedem | melwitt: right we'd blow up if id wasn't set on save | |
| 20:11:07 | melwitt | yeah | |
| 20:11:08 | mriedem | yeah we check on create | |
| 20:11:18 | mriedem | i was wondering if id changed though and we saved | |
| 20:11:51 | mriedem | i guess we don't really check for that anywhere | |
| 20:11:54 | dansmith | we pop id out of the changes, | |
| 20:12:03 | melwitt | even if we did, that doesn't explain the complete lack of an 'id' on a ComputeNode object in self.compute_nodes | |
| 20:12:04 | dansmith | but I guess we try to save anyway even if that was the only thing | |
| 20:12:50 | melwitt | I still don't see how we could lose that, even with the self.old_resources compare | |
| 20:13:23 | dansmith | if id is actually not set then it's either an object created with no id (not from the db) or someone del'd it off an objet | |
| 20:13:24 | mriedem | i don't either, i was wondering if we were getting a blank ComputeNode from old_resources which uses defaultdict and somehow shoved that blank one into self.compute_nodes | |
| 20:13:25 | mriedem | but we don't | |
| 20:13:51 | melwitt | yeah | |
| 20:16:17 | mriedem | there is a very small window where self.compute_nodes could have a ComputeNode in it without an id | |
| 20:16:20 | mriedem | in _init_compute_node | |
| 20:16:21 | mriedem | cn.create() | |
| 20:16:21 | mriedem | self.compute_nodes[nodename] = cn | |
| 20:16:25 | mriedem | b/c cn.create() is what sets the id | |
| 20:16:46 | mriedem | but, that code is all in a lock on the same host when we call update_available_resource | |
| 20:16:52 | mriedem | so not sure how anything could race and hit that | |
| 20:16:54 | melwitt | yeah, I wondered about that | |
| 20:17:46 | mriedem | tzumainn: did you by chance have multiple nova-compute services running on the same host? | |
| 20:17:59 | mriedem | no that still wouldn't do this | |
| 20:18:05 | mriedem | b/c the compute nodes map is in memory | |
| 20:18:19 | mriedem | i give up | |
| 20:18:20 | tzumainn | mriedem, I only have one, in any case | |
| 20:18:27 | tzumainn | mriedem, no worries, thanks for taking a look : ) | |