| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-18 | |||
| 11:10:38 | bauzas | sec, checking | |
| 11:10:44 | lyarwood | compute_api.parse_availability_zone also uses that | |
| 11:11:07 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L475 | |
| 11:11:34 | bauzas | lyarwood: but look | |
| 11:12:11 | bauzas | if you don't provide a tuple of 3 items (az, host, node) but only 2 (az, host) then you need to double ":" | |
| 11:12:16 | bauzas | lyarwood: ^ | |
| 11:12:31 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L488 | |
| 11:12:48 | bauzas | holy snap | |
| 11:12:55 | bauzas | I'm wrong | |
| 11:13:12 | lyarwood | yeah you only need 1 | |
| 11:13:17 | bauzas | "::" is for (az, host | |
| 11:13:21 | bauzas | damn | |
| 11:13:24 | bauzas | (az, node) | |
| 11:14:02 | bauzas | lyarwood: I guess you have nova service-list telling you that you have nova-compute on some random host literrally named 'unassigned-hostname' ? | |
| 11:14:52 | lyarwood | $ openstack compute service list --service nova-compute | |
| 11:14:53 | lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | |
| 11:14:55 | lyarwood | | ID | Binary | Host | Zone | Status | State | Updated At | | |
| 11:14:57 | lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | |
| 11:14:59 | lyarwood | | 2 | nova-compute | unassigned-hostname | nova | enabled | up | 2020-03-18T11:14:39.000000 | | |
| 11:15:01 | lyarwood | +----+--------------+---------------------+------+---------+-------+----------------------------+ | |
| 11:15:03 | lyarwood | yeah | |
| 11:16:49 | lyarwood | Mar 18 07:02:06 unassigned-hostname nova-scheduler[3674]: DEBUG nova.scheduler.manager [None req-27d6db0a-3f02-4350-8bab-7cc9f93639d0 admin admin] Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service. - ah | |
| 11:17:10 | lyarwood | maybe we are bailing on before we even get to filtering the hosts | |
| 11:20:17 | lyarwood | https://github.com/openstack/nova/blob/78e53bd388a5c2305da5249f1f217923d3fe2aac/nova/scheduler/manager.py#L134-L139 yup that's it | |
| 11:20:24 | bauzas | lyarwood: we don't go with checking the filters if you specify a target like this | |
| 11:20:39 | lyarwood | yeah it's failing before we get anywhere near there | |
| 11:20:44 | lyarwood | I think it's the hostname causing issues | |
| 11:20:50 | lyarwood | I'll unstack, set something valid and try again | |
| 11:21:10 | bauzas | weirdo | |
| 11:21:33 | lyarwood | yup indeed | |
| 11:45:03 | stephenfin | lyarwood: Sorry, meetings. Yup, I'll take a look | |
| 11:45:40 | stephenfin | and my lock was me cooped up at home, figuring out how traefik worked with a single beer | |
| 11:48:01 | stephenfin | gibi: replied on https://review.opendev.org/#/c/710276/ | |
| 11:48:27 | lyarwood | stephenfin: my deepest condolences | |
| 11:48:31 | stephenfin | I should have been clearer in the commit message - I'd forgotten half the reason I'd done some of that myself. Can edit it if you'd like | |
| 11:49:05 | lyarwood | bauzas: so I'm still getting no allocation candidates from placement | |
| 11:50:56 | stephenfin | yo, bauzas, can you hit https://review.opendev.org/#/c/710239 please? | |
| 11:52:30 | lyarwood | ah wait I'm being stupid | |
| 11:55:15 | openstackgerrit | Merged openstack/nova master: bug-fix: Reject live migration with vpmem https://review.opendev.org/708110 | |
| 12:09:01 | openstackgerrit | Merged openstack/nova master: VMware VMDK detach: get adapter type from instance VM https://review.opendev.org/653738 | |
| 12:14:58 | openstackgerrit | Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856 | |
| 12:21:21 | openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | |
| 12:42:21 | lyarwood | eh when did ubuntu make upgrade and dist-upgrade different? | |
| 12:42:51 | lyarwood | I hit this again https://bugs.launchpad.net/os-vif/+bug/1749972 because virt-builder is just running upgrade when you build with --update | |
| 12:42:51 | openstack | Launchpad bug 1749972 in os-vif "`brctl setageing $bridge 0` fails on Ubuntu 16.04 4.4.0-21-generic" [Undecided,Invalid] | |
| 12:43:06 | lyarwood | *sigh* | |
| 12:43:27 | sean-k-mooney | lyarwood: its always been different | |
| 12:43:42 | sean-k-mooney | lyarwood: dist-upgrade upgrades your kernel and all other packages | |
| 12:44:02 | sean-k-mooney | upgrade will hold back some packages that would require a hots reboot to take effect | |
| 12:44:39 | lyarwood | sean-k-mooney: ack thanks, I haven't used Ubuntu as a daily OS in something like 12 years so I guess I had just forgotten this | |
| 12:45:38 | sean-k-mooney | ya i use it daily so i swap between the two depening on what behavior i want | |
| 12:45:51 | jkulik | upgrade will also hold back packages needing new dependencies to be installed or other packages to be uninstalled. not only reboots. at least in debian. | |
| 12:46:07 | sean-k-mooney | jkulik: correctg | |
| 12:46:19 | sean-k-mooney | upgrade is more concervitive | |
| 12:47:05 | sean-k-mooney | lyarwood: how did that affect that bug | |
| 12:47:44 | sean-k-mooney | was brctl not upgraded? | |
| 12:49:20 | sean-k-mooney | lyarwood: also why are you using 16.04? aren't all stable brnaches that use that em at this point? | |
| 12:54:40 | lyarwood | sean-k-mooney: deploying stable/queens | |
| 12:54:58 | sean-k-mooney | i think queens runs fine on 18.04 as well | |
| 12:55:02 | sean-k-mooney | but ok | |
| 12:55:20 | lyarwood | sean-k-mooney: as you said in that bug it appears that something in the kernel changed during the lifecycle of that release | |
| 12:55:26 | lyarwood | sean-k-mooney: yeah I was just copying CI | |
| 12:55:26 | sean-k-mooney | i had completely forgoten about that bug report so thanks for marking it invalid | |
| 12:56:08 | lyarwood | np | |
| 13:01:17 | artom | gibi, thanks for the cleanup :) | |
| 13:02:12 | artom | I would have been even more agressive - like https://review.opendev.org/#/c/525335 is from 2 years ago with no response | |
| 13:02:27 | artom | But I won't tell you how to do your job ;) | |
| 13:03:31 | openstackgerrit | waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018 | |
| 13:04:29 | luyao | artom: Hi | |
| 13:04:29 | ygk_12345 | hi all | |
| 13:04:40 | ygk_12345 | i have rocky setup. | |
| 13:05:10 | ygk_12345 | when I try to launch an instance on a specific compute node , it is throwing this error in the nova-compute logs | |
| 13:05:12 | ygk_12345 | Failed to retrieve resource provider tree from placement API for UUID ea3af0ff-fb69-449a-b484-5ac722010a1f | |
| 13:07:29 | gibi | artom: you recently asked questions on https://review.opendev.org/#/c/525335 so I allow the author to answer. :) | |
| 13:07:39 | sean-k-mooney | ygk_12345: does the vm launch correctly | |
| 13:08:02 | ygk_12345 | sean-k-mooney no . it is going into error state | |
| 13:08:12 | sean-k-mooney | ygk_12345: i cant recall which release we made placmenert mandatory but have you deployed the placement service | |
| 13:08:38 | ygk_12345 | sean-k-mooney its rocky 18.1.9 branch in openstack ansible | |
| 13:09:08 | sean-k-mooney | did you delete the compute node and recreate it? | |
| 13:09:42 | ygk_12345 | sean-k-mooney i tried upgrading the working rocky setup to stein . it failed and so rolled back to rocky now | |
| 13:09:47 | ygk_12345 | yes | |
| 13:10:02 | ygk_12345 | sean-k-mooney how to delete the compute node ? | |
| 13:10:22 | ygk_12345 | sean-k-mooney i re ran the nova playbook | |
| 13:10:45 | sean-k-mooney | so this is not a nova issue. its and issue with osa and the operation you performed. | |
| 13:11:14 | ygk_12345 | sean-k-mooney so whats the workaround now ? | |
| 13:11:33 | sean-k-mooney | i think we have disucssed this before but if you delete the compute service and you restart the agent without altering the host name of the compute host then it will create a new compute service record in the db with a differnet uuid | |
| 13:12:11 | sean-k-mooney | but that will prevent the the agent form creating the resouce provider with the new uuid as the RP name will still be in use | |
| 13:12:35 | sean-k-mooney | can you validate that by checking if there is a placmenet RP with the hostname? | |
| 13:12:38 | ygk_12345 | sean-k-mooney i have altered the hostname and it created the new record in resource_providers db in nova_api table | |
| 13:12:46 | luyao | artom: If you see my message, could you help look at https://review.opendev.org/#/c/687856/15/nova/compute/manager.py@8336, I change the order of cleanup during post live migration. We do post_live_migration_at_destination first and then cleanup source host in existing code logic, if there are some specific reason ? Or if the order doesn't matter? | |
| 13:12:54 | ygk_12345 | sean-k-mooney how to do that ? | |
| 13:16:52 | artom | gibi, fair enough :) | |
| 13:17:01 | sean-k-mooney | you can use the plamcent openstack client plugin to list the resouce providers and grep for the hostname | |
| 13:17:42 | gibi | artom: ping me next week and I will close out the next patch of reviews including the one you commented :) | |
| 13:17:49 | ygk_12345 | sean-k-mooney is placement included at all in rocky ? | |
| 13:18:05 | artom | luyao, looking... | |
| 13:18:17 | artom | gibi, hehe, ack | |
| 13:18:35 | sean-k-mooney | ygk_12345: it was intoduced in pike maybe earlier | |
| 13:19:02 | sean-k-mooney | ygk_12345: so yes it should be included in rocky | |
| 13:19:42 | luyao | artom: thanks | |