| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-28 | |||
| 08:38:23 | opendevreview | Simon Li proposed openstack/nova-specs master: upport ironic instance change host while it's host not up https://review.opendev.org/c/openstack/nova-specs/+/815789 | |
| 10:16:29 | em__ | Hello. No matter which cloud-init box i start (debian11 genericcloud, debian11 nocloud, cirros 0.5.2) the boxes never get the ssh key i deployed. Interestingly, when looking at the logs for cirros is see this https://gist.github.com/EugenMayer/42a0f13ccf5f18076c4e2d84655bda66 - is this how a usual openstack instance retrieves its cloud-init config via HTTP (and it fails here) | |
| 10:50:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 11:33:56 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 14:00:20 | gibi | brinzhang: hi! regarding the owner trait discussion from the PTG. will you (or wenping) prepare the additional spec for nova to mark nova RPs with owner trait too? | |
| 14:49:12 | opendevreview | Merged openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 15:19:05 | gibi | melwitt: hi! Could you please look at https://review.opendev.org/c/openstack/nova/+/802060 , it is the nova-manage change for the packet rate feature and it has +2 from stephenfin already. Neutron is ready to land the related API extension | |
| 15:20:56 | melwitt | gibi: sure, I will add it to my queue | |
| 15:21:04 | gibi | thank you | |
| 15:39:38 | opendevreview | Mohammed Naser proposed openstack/nova master: Log failed metadata retrievals https://review.opendev.org/c/openstack/nova/+/815859 | |
| 16:02:04 | mnaser_ | ^ anyone ever ran into this... | |
| 16:02:14 | mnaser_ | i'm seeing random failures to look up instances | |
| 16:03:25 | clarkb | mnaser_: tempest jobs have a recurring failure due to metadata timeouts | |
| 16:03:33 | clarkb | not sure if related, but seems similar at least | |
| 16:03:42 | mnaser_ | i mean that change would help i think | |
| 16:03:49 | mnaser_ | at least uncover some of the potential issues | |
| 17:37:09 | mnaser_ | oh man | |
| 17:37:12 | mnaser_ | i think i've got a lead | |
| 17:37:40 | mnaser_ | since we churn VMs so fast at our cloud, and i guess nova or neutron (still looking) caches the instance id => ip address mapping | |
| 17:37:52 | mnaser_ | we get a request to metadata service 5 minutes later for an instance id that was already deleted | |
| 17:38:08 | mnaser_ | and my guess is that it's because neutron has cached the instance id => ip even if it has already changed | |
| 17:38:18 | mnaser_ | so metadata is like "wtf who are you, you don't exist" because the vm does indeed not exist | |
| 17:39:27 | clarkb | mnaser_: time to use config drive :) | |
| 17:39:40 | mnaser_ | https://github.com/openstack/neutron/blob/1ad9ca56b07ffdc9f7e0bc6a62af61961b9128eb/neutron/agent/metadata/agent.py#L86-L103 | |
| 17:39:44 | mnaser_ | ok so i guess it does retry if it gets a 404 | |
| 17:41:08 | mnaser_ | ok, https://github.com/openstack/neutron/commit/72a5b5b61f532bba6de502feda36dfc7d36cefc7 was fixed in train+, this cloud is stein | |
| 20:25:58 | prometheanfire | https://review.opendev.org/814889 oslo.concurrency-4.5.0 causes nova tests to fail | |
| 21:40:24 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova-specs master: Integration With Off-path Network Backends https://review.opendev.org/c/openstack/nova-specs/+/787458 | |
| 23:41:51 | opendevreview | Merged openstack/nova stable/train: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/791807 | |
| #openstack-nova - 2021-10-29 | |||
| 00:06:25 | opendevreview | Danil proposed openstack/nova master: Check if the InstanceNUMACell.pcpuset attribute exist prior to relying on it. This change enables support for the VM instances with with pinned CPUs (hw:cpu_policy=dedicated) that were created before upgrading the OpenStack Nova to Victoria release. https://review.opendev.org/c/openstack/nova/+/815923 | |
| 01:48:24 | opendevreview | melanie witt proposed openstack/nova stable/train: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/811822 | |
| 01:48:25 | opendevreview | melanie witt proposed openstack/nova stable/train: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/811823 | |
| 01:48:26 | opendevreview | melanie witt proposed openstack/nova stable/train: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/811824 | |
| 01:48:27 | opendevreview | melanie witt proposed openstack/nova stable/train: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811825 | |
| 06:09:16 | brinzhang_ | gibi: ack, wenpingsong__ will propose it later | |
| 06:59:15 | gibi | brinzhang: OK, cool, let me know if I can help | |
| 07:02:23 | brinzhang | gibi: ack | |
| 07:02:30 | brinzhang | wenpingsong: ^^ | |
| 07:05:04 | gibi | prometheanfire: ack, I remember looking at that failure and then got distracted, let me see if I can quickly fix it | |
| 07:40:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix unit test for oslo.concurrency 4.5 https://review.opendev.org/c/openstack/nova/+/815940 | |
| 07:41:47 | gibi | prometheanfire: ^^ | |
| 07:43:40 | gibi | bauzas: a small unit test fix ^^ to unblock the requirement bump | |
| 07:57:49 | bauzas | gibi : apologies but I'm off until Tuesday | |
| 07:58:22 | bauzas | (heading to Paris) | |
| 08:12:35 | gibi | bauzas: ohh, I remember now. Enjoy | |
| 08:31:12 | em_ | is there any way to add /remove/change the ssh-key of an cloud-init instance after the creation | |
| 08:31:23 | em_ | openstack server set --property? | |
| 09:32:13 | gibi | em_: I think you can only use rebuild to inject a new keypair | |
| 09:40:25 | em_ | does rebuild white the database? | |
| 09:40:28 | em_ | sorry disk | |
| 10:00:59 | gibi | yes it will rewrite the root disk of the VM from the original image | |
| 10:15:15 | em_ | so the concept is to never write data on the root disk, only on ephi? | |
| 10:16:06 | sean-k-mooney[m] | rebuild willl erase the ephemeral disks too | |
| 10:17:04 | sean-k-mooney[m] | rebuild is a distructive operation that should only be used to erase all data in the vm that is no stored in cinder volumes | |
| 10:17:35 | gibi | sean-k-mooney[m]: ohh, will rebuild replace ephemerals with an empty disk? | |
| 10:17:37 | sean-k-mooney[m] | cloud init by default does not run on each boot so there is no reliable way to change the ssh key pair | |
| 10:17:56 | gibi | ) | |
| 10:17:56 | gibi | (make sense though as the name is _ephemeral_ | |
| 10:18:31 | sean-k-mooney[m] | even if we updated it in the db cloud-init would not rerun on the next boot unless you altered its default behavior | |
| 10:19:21 | sean-k-mooney[m] | if you need to rotate an ssh key or add a new one you really need to do that by sshing in wiht the orignal and adding it manually | |
| 10:20:57 | sean-k-mooney[m] | gibi there used to be an api extention tat allowed you to ask for it to no be erased pre microversions | |
| 10:20:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: [nova-manage]support extended resource request https://review.opendev.org/c/openstack/nova/+/802060 | |
| 10:21:17 | sean-k-mooney[m] | but we never actully made that part of the api when we removed extentions | |
| 10:21:31 | sean-k-mooney[m] | and it was never supported by libvirt if i understand corectly | |
| 10:22:14 | gibi | ohh, OK, I learned new things today \o/ | |
| 10:23:27 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/liberty-eol/nova/api/openstack/compute/legacy_v2/contrib/preserve_ephemeral_rebuild.py | |
| 10:28:04 | sean-k-mooney[m] | oh i guess we still support tht | |
| 10:28:23 | sean-k-mooney[m] | its not documented anywhere | |
| 10:28:45 | sean-k-mooney[m] | it looks like you set preserve_ephemeral=true in the server metadata | |
| 10:29:40 | sean-k-mooney[m] | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/api/openstack/compute/helpers.py#L96-L103 | |
| 10:29:53 | sean-k-mooney[m] | i think this was only supported by ironic however | |
| 10:33:39 | sean-k-mooney[m] | yep its not supported int he default implemetation https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/compute/manager.py#L3234-L3238 | |
| 10:33:40 | sean-k-mooney[m] | so libvirt does not support it | |
| 10:34:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | |
| 10:47:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test nova-manage placement heal_allocations with new Neutron https://review.opendev.org/c/openstack/nova/+/815952 | |
| 11:10:22 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 14:41:22 | gibi | gmann: are you around? | |
| 14:41:35 | gmann | gibi: hi | |
| 14:41:38 | gibi | hi | |
| 14:42:05 | gibi | I'm trying to figure out the issue in the bandwidth tempest tests https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9ab/802060/18/check/nova-next/9ab4682/testr_results.html | |
| 14:42:24 | gibi | it seems that the tempest code now thinks that the new neutron API extension is enabled | |
| 14:42:35 | gibi | but that is not merged yet https://review.opendev.org/c/openstack/neutron/+/805637 | |
| 14:44:05 | gmann | gibi: ah, got it. | |
| 14:44:15 | gibi | api_extensions = all | |
| 14:44:15 | gibi | there is | |
| 14:44:17 | gibi | in the tempest conf | |
| 14:44:22 | gmann | gibi: I think we should make depends-on in otherway around | |
| 14:44:46 | gmann | tempest tests should be depends-on on neutron? or neutron patch fail on existing test? | |
| 14:45:11 | gibi | the neutron patch is OK to land as that adds the extension that makes the test passing | |
| 14:45:22 | gibi | I have another nova patch the fails now until the neutron patch lands | |
| 14:45:43 | gibi | and also the newer tempest patches adding pps coverage are failing unit the neutron patch lands | |
| 14:45:47 | gmann | gibi: you mean tempest existing test (before https://review.opendev.org/c/openstack/tempest/+/809168) fail ? | |
| 14:46:36 | gibi | I mean the test we adapeted to the new neutron API in https://review.opendev.org/c/openstack/tempest/+/809168 now fails for other patches as the neutron patch haven't landed yet | |
| 14:47:10 | gibi | which is due to the fact that tempest is configured that all the API extensions are available even the one that is not merged yet | |
| 14:47:19 | gmann | gibi: yeah. but if we revert 809168 and merge the series after neutron patch then is it fine? | |
| 14:47:32 | gmann | tempest seriesw | |
| 14:48:08 | gibi | I would keep the 809168 and wait for https://review.opendev.org/c/openstack/neutron/+/805637 to land clearing up the situation | |
| 14:48:13 | gmann | or existing assert which was modified in 809168 will fail if neutron patch merged as it change the response? | |
| 14:48:38 | gibi | yepp we need 809168 before we land the neutron API change as that would break existing tempest | |
| 14:48:44 | gibi | so I think we are OK as is now | |